Repository navigation
Keep invalid schedules from stopping workers and validate writes - #120
Merged
Merged
Conversation
Reject invalid schedule values before writing them. Keep unreadable stored rows from stopping workers, other schedules or queued tasks. Worker-stopping paths in versions 1.2.0-1.7.0 include the following. An interval above 62,135,596,800 seconds, combined with certain phases, ends every worker reading it. Restarting repeats the failure. Stored intervals can trigger this on SQLite. PostgreSQL, MySQL and MariaDB columns reject intervals this large. SCHEDULES entries fail before database access and are not limited to SQLite. The interval overflow was reproduced on SQLite. The other database conclusions for that interval come from code inspection. On PostgreSQL, an out-of-range stored bound can prevent every worker from starting and stop every dispatch pass of a running worker. Tasks already queued still ran in the PostgreSQL reproduction. An end time late on 9999-12-31 in a zone west of UTC, which is year 10000 in UTC, was accepted through create_schedule on every release from 1.2.0 through 1.7.0. Each Worker() then raised DataError during construction. Limit every_seconds to 62,135,596,800 on every write path. Reject bounds outside years 1 to 9999 in the database's time zone. Reject times with a time zone when USE_TZ is off. Skip and log stored rows that cannot be read or compared. Isolate tick and bound errors per schedule as schedule_dispatch_error. Database outages still stop the pass. Fall back to full table reads at SCHEDULE_RECONCILE_INTERVAL when the change marker is unreadable. Log skipped rows once, then at most once a minute per row per worker. Refuse oversized SCHEDULES intervals at check and worker start with django_ox.E002. Use E002 for every or phase outside timedelta's range. Cap phase_seconds and starting_deadline_seconds at 86,399,999,999,999. Allow update_schedule(row, enabled=False) with no other changes to pause a readable row that fails validation. Allow the admin Disable action for these rows. In 1.7.0, that action returned HTTP 500 for rows that failed validation. Pausing does not repair a row. Refuse to enable it until it is corrected. Report invalid rows as not runnable for "Run once now". Upgrade workers to a release containing this fix. Check SQLite intervals above the ceiling, stored bounds in the database's time zone, and SCHEDULES entries. Correct or pause readable invalid rows, or delete them. The PostgreSQL year-10000 row cannot be corrected or disabled through the admin or update_schedule. It also makes the admin changelist fail for all rows. Delete it with delete_schedule(OxSchedule(pk=schedule_pk)) or a plain SQL DELETE. The ceiling also rejects previously harmless oversized intervals. No migration is required. Tests cover each behavior the change adds or alters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keep invalid stored schedules from stopping workers, other schedules or queued tasks. Validate schedule writes, isolate unreadable rows, and allow readable rows that fail validation to be paused.
Worker-stopping paths in versions 1.2.0-1.7.0 include the following:
Both stop other schedules from firing. The interval path also stops queued-task processing. Tasks already queued still ran in the PostgreSQL bound reproduction.
Who is affected
create_schedule,update_schedule, admin add or change access, and direct table writes.SCHEDULESentry, regardless of database. This path requires settings access and fails before database access.create_scheduleandupdate_scheduleaccepted bounds in year 10000 UTC, such as an end time late on 9999-12-31 in a zone west of UTC.PostgreSQL, MySQL and MariaDB columns reject the triggering interval value. Those conclusions come from code inspection, not interval-overflow reproductions on those engines.
SQLite and MySQL refuse to store the out-of-range bound used in the PostgreSQL reproduction. This is separate from the SQLite interval overflow.
Changes
Validate schedule values
every_secondsat 62,135,596,800 on every write path.USE_TZis off. Apply these checks increate_schedule,update_scheduleandcreate_schedules.SCHEDULESintervals atmanage.py checkand worker start withdjango_ox.E002.everyorphasevalues outside Python'stimedeltarange asdjango_ox.E002, notOverflowError.phase_secondsandstarting_deadline_secondsat 86,399,999,999,999 seconds. SQLite previously accepted larger values that were skipped at every read.Isolate invalid stored rows
Skip and log rows the worker cannot read or compare. This includes driver conversion failures, impossible dates, out-of-range bounds, PostgreSQL infinity or BC dates, and MySQL zero dates. Skip intervals above the ceiling and phases that could put a tick before year 1.
Report ticks that cannot be derived and bounds that cannot be compared as
schedule_dispatch_error, per schedule, for any schedule source. Database outages still stop the pass. They are not treated as bad rows.If the change marker is unreadable, read the table in full at each
SCHEDULE_RECONCILE_INTERVAL, which defaults to 60 seconds. Log skipped rows once, then at most once a minute per row per worker while they remain skipped.Allow pausing without repair
Allow
update_schedule(row, enabled=False)with no other changes to pause a readable row that fails validation. The admin Disable action also works on these rows instead of returning HTTP 500, as it did in 1.7.0 for rows that failed validation.Pausing does not repair the row. Enabling it again is refused until it is corrected. "Run once now" reports an invalid stored row as not runnable instead of enqueueing it.
The interval ceiling also rejects previously harmless intervals, including an oversized interval with phase 0. Such a schedule fired at most once, with its next tick after the year 3939.
Operator action
Upgrade workers to a release containing this fix. No database migration is required.
On SQLite, inspect interval schedules with
every_secondsabove 62,135,596,800. Check stored start and end times against years 1 to 9999 in the database's time zone. CheckSCHEDULESentries too.Correct readable invalid rows in the admin change form or with
update_schedule, pause them, or delete them. Pausing does not repair a row. It cannot be enabled again until corrected.For the PostgreSQL year-10000 row, the admin changelist returns HTTP 500 and lists no rows, including healthy ones. The change form and Disable action also return HTTP 500.
update_schedule(row, enabled=False)raisesDataError. Delete the row withdelete_schedule(OxSchedule(pk=schedule_pk))or a plain SQLDELETE. The fixed writers refuse this bound with a field error, so such a row can only come from an affected release or SQL.Validation
create_scheduleaccepted an end time in year 10000 UTC on every release from 1.2.0 through 1.7.0. Constructing everyWorker()then raisedDataError. A running worker abandoned every dispatch pass. These runs used Django 6.0.8 and Python 3.12, not each release's own support matrix.USE_TZoff. In every tested case, the worker stayed alive, the queued task ran, and other schedules fired.Tests cover each behavior the change adds or alters.