Skip to content

Keep invalid schedules from stopping workers and validate writes - #120

Merged
PhiLily merged 1 commit into
mainfrom
fix/schedule-safety
Oct 4, 2026
Merged

PhiLily merged 1 commit into
mainfrom
fix/schedule-safety

Conversation

@PhiLily

@PhiLily PhiLily commented Oct 4, 2026

Copy link
Copy Markdown
Member

Summary

Keep invalid stored schedules from stopping workers, other schedules or queued tasks. Validate schedule writes, isolate unreadable rows, and allow readable rows that fail validation to be paused.

Worker-stopping paths in versions 1.2.0-1.7.0 include the following:

  • An interval above 62,135,596,800 seconds, combined with certain phases, ends every worker reading it on its first dispatch pass. Each restart fails again.
  • On PostgreSQL, a stored bound outside the supported date range can prevent every worker from starting. It can also stop every dispatch pass of a running worker.

Both stop other schedules from firing. The interval path also stops queued-task processing. Tasks already queued still ran in the PostgreSQL bound reproduction.

Who is affected

  • SQLite deployments with a triggering stored interval. Write paths include create_schedule, update_schedule, admin add or change access, and direct table writes.
  • Deployments with a triggering SCHEDULES entry, regardless of database. This path requires settings access and fails before database access.
  • PostgreSQL deployments with out-of-range stored schedule bounds. In versions 1.2.0-1.7.0, create_schedule and update_schedule accepted bounds in year 10000 UTC, such as an end time late on 9999-12-31 in a zone west of UTC.
  • Deployments with unreadable or incomparable stored schedule values.
  • Callers that need to pause readable rows that fail validation.
  • Deployments with previously harmless intervals above the new ceiling. These are also rejected or skipped.

PostgreSQL, MySQL and MariaDB columns reject the triggering interval value. Those conclusions come from code inspection, not interval-overflow reproductions on those engines.

SQLite and MySQL refuse to store the out-of-range bound used in the PostgreSQL reproduction. This is separate from the SQLite interval overflow.

Changes

Validate schedule values

  • Cap every_seconds at 62,135,596,800 on every write path.
  • Reject start and end bounds outside years 1 to 9999 in the database's time zone. Reject times with a time zone when USE_TZ is off. Apply these checks in create_schedule, update_schedule and create_schedules.
  • Refuse oversized SCHEDULES intervals at manage.py check and worker start with django_ox.E002.
  • Report settings every or phase values outside Python's timedelta range as django_ox.E002, not OverflowError.
  • Cap phase_seconds and starting_deadline_seconds at 86,399,999,999,999 seconds. SQLite previously accepted larger values that were skipped at every read.

Isolate invalid stored rows

Skip and log rows the worker cannot read or compare. This includes driver conversion failures, impossible dates, out-of-range bounds, PostgreSQL infinity or BC dates, and MySQL zero dates. Skip intervals above the ceiling and phases that could put a tick before year 1.

Report ticks that cannot be derived and bounds that cannot be compared as schedule_dispatch_error, per schedule, for any schedule source. Database outages still stop the pass. They are not treated as bad rows.

If the change marker is unreadable, read the table in full at each SCHEDULE_RECONCILE_INTERVAL, which defaults to 60 seconds. Log skipped rows once, then at most once a minute per row per worker while they remain skipped.

Allow pausing without repair

Allow update_schedule(row, enabled=False) with no other changes to pause a readable row that fails validation. The admin Disable action also works on these rows instead of returning HTTP 500, as it did in 1.7.0 for rows that failed validation.

Pausing does not repair the row. Enabling it again is refused until it is corrected. "Run once now" reports an invalid stored row as not runnable instead of enqueueing it.

The interval ceiling also rejects previously harmless intervals, including an oversized interval with phase 0. Such a schedule fired at most once, with its next tick after the year 3939.

Operator action

Upgrade workers to a release containing this fix. No database migration is required.

On SQLite, inspect interval schedules with every_seconds above 62,135,596,800. Check stored start and end times against years 1 to 9999 in the database's time zone. Check SCHEDULES entries too.

Correct readable invalid rows in the admin change form or with update_schedule, pause them, or delete them. Pausing does not repair a row. It cannot be enabled again until corrected.

For the PostgreSQL year-10000 row, the admin changelist returns HTTP 500 and lists no rows, including healthy ones. The change form and Disable action also return HTTP 500. update_schedule(row, enabled=False) raises DataError. Delete the row with delete_schedule(OxSchedule(pk=schedule_pk)) or a plain SQL DELETE. The fixed writers refuse this bound with a field error, so such a row can only come from an affected release or SQL.

Validation

  • The interval overflow was reproduced for stored rows and settings entries on every release from 1.2.0 through 1.7.0. These runs used SQLite, Django 6.1.1 and Python 3.14.6, not each release's own support matrix.
  • On PostgreSQL, create_schedule accepted an end time in year 10000 UTC on every release from 1.2.0 through 1.7.0. Constructing every Worker() then raised DataError. A running worker abandoned every dispatch pass. These runs used Django 6.0.8 and Python 3.12, not each release's own support matrix.
  • Fix tests and runs on SQLite, PostgreSQL and MySQL covered invalid stored shapes. Cases included SQLite text in integer columns, impossible dates, PostgreSQL infinity, -infinity, BC and year 10000, MySQL zero dates, and offset bounds with USE_TZ off. In every tested case, the worker stayed alive, the queued task ran, and other schedules fired.

Tests cover each behavior the change adds or alters.

Reject invalid schedule values before writing them. Keep unreadable
stored rows from stopping workers, other schedules or queued tasks.

Worker-stopping paths in versions 1.2.0-1.7.0 include the following. An interval above
62,135,596,800 seconds, combined with certain phases, ends every
worker reading it.
Restarting repeats the failure.

Stored intervals can trigger this on SQLite. PostgreSQL, MySQL and
MariaDB columns reject intervals this large. SCHEDULES entries fail
before database access and are not limited to SQLite. The interval
overflow was reproduced on SQLite. The other database conclusions
for that interval come from code inspection.

On PostgreSQL, an out-of-range stored bound can prevent every worker
from starting and stop every dispatch pass of a running worker.
Tasks already queued still ran in the PostgreSQL reproduction.
An end time late on 9999-12-31 in a zone west of UTC, which is year
10000 in UTC, was accepted through create_schedule on every release
from 1.2.0 through 1.7.0. Each Worker() then raised DataError during
construction.

Limit every_seconds to 62,135,596,800 on every write path. Reject
bounds outside years 1 to 9999 in the database's time zone. Reject
times with a time zone when USE_TZ is off.

Skip and log stored rows that cannot be read or compared. Isolate
tick and bound errors per schedule as schedule_dispatch_error.
Database outages still stop the pass. Fall back to full table reads
at SCHEDULE_RECONCILE_INTERVAL when the change marker is unreadable.
Log skipped rows once, then at most once a minute per row per worker.

Refuse oversized SCHEDULES intervals at check and worker start with
django_ox.E002. Use E002 for every or phase outside timedelta's range.
Cap phase_seconds and starting_deadline_seconds at
86,399,999,999,999.

Allow update_schedule(row, enabled=False) with no other changes to
pause a readable row that fails validation. Allow the admin Disable
action for these rows. In 1.7.0, that action returned HTTP 500 for rows
that failed validation. Pausing does not repair a row. Refuse to enable
it until it is corrected. Report invalid rows as not runnable for
"Run once now".

Upgrade workers to a release containing this fix. Check SQLite intervals
above the ceiling, stored bounds in the database's time zone, and
SCHEDULES entries. Correct or pause readable invalid rows, or delete
them. The PostgreSQL year-10000 row cannot be corrected or disabled
through the admin or update_schedule. It also makes the admin
changelist fail for all rows. Delete it with
delete_schedule(OxSchedule(pk=schedule_pk)) or a plain SQL DELETE.
The ceiling also rejects previously harmless oversized intervals.
No migration is required.

Tests cover each behavior the change adds or alters.
@PhiLily
PhiLily merged commit 5d8d85a into main Oct 4, 2026
35 checks passed
@PhiLily
PhiLily deleted the fix/schedule-safety branch October 4, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant