Repository navigation
Fail tasks whose stored retry callback is gone from the code base - #87
Merged
Merged
Conversation
The running reaper skipped a payload it could not deserialize. The claim script had already renewed the lease, so the task stayed in the running set and the broker logged the same ImportError every CLAIM_TTL. The inspector could not list or clear it. The reaper now fails such a task. It drops the unimportable retry callback, records an ImportError beside the lease timeout, and acknowledges the failure through the regular result path. The task leaves the running set, shows on the Failed tab, and can be requeued or dropped.
CONTRIBUTING.md says we do not guard against Redis data altered mid-flight, we fail loudly instead. The helper reads the task hash right after the claim returned that same key, so the guard is dead code that hides a failure instead of surfacing it.
Private methods in this module carry a single summary line. The explanation of the payload rewrite belongs in the user docs, which already describe the terminal failure.
The handler only had one caller and passing the ImportError as an argument read awkwardly. It now runs where the exception is caught.
codingjoe
added a commit
that referenced
this pull request
Oct 7, 2026
The reaper read the stored payload before deciding the fate of a task, so a payload the code can no longer read, a task function or retry callback that is gone, or a status a newer release wrote, raised on every pass. The claim script had already renewed the lease, so the task stayed in the running set and the same error repeated every `CLAIM_TTL`, and the inspector could not clear it. - Drop a claimed task the broker cannot read: log the read error with its traceback, and delete the stored payload. - Delete only the payload, not the running entry. A lease holder's late acknowledge still writes its result, and the reaper script sweeps the entry once the hash is gone. - Replace the import-error salvage from #87, which patched and published the payload, with that one drop. The source file is a net deletion. - Document the behavior in the README. Fixes #73
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_reap_running_queueskipped a payload whose stored retry callback was gone from the code base. The claim script had already renewed the lease, so the task stayed in the running set and the broker logged the sameImportErroreveryCLAIM_TTL. The inspector could not list or clear it.ImportErrorbeside theAcknowledgementTimeout, and acknowledge the failure through the regular result path.Fix #78