Skip to content

fix(redis): Prevent counter corruption from concurrent mark handled in Redis RQ#1878

Merged
vdusek merged 1 commit intoapify:masterfrom
Mantisus:redis-fix-negative-pending
May 7, 2026
Merged

fix(redis): Prevent counter corruption from concurrent mark handled in Redis RQ#1878
vdusek merged 1 commit intoapify:masterfrom
Mantisus:redis-fix-negative-pending

Conversation

@Mantisus
Copy link
Copy Markdown
Collaborator

@Mantisus Mantisus commented May 6, 2026

Description

  • Fixes counter corruption on concurrent mark_request_as_handled calls for the same request. When two coroutines concurrently called mark_request_as_handled for the same request, both could pass the hexists check before either executed the pipeline, causing counters to be updated multiple times for the same request, breaking the queue operation.

Issues

Testing

  • Added a test to verify that counters are updated correctly during concurrent execution.
  • Added a test to verify that the request is correctly restored to in_progress after failure.

Checklist

  • CI passed

@Mantisus Mantisus self-assigned this May 6, 2026
@Mantisus Mantisus requested review from janbuchar and vdusek May 6, 2026 21:59
@vdusek vdusek changed the title fix: Fix counter corruption on concurrent RedisRequestQueueClient.mark_request_as_handled calls for the same request fix: Prevent counter corruption from concurrent mark handled in Redis RQ May 7, 2026
@vdusek vdusek changed the title fix: Prevent counter corruption from concurrent mark handled in Redis RQ fix(redis): Prevent counter corruption from concurrent mark handled in Redis RQ May 7, 2026
Comment on lines +439 to +440
# If we fail to mark the request as handled after removing it from in_progress, we restore request in
# `in_progress` hash.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And that will result in a retry, correct?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if it's a Redis-related error, it will trigger a retry using @retry_on_error(RedisError).
If something unexpected happens, the error will be propagated.

@vdusek vdusek merged commit 50d70f0 into apify:master May 7, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Negative pending request count when using RedisRequestQueueClient

4 participants