perf(priority): use dedicated ZSET for prioritized jobs - #2818
Closed
papandreou wants to merge 1 commit into
Closed
Conversation
Adding a prioritized job used ZADD + ZCOUNT + LLEN + LINDEX/LINSERT to splice it into the 'wait' list at the right position. LINDEX/LINSERT on a Redis list is O(N), so addBulk against a queue with hundreds of thousands of waiting jobs turned into O(M*N) and pegged Redis CPU. Move prioritized jobs into their own 'prioritized' ZSET, scored by priority*2^32 + creation-order counter, so inserting one is a single O(log N) ZADD (mirrors taskforcesh/bullmq#1984). Jobs added before this change stay in 'wait' with entries in the legacy 'priority' ZSET and keep draining through the existing RPOPLPUSH/ZREM fallback path, so no migration step is required. Blocked workers previously woke up via BRPOPLPUSH(wait, active), which can't see jobs landing only in the new 'prioritized' ZSET. Add a dedicated 'marker' key that every job-adding script pushes to, and have Queue#getNextJob block on BRPOP(marker) instead, always re-fetching after waking (or timing out) so pre-existing/legacy backlogs are picked up too. The 'drained' event is now edge-triggered to prevent it firing more than once per idle transition.
Collaborator
|
hi @papandreou, instead of porting more bullmq functionalities into bull we are recommending users to migrate to bullmq. So pls consider it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We hit sustained high Redis CPU in production while
addBulkwas adding jobs into a queue that already held ~500K waiting jobs. I reproduced it locally with a standalone script that seeds a job backlog and times adding prioritized jobs on top. The per-jobadd()latency grows with backlog size, confirming an O(N) blowup:Root cause: Inserting a prioritized job used
ZADD+ZCOUNT+LLEN+LINDEX/LINSERTto splice it into the right spot in thewaitlist.LINDEX/LINSERTon a Redis list are O(N) (linked-list walk), so every prioritized insert against a large backlog turned into an expensive scan, andaddBulkof M jobs became O(M·N). A rediscovery of the findings in the Asymptotically faster priority jobs article from BullMQ.I had Claude help implement a fix modeled on how BullMQ solved the same problem (taskforcesh/bullmq#1984): prioritized jobs now live in their own
prioritizedZSET, scored bypriority * 2^32 + creation-order-counter, so inserting one is a single O(log N)ZADDinstead of an O(N) splice. Blocked workers previously woke viaBRPOPLPUSH(wait, active), which can't see jobs landing only in the new ZSET, so workers now block on a dedicatedmarkerkey instead and always re-check bothwaitandprioritizedon wake.Jobs already sitting in
waitfrom before this deploy keep draining through the existing fallback path (legacypriorityZSET +RPOPLPUSH/ZREM), so no migration step is required.