Skip to content

perf(priority): use dedicated ZSET for prioritized jobs - #2818

Closed
papandreou wants to merge 1 commit into
OptimalBits:developfrom
papandreou:feature/cheaperAddPrioritizedJob
Closed

perf(priority): use dedicated ZSET for prioritized jobs#2818
papandreou wants to merge 1 commit into
OptimalBits:developfrom
papandreou:feature/cheaperAddPrioritizedJob

Conversation

@papandreou

Copy link
Copy Markdown
Contributor

We hit sustained high Redis CPU in production while addBulk was adding jobs into a queue that already held ~500K waiting jobs. I reproduced it locally with a standalone script that seeds a job backlog and times adding prioritized jobs on top. The per-job add() latency grows with backlog size, confirming an O(N) blowup:

Backlog size Before (ms/job) After (ms/job)
0 0.32 0.33
10,000 0.53 0.25
50,000 1.33 0.22
100,000 2.35 0.20
200,000 4.71 0.31
400,000 8.70 0.31

Root cause: Inserting a prioritized job used ZADD + ZCOUNT + LLEN + LINDEX/LINSERT to splice it into the right spot in the wait list. LINDEX/LINSERT on a Redis list are O(N) (linked-list walk), so every prioritized insert against a large backlog turned into an expensive scan, and addBulk of M jobs became O(M·N). A rediscovery of the findings in the Asymptotically faster priority jobs article from BullMQ.

I had Claude help implement a fix modeled on how BullMQ solved the same problem (taskforcesh/bullmq#1984): prioritized jobs now live in their own prioritized ZSET, scored by priority * 2^32 + creation-order-counter, so inserting one is a single O(log N) ZADD instead of an O(N) splice. Blocked workers previously woke via BRPOPLPUSH(wait, active), which can't see jobs landing only in the new ZSET, so workers now block on a dedicated marker key instead and always re-check both wait and prioritized on wake.

Jobs already sitting in wait from before this deploy keep draining through the existing fallback path (legacy priority ZSET + RPOPLPUSH/ZREM), so no migration step is required.

Adding a prioritized job used ZADD + ZCOUNT + LLEN + LINDEX/LINSERT to
splice it into the 'wait' list at the right position. LINDEX/LINSERT
on a Redis list is O(N), so addBulk against a queue with hundreds of
thousands of waiting jobs turned into O(M*N) and pegged Redis CPU.

Move prioritized jobs into their own 'prioritized' ZSET, scored by
priority*2^32 + creation-order counter, so inserting one is a single

O(log N) ZADD (mirrors taskforcesh/bullmq#1984). Jobs added before
this change stay in 'wait' with entries in the legacy 'priority' ZSET
and keep draining through the existing RPOPLPUSH/ZREM fallback path,
so no migration step is required.

Blocked workers previously woke up via BRPOPLPUSH(wait, active), which
can't see jobs landing only in the new 'prioritized' ZSET. Add a
dedicated 'marker' key that every job-adding script pushes to, and
have Queue#getNextJob block on BRPOP(marker) instead, always
re-fetching after waking (or timing out) so pre-existing/legacy
backlogs are picked up too. The 'drained' event is now edge-triggered
to prevent it firing more than once per idle transition.
@roggervalf

Copy link
Copy Markdown
Collaborator

hi @papandreou, instead of porting more bullmq functionalities into bull we are recommending users to migrate to bullmq. So pls consider it

@roggervalf roggervalf closed this Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants