Repository navigation
fix(p2p): trim the simple priority queue on every push - #1596
Conversation
The simple priority queue trimmed its heap on push only when the signal channel was already pending. A push that refilled an empty signal skipped the trim, so with nobody draining, the heap ran past maxSize until the next 10ms tick. Instrumented under -race, a 2560-envelope flood at capacity 8 took the heap 7 to 18 envelopes past maxSize every time. TestLoadRouterSharedQueueOccupancyAndDrops checks that the queue retains no more than maxSize + cap(input) + cap(output) and failed when the drain caught that overshoot (85 and 87 retained against a ceiling of 84). The ceiling is right for the intended queue, so the queue now trims after every push and the ticker only re-signals. The test comment spells out why the envelope the run loop holds between input and heap cannot add to the count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
|
✅ Final review complete — no blockers (commit 4599c6a) · triage: low |
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 1 + Phase 2
The change correctly enforces the heap limit after every insertion while preserving delivery notifications and ticker-driven retries. One non-blocking performance suggestion remains concerning whole-queue sorting on every overflowing push. Verification was static at head 4599c6a; the supplied CI snapshot and current PR checks report successful build, test, lint, and vet checks.
🟡 1 suggestion(s)
Review provenance
Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
lowbygpt-6.1-sol(effort low) — The diff is a small, contained queue fix that makes trimming unconditional after each push, removes redundant ticker trimming, and clarifies a test comment. - Phase 1 reviewers:
glm-5.3-flash— general (completed, effort high); agentphase1-reviewer - Phase 1 model:
glm-5.3-flash— zai quota: 5h 93% left, weekly 48% left; passed overgemini-3.8-flash-high(antigravity quota lookup failed) - Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort medium); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort medium); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `internal/p2p/rqueue.go`:
- [SUGGESTION] internal/p2p/rqueue.go:81-86: Avoid whole-queue sorting for each overflowing push
The unconditional trim fixes the bound, but every overflowing insertion now invokes sort.Sort over the entire backlog to evict exactly one envelope. The shipped vote channel has RecvBufferCapacity 4096, making maxSize 16,777,216; a saturated insertion therefore sorts 16,777,217 entries. This adds whole-backlog sorting to the pushes that previously skipped trimming, increasing work precisely when the queue is backpressured. Consider an eviction structure that can locate and remove the lowest-ranked envelope without sorting the entire queue, while preserving the size/time ordering and NotifyDelivery behavior. This is non-blocking: sorting already existed, and O(n log n) is its worst-case bound, not a measured cost for each push; realistic saturation measurements should guide the optimization.
Out-of-scope follow-up suggestions (1)
These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.
- Revisit the simple-priority queue's squared capacity — The existing maxSize = size*size policy preallocates 16,777,216 pointer slots for the default vote queue alone—128 MiB on a 64-bit target, before envelope and message allocations. This also permits a very large backlog of untrusted inbound messages. The sizing policy predates this patch and is separate from enforcing its intended bound.
- Follow-up: Open a separate issue or follow-up PR to establish an explicit memory budget for simple-priority queues and evaluate byte-based capacity accounting.
| if len(pq) > q.maxSize { | ||
| sort.Sort(pq) | ||
| for _, dropped := range pq[q.maxSize:] { | ||
| dropped.envelope.NotifyDelivery() | ||
| } | ||
| pq = pq[:q.maxSize] |
There was a problem hiding this comment.
🟡 Suggestion: Avoid whole-queue sorting for each overflowing push
The unconditional trim fixes the bound, but every overflowing insertion now invokes sort.Sort over the entire backlog to evict exactly one envelope. The shipped vote channel has RecvBufferCapacity 4096, making maxSize 16,777,216; a saturated insertion therefore sorts 16,777,217 entries. This adds whole-backlog sorting to the pushes that previously skipped trimming, increasing work precisely when the queue is backpressured. Consider an eviction structure that can locate and remove the lowest-ranked envelope without sorting the entire queue, while preserving the size/time ordering and NotifyDelivery behavior. This is non-blocking: sorting already existed, and O(n log n) is its worst-case bound, not a measured cost for each push; realistic saturation measurements should guide the optimization.
source: glm-5.3-flash (phase1-reviewer: general)
|
Bots are done — your move: post |
Basic explanation
The router puts every inbound message for a channel into one shared queue, and that queue is supposed to hold at most
size*sizemessages in its heap, dropping the rest. It did not quite: it only trimmed the heap when a push found its wake-up signal already pending. A push that refilled an empty signal skipped the trim, so while nobody was draining, the heap could run past its limit until a 10ms ticker caught it.TestLoadRouterSharedQueueOccupancyAndDropspins that limit and failed intermittently in CI under-race("2560 envelopes offered, 85 retained (ceiling 84)"), including on #1582, which does not touchinternal/p2p. The test's ceiling was right; the queue was not holding to it. The queue now trims on every push.Detailed discussion
What was done
internal/p2p/rqueue.go:simpleQueue.runtrims the heap after every push instead of only in thedefaultarm of the signal send. The ticker's trim could no longer run and is removed; the ticker still re-signals so leftover envelopes are delivered.internal/p2p/load_router_queue_test.go: the comment on the ceiling now explains its arithmetic.maxSize + cap(input) + cap(output)covers the three places the queue holds envelopes. The one envelope the run loop holds between taking it from input and pushing it frees an input slot, but the push that follows trims the heap back tomaxSize, so it replaces an envelope already held rather than adding to the count.simple-priorityis the defaultp2p.queue-type, so this changes the default path: under a flood, the excess is dropped at push time instead of up to 10ms later. It also means every push that overflows the heap now sorts the whole heap, where the old code skipped the sort on some of them. The cost is measured under Testing.Testing
Run locally on macOS with
-race, capacity 8 as in the test (ceiling 84):-cpuWith temporary instrumentation (not committed) recording the heap's peak length, 200 floods of 2560 envelopes took the heap 7 to 18 envelopes past
maxSizeevery time before the change, and never past it after.The other queue and router tests in
internal/p2p(TestSimpleQueue,TestSimplePriorityQueueDoesNotPopWhenOutputIsFull, therouter_punishandrouter_inittests) pass under-race;gofmtandgo vetare clean.Cost when the queue is full
The vote channel's capacity of 4096 gives a limit of 16,777,216 envelopes, so sorting the heap is expensive there. These are scratch runs that are not committed: macOS on 30 cores, no
-race, empty votes, one run per cell over a 20-second window, and the queue already full at the start.Pushes accepted per second with nobody draining. This PR is about 1.3x to 1.6x slower here across two sets of runs:
Accepted and delivered per second with a consumer limited to 5,000 per second. There is no consistent difference here. Both that rate and the 50,000 per second flood are arbitrary choices:
The slowness itself predates this PR. #1597 addresses it and sets out the trade-off that comes with doing so.
Breaking changes
None.
Prior work
🤖 Generated with Claude Code
PR Hygiene ·
4599c6a/self-reviewedinternal/p2p/load_router_queue_test.go,internal/p2p/rqueue.go) — approved by lklimekWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.