Skip to content

fix(p2p): trim the simple priority queue on every push - #1596

Merged
lklimek merged 1 commit into
devfrom
test/p2p-shared-queue-ceiling
Nov 9, 2026
Merged

lklimek merged 1 commit into
devfrom
test/p2p-shared-queue-ceiling

Conversation

@QuantumExplorer

@QuantumExplorer QuantumExplorer commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Basic explanation

The router puts every inbound message for a channel into one shared queue, and that queue is supposed to hold at most size*size messages in its heap, dropping the rest. It did not quite: it only trimmed the heap when a push found its wake-up signal already pending. A push that refilled an empty signal skipped the trim, so while nobody was draining, the heap could run past its limit until a 10ms ticker caught it.

TestLoadRouterSharedQueueOccupancyAndDrops pins that limit and failed intermittently in CI under -race ("2560 envelopes offered, 85 retained (ceiling 84)"), including on #1582, which does not touch internal/p2p. The test's ceiling was right; the queue was not holding to it. The queue now trims on every push.

Detailed discussion

What was done

  • internal/p2p/rqueue.go: simpleQueue.run trims the heap after every push instead of only in the default arm of the signal send. The ticker's trim could no longer run and is removed; the ticker still re-signals so leftover envelopes are delivered.
  • internal/p2p/load_router_queue_test.go: the comment on the ceiling now explains its arithmetic. maxSize + cap(input) + cap(output) covers the three places the queue holds envelopes. The one envelope the run loop holds between taking it from input and pushing it frees an input slot, but the push that follows trims the heap back to maxSize, so it replaces an envelope already held rather than adding to the count.

simple-priority is the default p2p.queue-type, so this changes the default path: under a flood, the excess is dropped at push time instead of up to 10ms later. It also means every push that overflows the heap now sorts the whole heap, where the old code skipped the sort on some of them. The cost is measured under Testing.

Testing

Run locally on macOS with -race, capacity 8 as in the test (ceiling 84):

Code -cpu Runs Failures Max retained
before default 200 0 84 (13 runs)
before 2 300 1 87
after default 500 0 84 (77 runs)
after 2 300 0 84

With temporary instrumentation (not committed) recording the heap's peak length, 200 floods of 2560 envelopes took the heap 7 to 18 envelopes past maxSize every time before the change, and never past it after.

The other queue and router tests in internal/p2p (TestSimpleQueue, TestSimplePriorityQueueDoesNotPopWhenOutputIsFull, the router_punish and router_init tests) pass under -race; gofmt and go vet are clean.

Cost when the queue is full

The vote channel's capacity of 4096 gives a limit of 16,777,216 envelopes, so sorting the heap is expensive there. These are scratch runs that are not committed: macOS on 30 cores, no -race, empty votes, one run per cell over a 20-second window, and the queue already full at the start.

Pushes accepted per second with nobody draining. This PR is about 1.3x to 1.6x slower here across two sets of runs:

Capacity Limit Before After
128 16,384 1,350 989
1024 1,048,576 13.7 9.5
4096 16,777,216 0.4 (9 pushes) 0.3 (7 pushes)

Accepted and delivered per second with a consumer limited to 5,000 per second. There is no consistent difference here. Both that rate and the 50,000 per second flood are arbitrary choices:

Capacity Flood Before After
1024 unlimited 2,685 / 2,696 2,666 / 2,677
4096 unlimited 313 / 411 296 / 391
1024 50,000/s 3,230 / 3,220 3,348 / 3,351
4096 50,000/s 316 / 411 549 / 642

The slowness itself predates this PR. #1597 addresses it and sets out the trade-off that comes with doing so.

Breaking changes

None.

Prior work

🤖 Generated with Claude Code

PR Hygiene · 4599c6a

  • Bots — thepastaclaw ✓
  • Self-review — post /self-reviewed
  • Build green
  • Approvals
    • files with no dedicated owner (internal/p2p/load_router_queue_test.go, internal/p2p/rqueue.go) — approved by lklimek

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

The simple priority queue trimmed its heap on push only when the signal
channel was already pending. A push that refilled an empty signal skipped
the trim, so with nobody draining, the heap ran past maxSize until the
next 10ms tick. Instrumented under -race, a 2560-envelope flood at
capacity 8 took the heap 7 to 18 envelopes past maxSize every time.

TestLoadRouterSharedQueueOccupancyAndDrops checks that the queue retains
no more than maxSize + cap(input) + cap(output) and failed when the drain
caught that overshoot (85 and 87 retained against a ceiling of 84). The
ceiling is right for the intended queue, so the queue now trims after
every push and the ticker only re-signals. The test comment spells out
why the envelope the run loop holds between input and heap cannot add to
the count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thepastaclaw

thepastaclaw commented Nov 5, 2026 •

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit 4599c6a) · triage: low

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

The change correctly enforces the heap limit after every insertion while preserving delivery notifications and ticker-driven retries. One non-blocking performance suggestion remains concerning whole-queue sorting on every overflowing push. Verification was static at head 4599c6a; the supplied CI snapshot and current PR checks report successful build, test, lint, and vet checks.

🟡 1 suggestion(s)

Review provenance

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.1-sol (effort low) — The diff is a small, contained queue fix that makes trimming unconditional after each push, removes redundant ticker trimming, and clarifies a test comment.
  • Phase 1 reviewers: glm-5.3-flash — general (completed, effort high); agent phase1-reviewer
  • Phase 1 model: glm-5.3-flash — zai quota: 5h 93% left, weekly 48% left; passed over gemini-3.8-flash-high (antigravity quota lookup failed)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort medium); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `internal/p2p/rqueue.go`:
- [SUGGESTION] internal/p2p/rqueue.go:81-86: Avoid whole-queue sorting for each overflowing push
  The unconditional trim fixes the bound, but every overflowing insertion now invokes sort.Sort over the entire backlog to evict exactly one envelope. The shipped vote channel has RecvBufferCapacity 4096, making maxSize 16,777,216; a saturated insertion therefore sorts 16,777,217 entries. This adds whole-backlog sorting to the pushes that previously skipped trimming, increasing work precisely when the queue is backpressured. Consider an eviction structure that can locate and remove the lowest-ranked envelope without sorting the entire queue, while preserving the size/time ordering and NotifyDelivery behavior. This is non-blocking: sorting already existed, and O(n log n) is its worst-case bound, not a measured cost for each push; realistic saturation measurements should guide the optimization.
Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

  • Revisit the simple-priority queue's squared capacity — The existing maxSize = size*size policy preallocates 16,777,216 pointer slots for the default vote queue alone—128 MiB on a 64-bit target, before envelope and message allocations. This also permits a very large backlog of untrusted inbound messages. The sizing policy predates this patch and is separate from enforcing its intended bound.
    • Follow-up: Open a separate issue or follow-up PR to establish an explicit memory budget for simple-priority queues and evaluate byte-based capacity accounting.

Comment thread internal/p2p/rqueue.go
Comment on lines 81 to 86
if len(pq) > q.maxSize {
sort.Sort(pq)
for _, dropped := range pq[q.maxSize:] {
dropped.envelope.NotifyDelivery()
}
pq = pq[:q.maxSize]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: Avoid whole-queue sorting for each overflowing push

The unconditional trim fixes the bound, but every overflowing insertion now invokes sort.Sort over the entire backlog to evict exactly one envelope. The shipped vote channel has RecvBufferCapacity 4096, making maxSize 16,777,216; a saturated insertion therefore sorts 16,777,217 entries. This adds whole-backlog sorting to the pushes that previously skipped trimming, increasing work precisely when the queue is backpressured. Consider an eviction structure that can locate and remove the lowest-ranked envelope without sorting the entire queue, while preserving the size/time ordering and NotifyDelivery behavior. This is non-blocking: sorting already existed, and O(n log n) is its worst-case bound, not a measured cost for each push; realistic saturation measurements should guide the optimization.

source: glm-5.3-flash (phase1-reviewer: general)

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-self-review Waiting for the author to post /self-reviewed label Nov 6, 2026
@lklimek lklimek added this to the v2.0 milestone Nov 6, 2026
@lklimek
lklimek enabled auto-merge (squash) November 9, 2026 08:23
@lklimek
lklimek disabled auto-merge November 9, 2026 14:58
@lklimek
lklimek merged commit 0e08aa0 into dev Nov 9, 2026
23 of 24 checks passed
@lklimek
lklimek deleted the test/p2p-shared-queue-ceiling branch November 9, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants