Repository navigation
fix(dpns): stop re-reading finished name contests on every refresh - #1066
Conversation
A contest refresh pass asked Platform for the contenders of every contest the app had ever seen, finished ones included: one request per contest, about a thousand on Testnet, 24 at a time. Since #901 that pass also runs on arrival at Masternodes > Votes and from a background timer, so it fired every few minutes. The SDK sends requests to five endpoints at a time and each allows 150 a minute; one pass went past that, the endpoints answered ResourceExhausted and were banned, and every other action shared the loss. - A stored contest that already has its outcome (awarded or locked) is final: a refresh that lists it stamps it as current and no longer queues a contender query for it. A pass now costs the contest list plus the contests that are still open. - Only one refresh pass runs at a time. The timer and a user refresh used to overlap and each walk the same unfilled cache. - Contender queries of a pass start no closer than 125 ms apart, so the first load of a long history stays inside the request budget instead of arriving as one burst. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw
Pacing the contender queries made the first load of a long contest history take about two minutes, with the contests that can still be voted on sitting at a random alphabetical position in that queue. - The pass now waits for the contest end times (a query it already made) before it starts the contender queries, and reads the contests whose end time is in the future first. The unread history is back-filled after them. - The first 100 contender queries of a pass start at once; only the ones after them are paced. With 480 paced queries a minute that keeps a pass within 580 of the roughly 750 requests a minute the app has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 11 minutes. View limit details
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to Finished contests are skipped on later refreshes, while incomplete refreshes are retried. No merge-blocking issue is established; normal checks remain appropriate. Pre-merge checks |
|
|
🕓 Review not started yet because the new head is waiting for the 30-minute push debounce.
Commit 3056222. Normal review starts when eligible; priority review starts as soon as a slot is available. |
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 2 only (no Phase 1 for this repository)
The finished-contest cache filter and whole-pass refresh lock reduce repeated work, and prioritizing open contests improves cold-cache loading. However, the new pacing limits task creation rather than network-query admission, so backlog recovery can still exceed the endpoint request budget and undermine the PR's connection-safety goal. Verification was static; the supplied draft-PR CI snapshot shows Test Suite and Clippy skipped.
🔴 1 blocking
Review provenance
Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — The change modifies asynchronous contest-refresh scheduling, cache eligibility, and concurrency control across backend and context code, but does not change a critical surface such as consensus, funds movement, cryptography, or storage migrations. - Phase 1 reviewers: not run (disabled for this repository)
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `src/backend_task/contested_names/query_dpns_contested_resources.rs`:
- [BLOCKING] src/backend_task/contested_names/query_dpns_contested_resources.rs:102-105: Pace actual contender requests rather than queued task creation
`pace.tick()` spaces calls to `start(name)`, but that callback only spawns a task. Each task then waits for the 24-permit semaphore at lines 259–270 before entering `query_dpns_vote_contenders`. Sustained slow responses therefore build a queue behind the semaphore: replies taking about eight seconds admit only about three queries per second while this loop continues creating eight tasks per second, leaving hundreds queued during a long cold-cache load. If responses recover, those queued tasks start requests as quickly as permits become available, alongside newly scheduled work, without observing the 125 ms spacing. This can exceed the stated 750-request allowance in a later clock minute. The whole-pass mutex prevents overlapping passes but does not constrain this burst within one pass.
Acquire concurrency capacity in the dispatcher before consuming a pacing slot and launching the query, or enforce a shared pacing gate after semaphore acquisition immediately before the fetch. Add a regression test that holds requests in flight while paced tasks accumulate, then releases them and records actual fetch-start times. The existing elapsed-time test uses immediately failing mock queries and only proves that task creation takes long enough; it cannot detect backlog recovery bursts.
|
Waiting for bot review — coderabbitai not yet · thepastaclaw requested changes — dismiss the review or push a fix, 1 thread unresolved — resolve it. Wait for the missing reviews, or a writer can post |
|
Waiting for bot review — coderabbitai not yet · thepastaclaw requested changes — dismiss the review or push a fix, 1 thread unresolved — resolve it. Wait for the missing reviews, or a writer can post |
|
Waiting for bot review — coderabbitai not yet · thepastaclaw requested changes — dismiss the review or push a fix, 1 thread unresolved — resolve it. Wait for the missing reviews, or a writer can post |
|
Waiting for bot review — coderabbitai not yet · thepastaclaw requested changes — dismiss the review or push a fix, 1 thread unresolved — resolve it. Wait for the missing reviews, or a writer can post |
|
Waiting for bot review — coderabbitai not yet · thepastaclaw requested changes — dismiss the review or push a fix, 1 thread unresolved — resolve it. Wait for the missing reviews, or a writer can post |
The pace only spaced out the spawning of contender queries; each spawned query then waited for one of the 24 slots on its own. While answers were slow the pace kept spawning, so queries piled up behind the slots and all started at once when the answers arrived, well above the request allowance. The dispatcher now takes the slot first and waits for the pace second, so a query starts as soon as it is spawned and no sooner than one pace after the query before it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw
|
Waiting for bot review — coderabbitai not yet · thepastaclaw ✓. Wait for the missing reviews, or a writer can post |
thepastaclaw
left a comment
There was a problem hiding this comment.
Re-review — Final validation — Phase 2 only (no Phase 1 for this repository)
Reviewed the complete diff at head 6ac2ed3 and verified the refresh, persistence, dispatch, and regression-test claims against the source. No actionable in-scope findings remain; the prior semaphore-backlog pacing defect is fixed. This was a static review with no builds or tests executed; Clippy and Test Suite were still pending in the supplied CI snapshot.
🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)
Review provenance
Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — The diff changes asynchronous contest refresh scheduling, serialization, pacing, and cache eligibility across backend and context code, but does not change a critical surface such as consensus, funds movement, key handling, network deserialization, or storage migrations. - Phase 1 reviewers: not run (disabled for this repository)
- Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.
No unresolved findings remain from the prior review on this head.
|
Ready for review — files with no dedicated owner: lklimek. |
Conflict in CHANGELOG.md: both entries kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw
|
Waiting for bot review — coderabbitai ✓ · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
* fix(dpns): let operators vote while a long contest history loads On a first load the voting choices and the Votes badge waited for the whole paced back-fill, about two minutes for a thousand contests: both need each node's proved votes, and the pass read those only after its last contender query. Reading them earlier was not safe as it stood. A listed contest nobody has read yet has no end time and counts as open, so every unread finished contest would have shown as waiting for a vote. - While a pass runs, the names it has listed but not read, and has no end time to call open, are left out of the open contests. Each returns as soon as its contenders arrive, and whatever is left returns when the pass ends, however it ends. The set lives in memory only. - When a pass has such unread history, it proves the nodes' votes right after the open contests are read and refreshes the screen, then back-fills. The proof at the end of the pass is kept. A pass on a filled cache has nothing unread, takes neither step and sends the same requests as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw * fix(dpns): keep unread contest names out of the open list after a restart The names a first load had listed but not read were kept out of the open contests by a set held in memory for the length of the pass. The early vote proof, however, is stored. Quitting during a first load and starting again within the proof's lifetime — or a pass that ended with names still unread — left every unread name counted as an open contest the node had not voted on: a badge in the hundreds and enabled choices on finished contests. - A name that was only listed (never read, no end time, no contenders) is not an open contest, wherever the open contests are read. The rule lives on the contest itself, so it holds in a new process and after any pass. The in-memory set, its guard and its lock are gone. - While a pass is running and unread names remain, To decide and Voted say the contests are still loading instead of saying none are open. Tests that seeded such a bare name as an open contest now give it an end time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw * test(dpns): hold the contest back-fill by its reports instead of by time The first-load test sampled the vote state for a second and relied on the paced back-fill outlasting that. A loaded machine could stretch the sampling past the end of the pass and fail the test. The pass now reports to a channel with room for one report, so it gets only as far as the reports the test takes: the proof is awaited report by report and the back-fill stays held while the vote state is checked. A second test answers every contender read. It checks that the votes are not proved before the read of the open contest has finished, and that history read to its outcome does not show as waiting for a vote while the rest is still unread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw * test(dpns): take a fixed number of pass reports instead of polling for the proof The helper took reports until the stored proof became visible. Another worker could publish the proof right after an earlier report was taken, so the helper returned with a report still filling the one-slot channel. The pass then stayed blocked on its proof report and never began the back-fill, and the answered-history test failed with "no history was read". A pass sends its reports in a fixed order. Both tests now take exactly the reports up to and including the proof report, which puts the pass in the back-fill whichever task ran first, and then check the vote state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw --------- Co-authored-by: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — coderabbitai not yet · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
|
Waiting for bot review — coderabbitai not yet · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
|
/skip-bots |
Conflict in CHANGELOG.md: both entries kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw
|
Waiting for bot review — coderabbitai not yet · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
|
/skip-bots |
TL;DR
With a voting node loaded, the app re-read every name contest it had ever seen — finished ones included — each time contests were refreshed. On Testnet that is about a thousand requests in ten seconds, and the refresh runs by itself when Masternodes → Votes is opened and every few minutes in the background. The servers answered by refusing the app for a while, which could fail unrelated actions. Finished contests are now read once, and contests that are still open are read before the rest.
User story
As a masternode operator, I want to open my voting screen and its history as often as I like, so that checking past contests never makes the app lose its connection to the network for everything else I am doing.
Scenario
A Testnet profile with one wallet, several identities and one Evonode loaded with its voting key. Open Masternodes, choose Votes, then History. Leave the app open.
Actual behavior
Opening Votes starts a refresh that asks the network about every past contest again, roughly a thousand requests at once. The same refresh repeats every three minutes in the background on Testnet, and once more shortly after launch. Each time, the five servers the app is currently using reach their request limit together and are set aside; the Network screen shows fewer servers available (22 of 29 in the report). Nothing visibly fails on the Votes screen, but every other action draws on the same servers.
Expected behavior
A refresh asks only about contests that can still change: the ones still open and the ones not read yet. History keeps showing every finished contest with its outcome, and its "Last Updated" column still moves forward after a refresh. No server reaches its limit and none is set aside, however often Votes or History is opened.
The first time a long history is loaded, the contests still open are read first and appear within seconds; the finished ones fill in gradually over a couple of minutes instead of arriving in one burst. The voting choices and the count on the Votes badge are available as soon as the open contests have been read, while the history is still loading.
Detailed discussion
Root cause
The History view itself sends nothing. The cost is in the contest refresh pass (
query_dpns_contested_resources), which bothQueryDPNSContestsandRefreshContestsInBackgroundrun:insert_name_contests_as_normalized_namesreturned every stored name whose record was older than 30 s, whatever its state;So a pass cost one request per contest ever seen. That code is unchanged since before the last weekly build, but there only buttons started it. #901 added two automatic triggers: arrival at Masternodes → Votes (when the last completed refresh is at least 60 s old) and a background timer (3 min off Mainnet, 30 min on Mainnet) that runs while a voting node is loaded. Nothing kept two passes from overlapping.
The SDK rotates over a sticky set of five endpoints, and each allows 150 requests per clock minute, so the app has about 750 requests a minute. One pass on Testnet sent about 1,035. In the QA log every burst shows exactly five endpoints rate-limited at the same instant, with all in-flight requests spread evenly across them.
The change
AppContextgainsdpns_contest_refresh_pass, held for a whole pass and taken before any other lock. A second pass waits and then finds the cache filled.contender_queueputs the contests whose end time is in the future at the head of the queue. If the end times cannot be read, the previous order is used.start_in_orderstarts the first 100 contender queries of a pass at once and each later one at least 125 ms after the one before: at most 100 + 480 in a pass's first minute, inside the budget. The dispatcher takes one of the 24 concurrency slots before it waits for the pace, so a query that had to wait for slow answers still starts no sooner than 125 ms after the one before it. A pass with only a handful of open contests is not spaced at all.A pass now costs the contest list (one request per 100 contests), the end-times query, one request per contest that is still open or unread, and the unchanged per-node vote queries.
Not changed: the Scheduled view, the refresh triggers and their intervals. To decide, Voted and the attention badge change only as described under "First load of a long history".
First load of a long history
On an empty cache the first load still needs one request per contest. For about a thousand contests (estimates, not measured live):
The voting choices and the badge do not wait for the whole load. This part was reviewed as #1067 and is merged into this branch:
The limits and tests of this part are listed in #1067. Every later refresh takes seconds.
Testing
New unit tests, each confirmed failing before its fix:
a_long_finished_history_queues_only_open_and_new_contests— 600 finished contests, one open and one new are listed; exactly two are queued (before: all 602).a_listed_finished_contest_reads_as_just_updated— a listed finished contest is stamped as current and its outcome, end time and contenders are untouched.a_second_refresh_pass_waits_for_the_running_one— a pass does not run while another holds the gate.open_contests_are_dispatched_before_unread_history— on a cold cache with 400 listed names, the 20 open ones scattered among them are dispatched first, in listing order; asserted on the dispatch record, not on time.contender_queries_within_the_allowance_start_at_once— queries within the allowance do not wait for the pace.contender_queries_past_the_allowance_are_spaced_out— a real pass on a mock SDK with 106 new names spaces the six past the allowance.queries_that_waited_for_slow_answers_start_a_pace_apart— with every slot held by a slow answer and eight more queries waiting, each waiting query starts at least one pace after the one before it once answers arrive; asserted on recorded start times.Run locally:
cargo test --all-features --lib -- contested_names dpns masternodes(553 passed),cargo clippy --all-features --lib --profile test -- -D warnings,cargo fmt --all -- --check.Live check on Testnet (debug build of this branch, fresh profiles with an empty contest history, 3 sessions): no rate-limit errors and no endpoint bans in the log, against about 100 and 25 per refresh on
v1.0-dev. The first refresh started by itself about 38 s after launch and took a little over 2 minutes; a later one took about 40 s; no periodic bursts followed. During that first refresh the finished contests were listed under "To decide" with their choices disabled and the open contest could not be voted on for about 2 minutes, after which the list cleared by itself.Not fixed here: a fetch on Tools → Platform info started while that first refresh is running still shows no result (3 of 3), although nothing is rate limited any more. That loss has a separate cause and is being handled separately.
Not verified: the request rate itself. The application log has no line for a successful refresh, so the figures in the table still come from the code; the live check only shows that the limit was never hit. The ordering test composes the two units the pass uses; it does not run a whole pass, because the mock SDK cannot answer the time-dependent end-times query.
🤖 Generated with Claude Code
https://claude.ai/code/session_012YneRVSNLhVnGiedWDr3fw
🤖 Co-authored by Claudius the Magnificent AI Agent
PR Hygiene ·
3056222CHANGELOG.md,src/backend_task/contested_names/query_dpns_contested_resources.rs,src/context/contested_names_db.rsand 5 more) — lklimekWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.Summary by CodeRabbit