Repository navigation
Conversation
Add attempt-scoped worker admission, cancellation and finalization to BaseService. Migrate consensus, WAL, file groups, ticker and gossip; join child services and preserve durable commit completion. Co-Authored-By: OpenAI GPT-6 <noreply@openai.com> <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Track Router, PEX and MConnection work through BaseService; join connection, queue, subscription and request workers at their owners. Preserve the parent context for post-stop error callbacks and expose explicit rate-limit cleanup. Co-Authored-By: Codex GPT-6 <noreply@openai.com>
Migrate service workers and child ownership to the shared lifecycle. Preserve application finalization during block-sync handover and node shutdown, drain event observers before closing sinks, and keep failed startup interruptible. Co-Authored-By: Codex <noreply@openai.com> <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Close recheck admission under the mempool mutex and wait outside it for active and superseded batches, including application CheckTx and Flush calls. Co-Authored-By: Codex <noreply@openai.com> <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
|
⛔ Final review complete — 2 blocking finding(s) (commit 2b0f6c1) · triage: normal |
Port the focused lifecycle core from #1515, replacing the local State wait workaround while retaining other service ownership mechanisms. Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
* fix(consensus): wait for receive routine during shutdown Join the receive routine before State.Wait returns and cancel its context on explicit Stop. This prevents node cleanup racing in-flight consensus file writes. Add a regression test for the shutdown contract. Co-Authored-By: OpenAI Codex GPT-6 <noreply@openai.com> <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub> * fix(service)!: share worker lifecycle for consensus shutdown Port the focused lifecycle core from #1515, replacing the local State wait workaround while retaining other service ownership mechanisms. Co-Authored-By: OpenAI GPT-6 <noreply@openai.com> * fix(consensus): preserve in-flight work during manual shutdown Keep processing on the State parent's lifetime while canceling intake, and let handoff callers cancel their wait without abandoning reactor-owned work. Document the global cancellation contract, cleanup ownership and migration boundaries; harden context lookup and close logjack's group on return. Co-Authored-By: OpenAI GPT-6 <noreply@openai.com> * fix(cli): wait for node shutdown with a bounded grace period Join node cleanup after cancellation with a configurable shutdown deadline. Return an error on timeout without bypassing service joins or store ordering. Organize BaseService startup into lock-scoped lifecycle phases. <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub> Co-Authored-By: Codex <noreply@openai.com> * fix(service): cancel the attempt before deferred shutdown hooks Stop during startup publishes the stop request, releases the lifecycle lock and only then cancels the attempt context. If OnStart returned in that window, Start ran OnStop on a still-live context, breaking the cancellation-before-OnStop contract. finishStop now cancels first; cancellation is idempotent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(blocksync): join the synchronizer drain before reactor shutdown completes On node cancellation the Synchronizer's own stop can win the race and block in OnStop until in-flight block application returns. The reactor's Stop call then returned immediately, so reactor Wait could release store teardown while Apply was still running. The reactor now waits for the Synchronizer after stopping it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(blocksync): join the synchronizer regardless of the sync flag poolRoutine clears blockSyncFlag after a Stop that returns early when the synchronizer's own cancellation-driven stop is still draining, so a flag-gated join in reactor OnStop could be skipped and reactor Wait could release store teardown before the consumer finished. OnStop now stops and waits for any constructed synchronizer; both calls are no-ops for one that never started. The regression now also runs poolRoutine through the handover before reactor shutdown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(privval): make the signer's initial connect request cancellation-aware SignerListenerEndpoint.OnStart sent its first connection request on an unbuffered channel unconditionally. With Stop now cancelling OnStart's context, serviceLoop can exit on ctx.Done before taking the request, leaving Start blocked forever and OnStop never run. The send now also selects on ctx.Done and returns nil, so the regular shutdown path still releases the listener and ping ticker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(consensus): close the unstarted WAL group in TestWALSearchForEndHeight OpenGroup detaches the head AutoFile worker from the caller's context, so the never-started WAL leaked its timer/SIGHUP worker. Register the group's Close as cleanup and take the leak snapshot at test start so the check covers the WAL's lifetime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(service): reject canceled contexts at worker admission Parent cancellation closes the attempt context before the async watcher takes the lifecycle lock and publishes stopping. In that window Go still admitted work, e.g. a consensus handoff that could reach State.Start during shutdown. Go now also refuses a canceled ctx under the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: OpenAI GPT-6 <noreply@openai.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Resolve against the merged lifecycle core from #1513; its BaseService, lifecycle tests, ticker and signer-listener startup path win. #1515's remaining service migrations are adapted to that API. - libs/service: take base service.go and lifecycle tests (superset). - blocksync: Synchronizer handlers stay on Go() with base's application context fallback; reactor keeps base's OnStop join and an OnDrain join for a synchronizer started by an admitted handoff. Adapt TestReactorWaitJoinsSynchronizerDrain: poolRoutine now joins the draining synchronizer itself, so run it concurrently and assert it stays blocked until application returns. - consensus reactor: keep #1515's managed workers and OnDrain helper cleanup; take base's cancellable SwitchToConsensus wait. - consensus commit: keep base's contract (parent cancellation aborts an in-flight commit); drop #1515's WithoutCancel finalize and turn TestNodeWaitPreservesFinalization into TestNodeWaitJoinsFinalization. - node: keep #1515's OnStop ordering (reactors, rechecks, event bus, dependency context, indexer, proxy) which subsumes base's reactors-before-sinks join. - docs/UPGRADING: merge both lifecycle documents; keep base's durable-finalization wording; add the WSRPCConnection.Go requirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
newStateWithConfigAndBlockStore started an EventBus on the test context and never joined it. Its asynchronous shutdown logged through the testing logger after the test returned, which the race detector reports (pre-existing on v1.8-dev, e.g. under TestPeerVoteVerificationDraws). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Solid lifecycle refactor with genuinely non-tautological tests backing it — nothing here blocks merge. One real regression worth a look: the pprof/Prometheus admin servers lost their shutdown timeout, so a single stalled client can hang node shutdown forever. The rest is documentation drift and teardown-logic duplication that's safe to leave for a fast-follow, not a gate.
4 finding(s) posted: 4 inline, 0 in this body.
🤖 Co-authored by Claudius the Magnificent AI Agent
📊 View full HTML review report
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
The base dropped cometbft-db (#1566); point the p2p and indexer lifecycle tests at libs/db so they compile against v1.8-dev. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
… write errcheck flags the discarded error received from Serve on the timeout path. Include it in the failure message instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lients The auxiliary HTTP servers shut down with an unbounded Shutdown that the node joins, so one stalled client (e.g. a long pprof profile) blocked node Wait forever. Node and seed OnStop also called Shutdown without a deadline. Serve them like the RPC server: on cancellation reject new requests, cancel request contexts, close connections and join active handlers. Drop the redundant OnStop shutdowns; the owned worker already stops the server. Bind failures are now logged instead of silently dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OnStart's rollback hand-copied OnStop's teardown and the copies had started to differ in ordering. Both now call joinChildren. Waiting on a child that never started returns immediately, so the rollback no longer needs to track whether the router started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
thepastaclaw
left a comment
There was a problem hiding this comment.
Re-review — Final validation — Phase 1 + Phase 2
Static verification at 6fb4a4c confirms eleven of thirteen prior findings are fixed; failed disconnect cleanup still removes unrelated subscribers, and caller-owned HTTP transport handling remains a nonblocking ownership gap. Current CI has no failed code checks: build, lint, vet and completed tests pass, while unit-test shards 00, 01 and 05 and the rotation end-to-end test remain pending. The base is two commits ahead with no reported conflicts or overlapping changed files.
🔴 1 blocking | 🟡 1 suggestion(s)
2 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — The change is large and intricate but modifies service lifecycle, worker ownership and shutdown ordering rather than consensus rules, funds movement, cryptography, signing or key handling, peer-facing deserialization, or storage migrations. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `rpc/client/http/ws.go`:
- [BLOCKING] rpc/client/http/ws.go:159-166: Scope failed disconnect cleanup to the disconnected subscriber
(existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127721)
This changes failed cleanup, not just the existing successful-cleanup isolation bug. If downstream peer A disconnects while the shared upstream writer is reconnecting, the proxy passes A's address here, but this method immediately removes connected peer B's query and broadcasts `cleared`. When the bounded upstream send times out, B's delivery state is already gone: incoming events are dropped and reconnect restoration cannot find its query. The base returned on that send error before clearing the map.
Clearing A's abandoned channel is necessary to prevent it from stalling delivery, but clearing B's live channel extends the defect into a previously preserved failure path. Track subscriber delivery ownership in `wsEvents`, retain upstream queries needed by other subscribers, and keep network sends outside the map lock. The current reconnect regression subscribes `other` after cleanup; add a case where B is already subscribed before A disconnects and verify B survives failed cleanup and reconnect.
In `light/rpc/client.go`:
- [SUGGESTION] light/rpc/client.go:111-117: Light client still joins a caller-owned HTTP event client
(existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127750)
The ownership guard protects the local client's running EventBus but cannot recognize an already-started `*http.HTTP`: its embedded `wsEvents` exposes neither `IsRunning` nor `Wait`. The unconditional `next.Start(ctx)` therefore restarts that live transport. `wsEvents.Start` replaces its lifetime and listener completion fields, and `WSClient.Start` dials again and replaces its transport channels; this is not an idempotent adoption of the caller's existing lifetime. The new `OnDrain` then invokes its joining `Stop` method as though the wrapper owned it.
Keep this nonblocking because no in-repository production caller shares the HTTP client, but the newly added ownership handling still misses this public implementation. Expose lifecycle state or explicit ownership so caller-owned transports are neither restarted nor joined. If that support is deferred, link a tracking issue or follow-up PR; the known-limitations paragraph currently provides no tracking reference.
|
Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix. |
Remaining blocker (wsEvents.UnsubscribeAll per-subscriber cleanup) is a pre-existing design limitation outside this PR scope, documented under Known limitations. Dismissed by maintainer decision.
Preserve managed connection shutdown and peer-input penalties, including receive completion when cancellation rejects worker admission. <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub> Co-Authored-By: Codex GPT-6 <noreply@openai.com>
thepastaclaw
left a comment
There was a problem hiding this comment.
Re-review — Final validation — Phase 1 + Phase 2
At head 564a3bc, eleven of the thirteen prior findings are fixed. Failed disconnect cleanup still deletes unrelated subscribers' delivery state, which blocks approval; caller-owned HTTP transport handling remains a nonblocking ownership gap. Validation was static only: the supplied head-specific CI snapshot and current PR checks report successful build, test, lint, vet, and e2e checks.
🔴 1 blocking | 🟡 1 suggestion(s)
2 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — This is a large, intricate concurrency and shutdown change, but the described diff changes worker ownership and teardown rather than consensus rules, cryptography, key handling, peer-facing deserialization, or storage migrations. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `rpc/client/http/ws.go`:
- [BLOCKING] rpc/client/http/ws.go:159-166: Scope failed disconnect cleanup to the disconnected subscriber
(existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127721)
This changes failed cleanup from preserving local subscriptions to deleting all of them. With downstream peers A and B subscribed to different queries, A's disconnect invokes the proxy's bounded cleanup with A's address, but this method ignores that address, empties the shared map, and closes the shared delivery-interruption signal before attempting the upstream send. If the upstream writer is unavailable during reconnect and the send times out, B remains connected but loses its local delivery state and its query is unavailable for restoration. The base branch already had the successful-UnsubscribeAll isolation defect, but its failed-send path returned before clearing the map; making failures destructive is a regression, not merely the same existing limitation. Preventing abandoned channels from blocking delivery is necessary, but cleanup must identify the departing subscriber and preserve queries and delivery state belonging to live subscribers. Implement that ownership in wsEvents, keep network sends outside the map lock, and add a two-subscriber regression covering timed-out cleanup followed by reconnect. TestUnsubscribeAllDuringResubscribe uses an empty subscriber and expects existing queries to disappear, so it does not cover this isolation requirement.
In `light/rpc/client.go`:
- [SUGGESTION] light/rpc/client.go:111-137: Light client still joins a caller-owned HTTP event client
(existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127750)
The ownership guard cannot recognize an independently started *http.HTTP: its embedded wsEvents exposes neither IsRunning nor Wait. OnStart therefore leaves callerOwnsNext false and unconditionally calls Start again. wsEvents.Start replaces its lifetime and listener-completion fields, while WSClient.Start redials and replaces transport channels without an already-running guard. OnDrain subsequently calls Stop as though the wrapper owned the transport; that method joins rather than cancels the transport, so it does not establish ownership of the caller's original workers. This PR's new ownership and drain logic therefore protects a shared local EventBus but not the primary HTTP implementation. Expose lifecycle state or an explicit ownership option in the HTTP-client owning layer, and skip both restart and drain for caller-owned clients. Add a regression using an independently started HTTP client. No in-repository production caller currently shares this transport, so this remains nonblocking; if deferred, the disclosed limitation needs a linked tracking issue or follow-up PR.
|
Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix. |
Resolve conflicts with dev's blocksync askEveryPeer handler (#1618) and consensus round-state re-announce (#1598): keep this branch's managed workers (Go/Wait) and dev's new behaviour. Semantic follow-ups needed by the merge: - start the reactor's sendLiveness/sendState workers with r.Go so shutdown joins them; - drive TestReactorRefreshesRoundStepAfterLostAnnouncement through a started reactor, since peerUp now starts gossip only within a running service and no longer takes a retry count. <sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
thepastaclaw
left a comment
There was a problem hiding this comment.
Re-review — Final validation — Phase 1 + Phase 2
The lifecycle migration is broadly sound, and the supplied build, test, lint, vet, protocol, and vulnerability checks all pass for this head; only review and PR-hygiene checks remain pending. Two additional startup-error paths still bypass cleanup: validator construction can leak resources after identity lookup fails, and the proxy can retain a partially started ABCI client.
🔴 2 blocking | 🟡 1 suggestion(s)
Still open from earlier rounds (2)
- 🟡 Suggestion: Light client still joins a caller-owned HTTP event client —
light/rpc/client.go:111-137(thread) - 🔴 Blocking: Scope failed disconnect cleanup to the disconnected subscriber —
rpc/client/http/ws.go:159-166(thread)
Follow-up candidates (optional) (1)
New non-blocking observations on code unchanged since the last full review. No inline threads are opened for them and later reviews do not repeat them: fix forward or track separately if they are worth it.
🟡 Suggestion: Release node resources when validator identity lookup fails
node/node.go:218-227
After createPrivval succeeds, its Close method is registered in closers, but a GetProTxHash failure returns directly instead of closing that closer chain. The remote signer connection, database handles, and startup context therefore remain live when validator construction fails at this point. This is a startup rollback path introduced by the new resource-registration scheme and must use the same cleanup as the surrounding constructor errors.
proTxHash, err = privValidator.GetProTxHash(ctx)
if err != nil {
return nil, combineCloseError(err, makeCloser(closers))
}
source: gpt-6.1-sol (phase2-reviewer: general)
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — This is a large, cross-cutting concurrency and shutdown lifecycle change spanning consensus, P2P, ABCI, RPC, and node services, but it does not itself alter consensus rules, cryptography, signatures, peer-facing deserialization, or storage migrations. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Code unchanged since the last final review (
564a3bcd): 1 new blocker(s) went to an independent confirmation pass (the final verifier's model in a fresh session, without its reasoning): 1 not confirmed (now follow-up candidates) - Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.
In `internal/proxy/client.go`:
- [BLOCKING] internal/proxy/client.go:98-101: Release the ABCI client when proxy startup fails
If the wrapped ABCI client's Start returns an error after allocating a connection or starting background work, OnStart now returns without calling Stop or Wait. BaseService does not invoke OnStop after a failed OnStart, so a partially started custom or public abciclient.Client can retain its connection or workers while node startup rolls back. The previous deferred cleanup was removed by this migration; failed Start must stop and join the client before returning the error.
In `light/rpc/client.go`:
- [SUGGESTION] light/rpc/client.go:111-137: Light client still joins a caller-owned HTTP event client
(carried; thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127750)
In `rpc/client/http/ws.go`:
- [BLOCKING] rpc/client/http/ws.go:159-166: Scope failed disconnect cleanup to the disconnected subscriber
(carried; thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127721)
| func (app *proxyClient) OnStart(ctx context.Context) error { | ||
| var err error | ||
| defer func() { | ||
| if err != nil { | ||
| tryCallStop(app.client) | ||
| } | ||
| }() | ||
| if err := app.client.Start(ctx); err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
🔴 Blocking: Release the ABCI client when proxy startup fails
If the wrapped ABCI client's Start returns an error after allocating a connection or starting background work, OnStart now returns without calling Stop or Wait. BaseService does not invoke OnStop after a failed OnStart, so a partially started custom or public abciclient.Client can retain its connection or workers while node startup rolls back. The previous deferred cleanup was removed by this migration; failed Start must stop and join the client before returning the error.
| func (app *proxyClient) OnStart(ctx context.Context) error { | |
| var err error | |
| defer func() { | |
| if err != nil { | |
| tryCallStop(app.client) | |
| } | |
| }() | |
| if err := app.client.Start(ctx); err != nil { | |
| return err | |
| } | |
| if err := app.client.Start(ctx); err != nil { | |
| tryCallStop(app.client) | |
| app.client.Wait() | |
| return err | |
| } |
source: gpt-6.1-sol (phase2-reviewer: tenderdash-consensus-security)
|
Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them. |
TL;DR: every remaining Tenderdash service now registers its background work with
BaseService.Go, soWaitreturns only after P2P, ABCI, RPC, mempool, indexing and node workers have actually stopped.Issue being fixed or feature implemented
As a node operator, I want
tenderdashto finish shutting down only after every service has stopped touching files, stores and application connections, so a restart never races leftover workers.Scenario: SIGTERM arrives while peers are connected, a WebSocket client is subscribed and the mempool is rechecking transactions. Shutdown should close RPC admission, join handlers, peer sessions, rechecks and reactor workers, and only then release the ABCI connections and stores.
This builds on #1513 (merged), which added the lifecycle core (attempt-scoped contexts,
Goadmission, cancel-before-OnStop,OnDrain) and migrated consensus State/ticker/WAL/autofile. This PR migrates the services #1513 left on plain goroutines.What was done?
Go; connections are closed and joined by their owner.MConnectionsend/receive routines and callbacks are joined byWait.PeerUpdates.CloseandRateLimit.Closejoin their workers. PEX, evidence, statesync, mempool and blocksync reactors register channel and peer-update workers.OnDrainjoins State, then closes peer updates and rate limiters. Peer gossip workers are joined per peer.Go;poolRoutinejoins the synchronizer before reading final state.SwitchToBlockSyncruns as an admitted reactor worker and is rejected after stop;OnDrainjoins a synchronizer started by an admitted handoff.Wait. The routed client stops and joins only the clients its own start attempt started, including on startup failure.rpccore.Environmentowns the RPC service (StartService/StopService) and drains async broadcasts. Light proxy and light RPC unsubscribe cleanup is bounded by a one-second timeout within the proxy lifetime and the request context. The HTTP RPC client'sStopalso joins its event listener and resubscriptions.OnDrain.TxMempool.StopRecheckspermanently closes recheck admission and joins all admitted batches.docs/architecture/service-lifecycle.mdcovers the full migration, ownership/durability rules and the cancellation audit.UPGRADING.mdlists the new interface requirement.Known limitation: durable finalization across node shutdown is not guaranteed. Following the #1513 contract, reactor/node shutdown cancels the context passed to
State.Startand can interrupt an in-flight commit (WAL recovery handles it); nodeWaitstill joins that commit before releasing stores. A durable-finalization mode is a possible follow-up.Known limitation: the light RPC client treats a wrapped client as its own unless that client reports it was already running. The HTTP client does not report this, so one the caller started first is still joined.
Known limitation: a read-only
WSClientwhose connection is reset does not reconnect until its next write, so its events stop until then.Known limitation:
wsEvents.UnsubscribeAllignores its subscriber, so one light proxy peer disconnecting drops every peer's subscriptions. A disconnect cleanup that times out can leave its query subscribed upstream until the next reconnect.Known limitation: an RPC listener bind failure stays retryable and keeps the stores open, and
service.Servicehas no call to release them if the caller gives up. The Dash Core client has no HTTP timeout:Closewaits 5s, but a reactor stuck in a Core call blocksWaitand failed-Startrollback.--shutdown-timeoutbounds onlystart's wait. Rebuilding a node in the same process panics on duplicate metrics registration for a seed, or for a node with Prometheus enabled, unless the namespace differs.How Has This Been Tested?
TestRouterWaitDrainsAcceptedConnection,TestMConnectionWaitDrainsCallback,TestPeerGossipWorkerWaitDrainsHandler,TestStopRechecksJoinsSupersededBatches,TestStopRechecksRacesAdmission,TestServeJoinsActiveHandler,TestServeJoinsWebsocketSessionWork,TestRPCServiceRetryAndEventLogDrain,TestSeedFailedStartDrainsRouter,TestNodeWaitJoinsFinalization,TestOwnedHTTPServerStopEvictsStalledClient,TestSocketServerStopUnblocksStalledConnection,TestRoutedClientFailedStartReleasesOnlyItsChildren,TestListenerStopInterruptsPendingRequest,TestProxyShutdownWithStoppedUpstreamClient,TestUnsubscribeWSObservesCallerCancellation,TestNodeCancellationInterruptsApplicationStartup,TestSwitchToBlockSyncRejectsAfterStopandTestIndexerStopsWhileEventBusRemainsLive.TestReactorWaitJoinsSynchronizerDrainalso checks thatpoolRoutinestays blocked until synchronizer application finishes.go build ./...andgo vet ./...are clean; changed files aregofmt-clean.go test -race -tags=deadlock -count=1passes on every affected package: libs/service, node, internal/consensus, internal/blocksync, internal/p2p/..., privval, internal/evidence, internal/mempool, rpc/..., internal/rpc/core, internal/state/indexer/..., internal/statesync, internal/eventbus, internal/pubsub, internal/proxy, internal/libs/autofile, abci/client, abci/server, dash/quorum and light/rpc.Breaking Changes
rpc/jsonrpc/server.Serve/ServeTLSalways close the listener, cancel request contexts, close active connections instead of a 1-second graceful shutdown, and return only after every handler, including WebSocket sessions, finishes. The startup log line is gone.WSRPCConnectionimplementations must provideGo(func(context.Context)) boolso the server owns subscription workers.Starton that object returns an error. A failed seed start is final in the same way.Waiton migrated services (MConnection, Router, Synchronizer, ABCI clients/servers, RPC servers) now blocks until their workers and callbacks return. Callers that calledStopalone must follow it withWaitbefore releasing shared resources.Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code
PR Hygiene ·
2b0f6c1/self-reviewedWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.