Skip to content

fix(service)!: join remaining service workers before shutdown completes - #1515

Open
lklimek wants to merge 40 commits into
devfrom
fix/base-service-stop
Open

lklimek wants to merge 40 commits into
devfrom
fix/base-service-stop

Conversation

@lklimek

@lklimek lklimek commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR: every remaining Tenderdash service now registers its background work with BaseService.Go, so Wait returns only after P2P, ABCI, RPC, mempool, indexing and node workers have actually stopped.

Issue being fixed or feature implemented

As a node operator, I want tenderdash to finish shutting down only after every service has stopped touching files, stores and application connections, so a restart never races leftover workers.

Scenario: SIGTERM arrives while peers are connected, a WebSocket client is subscribed and the mempool is rechecking transactions. Shutdown should close RPC admission, join handlers, peer sessions, rechecks and reactor workers, and only then release the ABCI connections and stores.

This builds on #1513 (merged), which added the lifecycle core (attempt-scoped contexts, Go admission, cancel-before-OnStop, OnDrain) and migrated consensus State/ticker/WAL/autofile. This PR migrates the services #1513 left on plain goroutines.

What was done?

  • P2P: Router channel, accept, dial and peer-routing work runs through Go; connections are closed and joined by their owner. MConnection send/receive routines and callbacks are joined by Wait. PeerUpdates.Close and RateLimit.Close join their workers. PEX, evidence, statesync, mempool and blocksync reactors register channel and peer-update workers.
  • Consensus reactor: channel, peer and gossip workers are registered; OnDrain joins State, then closes peer updates and rate limiters. Peer gossip workers are joined per peer.
  • Blocksync: Synchronizer handlers use Go; poolRoutine joins the synchronizer before reading final state. SwitchToBlockSync runs as an admitted reactor worker and is rejected after stop; OnDrain joins a synchronizer started by an admitted handoff.
  • ABCI: socket/gRPC clients and servers, the routed client and the proxy wrapper join their workers and children. The socket server closes accepted connections on shutdown, so a client that stops reading cannot block Wait. The routed client stops and joins only the clients its own start attempt started, including on startup failure.
  • RPC: HTTP and WebSocket servers close admission and join active handlers, hijacked sessions and subscription workers. rpccore.Environment owns the RPC service (StartService/StopService) and drains async broadcasts. Light proxy and light RPC unsubscribe cleanup is bounded by a one-second timeout within the proxy lifetime and the request context. The HTTP RPC client's Stop also joins its event listener and resubscriptions.
  • Events and indexing: EventBus, pubsub and the indexer use managed workers; the indexer closes sinks in OnDrain.
  • Mempool: TxMempool.StopRechecks permanently closes recheck admission and joins all admitted batches.
  • Node and seed: application connections, event bus and indexer run on a dependency context that is canceled only after reactors are joined. Failed startup joins every started child, then releases node-owned resources exactly once; normal shutdown also closes each store once. Prometheus/pprof servers are owned workers that close stalled connections and join handlers on shutdown. Shutdown and failed startup share one child-teardown path. The signer listener interrupts in-flight signer I/O on stop. Privval listener, signer services, light RPC client and the quorum executor are migrated too.
  • Docs: docs/architecture/service-lifecycle.md covers the full migration, ownership/durability rules and the cancellation audit. UPGRADING.md lists the new interface requirement.

Known limitation: durable finalization across node shutdown is not guaranteed. Following the #1513 contract, reactor/node shutdown cancels the context passed to State.Start and can interrupt an in-flight commit (WAL recovery handles it); node Wait still joins that commit before releasing stores. A durable-finalization mode is a possible follow-up.

Known limitation: the light RPC client treats a wrapped client as its own unless that client reports it was already running. The HTTP client does not report this, so one the caller started first is still joined.

Known limitation: a read-only WSClient whose connection is reset does not reconnect until its next write, so its events stop until then.

Known limitation: wsEvents.UnsubscribeAll ignores its subscriber, so one light proxy peer disconnecting drops every peer's subscriptions. A disconnect cleanup that times out can leave its query subscribed upstream until the next reconnect.

Known limitation: an RPC listener bind failure stays retryable and keeps the stores open, and service.Service has no call to release them if the caller gives up. The Dash Core client has no HTTP timeout: Close waits 5s, but a reactor stuck in a Core call blocks Wait and failed-Start rollback. --shutdown-timeout bounds only start's wait. Rebuilding a node in the same process panics on duplicate metrics registration for a seed, or for a node with Prometheus enabled, unless the namespace differs.

How Has This Been Tested?

  • New regressions for the migrated services, including TestRouterWaitDrainsAcceptedConnection, TestMConnectionWaitDrainsCallback, TestPeerGossipWorkerWaitDrainsHandler, TestStopRechecksJoinsSupersededBatches, TestStopRechecksRacesAdmission, TestServeJoinsActiveHandler, TestServeJoinsWebsocketSessionWork, TestRPCServiceRetryAndEventLogDrain, TestSeedFailedStartDrainsRouter, TestNodeWaitJoinsFinalization, TestOwnedHTTPServerStopEvictsStalledClient, TestSocketServerStopUnblocksStalledConnection, TestRoutedClientFailedStartReleasesOnlyItsChildren, TestListenerStopInterruptsPendingRequest, TestProxyShutdownWithStoppedUpstreamClient, TestUnsubscribeWSObservesCallerCancellation, TestNodeCancellationInterruptsApplicationStartup, TestSwitchToBlockSyncRejectsAfterStop and TestIndexerStopsWhileEventBusRemainsLive.
  • TestReactorWaitJoinsSynchronizerDrain also checks that poolRoutine stays blocked until synchronizer application finishes.
  • go build ./... and go vet ./... are clean; changed files are gofmt-clean.
  • The consensus test harness now joins its event bus before a test ends, fixing an existing race-detector report from late shutdown logging.
  • go test -race -tags=deadlock -count=1 passes on every affected package: libs/service, node, internal/consensus, internal/blocksync, internal/p2p/..., privval, internal/evidence, internal/mempool, rpc/..., internal/rpc/core, internal/state/indexer/..., internal/statesync, internal/eventbus, internal/pubsub, internal/proxy, internal/libs/autofile, abci/client, abci/server, dash/quorum and light/rpc.

Breaking Changes

  • rpc/jsonrpc/server.Serve/ServeTLS always close the listener, cancel request contexts, close active connections instead of a 1-second graceful shutdown, and return only after every handler, including WebSocket sessions, finishes. The startup log line is gone.
  • WSRPCConnection implementations must provide Go(func(context.Context)) bool so the server owns subscription workers.
  • After a node startup failure that happens past RPC listener reservation, build a fresh node graph. Rollback permanently closes non-service resources such as mempool recheck admission and the stores, and a later Start on that object returns an error. A failed seed start is final in the same way.
  • Wait on migrated services (MConnection, Router, Synchronizer, ABCI clients/servers, RPC servers) now blocks until their workers and callbacks return. Callers that called Stop alone must follow it with Wait before releasing shared resources.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

PR Hygiene · 2b0f6c1

  • Bots — thepastaclaw requested changes — dismiss the review or push a fix, 1 thread unresolved — resolve it
  • Self-review — post /self-reviewed
  • Build failed
  • Approvals — you own every area touched; none needed

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

lklimek and others added 8 commits September 24, 2026 18:01
Add attempt-scoped worker admission, cancellation and finalization to
BaseService. Migrate consensus, WAL, file groups, ticker and gossip;
join child services and preserve durable commit completion.

Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Track Router, PEX and MConnection work through BaseService; join connection, queue, subscription and request workers at their owners. Preserve the parent context for post-stop error callbacks and expose explicit rate-limit cleanup.

Co-Authored-By: Codex GPT-6 <noreply@openai.com>
Migrate service workers and child ownership to the shared lifecycle. Preserve
application finalization during block-sync handover and node shutdown, drain
event observers before closing sinks, and keep failed startup interruptible.

Co-Authored-By: Codex <noreply@openai.com>

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Close recheck admission under the mempool mutex and wait outside it for
active and superseded batches, including application CheckTx and Flush calls.

Co-Authored-By: Codex <noreply@openai.com>

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thepastaclaw

thepastaclaw commented Sep 24, 2026 •

Copy link
Copy Markdown

⛔ Final review complete — 2 blocking finding(s) (commit 2b0f6c1) · triage: normal

@lklimek lklimek added the claudius-review Trigger Claudius AI code review on this PR label Sep 25, 2026
lklimek added a commit that referenced this pull request Sep 25, 2026
Port the focused lifecycle core from #1515, replacing the local State wait workaround while retaining other service ownership mechanisms.

Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
lklimek added a commit that referenced this pull request Oct 23, 2026
* fix(consensus): wait for receive routine during shutdown

Join the receive routine before State.Wait returns and cancel its context
on explicit Stop. This prevents node cleanup racing in-flight consensus
file writes. Add a regression test for the shutdown contract.

Co-Authored-By: OpenAI Codex GPT-6 <noreply@openai.com>

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>

* fix(service)!: share worker lifecycle for consensus shutdown

Port the focused lifecycle core from #1515, replacing the local State wait workaround while retaining other service ownership mechanisms.

Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>

* fix(consensus): preserve in-flight work during manual shutdown

Keep processing on the State parent's lifetime while canceling intake, and
let handoff callers cancel their wait without abandoning reactor-owned work.
Document the global cancellation contract, cleanup ownership and migration
boundaries; harden context lookup and close logjack's group on return.

Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>

* fix(cli): wait for node shutdown with a bounded grace period

Join node cleanup after cancellation with a configurable shutdown deadline.
Return an error on timeout without bypassing service joins or store ordering.
Organize BaseService startup into lock-scoped lifecycle phases.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>

Co-Authored-By: Codex <noreply@openai.com>

* fix(service): cancel the attempt before deferred shutdown hooks

Stop during startup publishes the stop request, releases the lifecycle
lock and only then cancels the attempt context. If OnStart returned in
that window, Start ran OnStop on a still-live context, breaking the
cancellation-before-OnStop contract. finishStop now cancels first;
cancellation is idempotent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(blocksync): join the synchronizer drain before reactor shutdown completes

On node cancellation the Synchronizer's own stop can win the race and
block in OnStop until in-flight block application returns. The reactor's
Stop call then returned immediately, so reactor Wait could release store
teardown while Apply was still running. The reactor now waits for the
Synchronizer after stopping it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(blocksync): join the synchronizer regardless of the sync flag

poolRoutine clears blockSyncFlag after a Stop that returns early when
the synchronizer's own cancellation-driven stop is still draining, so a
flag-gated join in reactor OnStop could be skipped and reactor Wait
could release store teardown before the consumer finished. OnStop now
stops and waits for any constructed synchronizer; both calls are no-ops
for one that never started. The regression now also runs poolRoutine
through the handover before reactor shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(privval): make the signer's initial connect request cancellation-aware

SignerListenerEndpoint.OnStart sent its first connection request on an
unbuffered channel unconditionally. With Stop now cancelling OnStart's
context, serviceLoop can exit on ctx.Done before taking the request,
leaving Start blocked forever and OnStop never run. The send now also
selects on ctx.Done and returns nil, so the regular shutdown path still
releases the listener and ping ticker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(consensus): close the unstarted WAL group in TestWALSearchForEndHeight

OpenGroup detaches the head AutoFile worker from the caller's context,
so the never-started WAL leaked its timer/SIGHUP worker. Register the
group's Close as cleanup and take the leak snapshot at test start so
the check covers the WAL's lifetime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(service): reject canceled contexts at worker admission

Parent cancellation closes the attempt context before the async watcher
takes the lifecycle lock and publishes stopping. In that window Go still
admitted work, e.g. a consensus handoff that could reach State.Start
during shutdown. Go now also refuses a canceled ctx under the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: OpenAI GPT-6 <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lklimek and others added 2 commits October 23, 2026 08:22
Resolve against the merged lifecycle core from #1513; its BaseService,
lifecycle tests, ticker and signer-listener startup path win. #1515's
remaining service migrations are adapted to that API.

- libs/service: take base service.go and lifecycle tests (superset).
- blocksync: Synchronizer handlers stay on Go() with base's application
  context fallback; reactor keeps base's OnStop join and an OnDrain join
  for a synchronizer started by an admitted handoff. Adapt
  TestReactorWaitJoinsSynchronizerDrain: poolRoutine now joins the
  draining synchronizer itself, so run it concurrently and assert it
  stays blocked until application returns.
- consensus reactor: keep #1515's managed workers and OnDrain helper
  cleanup; take base's cancellable SwitchToConsensus wait.
- consensus commit: keep base's contract (parent cancellation aborts an
  in-flight commit); drop #1515's WithoutCancel finalize and turn
  TestNodeWaitPreservesFinalization into TestNodeWaitJoinsFinalization.
- node: keep #1515's OnStop ordering (reactors, rechecks, event bus,
  dependency context, indexer, proxy) which subsumes base's
  reactors-before-sinks join.
- docs/UPGRADING: merge both lifecycle documents; keep base's
  durable-finalization wording; add the WSRPCConnection.Go requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
newStateWithConfigAndBlockStore started an EventBus on the test context
and never joined it. Its asynchronous shutdown logged through the
testing logger after the test returned, which the race detector reports
(pre-existing on v1.8-dev, e.g. under TestPeerVoteVerificationDraws).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lklimek lklimek changed the title fix(service)!: join background work before shutdown completes fix(service)!: join remaining service workers before shutdown completes Oct 23, 2026

@claudbot claudbot Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid lifecycle refactor with genuinely non-tautological tests backing it — nothing here blocks merge. One real regression worth a look: the pprof/Prometheus admin servers lost their shutdown timeout, so a single stalled client can hang node shutdown forever. The rest is documentation drift and teardown-logic duplication that's safe to leave for a fast-follow, not a gate.

4 finding(s) posted: 4 inline, 0 in this body.

🤖 Co-authored by Claudius the Magnificent AI Agent
📊 View full HTML review report

Comment thread node/setup.go Resolved
Comment thread docs/architecture/service-lifecycle.md Resolved
Comment thread UPGRADING.md Resolved
Comment thread node/node.go Resolved
@claudbot claudbot Bot removed the claudius-review Trigger Claudius AI code review on this PR label Oct 23, 2026
@lklimek
lklimek marked this pull request as ready for review October 23, 2026 09:09
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Oct 23, 2026
The base dropped cometbft-db (#1566); point the p2p and indexer
lifecycle tests at libs/db so they compile against v1.8-dev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

lklimek and others added 4 commits October 23, 2026 09:39
… write

errcheck flags the discarded error received from Serve on the timeout
path. Include it in the failure message instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lients

The auxiliary HTTP servers shut down with an unbounded Shutdown that the
node joins, so one stalled client (e.g. a long pprof profile) blocked
node Wait forever. Node and seed OnStop also called Shutdown without a
deadline.

Serve them like the RPC server: on cancellation reject new requests,
cancel request contexts, close connections and join active handlers.
Drop the redundant OnStop shutdowns; the owned worker already stops the
server. Bind failures are now logged instead of silently dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OnStart's rollback hand-copied OnStop's teardown and the copies had
started to differ in ordering. Both now call joinChildren. Waiting on a
child that never started returns immediately, so the rollback no longer
needs to track whether the router started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 5, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

Static verification at 6fb4a4c confirms eleven of thirteen prior findings are fixed; failed disconnect cleanup still removes unrelated subscribers, and caller-owned HTTP transport handling remains a nonblocking ownership gap. Current CI has no failed code checks: build, lint, vet and completed tests pass, while unit-test shards 00, 01 and 05 and the rotation end-to-end test remain pending. The base is two commits ahead with no reported conflicts or overlapping changed files.

🔴 1 blocking | 🟡 1 suggestion(s)

2 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — The change is large and intricate but modifies service lifecycle, worker ownership and shutdown ordering rather than consensus rules, funds movement, cryptography, signing or key handling, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `rpc/client/http/ws.go`:
- [BLOCKING] rpc/client/http/ws.go:159-166: Scope failed disconnect cleanup to the disconnected subscriber
  (existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127721)
  This changes failed cleanup, not just the existing successful-cleanup isolation bug. If downstream peer A disconnects while the shared upstream writer is reconnecting, the proxy passes A's address here, but this method immediately removes connected peer B's query and broadcasts `cleared`. When the bounded upstream send times out, B's delivery state is already gone: incoming events are dropped and reconnect restoration cannot find its query. The base returned on that send error before clearing the map.

  Clearing A's abandoned channel is necessary to prevent it from stalling delivery, but clearing B's live channel extends the defect into a previously preserved failure path. Track subscriber delivery ownership in `wsEvents`, retain upstream queries needed by other subscribers, and keep network sends outside the map lock. The current reconnect regression subscribes `other` after cleanup; add a case where B is already subscribed before A disconnects and verify B survives failed cleanup and reconnect.

In `light/rpc/client.go`:
- [SUGGESTION] light/rpc/client.go:111-117: Light client still joins a caller-owned HTTP event client
  (existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127750)
  The ownership guard protects the local client's running EventBus but cannot recognize an already-started `*http.HTTP`: its embedded `wsEvents` exposes neither `IsRunning` nor `Wait`. The unconditional `next.Start(ctx)` therefore restarts that live transport. `wsEvents.Start` replaces its lifetime and listener completion fields, and `WSClient.Start` dials again and replaces its transport channels; this is not an idempotent adoption of the caller's existing lifetime. The new `OnDrain` then invokes its joining `Stop` method as though the wrapper owned it.

  Keep this nonblocking because no in-repository production caller shares the HTTP client, but the newly added ownership handling still misses this public implementation. Expose lifecycle state or explicit ownership so caller-owned transports are neither restarted nor joined. If that support is deferred, link a tracking issue or follow-up PR; the known-limitations paragraph currently provides no tracking reference.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 5, 2026
@lklimek
lklimek dismissed thepastaclaw’s stale review November 5, 2026 08:54

Remaining blocker (wsEvents.UnsubscribeAll per-subscriber cleanup) is a pre-existing design limitation outside this PR scope, documented under Known limitations. Dismissed by maintainer decision.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 5, 2026
@Claudius-Maginificent Claudius-Maginificent added the priority: medium Medium priority label Nov 5, 2026
@github-actions github-actions Bot removed the waiting-bots Waiting for the review bots to report on this head label Nov 5, 2026
Preserve managed connection shutdown and peer-input penalties, including
receive completion when cancellation rejects worker admission.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>

Co-Authored-By: Codex GPT-6 <noreply@openai.com>

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

At head 564a3bc, eleven of the thirteen prior findings are fixed. Failed disconnect cleanup still deletes unrelated subscribers' delivery state, which blocks approval; caller-owned HTTP transport handling remains a nonblocking ownership gap. Validation was static only: the supplied head-specific CI snapshot and current PR checks report successful build, test, lint, vet, and e2e checks.

🔴 1 blocking | 🟡 1 suggestion(s)

2 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large, intricate concurrency and shutdown change, but the described diff changes worker ownership and teardown rather than consensus rules, cryptography, key handling, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `rpc/client/http/ws.go`:
- [BLOCKING] rpc/client/http/ws.go:159-166: Scope failed disconnect cleanup to the disconnected subscriber
  (existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127721)
  This changes failed cleanup from preserving local subscriptions to deleting all of them. With downstream peers A and B subscribed to different queries, A's disconnect invokes the proxy's bounded cleanup with A's address, but this method ignores that address, empties the shared map, and closes the shared delivery-interruption signal before attempting the upstream send. If the upstream writer is unavailable during reconnect and the send times out, B remains connected but loses its local delivery state and its query is unavailable for restoration. The base branch already had the successful-UnsubscribeAll isolation defect, but its failed-send path returned before clearing the map; making failures destructive is a regression, not merely the same existing limitation. Preventing abandoned channels from blocking delivery is necessary, but cleanup must identify the departing subscriber and preserve queries and delivery state belonging to live subscribers. Implement that ownership in wsEvents, keep network sends outside the map lock, and add a two-subscriber regression covering timed-out cleanup followed by reconnect. TestUnsubscribeAllDuringResubscribe uses an empty subscriber and expects existing queries to disappear, so it does not cover this isolation requirement.

In `light/rpc/client.go`:
- [SUGGESTION] light/rpc/client.go:111-137: Light client still joins a caller-owned HTTP event client
  (existing thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127750)
  The ownership guard cannot recognize an independently started *http.HTTP: its embedded wsEvents exposes neither IsRunning nor Wait. OnStart therefore leaves callerOwnsNext false and unconditionally calls Start again. wsEvents.Start replaces its lifetime and listener-completion fields, while WSClient.Start redials and replaces transport channels without an already-running guard. OnDrain subsequently calls Stop as though the wrapper owned the transport; that method joins rather than cancels the transport, so it does not establish ownership of the caller's original workers. This PR's new ownership and drain logic therefore protects a shared local EventBus but not the primary HTTP implementation. Expose lifecycle state or an explicit ownership option in the HTTP-client owning layer, and skip both restart and drain for caller-owned clients. Add a regression using an independently started HTTP client. No in-repository production caller currently shares this transport, so this remains nonblocking; if deferred, the disclosed limitation needs a linked tracking issue or follow-up PR.

@Claudius-Maginificent Claudius-Maginificent added priority: high High priority — address soon and removed priority: medium Medium priority labels Nov 5, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-self-review Waiting for the author to post /self-reviewed label Nov 6, 2026
@lklimek lklimek modified the milestones: v3.0, v2.1 Nov 6, 2026
Resolve conflicts with dev's blocksync askEveryPeer handler (#1618) and
consensus round-state re-announce (#1598): keep this branch's managed
workers (Go/Wait) and dev's new behaviour.

Semantic follow-ups needed by the merge:
- start the reactor's sendLiveness/sendState workers with r.Go so
  shutdown joins them;
- drive TestReactorRefreshesRoundStepAfterLostAnnouncement through a
  started reactor, since peerUp now starts gossip only within a running
  service and no longer takes a retry count.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 17, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The lifecycle migration is broadly sound, and the supplied build, test, lint, vet, protocol, and vulnerability checks all pass for this head; only review and PR-hygiene checks remain pending. Two additional startup-error paths still bypass cleanup: validator construction can leak resources after identity lookup fails, and the proxy can retain a partially started ABCI client.

🔴 2 blocking | 🟡 1 suggestion(s)

Still open from earlier rounds (2)

  • 🟡 Suggestion: Light client still joins a caller-owned HTTP event client — light/rpc/client.go:111-137 (thread)
  • 🔴 Blocking: Scope failed disconnect cleanup to the disconnected subscriber — rpc/client/http/ws.go:159-166 (thread)
Follow-up candidates (optional) (1)

New non-blocking observations on code unchanged since the last full review. No inline threads are opened for them and later reviews do not repeat them: fix forward or track separately if they are worth it.

🟡 Suggestion: Release node resources when validator identity lookup fails
node/node.go:218-227

After createPrivval succeeds, its Close method is registered in closers, but a GetProTxHash failure returns directly instead of closing that closer chain. The remote signer connection, database handles, and startup context therefore remain live when validator construction fails at this point. This is a startup rollback path introduced by the new resource-registration scheme and must use the same cleanup as the surrounding constructor errors.

proTxHash, err = privValidator.GetProTxHash(ctx)
if err != nil {
	return nil, combineCloseError(err, makeCloser(closers))
}

source: gpt-6.1-sol (phase2-reviewer: general)

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large, cross-cutting concurrency and shutdown lifecycle change spanning consensus, P2P, ABCI, RPC, and node services, but it does not itself alter consensus rules, cryptography, signatures, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Code unchanged since the last final review (564a3bcd): 1 new blocker(s) went to an independent confirmation pass (the final verifier's model in a fresh session, without its reasoning): 1 not confirmed (now follow-up candidates)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `internal/proxy/client.go`:
- [BLOCKING] internal/proxy/client.go:98-101: Release the ABCI client when proxy startup fails
  If the wrapped ABCI client's Start returns an error after allocating a connection or starting background work, OnStart now returns without calling Stop or Wait. BaseService does not invoke OnStop after a failed OnStart, so a partially started custom or public abciclient.Client can retain its connection or workers while node startup rolls back. The previous deferred cleanup was removed by this migration; failed Start must stop and join the client before returning the error.

In `light/rpc/client.go`:
- [SUGGESTION] light/rpc/client.go:111-137: Light client still joins a caller-owned HTTP event client
  (carried; thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127750)

In `rpc/client/http/ws.go`:
- [BLOCKING] rpc/client/http/ws.go:159-166: Scope failed disconnect cleanup to the disconnected subscriber
  (carried; thread: https://github.com/dashpay/tenderdash/pull/1515#discussion_r4471127721)

Comment thread internal/proxy/client.go
Comment on lines 98 to +101
func (app *proxyClient) OnStart(ctx context.Context) error {
var err error
defer func() {
if err != nil {
tryCallStop(app.client)
}
}()
if err := app.client.Start(ctx); err != nil {
return err
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Release the ABCI client when proxy startup fails

If the wrapped ABCI client's Start returns an error after allocating a connection or starting background work, OnStart now returns without calling Stop or Wait. BaseService does not invoke OnStop after a failed OnStart, so a partially started custom or public abciclient.Client can retain its connection or workers while node startup rolls back. The previous deferred cleanup was removed by this migration; failed Start must stop and join the client before returning the error.

Suggested change
func (app *proxyClient) OnStart(ctx context.Context) error {
var err error
defer func() {
if err != nil {
tryCallStop(app.client)
}
}()
if err := app.client.Start(ctx); err != nil {
return err
}
if err := app.client.Start(ctx); err != nil {
tryCallStop(app.client)
app.client.Wait()
return err
}

source: gpt-6.1-sol (phase2-reviewer: tenderdash-consensus-security)

@github-actions

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: high High priority — address soon waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants