Skip to content

fix(service)!: share worker lifecycle for consensus shutdown - #1513

Merged
lklimek merged 15 commits into
v1.8-devfrom
fix/consensus-wait-shutdown
Oct 23, 2026
Merged

lklimek merged 15 commits into
v1.8-devfrom
fix/consensus-wait-shutdown

Conversation

@lklimek

@lklimek lklimek commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Consensus shutdown now waits for its receive loop and file workers before node teardown releases their resources. This is the focused lifecycle-core extraction from #1515.

Issue being fixed or feature implemented

Node shutdown could return while consensus still used its data directory, causing teardown failures such as directory not empty. A shared worker lifecycle replaces the consensus-only cancellation and WaitGroup workaround.

What was done?

  • The CLI waits for node shutdown after SIGINT/SIGTERM. --shutdown-timeout defaults to 30 seconds and must be positive; expiry returns an explicit error and the binary exits nonzero. Internal reactor joins and store-close ordering remain intact.

  • BaseService startup uses lock-scoped lifecycle phases, preserving admission, retry, and concurrent Stop behavior. CLI deadline and supervisor/embedding requirements are documented.

  • Added attempt-scoped contexts, synchronized worker admission, cancellation and drain to BaseService. Optional OnDrain runs after registered work finishes.

  • Migrated consensus State, timeout ticker, WAL and autofile group; removed their redundant lifecycle fields and Wait overrides. Explicitly joined AutoFile timer/signal work.

  • Joined consensus handoffs so they cannot start State after reactor shutdown. A caller may cancel its wait after admission; the reactor continues to own and join startup. Node waits for reactors before closing event sinks and stores.

  • Preserved parent lifetimes for blocksync application during handover, connection error callbacks, and consensus processing already in flight during direct State.Stop. Parent cancellation still aborts this work.

  • BaseService cancels the attempt before running deferred shutdown hooks, so OnStop always observes a canceled context, even when Stop races startup. The blocksync reactor waits for its Synchronizer after stopping it, so reactor shutdown cannot release store teardown while block application is still draining.

  • Removed the node test's extra leaktest shutdown barrier. Added upgrade guidance, an in-tree cancellation-hook audit, explicit autofile Close ownership, and documented best-effort router shutdown notifications.

This PR is independent of #1515 and carries only its core. RPC/WebSocket ownership, mempool recheck drain, broad P2P/ABCI/reactor migrations and the broader durable-finalization/dependency-lifetime changes remain outside this PR. Plain goroutines are not automatically tracked; this does not promise that Node.Wait joins every repository helper.

How Has This Been Tested?

  • The CLI regression first reproduced premature return before final cleanup, then passed with the fix. CLI cases and the BaseService lifecycle suite passed with race/deadlock detection, including successful drain, hung workers/shutdown hooks, startup failure/cancellation, invalid deadlines, and concurrent lifecycle transitions. Scoped golangci-lint v2.15.0 and goimports passed for these changes; no full repository or E2E suite was run for this follow-up.

  • Confirmed red then green regressions for shared manual-stop cancellation, late consensus handoff and callback context preservation.

  • Existing blocksync handover regressions exposed the context compatibility issue and passed after its focused correction.

  • Affected packages passed go test -race -tags=deadlock -p 1: service, autofile, consensus, node, blocksync and P2P connections.

  • Both original node shutdown regressions passed 100 repetitions with GOMAXPROCS=2 and race/deadlock enabled. Late handoff passed 10 repetitions; existing blocksync handover cases passed 5 repetitions.

  • Goimports formatting and scoped golangci-lint passed with 0 issues.

  • Blocked FinalizeBlock and handoff cancellation regressions failed before the fixes and passed afterward. They verify durable state after direct State.Stop, parent cancellation, message metadata, and reactor ownership after a caller stops waiting.

  • Follow-up race/deadlock tests passed for consensus, node, blocksync, autofile, service, and the router's canceled-context peer cleanup. The lifecycle regression subset also passed 10 repetitions. Scoped lint passed with 0 issues.

  • CI and both e2e networks (dashcore, rotate) passed on 8e431fc. CI for the current head is pending.

Breaking Changes

The CLI now waits for shutdown after cancellation, up to --shutdown-timeout (default 30s). Configure the supervisor's termination grace period longer than this deadline. Embedded command callers must terminate on a shutdown-timeout error because cleanup may still be running. This deadline applies after startup returns.

Manual Stop cancels the context passed to OnStart. Stop invokes OnStop synchronously but registered work is joined by Wait; callers must Stop then Wait before releasing worker resources. Hooks run outside the lifecycle lock and must not wait for their own registered work. TimeoutTicker implementations now require Wait. Failed startup may retry after worker cleanup; successfully stopped services cannot restart.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Co-authored by Claudius the Magnificent AI Agent

PR Hygiene · 2e7f2e3

  • Bots — thepastaclaw ✓
  • Self-review — post /self-reviewed
  • Build green
  • Approvals — you own every area touched; none needed

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Join the receive routine before State.Wait returns and cancel its context
on explicit Stop. This prevents node cleanup racing in-flight consensus
file writes. Add a regression test for the shutdown contract.

Co-Authored-By: OpenAI Codex GPT-6 <noreply@openai.com>

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
@lklimek lklimek added the claudius-review Trigger Claudius AI code review on this PR label Sep 24, 2026
@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@thepastaclaw

thepastaclaw commented Sep 24, 2026 •

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit 2e7f2e3) · triage: normal

lklimek and others added 2 commits September 25, 2026 09:15
Port the focused lifecycle core from #1515, replacing the local State wait workaround while retaining other service ownership mechanisms.

Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
@lklimek lklimek changed the title fix(consensus): wait for receive routine during shutdown fix(service)!: share worker lifecycle for consensus shutdown Sep 25, 2026

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claudius review — verdict: Request changes. 15 findings across 3 reviewers (0 CRITICAL, 0 HIGH, 5 MEDIUM, 8 LOW, 2 INFO). One LOW-severity finding is classified blocking because it plausibly trips the G-GROWTH gate (unbounded resource growth). The fourth planned reviewer (project-reviewer-adams) was skipped per this pipeline's early-stop rule once the security reviewer's own gate returned a blocking candidate.

Blocking — please confirm before merge:

  • SEC-008 — internal/libs/autofile/group.go:82-86: OpenGroup's head AutoFile is now opened with context.WithoutCancel, so its goroutine, file descriptor, and SIGHUP registration are released only by an explicit Close(). In-tree WAL usage looks covered (traced every OnDrain/Wait call site, found no dropped Group), but if any rotation or reopen path ever skips Close(), this leaks per-occurrence and grows unboundedly over a long-running validator's life. Please confirm Close() is unconditionally reached on every group rotation before merging — this is exactly the kind of thing static reading alone can't fully settle.

Worth a maintainer's close look (non-blocking, but not to be waved away):

  • SEC-001 — the refactor now cancels a service's context before calling OnStop() (previously the reverse). This defeats the pre-PR safeguard that let an in-progress consensus ApplyCommit finish before the timeout ticker stopped, with no replacement guard added. Caveat, in fairness: the guard may already have been partly defeated by pre-existing Reactor.OnStop ordering, so this could narrow an existing gap rather than open a brand-new one — but it's exactly the kind of shutdown-correctness regression a PR titled "fix consensus shutdown" shouldn't be introducing.
  • CALL-001 — that same cancel-before-OnStop reordering is a contract change hitting every service.Implementation in the tree (~30 embedders), of which this PR audited and bridged only 2 (blocksync synchronizer, p2p connection). SEC-006/SEC-007 found a third unmigrated call site (p2p/router.go's routePeer) with a real, if low-impact-in-production, behavioral delta.
  • DOC-001 — this is a breaking change (fix(service)!:) to a public package's runtime contract with no UPGRADING.md entry.
  • SEC-002 — an unchecked type assertion in blocksync/synchronizer.go:255 that would panic if OnStart is ever invoked outside its Start() wrapper; the same PR already uses the safe comma-ok idiom for the equivalent case in connection.go, so this is inconsistency more than a proven live bug.
  • SEC-003 — SwitchToConsensus now blocks uninterruptibly because the caller's context is discarded after admission; a wedged WAL replay can no longer be cancelled by its caller.

Also flagged (LOW/INFO, no action required to merge): two stale godocs left behind by the refactor (ticker.go, wal.go), an architecture-doc gap on the Stop-during-startup case, an unsynchronized field write in State.OnStart's rollback path, a nil-channel-block edge case in the new Stopping(), and — credit where due — the new lifecycle_test.go suite is genuinely solid engineering, not padding.

Full detail in the report.json/report.html CI artifacts.
📊 View full HTML review report

Comment thread libs/service/service.go Resolved
Comment thread libs/service/service.go Resolved
Comment thread internal/consensus/state.go Resolved
Comment thread internal/blocksync/synchronizer.go Resolved
Comment thread internal/consensus/reactor.go Resolved
@github-actions github-actions Bot removed the claudius-review Trigger Claudius AI code review on this PR label Sep 25, 2026
@lklimek

lklimek commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Checked all five open threads and the full 15-finding report against 8e431fc. No code changes in this verification pass. The five threads remain open; individual assessments are in replies.

The reported rotation leak is not substantiated: Group.rotateFile retains the same AutoFile, closes its current descriptor under lock, and reopens lazily. It does not allocate a new worker or SIGHUP registration per rotation. WAL OnDrain calls Group.Close; failed WAL startup and repair also close/join their old group. Explicit Close ownership should be documented, and logjack should defer Close, but its process-exit cleanup gap does not establish unbounded validator growth or a merge blocker.

Remaining small items: stale ticker/WAL hook comments and the architecture document's missing startup-stop exception are valid. The synchronizer comment should say parent lifetime: the captured parent can be the blocksync reactor, so manual reactor shutdown also cancels application.

The rollback data-race claim is not demonstrated by a concurrent reader: startup failure precedes receiveRoutine admission, and BaseService defers OnStop while startup is active. A custom successfully stopped ticker must be recreated for retry, as the architecture document already requires for composite children. A lock around assignments alone would not establish reader synchronization. The Stopping nil-channel behavior is documented; a blocked ScheduleTimeout on an unstarted ticker is an API edge case requiring a concrete supported call path before treating it as a regression.

The substantive remaining concerns are the global cancellation-contract audit (including router shutdown notifications), in-flight consensus finalization policy, handoff caller-cancellation policy, and upgrade guidance. Static verification only; no new tests were run.

🤖 Co-authored by Claudius the Magnificent AI Agent

Keep processing on the State parent's lifetime while canceling intake, and
let handoff callers cancel their wait without abandoning reactor-owned work.
Document the global cancellation contract, cleanup ownership and migration
boundaries; harden context lookup and close logjack's group on return.

Co-Authored-By: OpenAI GPT-6 <noreply@openai.com>
@lklimek

lklimek commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the actionable feedback in ec8d306:

  • Direct State.Stop cancels intake while already-running consensus processing retains the State parent's context. The regression blocks actual FinalizeBlock, verifies Wait stays blocked, then verifies the committed height is persisted. Parent cancellation still aborts finalization; reactor/node shutdown may require WAL recovery. This deliberately does not pull in the detached dependency graph from fix(service)!: join remaining service workers before shutdown completes #1515.
  • Canceling a handoff caller releases its wait; admitted startup remains reactor-owned. The regression verifies both that startup remains live after caller cancellation and that reactor shutdown joins it.
  • Added UPGRADING.md migration guidance and an inventory of all in-tree stop-hook groups. Router bookkeeping cleanup still runs, while shutdown broadcasts are explicitly best effort; existing canceled-context peer-manager tests pass.
  • Made the synchronizer context assertion safe, corrected hook/startup documentation, documented explicit autofile Close ownership, and added logjack's deferred Close.

The two main regressions were red before the fix, then green. Affected package tests pass with race/deadlock enabled; the lifecycle regression subset passed 10 repetitions. Final consensus/node tests and scoped lint pass (0 lint issues). New CI is pending.

The speculative per-rotation resource-growth finding is not reproduced: rotation reuses one AutoFile and closes its descriptor; final Close joins its worker. No speculative rollback locks or Stopping contract changes were introduced.

🤖 Co-authored by Claudius the Magnificent AI Agent

@lklimek
lklimek marked this pull request as ready for review September 30, 2026 12:06
@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Sep 30, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

The lifecycle changes correctly coordinate worker admission, cancellation, draining, consensus handoff ownership, and file cleanup within the PR's stated partial-migration scope. Source inspection and the shutdown regressions support the supplied clean findings; no actionable in-scope defects were confirmed. All seven affected packages passed a fresh race/deadlock-enabled test run using a compatible BLS build, and the worktree remains unchanged.

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: muse-spark-1.3-contributor (agent: phase1-reviewer, role: tenderdash-consensus-security); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — The diff introduces intricate, cross-cutting worker lifecycle and shutdown synchronization changes in libs/service/service.go and consensus integration, but does not change consensus rules, cryptography, funds handling, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer, muse-spark-1.3-contributor — tenderdash-consensus-security (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity below 15% reserve: weekly 13% left, 5h 100% left), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer

@github-actions

Copy link
Copy Markdown

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Sep 30, 2026
Retain the service-lifecycle upgrade guidance alongside the base's ABCI
commit-verification and mempool upgrade instructions. Preserve the base's
consensus test fixtures with the shutdown lifecycle regressions.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>

Co-Authored-By: Codex GPT-6 <noreply@openai.com>
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Oct 21, 2026
Comment thread node/node.go Resolved
Comment thread libs/service/service.go Resolved
Join node cleanup after cancellation with a configurable shutdown deadline.
Return an error on timeout without bypassing service joins or store ordering.
Organize BaseService startup into lock-scoped lifecycle phases.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>

Co-Authored-By: Codex <noreply@openai.com>
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@lklimek lklimek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/self-reviewed

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Oct 22, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

Static verification confirms one blocking signer-startup cancellation race and one test-owned WAL resource leak. The three prior shutdown-ordering findings are fixed: deferred shutdown hooks establish cancellation first, and the blocksync reactor joins its synchronizer even after the sync flag is cleared. Current-head CI reports passing build, unit tests, lint, vet, and dashcore E2E checks; rotate E2E remains pending, and no local builds or tests were run.

🔴 1 blocking | 🟡 1 suggestion(s)

2 finding(s) not shown inline (the lines are not part of this PR's diff)

🔴 Blocking: Make the signer's initial connection request cancellation-aware
privval/signer_listener_endpoint.go:77-80

The initial connection request is an unconditional send on an unbuffered channel. BaseService now allows Stop during OnStart and immediately cancels the hook's context. If serviceLoop selects ctx.Done before accepting this request, it exits and leaves OnStart permanently blocked on the send. BaseService defers OnStop until startup returns, so Start and Wait cannot complete, and the listener and ticker never reach their shutdown cleanup. Make the send cancellation-aware and cover Stop racing startup before the first request is accepted. Returning nil after the cancellation branch preserves BaseService's successful-start shutdown path; returning ctx.Err() instead requires explicit resource rollback because failed startup does not invoke OnStop.

	go sl.serviceLoop(ctx)
	go sl.pingLoop(ctx)

	select {
	case sl.connectRequestCh <- struct{}{}:
	case <-ctx.Done():
	}

source: muse-spark-1.3-contributor (phase1-reviewer: general); gpt-6.1-sol (phase2-reviewer: general)

🟡 Suggestion: Close the unstarted WAL in the search test
internal/consensus/wal_test.go:215-216

This test constructs a WAL but never starts it or closes its group. OpenGroup now passes context.WithoutCancel(ctx) to OpenAutoFile, so the deferred cancel no longer terminates the head's timer/signal worker or removes its SIGHUP registration. Closing the search reader only closes its separate reader file, and Stop/Wait on an unstarted WAL cannot invoke OnDrain. Register Group.Close immediately after construction to release the worker and registration. The descriptor is periodically closed by AutoFile, so the persistent leak here is the worker and signal registration rather than a permanently open descriptor.

	wal, err := NewWAL(ctx, logger, walFile)
	require.NoError(t, err)
	t.Cleanup(wal.Group().Close)

source: muse-spark-1.3-contributor (phase1-reviewer: general); gpt-6.1-sol (phase2-reviewer: general, tenderdash-consensus-security)

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — The change is intricate and cross-cutting but modifies shutdown lifecycle, cancellation, and resource-drain ordering rather than consensus rules, cryptography, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `privval/signer_listener_endpoint.go`:
- [BLOCKING] privval/signer_listener_endpoint.go:77-80: Make the signer's initial connection request cancellation-aware
  The initial connection request is an unconditional send on an unbuffered channel. BaseService now allows Stop during OnStart and immediately cancels the hook's context. If serviceLoop selects ctx.Done before accepting this request, it exits and leaves OnStart permanently blocked on the send. BaseService defers OnStop until startup returns, so Start and Wait cannot complete, and the listener and ticker never reach their shutdown cleanup. Make the send cancellation-aware and cover Stop racing startup before the first request is accepted. Returning nil after the cancellation branch preserves BaseService's successful-start shutdown path; returning ctx.Err() instead requires explicit resource rollback because failed startup does not invoke OnStop.

In `internal/consensus/wal_test.go`:
- [SUGGESTION] internal/consensus/wal_test.go:215-216: Close the unstarted WAL in the search test
  This test constructs a WAL but never starts it or closes its group. OpenGroup now passes context.WithoutCancel(ctx) to OpenAutoFile, so the deferred cancel no longer terminates the head's timer/signal worker or removes its SIGHUP registration. Closing the search reader only closes its separate reader file, and Stop/Wait on an unstarted WAL cannot invoke OnDrain. Register Group.Close immediately after construction to release the worker and registration. The descriptor is periodically closed by AutoFile, so the persistent leak here is the worker and signal registration rather than a permanently open descriptor.

@github-actions

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Oct 22, 2026
lklimek and others added 2 commits October 22, 2026 17:05
…aware

SignerListenerEndpoint.OnStart sent its first connection request on an
unbuffered channel unconditionally. With Stop now cancelling OnStart's
context, serviceLoop can exit on ctx.Done before taking the request,
leaving Start blocked forever and OnStop never run. The send now also
selects on ctx.Done and returns nil, so the regular shutdown path still
releases the listener and ping ticker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Height

OpenGroup detaches the head AutoFile worker from the caller's context,
so the never-started WAL leaked its timer/SIGHUP worker. Register the
group's Close as cleanup and take the leak snapshot at test start so
the check covers the WAL's lifetime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Claudius-Maginificent

Copy link
Copy Markdown
Collaborator

Re: review of 07d627c — both findings confirmed and fixed.

  • Signer startup cancellation (privval/signer_listener_endpoint.go): the initial connect request now also selects on ctx.Done, so Stop or cancellation during startup can no longer leave OnStart blocked once serviceLoop has exited. It still returns nil, so OnStop releases the listener and ticker (5584849). TestSignerListenerEndpointStartupCanceledBeforeConnectRequest failed before the fix and passes now.
  • WAL search test (internal/consensus/wal_test.go): TestWALSearchForEndHeight now closes the unstarted WAL group, and its leak check is registered first so it covers the test's whole lifetime; with only the check moved, it caught the leaked AutoFile worker (e7fd99d).

A syntactic sweep of non-test OnStart bodies found no other unconditional blocking channel operation.

Co-authored by Claudius the Magnificent AI Agent

@github-actions

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Oct 22, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The lifecycle extraction and the five previously reported fixes are correct at the reviewed head. One blocking admission race remains: a canceled reactor context can be admitted as a worker before the asynchronous shutdown watcher marks the service attempt as stopping, allowing consensus handoff code to run during shutdown. The supplied exact-head CI snapshot reports successful build, tests, lint, vet, vulnerability checks, and E2E checks; PR Hygiene remains pending.

🔴 1 blocking

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large, intricate cross-cutting lifecycle and concurrency change affecting node shutdown, consensus worker ownership, and storage teardown, but it does not change consensus rules, cryptography, signatures, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `libs/service/service.go`:
- [BLOCKING] libs/service/service.go:150-152: Reject canceled contexts during worker admission
  BaseService.Go synchronizes admission against the attempt and a.stopping, but it does not check whether the supplied context is already canceled. SwitchToConsensus checks its caller context before calling Go, then admits the worker with r.ctx; those contexts can diverge. If parent cancellation closes r.ctx before the asynchronous BaseService watcher acquires the lifecycle lock and marks the attempt as stopping, Go still admits the handoff because a.stopping is false. The callback can then update consensus state and reach State.Start with a canceled context while shutdown is already in progress, racing State.Stop and violating the lifecycle admission invariant. Check ctx.Err() at the synchronized admission point and add a regression covering cancellation between the caller precheck and Go admission.

Comment thread libs/service/service.go Resolved
@github-actions

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Oct 22, 2026
Parent cancellation closes the attempt context before the async watcher
takes the lifecycle lock and publishes stopping. In that window Go still
admitted work, e.g. a consensus handoff that could reach State.Start
during shutdown. Go now also refuses a canceled ctx under the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Oct 22, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

Verified the complete diff at head 2e7f2e3 and independently confirmed that all six prior findings are fixed. No additional actionable in-scope issues were identified. This was a static review with no local builds or tests; current CI shows the build, all six unit-test shards, both E2E networks, ABCI/application tests, vet, and lint passing, with only PR Hygiene pending.

🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large, intricate cross-cutting lifecycle and shutdown change affecting consensus and storage-adjacent concurrency, but it does not change consensus rules, cryptography, signatures, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.

No unresolved findings remain from the prior review on this head.

@github-actions

Copy link
Copy Markdown

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Oct 22, 2026

@lklimek lklimek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/self-reviewed

@lklimek
lklimek enabled auto-merge (squash) October 23, 2026 07:16
@lklimek
lklimek disabled auto-merge October 23, 2026 07:26
@lklimek
lklimek merged commit 40aeb0b into v1.8-dev Oct 23, 2026
27 of 28 checks passed
@lklimek
lklimek deleted the fix/consensus-wait-shutdown branch October 23, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants