Skip to content

fix(docker): wait for tenderdash to exit after forwarding a signal - #1607

Merged
lklimek merged 5 commits into
devfrom
fix/docker-entrypoint-wait-shutdown
Nov 10, 2026
Merged

lklimek merged 5 commits into
devfrom
fix/docker-entrypoint-wait-shutdown

Conversation

@Claudius-Maginificent

@Claudius-Maginificent Claudius-Maginificent commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR: Stopping a Tenderdash container now lets Tenderdash finish its shutdown instead of killing it halfway.

User story

As a node operator (for example through dashmate), I want docker stop and restarts to shut Tenderdash down cleanly, to avoid every stop, restart or update behaving like a crash.

Scenario

Base flow

An operator stops or restarts a Tenderdash container. Docker sends SIGTERM to the container's main process, the entrypoint script, which forwards it to Tenderdash.

Actual behavior

The entrypoint exits right after forwarding the signal, without waiting for Tenderdash. Because it is the container's main process, Docker then ends the container and Tenderdash is killed in the middle of its shutdown. docker stop -t 40 returns in about 0.2 s.

Expected behavior

The entrypoint waits until Tenderdash has finished shutting down (up to its 30 s --shutdown-timeout, within Docker's stop grace period) and exits with Tenderdash's exit code.

Detailed discussion

What was done

In DOCKER/docker-entrypoint.sh, a trapped signal makes bash's wait "$child" return early with status 128+signal while Tenderdash is still running. The signal handler now also sets interrupted, and the script repeats wait until one completes without interruption, so it exits only after Tenderdash does and with Tenderdash's own status. If Tenderdash has already exited by then, bash returns its saved status. A repeated signal is forwarded again, as before. The 10-second sleep after a non-signal failure (#845) is unchanged.

DOCKER/test-entrypoint.sh runs the entrypoint with a fake tenderdash on PATH and checks:

  • a slow shutdown: the entrypoint waits for it and keeps its exit status;
  • a second signal during the shutdown;
  • an immediate exit on the signal, repeated 20 times: the status is 0, not 143;
  • a plain non-zero exit without a signal.

The Release tooling workflow runs it when the entrypoint or the test changes.

Testing

  • DOCKER/test-entrypoint.sh passes; on the original entrypoint from dev it fails (the entrypoint exits before Tenderdash finishes).
  • shellcheck is clean on both scripts.

Breaking changes

None. Containers stop as late as Tenderdash's shutdown takes, bounded by Docker's stop timeout (default 10 s; dashmate sets stop_grace_period: 40s, dashpay/platform#5836).

Prior work

Attribution

🤖 Generated with Claude Code

🤖 Co-authored by Claudius the Magnificent AI Agent

PR Hygiene · dd97674

  • Bots — thepastaclaw ✓
  • Self-review — not asked of a bot author
  • Build green
  • Approvals
    • github (.github/workflows/release.yml) — approved by lklimek
    • files with no dedicated owner (DOCKER/docker-entrypoint.sh, DOCKER/test-entrypoint.sh) — approved by lklimek

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

A signal trapped by the entrypoint interrupts `wait`, so the script (PID 1)
exited right after forwarding SIGTERM and the kernel killed tenderdash
mid-shutdown. `docker stop` returned within milliseconds and the 30 s
--shutdown-timeout never ran. The entrypoint now keeps waiting until
tenderdash has exited and returns its exit code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thepastaclaw

thepastaclaw commented Nov 10, 2026 •

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit dd97674) · triage: low

@lklimek lklimek added this to the v2.0 milestone Nov 10, 2026
@lklimek
lklimek marked this pull request as ready for review November 10, 2026 10:22
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Nov 10, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 2 only (queue backlog)

The signal-forwarding change correctly keeps the container process alive during Tenderdash shutdown, and all reported code checks passed. However, the liveness-gated retry has a race that can preserve Bash's signal-interrupted status instead of Tenderdash's final exit status, so the PR still has one blocking issue. The lifecycle-sensitive behavior also lacks a committed regression harness.

🔴 1 blocking | 🟡 1 suggestion(s)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.1-sol (effort low) — This is a small, self-contained shell-script behavior fix in the Docker entrypoint, and its correctness is straightforward to verify without touching a critical code surface.
  • Phase 1 reviewers: not run (skipped for throughput: 13 PRs queued, above the 10 limit)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `DOCKER/docker-entrypoint.sh`:
- [BLOCKING] DOCKER/docker-entrypoint.sh:59-62: Collect the child status even when it exits before the liveness check
  When a trapped signal interrupts the first `wait`, `exit_code` contains Bash's signal status (for example, 143), not Tenderdash's status. If Tenderdash exits and Bash reaps the child between that interrupted `wait` and `kill -0`, the liveness check fails and the loop is skipped, leaving 143 as the script's result even when Tenderdash exited successfully or with another status. The same race can occur after a later interrupted wait. Do not use a failed `kill -0` as proof that the preceding `wait` collected the child's final status; track whether each wait was interrupted and perform a final status-bearing wait after shutdown completion.
- [SUGGESTION] DOCKER/docker-entrypoint.sh:53-62: Commit a regression harness for signal-handling behavior
  The entrypoint now contains lifecycle-sensitive signal and exit-status logic, but the regression harness described in the PR is not committed. The repository's existing e2e Docker entrypoints do not exercise this production entrypoint. Add a lightweight automated harness covering delayed cleanup, repeated signals, ordinary nonzero exits, and a child that exits immediately after receiving the signal, including the exit-status race identified above, so future changes cannot silently reintroduce premature exits or status loss.

Comment thread DOCKER/docker-entrypoint.sh Resolved
Comment thread DOCKER/docker-entrypoint.sh Resolved
@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 10, 2026
lklimek and others added 2 commits November 10, 2026 10:45
…ignal handler

The entrypoint checked `kill -0` to decide whether to wait again after a
trapped signal interrupted `wait`. If tenderdash exited, and was reaped,
between the interrupted wait and that check, the script exited with the
interrupted wait's 143 instead of tenderdash's status. The signal handler
now marks the wait as interrupted, and the script waits again until a wait
completes uninterrupted; bash returns the saved status of a child that
has already exited.

DOCKER/test-entrypoint.sh checks a slow shutdown, a repeated signal, an
immediate exit on the signal and a plain non-zero exit with a fake
tenderdash; the release tooling workflow runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 10, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The entrypoint fix, regression harness, and release-workflow integration are correct, and the supplied CI snapshot shows all code checks succeeding. One test-robustness nitpick remains: a wedged entrypoint can make the harness wait indefinitely instead of failing within a bounded time; PR Hygiene and this review are still pending, but no code checks have failed.

💬 1 nitpick(s)

Approved: nothing here blocks the merge. The non-blocking findings are optional: fix them forward, in this PR or a follow-up, without waiting for another review round.

Review provenance

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.1-sol (effort low) — This is a small, self-contained Docker entrypoint signal-handling fix with focused tests and no consensus, funds, cryptography, networking, or storage changes.
  • Phase 1 reviewers: glm-5.3-flash — general (completed, effort high); agent phase1-reviewer
  • Phase 1 model: glm-5.3-flash — zai quota: 5h 99% left, weekly 91% left; passed over gemini-3.8-flash-high (antigravity quota lookup failed)
  • Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `DOCKER/test-entrypoint.sh`:
- [NITPICK] DOCKER/test-entrypoint.sh:23-24: Bound each entrypoint run with timeout so a wedged entrypoint fails instead of hanging CI
  `run_entrypoint` backgrounds the production entrypoint and then waits for it without a deadline. If a future signal-handling regression causes the entrypoint or its fake Tenderdash process to remain alive indefinitely, the release-tooling job hangs until its job-level timeout rather than reporting a focused test failure. Wrap the invocation with a generous timeout so a wedged run returns a failure status while preserving the existing signal-forwarding behavior.

Comment thread DOCKER/test-entrypoint.sh Resolved
@github-actions github-actions Bot removed the waiting-bots Waiting for the review bots to report on this head label Nov 10, 2026
@github-actions

Copy link
Copy Markdown

Ready for review — github: ktechmidas or lklimek or shumkov · files with no dedicated owner: lklimek or shumkov.
Full checklist in the description.

@github-actions github-actions Bot added the ready-for-human PR Hygiene: bots done, author self-reviewed, awaiting human review label Nov 10, 2026
lklimek and others added 2 commits November 10, 2026 12:53
…eout

A wedged entrypoint now fails the check instead of hanging the job.
--foreground keeps timeout from signalling the fake tenderdash directly,
so the test still proves the entrypoint forwards the signal, and
--preserve-status reports the entrypoint's own exit status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed ready-for-human PR Hygiene: bots done, author self-reviewed, awaiting human review labels Nov 10, 2026
@lklimek
lklimek enabled auto-merge (squash) November 10, 2026 13:43

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The signal-forwarding implementation is correct at the reviewed head, and the committed regression harness is wired into the release workflow. One maintainability suggestion remains: document the Bash saved-status behavior that makes the repeated wait safe after an interrupted wait.

🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)

Approved: nothing here blocks the merge. The non-blocking findings are optional: fix them forward, in this PR or a follow-up, without waiting for another review round.

Follow-up candidates (optional) (1)

New non-blocking observations on code unchanged since the last full review. No inline threads are opened for them and later reviews do not repeat them: fix forward or track separately if they are worth it.

🟡 Suggestion: Document Bash's saved-status behavior in the re-wait loop
DOCKER/docker-entrypoint.sh:59-64

The loop is correct today because the entrypoint intentionally does not enable set -e and immediately captures wait's status. Its other non-obvious invariant is that Bash returns the saved child status when wait is called again after the child has already exited; documenting that behavior will help prevent a future maintainer from replacing the loop with a liveness check or otherwise reintroducing the status race.

while :; do
	interrupted=false
	# Bash re-reports the saved child status when wait is called again after
	# the child has exited; the interrupted status is consumed below.
	wait "$child"
	exit_code=$?
	[ "$interrupted" = true ] || break
done

source: glm-5.3-flash (phase1-reviewer: general)

Review provenance

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.1-sol (effort low) — This is a small, well-contained Docker entrypoint signal-handling change with focused tests and no critical consensus, funds, cryptography, or storage impact.
  • Phase 1 reviewers: glm-5.3-flash — general (completed, effort high); agent phase1-reviewer
  • Phase 1 model: glm-5.3-flash — zai quota: 5h 96% left, weekly 90% left; passed over gemini-3.8-flash-high (antigravity quota lookup failed)
  • Code unchanged since the last final review (8370b72b): 1 new suggestion(s) listed as follow-up candidates, not posted inline
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.

No unresolved findings remain from the prior review on this head.

@github-actions

Copy link
Copy Markdown

Policy satisfied — this can merge.
Full checklist in the description.

@github-actions github-actions Bot removed the waiting-bots Waiting for the review bots to report on this head label Nov 10, 2026
@lklimek
lklimek merged commit 7ef1a84 into dev Nov 10, 2026
31 checks passed
@lklimek
lklimek deleted the fix/docker-entrypoint-wait-shutdown branch November 10, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants