Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,39 @@ This guide provides instructions for upgrading to specific versions of Tenderdas

## Unreleased

### Service lifecycle

All implementations embedding `libs/service.BaseService` receive a context in
`OnStart` that is canceled by either the caller or manual `Stop`. Cancellation
now precedes `OnStop`, including for implementations that have not adopted
managed workers. Audit shutdown callbacks that require a live context; preserve
the appropriate owner context explicitly instead of using the canceled work context.

Register owned workers with `BaseService.Go`. `OnStop` must unblock them, not
wait for them; move shared-resource cleanup to optional `OnDrain`, which runs
after those workers exit. Call `Stop` then `Wait` before releasing resources.
Plain goroutines and their existing joins remain the implementation's responsibility.
`TimeoutTicker` implementations must now provide `Wait`.

A stop requested during startup cancels work immediately but defers `OnStop`
until successful startup returns. Failed startup must roll back its resources;
BaseService joins registered work without calling shutdown hooks. Retry requires
recreating any child that already started successfully and then stopped.

Direct consensus `State.Stop` drains a commit already in progress using the
context originally passed to `State.Start`. Cancel that parent to abort it;
reactor/node shutdown still cancels that parent and can require WAL recovery.
This is not an unconditional durable-finalization guarantee for node shutdown.
Once admitted, a consensus handoff belongs to its reactor. Canceling the caller
releases its wait; reactor shutdown cancels and joins the admitted handoff.
Router disconnect notifications during shutdown are best effort; subscribers
must terminate on their own context, not wait for a final `PeerStatusDown`.

Autofile groups require explicit `Close`, even if never started. Canceling the
opening context or calling `Stop`/`Wait` alone does not close the head file.
See [service lifecycle](docs/architecture/service-lifecycle.md) for ownership
and migration boundaries.

### ABCI: commit vote-extension verification

Tenderdash now calls `VerifyVoteExtension` after `ProcessProposal` and before it
Expand Down
24 changes: 23 additions & 1 deletion cmd/tenderdash/commands/run_node.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,13 @@ package commands
import (
"bytes"
"crypto/sha256"
"errors"
"fmt"
"io"
"os"
"os/signal"
"syscall"
"time"

"github.com/spf13/cobra"

Expand Down Expand Up @@ -89,12 +91,18 @@ func addDBFlags(cmd *cobra.Command, conf *cfg.Config) {

// NewRunNodeCmd returns the command that allows the CLI to start a node.
// It can be used with a custom PrivValidator and in-process ABCI application.
// A shutdown timeout leaves cleanup running; callers must terminate the process
// on that error rather than release resources still owned by the node.
func NewRunNodeCmd(nodeProvider cfg.ServiceProvider, conf *cfg.Config, logger log.Logger) *cobra.Command {
var shutdownTimeout time.Duration
cmd := &cobra.Command{
Use: "start",
Aliases: []string{"node", "run"},
Short: "Run the tendermint node",
RunE: func(cmd *cobra.Command, _args []string) error {
if shutdownTimeout <= 0 {
return errors.New("shutdown timeout must be positive")
}
if err := checkGenesisHash(conf); err != nil {
return err
}
Expand All @@ -114,10 +122,24 @@ func NewRunNodeCmd(nodeProvider cfg.ServiceProvider, conf *cfg.Config, logger lo
logger.Info("started node", "chain", conf.ChainID())

<-ctx.Done()
return nil
timer := time.NewTimer(shutdownTimeout)
defer timer.Stop()
done := make(chan struct{})
go func() {
n.Wait()
close(done)
}()
select {
case <-done:
return nil
case <-timer.C:
return fmt.Errorf("node shutdown timed out after %s; cleanup is incomplete", shutdownTimeout)
}
},
}

cmd.Flags().DurationVar(&shutdownTimeout, "shutdown-timeout", 30*time.Second,
"maximum wait for node shutdown before exiting with an error")
AddNodeFlags(cmd, conf)
return cmd
}
Expand Down
157 changes: 157 additions & 0 deletions cmd/tenderdash/commands/run_node_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
package commands

import (
"context"
"errors"
"testing"
"time"

"github.com/stretchr/testify/require"

cfg "github.com/dashpay/tenderdash/config"
"github.com/dashpay/tenderdash/libs/log"
"github.com/dashpay/tenderdash/libs/service"
)

type commandLifecycle struct {
start func(context.Context) error
stop func()
drain func()
}

func (s commandLifecycle) OnStart(ctx context.Context) error { return s.start(ctx) }
func (s commandLifecycle) OnStop() {
if s.stop != nil {
s.stop()
}
}
func (s commandLifecycle) OnDrain() {
if s.drain != nil {
s.drain()
}
}

func TestRunNodeWaitsForShutdown(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
started := make(chan struct{})
draining := make(chan struct{})
release := make(chan struct{})
n := service.NewBaseService(log.NewNopLogger(), "command-test", commandLifecycle{
start: func(context.Context) error { close(started); return nil },
drain: func() { close(draining); <-release },
})
defer n.Wait()
defer close(release)
cmd := NewRunNodeCmd(func(context.Context, *cfg.Config, log.Logger) (service.Service, error) {
return n, nil
}, cfg.DefaultConfig(), log.NewNopLogger())
cmd.SetContext(ctx)
result := make(chan error, 1)
go func() { result <- cmd.RunE(cmd, nil) }()
<-started
cancel()
select {
case <-draining:
case <-time.After(time.Second):
t.Fatal("node did not begin draining")
}
select {
case err := <-result:
t.Fatalf("command returned before final cleanup: %v", err)
case <-time.After(20 * time.Millisecond):
}
// Release final cleanup without closing the channel twice in deferred cleanup.
release <- struct{}{}
select {
case err := <-result:
require.NoError(t, err)
case <-time.After(time.Second):
t.Fatal("command did not return after node drain")
}
}

func TestRunNodeShutdownTimeout(t *testing.T) {
for _, blocked := range []string{"worker", "stop-hook"} {
t.Run(blocked, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
started := make(chan struct{})
release := make(chan struct{})
var n *service.BaseService
hooks := commandLifecycle{start: func(ctx context.Context) error {
if blocked == "worker" {
n.Go(ctx, func(context.Context) { <-release })
}
close(started)
return nil
}}
if blocked == "stop-hook" {
hooks.stop = func() { <-release }
}
n = service.NewBaseService(log.NewNopLogger(), "command-test", hooks)
defer n.Wait()
defer close(release)
cmd := NewRunNodeCmd(func(context.Context, *cfg.Config, log.Logger) (service.Service, error) {
return n, nil
}, cfg.DefaultConfig(), log.NewNopLogger())
cmd.SetContext(ctx)
require.NoError(t, cmd.Flags().Set("shutdown-timeout", "50ms"))
result := make(chan error, 1)
go func() { result <- cmd.RunE(cmd, nil) }()
<-started
cancel()
select {
case err := <-result:
require.ErrorContains(t, err, "node shutdown timed out after 50ms")
case <-time.After(2 * time.Second):
t.Fatal("shutdown deadline blocked behind node cleanup")
}
})
}
}

func TestRunNodeStartupErrors(t *testing.T) {
for _, failure := range []string{"provider", "start", "canceled-start"} {
t.Run(failure, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
startupErr := errors.New("startup failed")
n := service.NewBaseService(log.NewNopLogger(), "command-test", commandLifecycle{
start: func(ctx context.Context) error {
if failure == "canceled-start" {
cancel()
return ctx.Err()
}
return startupErr
},
})
cmd := NewRunNodeCmd(func(context.Context, *cfg.Config, log.Logger) (service.Service, error) {
if failure == "provider" {
return nil, startupErr
}
return n, nil
}, cfg.DefaultConfig(), log.NewNopLogger())
cmd.SetContext(ctx)
err := cmd.RunE(cmd, nil)
if failure == "canceled-start" {
require.ErrorIs(t, err, context.Canceled)
} else {
require.ErrorIs(t, err, startupErr)
}
})
}
}

func TestRunNodeRejectsInvalidShutdownTimeout(t *testing.T) {
for _, timeout := range []string{"0s", "-1s"} {
t.Run(timeout, func(t *testing.T) {
cmd := NewRunNodeCmd(func(context.Context, *cfg.Config, log.Logger) (service.Service, error) {
t.Fatal("invalid deadline must be rejected before creating a node")
return nil, nil
}, cfg.DefaultConfig(), log.NewNopLogger())
require.NoError(t, cmd.Flags().Set("shutdown-timeout", timeout))
require.ErrorContains(t, cmd.RunE(cmd, nil), "shutdown timeout must be positive")
})
}
}
92 changes: 92 additions & 0 deletions docs/architecture/service-lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Service lifecycle core

`BaseService.Start` gives `OnStart` an attempt-scoped child context. Parent
cancellation and manual `Stop` both cancel it. `Go(ctx, fn)` registers workers
using that context or a derived context; registration is synchronized with
shutdown and rejected for a different attempt or after shutdown begins.

Shutdown closes admission, marks `IsRunning` false, cancels the context and
calls `OnStop`. It then joins registered workers, calls optional `OnDrain`, and
releases `Wait`. `Stopping()` signals cancellation separately from completion.
`Stop` invokes `OnStop` synchronously but does not join registered workers.
Concurrent or recursive calls to `Stop` return immediately. During `OnStart`,
`Stop` cancels immediately and defers the shutdown hook until successful startup
returns; `Wait` still joins the attempt.

Hooks run outside the lifecycle mutex. `OnStop` must unblock workers without
waiting for its own registered work. Workers may call `Stop`; neither hooks nor
workers may call their own service's `Wait`. Release resources shared with
workers in `OnDrain`, or after `Stop` followed by `Wait`.

`IsRunning` is false during startup and shutdown. `Wait` joins the current
attempt, including startup and cleanup; it returns immediately before startup.
Failed `OnStart` cancels and joins its workers before retry becomes possible.
The implementation must roll back acquired resources: failed attempts do not
invoke `OnStop` or `OnDrain`. Successfully stopped services cannot restart.
Composite retries also require recreating children already successfully stopped.

## First migration stage

Consensus State registers its receive loop and joins its queue fan-in and WAL
before returning. The timeout ticker, WAL and autofile group use shared worker
tracking. AutoFile explicitly joins its timer/signal worker. WAL ownership lasts
until the receive loop's final write. Consensus handoff admission is also joined,
so a late blocksync handoff cannot start State after reactor shutdown. The caller
may cancel its wait after admission, but the reactor still owns and joins the
handoff. Startup and the resulting State use the reactor's lifetime.

Node waits for reactors before closing event sinks and stores. The blocked
receive-loop regression and original node teardown tests verify this path
without using leaktest as an extra production-shutdown barrier.

This is a partial migration. Plain goroutines are not tracked automatically.
RPC/WebSocket handlers, mempool rechecks, reactor gossip and other service
workers retain their existing ownership mechanisms. This change does not claim
that Node.Wait joins every helper or introduce the broader durable-finalization
and dependency-lifetime changes from #1515.

Three compatibility bridges preserve parent-context lifetimes: blocksync
application during handoff, connection error callbacks after I/O shutdown, and
an admitted consensus commit during direct State.Stop. Parent cancellation still
aborts these operations. In particular, reactor/node shutdown can interrupt
finalization and require WAL recovery. Their existing ownership mechanisms remain
in place; this does not detach application work from node shutdown.

## Cancellation compatibility audit

Cancellation precedes OnStop for every BaseService implementation, not only the
managed-worker migrations. The in-tree hook/callback audit covers these groups:

| Implementations | Cancellation and cleanup policy |
| --- | --- |
| Consensus State, reactor, ticker, WAL; autofile Group | Managed work is joined; commit uses the State parent context; file ownership ends with explicit Close. |
| Blocksync reactor and Synchronizer | Handoff preserves application work across Synchronizer.Stop; stopping its parent reactor may cancel it. Reactor OnStop stops and waits for the Synchronizer, so reactor Wait covers in-flight application. |
| P2P Router and MConnection | Connection callbacks retain the parent. Router removes peer bookkeeping even with a canceled context; final PeerStatusDown broadcasts are best effort. |
| PEX, mempool, statesync and evidence reactors | Processing uses the work context; stop hooks close resources/dispatchers or require no extra signal. Existing plain-worker ownership is unchanged. |
| EventBus, pubsub Server and indexer Service | Event processing is cancellation-driven; indexer closes sinks in its existing hook. No guarantee of a live cleanup callback context. |
| ABCI local, socket, gRPC and routed clients; proxy; socket/gRPC servers | Stop hooks close transports, drain requests or stop children without requiring a live work context. |
| SignerServer, signer listener/dialer endpoints | Hooks close endpoints/listeners; dial/retry work observes cancellation. |
| ValidatorConnExecutor and light RPC Client | Unsubscribe uses a separate bounded cleanup context; client hooks invoke owned closers. |
| Node and seed node | Parent cancellation stops children; hooks wait for existing service completion before closing owned stores. |

This is a cancellation compatibility audit, not a claim that unmigrated plain
workers are all joined. Subscribers must stop with their owner, without relying
on a final router broadcast. Avoid detaching broadcasts from cancellation:
unresponsive subscribers could otherwise block shutdown.

Group rotation reuses its AutoFile and closes the current descriptor before
renaming; it does not create a worker or signal registration per rotation. The
owner must Close the group, including when it never started, to join the head's
single timer/signal worker. Context cancellation alone does not release the head.

## CLI shutdown deadline

The `start` command cancels the node on SIGINT or SIGTERM and waits for
`Node.Wait` before returning. `--shutdown-timeout` controls this wait (default
30 seconds; must be positive). Set the deployment supervisor's termination grace
period longer than this timeout to allow Tenderdash to report incomplete cleanup.
If the deadline expires, the command returns an error and the Tenderdash binary
exits nonzero. This does not skip reactor joins or close stores while workers
remain active. An embedded command caller must also terminate on this error;
cleanup may still be running. The deadline bounds shutdown after startup returns,
not a blocked startup hook.
Loading
Loading