failover: on-demand operator promotion

book/api/cli.md

@@ -190,6 +190,16 @@ argument is required and must be the path to an Agave style
 `identity.json` keypair file. If the path is specified as `-` the key
 will instead be read from `stdin`.
 
+::: tip NOTE
+
+Between two Firedancer validators, Firedancer supports a native
+[failover](#failover-promote) mechanism that is faster and easier than
+`set-identity`, and using `set-identity` between them is heavily
+discouraged. Use `set-identity` primarily for failing over between Agave
+and Firedancer.
+
+:::
+
 It is not generally safe to call `set-identity`, as another validator
 might be running with the same identity, and if they both produce a
 block or vote concurrently, the validator may violate consensus and be
@@ -333,6 +343,113 @@ than 16).
 
 <<< @/snippets/commands/add-authorized-voter.ansi
 
+## `failover promote`
+This command asks a remote active validator to hand off its identity and
+vote history to the local running standby. Run this command on the
+standby. The remote validator must have `[failover.enabled]` set to
+`true` to accept the request. Both validators must use the same vote
+account.
+
+```sh
+firedancer failover promote [<keypair>] [--name <name>] [--yes] [--force] [--address <address>] [--port <port>]
+```
+
+The command attaches to the local running validator and discovers it
+automatically. If more than one is running, pass `--name <name>` to
+select one (see [`ps`](#ps)). It asks for confirmation when run from a
+terminal unless `--yes` is given.
+
+The local validator finds the active's address through gossip. Use
+`--address` to give the remote validator's IPv4 address or host name
+instead, for example when connecting over a private network. Use
+`--port` if its failover listener does not use the default port 8010.
+
+For example, to connect to a known failover listener over a private
+network, run the following on the local standby.
+
+```sh
+firedancer failover promote --address 192.0.2.10 --port 8010
+```
+
+### Setup
+
+Set `[failover.enabled]` to `true` on both validators. Configure the same
+staked identity in `[paths.identity_key]` and the same vote account in
+`[paths.vote_account]`. Firedancer generates a fresh junk identity in memory
+on each startup. Both validators boot under a junk identity without voting
+and listen for handoff requests.
+
+Once one validator is active, run the following on the standby to request
+a handoff. The staked identity keypair is already loaded from the
+configuration, so no keypair argument is needed.
+
+```sh
+firedancer failover promote --name standby
+firedancer failover status --name standby
+```
+
+For example, an accepted request prints the handoff ID and the status
+needed to confirm completion.
+
+<<< @/snippets/commands/failover-promote.ansi
+
+After the handoff, the old active can request the identity back with
+`failover promote`. If both validators are still standbys, first promote
+one with `--force` as described below. An optional keypair argument must
+match the standby's current failover target.
+
+### Forced promotion
+
+With `--force`, the local standby takes the identity without contacting
+the remote validator. This is also needed for first use when both
+machines are standbys.
+
+::: warning WARNING
+
+Only use `--force` after ensuring every other machine with the identity
+cannot sign. It skips the checks on the other machine and can leave votes
+and leader windows unaccounted for by accepting incomplete or empty history.
+Never force promotion on both machines at once.
+
+:::
+
+| Arguments             | Description |
+|-----------------------|-------------|
+| `<keypair>`           | Optionally provide the target identity keypair file. It must match the standby's current failover target. |
+| `--name <name>`       | Select the validator instance to attach to if more than one is running on this host. |
+| `--config <path>`     | Optionally use the configuration TOML file the validator was started with. |
+| `--yes`               | Skip the confirmation prompt. |
+| `--force`             | Take the identity without cooperation from the remote validator, allowing incomplete or empty vote history. |
+| `--address <address>` | Use this IPv4 address or host name for the remote validator instead of the address found through gossip. |
+| `--port <port>`       | Set the remote validator's failover TCP port. The default is 8010. |
+
+## `failover status`
+This command prints the local running validator's failover status. Use
+it to check progress after [`failover promote`](#failover-promote).
+
+```sh
+firedancer failover status [--name <name>]
+```
+
+The command discovers the local running validator automatically. If more
+than one is running, pass `--name <name>` to select one (see [`ps`](#ps)).
+It exits with code 0 if status was retrieved, including when failover
+is disabled. Otherwise it exits with a non-zero code and prints the
+reason to `stderr`.
+
+The following example shows a completed handoff. The last handoff ID
+matches the ID printed by `failover promote` above. The connection closes
+after completion, so `peer role` is unknown until another authenticated
+session is established. The new standby reconnects when it discovers the
+new active through gossip.
+
+<<< @/snippets/commands/failover-status.ansi
+
+| Arguments             | Description |
+|-----------------------|-------------|
+| `--name <name>`       | Select the validator instance to attach to if more than one is running on this host. |
+| `--config <path>`     | Optionally use the configuration TOML file the validator was started with. |
+
 ## `remove-all-authorized-voters`
 Removes all authorized voters from the running validator, including any
 seeded from `[paths.authorized_voter_paths]` at startup as well as any

book/snippets/commands/failover-promote.ansi

@@ -0,0 +1,5 @@
+$ firedancer failover promote --name standby --yes
+failover promote       accepted
+role:                  standby
+action:                handoff, requesting the active's final vote state
+handoff:               123456789, done when `failover status` here shows role: active and last handoff: 123456789, taken

book/snippets/commands/failover-status.ansi

@@ -0,0 +1,13 @@
+$ firedancer failover status --name standby
+failover:              enabled
+role:                  active
+action:                idle
+stuck:                 no
+link:                  listening
+peer role:             unknown, no authenticated failover session
+peer boot id:          0000000000000001
+peer address:          192.0.2.10:8010, from gossip
+last handoff:          123456789, taken
+other holder:          n/a, this machine holds the identity
+takeover adopts:       an empty history (requires --force)
+coverage floor:        slot 300

src/app/firedancer-dev/main.h

@@ -117,6 +117,7 @@ extern fd_topo_run_tile_t fd_tile_udpecho;
 extern fd_topo_run_tile_t fd_tile_genesi;
 extern fd_topo_run_tile_t fd_tile_ipecho;
 extern fd_topo_run_tile_t fd_tile_admin;
+extern fd_topo_run_tile_t fd_tile_failov;
 
 extern fd_topo_run_tile_t fd_tile_gossvf;
 extern fd_topo_run_tile_t fd_tile_gossip;
@@ -185,6 +186,7 @@ fd_topo_run_tile_t * TILES[] = {
   &fd_tile_genesi,
   &fd_tile_ipecho,
   &fd_tile_admin,
+  &fd_tile_failov,
   &fd_tile_solcap,
   &fd_tile_snapmk,
   &fd_tile_snapzp,
@@ -203,6 +205,7 @@ extern action_t fd_action_mem;
 extern action_t fd_action_netconf;
 extern action_t fd_action_set_identity;
 extern action_t fd_action_get_identity;
+extern action_t fd_action_failover;
 extern action_t fd_action_ps;
 extern action_t fd_action_version;
 extern action_t fd_action_bench;
@@ -244,6 +247,7 @@ action_t * ACTIONS[] = {
   &fd_action_netconf,
   &fd_action_set_identity,
   &fd_action_get_identity,
+  &fd_action_failover,
   &fd_action_ps,
   &fd_action_help,
   &fd_action_metrics,

src/app/firedancer/Local.mk

@@ -24,6 +24,7 @@ $(call add-objs,commands/shred_version,fd_firedancer)
 $(call add-objs,commands/snapshot_create,fd_firedancer)
 $(call add-objs,commands/set_identity,fd_firedancer)
 $(call add-objs,commands/get_identity,fd_firedancer)
+$(call add-objs,commands/failover,fd_firedancer)
 $(call add-objs,commands/adminctl_client,fd_firedancer)
 $(call add-objs,commands/monitor_gossip/monitor_gossip commands/monitor_gossip/gossip_diag,fd_firedancer)
 $(call add-objs,commands/wait,fd_firedancer)