failover: on-demand operator promotion by cmoyes-jump · Pull Request #11769 · firedancer-io/firedancer

cmoyes-jump · 2026-09-29T02:40:05Z

Adds manual operated-initiated failover between validators over authenticated connections. Both validators boot as standby under a junk identity.

github-actions · 2026-09-29T02:41:59Z

 ┌─ ⚡ PERF · 1d18ba2 vs main@ccf8d76 ─────────────────────────────────
 │ SUITE                               BASELINE          NEW         Δ
 │ replay tps, mainnet               28,652 tps   28,451 tps  ·  -0.70%
 │ bench tps, localnet              667,216 tps  667,093 tps  ·  -0.02%
 │ snapshot load, testnet                3.79 s       3.78 s  ·  -0.36%
!│ mem total, mainnet                146.36 GiB   146.36 GiB  ▲   0.00%
!│ mem total, testnet                 87.25 GiB    87.25 GiB  ▲   0.00%
!│ mem total, ag mainnet             147.17 GiB   147.17 GiB  ▲   0.00%
 │ clean compile, firedancer             3.44 s       3.47 s  ·  +0.93%
!│ binary size, firedancer             59.94 MB     60.44 MB  ▲  +0.83%
 ├─────────────────────────────────────────────────────────────────────
@@ 0 REGRESSIONS · 4 WARNINGS · 0 IMPROVED · 4 NOISE @@
 └─────────────────────────────────────────────────────────────────────

history · 11 pushes

 ┌─ HISTORY · Δ vs main, per push, newest first ──────────────────────────────────
 │ HEAD         TPS    BENCH     SNAP    MEM·M    MEM·T     AG·M  COMPILE   BINARY
!│ 1d18ba2   -0.70%   -0.02%   -0.36%    0.00%    0.00%    0.00%   +0.93%   +0.83%
!│ 0799d1b   +0.13%   -1.21%   +0.66%    0.00%    0.00%    0.00%   -0.25%   +0.83%
!│ 8817682   -0.41%   +0.82%   +0.20%    0.00%    0.00%    0.00%   +0.14%   +0.83%
!│ 39bd823   +0.04%   +0.24%   -0.48%    0.00%    0.00%    0.00%   +0.54%   +0.83%
!│ ce00201   +0.44%   -0.16%   -0.05%    0.00%    0.00%    0.00%   +0.83%   +0.83%
!│ b653c3b   -0.74%   -0.38%   +0.06%    0.00%    0.00%    0.00%   +0.71%   +0.84%
!│ cf19891    0.00%   +0.50%   +0.01%    0.00%    0.00%    0.00%   -0.04%   +0.82%
!│ edc8ff3   -0.48%   +0.20%   -0.11%    0.00%    0.00%    0.00%   +1.03%   +0.82%
!│ dc01143   +0.03%   -1.64%   +0.36%    0.00%    0.00%    0.00%   +0.56%   +0.80%
!│ e7ea67d   -0.64%   +2.51%   +1.25%    0.00%    0.00%    0.00%        …   +0.74%
!│ abe1cac   +0.90%   +0.29%   +1.80%    0.00%    0.00%    0.00%   +1.72%   +0.74%
 └────────────────────────────────────────────────────────────────────────────────

Copilot

Copilot review overview

🟡 Changes recommended

Voting status and command audit data can currently report incorrect failover state.

Review effort: Balanced
Findings: 2

Open (2)

Propagate --yes to all accepted commands · New
Reflect publication guards in is_voting status · New

What changed in this PR

Adds authenticated, operator-driven validator failover with Tower handoff and safe identity switching.

Changes:

Adds failover protocol, TLS channel, controller tile, and CLI commands.
Adds Tower adoption and failover-aware signing safeguards.
Integrates configuration, topology, admin control, seccomp, events, and tests.

File	Description
`src/discof/tower/test_tower_tile.c`	Tests Tower adoption.
`src/discof/tower/fd_tower_tile.h`	Defines adoption results.
`src/discof/tower/fd_tower_tile.c`	Implements adoption and voting guards.
`src/discof/tower/fd_tower_tile_private.h`	Adds adoption state.
`src/discof/failover/test_failover_wire.c`	Tests framing and replay rejection.
`src/discof/failover/test_failover_tile.c`	Tests handoff controller flows.
`src/discof/failover/test_failover_proto.c`	Tests HELLO validation.
`src/discof/failover/test_failover_channel.c`	Tests authenticated transport.
`src/discof/failover/Local.mk`	Builds failover components and tests.
`src/discof/failover/generated/fd_failover_tile_seccomp.h`	Provides generated syscall filter.
`src/discof/failover/fd_failover_wire.h`	Declares framed wire protocol.
`src/discof/failover/fd_failover_wire.c`	Implements framing and sequencing.
`src/discof/failover/fd_failover_tls.h`	Declares TLS transport.
`src/discof/failover/fd_failover_tls.c`	Implements TLS transport.
`src/discof/failover/fd_failover_tile.seccomppolicy`	Defines failover syscall policy.
`src/discof/failover/fd_failover_proto.h`	Defines failover messages and states.
`src/discof/failover/fd_failover_proto.c`	Implements protocol validation.
`src/discof/failover/fd_failover_channel.h`	Declares authenticated channel API.
`src/discof/failover/fd_failover_channel.c`	Implements connection management.
`src/discof/failover/fd_failover_tile.c`	Implements failover orchestration.
`src/discof/failover/fd_failover_bus.h`	Defines admin/failover IPC.
`src/discof/admin/fd_adminctl.h`	Adds failover admin ABI.
`src/discof/admin/fd_adminctl.c`	Strengthens payload bounds and clearing.
`src/discof/admin/fd_admin_tile.c`	Routes failover commands and switches.
`src/disco/topo/fd_topob.c`	Registers the failover tile.
`src/disco/topo/fd_topo.h`	Adds failover tile configuration.
`src/disco/topo/fd_topo.c`	Accounts for failover key memory.
`src/disco/keyguard/test_keyguard.c`	Tests member-certificate authorization.
`src/disco/keyguard/fuzz_keyguard.c`	Adds failover fuzz role mapping.
`src/disco/keyguard/fd_sign_tile.c`	Supports preloaded failover keys.
`src/disco/keyguard/fd_keyswitch.h`	Adds public-key identity selection.
`src/disco/keyguard/fd_keyguard.h`	Defines failover signing role.
`src/disco/keyguard/fd_keyguard_match.c`	Matches certificate payloads.
`src/disco/keyguard/fd_keyguard_authorize.c`	Authorizes certificate signing.
`src/disco/events/schema/events.proto`	Adds failover event type.
`src/disco/events/schema/admin_command.json`	Extends event schema.
`src/disco/events/generated/fd_event_gen.h`	Updates generated event constants.
`src/choreo/tower/test_tower.c`	Tests received-Tower replacement.
`src/choreo/tower/test_tower_serdes.c`	Tests strict decoding and expansion.
`src/choreo/tower/fd_tower.h`	Declares Tower adoption API.
`src/choreo/tower/fd_tower.c`	Implements Tower adoption.
`src/choreo/tower/fd_tower_serdes.h`	Declares strict Tower decoding.
`src/choreo/tower/fd_tower_serdes.c`	Implements validation and expansion.
`src/app/shared/test_config_parse.c`	Tests failover configuration parsing.
`src/app/shared/test_config_json.c`	Tests configuration redaction.
`src/app/shared/fd_config.h`	Adds failover settings.
`src/app/shared/fd_config.c`	Derives and validates failover mode.
`src/app/shared/fd_config_parse.c`	Parses failover settings.
`src/app/shared/fd_config_json.c`	Serializes redacted settings.
`src/app/shared/fd_action.h`	Adds failover CLI arguments.
`src/app/shared/commands/run/run.c`	Validates keys and port permissions.
`src/app/firedancer/topology.c`	Wires failover into Firedancer.
`src/app/firedancer/main.c`	Registers tile and command.
`src/app/firedancer/Local.mk`	Builds the failover command.
`src/app/firedancer/config/default.toml`	Documents failover configuration.
`src/app/firedancer/commands/set_identity.c`	Refuses manual identity switching.
`src/app/firedancer/commands/failover.c`	Implements operator commands.
`src/app/firedancer-dev/main.h`	Registers failover in development mode.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🔵 Needs a closer look

Consensus-history transfer and signing-authority transitions require human safety review and integration validation; builds and tests were not run.

0 open findings

🧠 Review effort: Balanced

Copilot

🔵 Needs a closer look

Authentication, signing-key transitions, and consensus-history adoption require human review and end-to-end sandboxed validation.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Authentication, signing-authority transitions, and consensus-history handoff require human review and end-to-end validation.

1 open issue

Rename handshake gauge and document its absent-value behavior · New

_{Review effort: Balanced}

+    <counter name="FrameProtocolRejected" summary="Authenticated frames rejected by the controller" />
+    <counter name="ConnAuthenticated" summary="Completed mutual admission, TLS and HELLO authentications" />
+    <counter name="HandshakeDurationNanos" summary="Cumulative time from candidate allocation to completed authentication" />
+    <gauge name="HandshakeLatestDurationNanos" summary="Duration of the most recently completed authentication" />


Copilot

🔵 Needs a closer look

Consensus handoff, authentication, and signing-key transitions require human safety review backed by end-to-end validation.

1 open issue

Rename handshake gauge and document its absent-value behavior

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Authenticated identity switching, private-key lifecycle changes, and consensus-history adoption require specialist human review and end-to-end validation.

1 open issue

Rename handshake gauge and document its absent-value behavior

_{Review effort: Balanced}

…ytes

…remove demote

Copilot

🔵 Needs a closer look

Consensus handoff and signing-key isolation require specialist review and runtime validation, with an unresolved adoption failure remaining.

2 open issues

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Authentication, private-key handling, and consensus-history transfer require human safety review and end-to-end runtime validation.

1 open issue

Rename handshake gauge and document its absent-value behavior

1 resolved since last review

Preflight history slot capacity before consensus state adoption

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Consensus-history adoption and signing-key transitions need specialist review and integration validation before approval.

2 open issues

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Consensus-history adoption, cross-tile identity switching, and authentication require specialist human review and end-to-end runtime validation.

1 open issue

Rename handshake gauge and document its absent-value behavior

1 resolved since last review

Unsigned standby marks can enable invalid FINAL signing after promotion

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Authentication, private-key handling, and consensus transfer need human validation, and a history-adoption capacity issue remains unresolved.

2 open issues

_{Review effort: Balanced}

+    free--;
+  }
+
+  advance_root( self, hist->anchor );


Copilot

🔵 Needs a closer look

Authentication, signing-key custody, and consensus-history adoption cross safety boundaries that require human review and end-to-end validation.

2 open issues

_{Review effort: Balanced}

cmoyes-jump self-assigned this Sep 29, 2026

Copilot AI balanced review requested due to automatic review settings September 29, 2026 02:40

cmoyes-jump marked this pull request as draft September 29, 2026 02:40

Copilot started reviewing on behalf of cmoyes-jump September 29, 2026 02:40

View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Comment thread src/app/firedancer/commands/failover.c Outdated

Comment thread src/discof/tower/fd_tower_tile.c

cmoyes-jump force-pushed the failover/p0-1-proto branch 3 times, most recently from fd4bd6e to af8caec Compare September 29, 2026 03:59

mmcgee-jump reviewed Sep 29, 2026

View reviewed changes

Comment thread src/discof/admin/fd_admin_tile.c Outdated

mmcgee-jump reviewed Sep 29, 2026

View reviewed changes

Comment thread src/discof/admin/fd_admin_tile.c Outdated

cmoyes-jump force-pushed the failover/p0-1-proto branch 15 times, most recently from b011343 to 940586f Compare October 1, 2026 01:02

cmoyes-jump changed the title ~~failover: mini-MVP with on-demand operator promotion via Tower handoff~~ failover: on-demand operator promotion via Tower handoff Oct 1, 2026

cmoyes-jump force-pushed the failover/p0-1-proto branch 3 times, most recently from aed268c to 1f3a6ce Compare October 1, 2026 19:44

Copilot AI reviewed Oct 15, 2026

View reviewed changes

Copilot AI reviewed Oct 19, 2026

View reviewed changes

Copilot AI reviewed Oct 20, 2026

View reviewed changes

Copilot AI reviewed Oct 21, 2026

View reviewed changes

cmoyes-jump added 17 commits October 21, 2026 17:14

failover: add the HELLO message and the session state machine

34eaa9d

failover: frame the protocol messages on the wire

b922945

failover: add the [failover] config and derive the junk identity path

95f4c77

tower: add an exact compact Tower sync decode that rejects trailing b…

3e1ab5d

…ytes

tower: add explicit adoption of received vote history

cbd5550

failover: generate the junk identity and boot every tile under it

0db56ac

failover: preload both identities and sign the member cert and pair TLS

056d88b

failover: add the handoff messages and validate the final Tower

c7b9b2a

failover: add the authenticated on-demand pair channel

774976b

failover: add identity switching and failover commands to the admin bus

f5eaf10

tower: add explicit adoption, shadow voting and a vote drain

d1e5258

failover: add the on-demand handoff controller

7e95921

failover: add the promote, demote and status commands

c06799c

failover: document the controller, channel and protocol states

623113a

failover: make set-identity restart failover on the new identity and …

ba4817c

…remove demote

Harden failover authentication and bound public listener work

993daa9

Generate failover junk identities in memory

7faba1b

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread src/choreo/votor/ag_votor.c

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread src/choreo/votor/ag_votor.c

failover: make handoffs Alpenglow-only

0799d1b

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread src/choreo/votor/ag_votor.c

free--;

}

advance_root( self, hist->anchor );

failover: add operator guidance to the duplicate instance abort

1d18ba2

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Conversation

cmoyes-jump commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cmoyes-jump commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading