Skip to content

fix: make the validator service start on Prysm v7 - #30

Merged
inf-1oop merged 1 commit into
mainfrom
fix/validator-v7-flags
Sep 16, 2026
Merged

inf-1oop merged 1 commit into
mainfrom
fix/validator-v7-flags

Conversation

@inf-1oop

Copy link
Copy Markdown
Contributor

main currently ships a validator that cannot start

#26 set PRYSM_VERSION=v7.1.8, but the validator command still passes --db-backup-output-dir, which Prysm removed in v7:

Incorrect Usage: flag provided but not defined: -db-backup-output-dir

The container then crash-loops. This is not latent — it affects anyone who pulls main and starts a validator today. I hit it for real upgrading contango-validator-1, where it cost ~2.5 minutes of missed duties.

Reproduced against gcr.io/prysmaticlabs/prysm/validator:v7.1.8 with the exact flag set main renders:

flag set result on v7.1.8
main today Incorrect Usage: flag provided but not defined: -db-backup-output-dir
this PR INFO main: Prysm Validator started version=Prysm/v7.1.8/51b5a75...

Three fixes, all in docker-compose.yml

1. Drop the two dead backup flags

--db-backup-output-dir is gone in v7. --enable-db-backup-webhook goes with it — v7 keeps it only in config/features/deprecated_flags.go as a hidden no-op, so it logs is deprecated and has no effect on every start and does nothing.

2. validator-backup actually backs something up

The service POSTs to the validator's /db/backup webhook — which is precisely what the deprecated flag used to enable. The call returns success while backing up nothing.

That matters more than a normal dead feature: the README tells operators to run it before upgrading, and slashing protection history lives in that database. A backup step that silently no-ops is worse than no backup step, because it buys false confidence before the one operation where you need the file.

It now runs slashing-protection-history export, which writes the standard EIP-3076 file that any client can import.

One constraint, spelled out in a comment in the file: it must run with the validator stopped, because validator.db is a bolt database with a single-writer lock.

docker compose stop validator
docker compose --profile backup run --rm validator-backup
docker compose --profile node --profile validator --profile init up -d validator

3. Restart policy for geth, beacon, validator

Only stats and caddy used the x-restart anchor. Without it, an OOM kill or crash stops attesting until a human notices — an operator fleet has already hit this, with a geth OOM-killed during an archive sync sitting dead for two days while the node still reported "up".

Verification

  • main's rendered flag set fails immediately on v7.1.8 with the error above
  • this PR's flag set reaches Prysm Validator started
  • validator slashing-protection-history export exists in v7.1.8 and accepts --slashing-protection-export-dir
  • docker compose config clean across the node, validator, init and backup profiles
  • the running upgrade on contango-validator-1 (canonical validator v7.1.8 + v7.1.8-vega.1 beacon) has been performing all duties — attestations, aggregates, sync committee messages and contributions, and block proposals — for 4+ hours

Deliberately not in this PR

These came out of the same upgrade but need testing rather than review, so they belong to the validator rollout session:

  • --enable-doppelganger is absent from the repo. It is a real anti-double-signing control and operators are hand-rolling it in overrides. Shipping it costs ~2 epochs of missed attestations per start, which deserves a deliberate decision.
  • The override trap. Compose replaces command: rather than merging, so an operator override restating the validator command silently bypasses every future fix here — fix: point the validator REST endpoint at the beacon HTTP port #28's REST fix was in the repo and inert on the node for exactly this reason. Fixes 3 (and doppelganger) reduce the reason to keep an override at all; the README should also state the rule.
  • v5→v7 renamed log strings (Submitted new sync message → Submitted sync committee messages). Any alerting that greps validator logs goes quiet and looks healthy. This fooled my own checks twice.
  • No rollback path. A v7→v5 validator downgrade is unsupported and untested.

🤖 Generated with Claude Code

main currently ships a validator that cannot start. #26 set PRYSM_VERSION=v7.1.8, but
the validator command still passes --db-backup-output-dir, which Prysm removed in v7:

    Incorrect Usage: flag provided but not defined: -db-backup-output-dir

The container then crash-loops. Reproduced against gcr.io/prysmaticlabs/prysm/validator:
v7.1.8 with the exact flag set main renders, and hit for real while upgrading
contango-validator-1, where it cost about two and a half minutes of missed duties.

Three fixes, all in docker-compose.yml.

--db-backup-output-dir is removed outright. --enable-db-backup-webhook goes with it: v7
keeps it only in config/features/deprecated_flags.go as a hidden no-op, so it logs
"is deprecated and has no effect" on every start and does nothing.

validator-backup no longer POSTs to the /db/backup webhook, because that webhook is what
the deprecated flag used to enable -- the call returns success while backing up nothing,
which is worse than having no backup step at all given the README tells operators to run
it before upgrading and slashing protection lives in that database. It now runs
`slashing-protection-history export`, which writes the standard EIP-3076 file that any
client can import. That has to run with the validator stopped, since validator.db is a
bolt database with a single-writer lock, so the comment spells out the three commands.

geth, beacon and validator get the restart policy that until now only stats and caddy
had. Without it an OOM kill or a crash stops attesting until a person notices; an
operator fleet has already hit this, with a geth OOM-killed during an archive sync
sitting dead for two days while the node still reported up.

Verified: the flag set main renders today fails immediately on v7.1.8 with the error
above, the flag set after this change gets past flag parsing, and
`slashing-protection-history export --slashing-protection-export-dir` exists in v7.1.8.
`docker compose config` is clean across the node, validator, init and backup profiles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@inf-1oop
inf-1oop merged commit 3df637e into main Sep 16, 2026
3 checks passed
@inf-1oop
inf-1oop deleted the fix/validator-v7-flags branch September 16, 2026 02:29
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 2.4.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant