Repository navigation
fix: make the validator service start on Prysm v7 - #30
Merged
Merged
Conversation
main currently ships a validator that cannot start. #26 set PRYSM_VERSION=v7.1.8, but the validator command still passes --db-backup-output-dir, which Prysm removed in v7: Incorrect Usage: flag provided but not defined: -db-backup-output-dir The container then crash-loops. Reproduced against gcr.io/prysmaticlabs/prysm/validator: v7.1.8 with the exact flag set main renders, and hit for real while upgrading contango-validator-1, where it cost about two and a half minutes of missed duties. Three fixes, all in docker-compose.yml. --db-backup-output-dir is removed outright. --enable-db-backup-webhook goes with it: v7 keeps it only in config/features/deprecated_flags.go as a hidden no-op, so it logs "is deprecated and has no effect" on every start and does nothing. validator-backup no longer POSTs to the /db/backup webhook, because that webhook is what the deprecated flag used to enable -- the call returns success while backing up nothing, which is worse than having no backup step at all given the README tells operators to run it before upgrading and slashing protection lives in that database. It now runs `slashing-protection-history export`, which writes the standard EIP-3076 file that any client can import. That has to run with the validator stopped, since validator.db is a bolt database with a single-writer lock, so the comment spells out the three commands. geth, beacon and validator get the restart policy that until now only stats and caddy had. Without it an OOM kill or a crash stops attesting until a person notices; an operator fleet has already hit this, with a geth OOM-killed during an archive sync sitting dead for two days while the node still reported up. Verified: the flag set main renders today fails immediately on v7.1.8 with the error above, the flag set after this change gets past flag parsing, and `slashing-protection-history export --slashing-protection-export-dir` exists in v7.1.8. `docker compose config` is clean across the node, validator, init and backup profiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🎉 This PR is included in version 2.4.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
maincurrently ships a validator that cannot start#26 set
PRYSM_VERSION=v7.1.8, but the validator command still passes--db-backup-output-dir, which Prysm removed in v7:The container then crash-loops. This is not latent — it affects anyone who pulls
mainand starts a validator today. I hit it for real upgradingcontango-validator-1, where it cost ~2.5 minutes of missed duties.Reproduced against
gcr.io/prysmaticlabs/prysm/validator:v7.1.8with the exact flag setmainrenders:maintodayIncorrect Usage: flag provided but not defined: -db-backup-output-dirINFO main: Prysm Validator started version=Prysm/v7.1.8/51b5a75...Three fixes, all in
docker-compose.yml1. Drop the two dead backup flags
--db-backup-output-diris gone in v7.--enable-db-backup-webhookgoes with it — v7 keeps it only inconfig/features/deprecated_flags.goas a hidden no-op, so it logsis deprecated and has no effecton every start and does nothing.2.
validator-backupactually backs something upThe service POSTs to the validator's
/db/backupwebhook — which is precisely what the deprecated flag used to enable. The call returns success while backing up nothing.That matters more than a normal dead feature: the README tells operators to run it before upgrading, and slashing protection history lives in that database. A backup step that silently no-ops is worse than no backup step, because it buys false confidence before the one operation where you need the file.
It now runs
slashing-protection-history export, which writes the standard EIP-3076 file that any client can import.One constraint, spelled out in a comment in the file: it must run with the validator stopped, because
validator.dbis a bolt database with a single-writer lock.3. Restart policy for
geth,beacon,validatorOnly
statsandcaddyused thex-restartanchor. Without it, an OOM kill or crash stops attesting until a human notices — an operator fleet has already hit this, with a geth OOM-killed during an archive sync sitting dead for two days while the node still reported "up".Verification
main's rendered flag set fails immediately on v7.1.8 with the error abovePrysm Validator startedvalidator slashing-protection-history exportexists in v7.1.8 and accepts--slashing-protection-export-dirdocker compose configclean across thenode,validator,initandbackupprofilescontango-validator-1(canonical validator v7.1.8 +v7.1.8-vega.1beacon) has been performing all duties — attestations, aggregates, sync committee messages and contributions, and block proposals — for 4+ hoursDeliberately not in this PR
These came out of the same upgrade but need testing rather than review, so they belong to the validator rollout session:
--enable-doppelgangeris absent from the repo. It is a real anti-double-signing control and operators are hand-rolling it in overrides. Shipping it costs ~2 epochs of missed attestations per start, which deserves a deliberate decision.command:rather than merging, so an operator override restating the validator command silently bypasses every future fix here — fix: point the validator REST endpoint at the beacon HTTP port #28's REST fix was in the repo and inert on the node for exactly this reason. Fixes 3 (and doppelganger) reduce the reason to keep an override at all; the README should also state the rule.Submitted new sync message→Submitted sync committee messages). Any alerting that greps validator logs goes quiet and looks healthy. This fooled my own checks twice.🤖 Generated with Claude Code