Repository navigation
fix: point the validator REST endpoint at the beacon HTTP port - #28
Merged
Merged
Conversation
--beacon-rest-api-provider was set to ${EXTERNAL_IP}:${RPC_PORT}, which renders as
"0.0.0.0:4000". That is wrong three ways: a REST endpoint is a URL and needs a scheme,
the gRPC dial target must not have one, and 4000 is the gRPC port while the HTTP API is
on 3500.
It did not matter until now. On Prysm v5.1.0 the REST API is enabled only by the
explicit --enable-beacon-rest-api boolean, so this flag was inert and the validator
used gRPC:
if ctx.Bool(EnableBeaconRESTApi.Name) {
v7.1.8 also enables it implicitly whenever the provider flag is set:
if ctx.Bool(EnableBeaconRESTApi.Name) || ctx.IsSet(BeaconRESTApiProviderFlag.Name) {
so the value starts being used. Reproduced against validator:v7.1.8, where it fails
every 12 seconds and the validator performs no duties:
INFO rest: Initialized REST connection provider endpoints=[[invalid endpoint]]
ERROR beacon-api: failed to get health of node ...
parse "0.0.0.0:4000/eth/v1/node/health": first path segment in URL cannot
contain colon
WARN fallback: No responsive beacon node found tried=[[invalid endpoint]]
The same run on v5.1.0 initialises no REST provider at all.
This matters here because the commit that moves PRYSM_VERSION to v7.1.8 is what makes
the flag live, so without this fix that bump costs every operator their attestations.
Verified with isolated datadirs on v7.1.8: http://beacon:3500 initialises as
endpoints=[http://beacon:3500] with no parse errors, against four with the old value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
maciejwitowski
added a commit
that referenced
this pull request
Sep 14, 2026
`npx semantic-release@25.0.9` pinned only the top package. semantic-release loads its plugins from its own node_modules tree, and every dependency below it (release-notes-generator ^14.1.0, conventional-changelog-writer ^8.0.0, the GitHub plugin's Octokit stack) resolved fresh on each run with no lockfile. Any new npm release in that tree reached a job that holds a contents:write token, unreviewed. Move the toolchain into .github/release/package.json with a committed package-lock.json: semantic-release 25.0.9, commit-analyzer 13.0.1, release-notes-generator 14.1.1, github 12.0.9, conventionalcommits preset 9.3.1. The job installs it with `npm ci --ignore-scripts` (no package in the tree has an install script) and runs that binary from the repo root. It lives under .github/ so the operator checkout gets no root package.json. Node is pinned to 24, the version this was verified on. Verified on Node 24 against main at 47db176, running the two workflow steps as written with the full .releaserc and a GitHub token, `--dry-run --no-ci`, no root node_modules: next version 2.3.2, notes render Bug Fixes with #26, #27 and #28. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dqz4KWjLzxNEJuea75Avb7
inf-1oop
added a commit
that referenced
this pull request
Sep 16, 2026
## `main` currently ships a validator that cannot start #26 set `PRYSM_VERSION=v7.1.8`, but the validator command still passes `--db-backup-output-dir`, which Prysm **removed** in v7: ``` Incorrect Usage: flag provided but not defined: -db-backup-output-dir ``` The container then crash-loops. This is not latent — it affects anyone who pulls `main` and starts a validator today. I hit it for real upgrading `contango-validator-1`, where it cost ~2.5 minutes of missed duties. Reproduced against `gcr.io/prysmaticlabs/prysm/validator:v7.1.8` with the exact flag set `main` renders: | flag set | result on v7.1.8 | |---|---| | `main` today | `Incorrect Usage: flag provided but not defined: -db-backup-output-dir` | | this PR | `INFO main: Prysm Validator started version=Prysm/v7.1.8/51b5a75...` | ## Three fixes, all in `docker-compose.yml` ### 1. Drop the two dead backup flags `--db-backup-output-dir` is gone in v7. `--enable-db-backup-webhook` goes with it — v7 keeps it only in `config/features/deprecated_flags.go` as a hidden no-op, so it logs `is deprecated and has no effect` on every start and does nothing. ### 2. `validator-backup` actually backs something up The service POSTs to the validator's `/db/backup` webhook — which is precisely what the deprecated flag used to enable. **The call returns success while backing up nothing.** That matters more than a normal dead feature: the README tells operators to run it before upgrading, and **slashing protection history lives in that database**. A backup step that silently no-ops is worse than no backup step, because it buys false confidence before the one operation where you need the file. It now runs `slashing-protection-history export`, which writes the standard **EIP-3076** file that any client can import. One constraint, spelled out in a comment in the file: it must run with the validator **stopped**, because `validator.db` is a bolt database with a single-writer lock. ```bash docker compose stop validator docker compose --profile backup run --rm validator-backup docker compose --profile node --profile validator --profile init up -d validator ``` ### 3. Restart policy for `geth`, `beacon`, `validator` Only `stats` and `caddy` used the `x-restart` anchor. Without it, an OOM kill or crash stops attesting until a human notices — an operator fleet has already hit this, with a geth OOM-killed during an archive sync sitting dead for two days while the node still reported "up". ## Verification - `main`'s rendered flag set fails immediately on v7.1.8 with the error above - this PR's flag set reaches `Prysm Validator started` - `validator slashing-protection-history export` exists in v7.1.8 and accepts `--slashing-protection-export-dir` - `docker compose config` clean across the `node`, `validator`, `init` and `backup` profiles - the running upgrade on `contango-validator-1` (canonical validator v7.1.8 + `v7.1.8-vega.1` beacon) has been performing all duties — attestations, aggregates, sync committee messages and contributions, and block proposals — for 4+ hours ## Deliberately not in this PR These came out of the same upgrade but need testing rather than review, so they belong to the validator rollout session: - **`--enable-doppelganger`** is absent from the repo. It is a real anti-double-signing control and operators are hand-rolling it in overrides. Shipping it costs ~2 epochs of missed attestations per start, which deserves a deliberate decision. - **The override trap.** Compose *replaces* `command:` rather than merging, so an operator override restating the validator command silently bypasses every future fix here — #28's REST fix was in the repo and inert on the node for exactly this reason. Fixes 3 (and doppelganger) reduce the *reason* to keep an override at all; the README should also state the rule. - **v5→v7 renamed log strings** (`Submitted new sync message` → `Submitted sync committee messages`). Any alerting that greps validator logs goes quiet and looks healthy. This fooled my own checks twice. - **No rollback path.** A v7→v5 validator downgrade is unsupported and untested. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
maciejwitowski
added a commit
that referenced
this pull request
Sep 24, 2026
## Problem The Release workflow fails at `generateNotes` on every push that should cut a release. No GitHub release has been published since v2.3.1 (2026-08-03), so #27, #26 and #28 are on `main` with no release. And v2.3.1 itself shipped with an empty body. | Run | Commit | Result | |---|---|---| | 2026-08-03 | ffd802d #21 | success, but v2.3.1 notes are the version header only | | 2026-08-12 → 08-19 | chore/ci commits | success (no release due, notes step never ran) | | 2026-09-04 | a4cb188 #27 | failure | | 2026-09-14 | a409872 #26 | failure | | 2026-09-14 | 47db176 #28 | failure | ``` Failed step "generateNotes" of plugin "@semantic-release/release-notes-generator" Error: Missing helper: "conventional-changelog-conventionalcommits requires conventional-changelog-writer@9 or newer ..." ``` ## Cause The workflow installed the release tooling unpinned, with no lockfile: - `npx semantic-release` resolves to 25.0.9. Its `@semantic-release/release-notes-generator` 14.1.1 (the latest) depends on `conventional-changelog-writer ^8`. - `npm install --no-save conventional-changelog-conventionalcommits` resolves to the latest preset. Preset **v10** targets writer 9: - 10.0.0–10.3.0 (from 2026-06-26): no error, but writer 8 renders only the version header. That is why the [v2.3.1 release](https://github.com/vana-com/vana/releases/tag/v2.3.1) has no notes, while v2.3.0 (January, preset 9.x) does. - 10.4.0 (2026-08-18): added an explicit guard, which turned the silent empty output into the hard failure above. Pinning only the top-level packages does not close this. semantic-release loads its plugins from its own `node_modules` tree, and every range below it (`release-notes-generator ^14.1.0`, `conventional-changelog-writer ^8.0.0`, the GitHub plugin's Octokit stack) resolves fresh on each run. The job holds a `contents: write` token, so any new npm release in that tree runs with it, unreviewed. ## Fix - New `.github/release/package.json` + committed `package-lock.json` for the toolchain: | Package | Version | |---|---| | `semantic-release` | 25.0.9 | | `@semantic-release/commit-analyzer` | 13.0.1 | | `@semantic-release/release-notes-generator` | 14.1.1 | | `@semantic-release/github` | 12.0.9 | | `conventional-changelog-conventionalcommits` | 9.3.1, the last major built for writer 8 | - The job installs it with `npm ci --ignore-scripts --prefix .github/release` (no package in the tree has an install script) and runs `.github/release/node_modules/.bin/semantic-release` from the repo root, so `.releaserc.yaml` is still the config. - It lives under `.github/` so the operator checkout gets no root `package.json`. `.gitignore` covers its `node_modules`. - Node pinned to `24`, the version this was verified on. - A comment in the workflow says when to bump the preset: once the notes generator ships writer 9. ## Verification Ran the two workflow steps as written, on Node 24 / npm 11, on a checkout of `main` at 47db176 with no root `node_modules`, the full `.releaserc.yaml` and a GitHub token, `--dry-run --no-ci`: - `npm ci` installs one deduped copy of each package; every plugin and the preset resolve from `.github/release/node_modules`. - GitHub `verifyConditions` passes, next version **2.3.2**, notes render `### Bug Fixes` with #26, #27, #28, same format as v2.3.0. Preset comparison (same command pair, earlier run): | Preset | Result | |---|---| | 9.3.1 | notes render Bug Fixes | | 10.3.0 | notes are the header only | | 10.4.0 | fails with the same "Missing helper" error as CI | Cosmetic: #26's commit body contains the word "closes" before "compatibility", so its line gets a bogus `closes #compatibility` link. Harmless; editing the release body after publish fixes it if wanted. ## Heads-up on merging Merging this triggers the workflow on `main`, which will **publish v2.3.2** as a public GitHub release containing #26, #27 and #28 (Prysm v7 config keys, the deposit contract switch config, the validator REST fix). The GitHub plugin also comments "included in version 2.3.2" on those PRs and labels them `released`, as it did on #21 for v2.3.1. Anyone watching this repo's releases gets notified, so merge when that announcement is fine to go out. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Dqz4KWjLzxNEJuea75Avb7 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🎉 This PR is included in version 2.4.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
docker-compose.ymlpasses--beacon-rest-api-provider=${EXTERNAL_IP}:${RPC_PORT}, which renders as0.0.0.0:4000. That value is wrong three ways:4000is the gRPC port. The HTTP API is on3500(--grpc-gateway-port, now an alias for--http-port).Why it is breaking now, and not before
On Prysm v5.1.0 the REST API is enabled only by the explicit boolean, so the provider flag was dead config and the validator used gRPC:
v7.1.8 also enables it implicitly whenever the provider flag is set:
So the value stops being ignored and starts deciding how the validator reaches the beacon.
#26 moved
PRYSM_VERSIONfromv5.1.0tov7.1.8and is already merged, so this is live onmaintoday. Any operator who pullsmainand restarts their validator loses attestations.Reproduction
Against
gcr.io/prysmaticlabs/prysm/validator:v7.1.8with the valuemaincurrently produces, repeating every 12 seconds:The same run on
v5.1.0initialises no REST provider at all. Note the gRPC provider is still constructed (endpoints=[beacon:4000]) — the validator does not discard it, but the health check runs over REST, fails to build a request, and the fallback layer concludes no beacon is responsive. Net effect: zero duties.Fix
Verified on
v7.1.8with isolated datadirs per container, so no bolt-lock contention between runs:0.0.0.0:4000(currentmain)[invalid endpoint]http://beacon:3500(this PR)http://beacon:3500http://127.0.0.1:3500(default, unused)Note on the approach
Dropping the flag was the other option: it restores v5.1.0's behaviour exactly, since the feature flag then stays off and gRPC carries the duties. Repointing it instead puts the validator on the standardised Beacon API, which is where Prysm is heading and which works against any consensus client — but it is a transport change, not only a syntax fix, and deserves a validator-in-the-loop test before the fleet moves.
What is verified here is that the endpoint parses and initialises correctly. What is not verified is that duties actually flow over REST end to end — that needs a live beacon and real keys.
Scope
docker-compose.ymlonly. The Caddyfile is untouched; public endpoint exposure was handled separately in #27.🤖 Generated with Claude Code