Skip to content

fix: point the validator REST endpoint at the beacon HTTP port - #28

Merged
inf-1oop merged 1 commit into
mainfrom
fix/validator-rest-endpoint
Sep 14, 2026
Merged

inf-1oop merged 1 commit into
mainfrom
fix/validator-rest-endpoint

Conversation

@inf-1oop

Copy link
Copy Markdown
Contributor

Problem

docker-compose.yml passes --beacon-rest-api-provider=${EXTERNAL_IP}:${RPC_PORT}, which renders as 0.0.0.0:4000. That value is wrong three ways:

No scheme A REST endpoint is a URL. The gRPC provider on the line above is a dial target and must not have one — they look alike but are parsed by different code.
Wrong port 4000 is the gRPC port. The HTTP API is on 3500 (--grpc-gateway-port, now an alias for --http-port).
Silently inert on v5 Which is why nobody noticed.

Why it is breaking now, and not before

On Prysm v5.1.0 the REST API is enabled only by the explicit boolean, so the provider flag was dead config and the validator used gRPC:

// v5.1.0  config/features/config.go:296
if ctx.Bool(EnableBeaconRESTApi.Name) {

v7.1.8 also enables it implicitly whenever the provider flag is set:

// v7.1.8  config/features/config.go:372
if ctx.Bool(EnableBeaconRESTApi.Name) || ctx.IsSet(validatorflags.BeaconRESTApiProviderFlag.Name) {

So the value stops being ignored and starts deciding how the validator reaches the beacon.

#26 moved PRYSM_VERSION from v5.1.0 to v7.1.8 and is already merged, so this is live on main today. Any operator who pulls main and restarts their validator loses attestations.

Reproduction

Against gcr.io/prysmaticlabs/prysm/validator:v7.1.8 with the value main currently produces, repeating every 12 seconds:

WARN  flags: Enabled feature flag enable-beacon-rest-api=...
INFO  rest: Initialized REST connection provider count=1 endpoints=[[invalid endpoint]]
ERROR beacon-api: failed to get health of node error=failed to create request for
      endpoint [invalid endpoint]: parse "0.0.0.0:4000/eth/v1/node/health":
      first path segment in URL cannot contain colon  url=0.0.0.0:4000
WARN  fallback: No responsive beacon node found tried=[[invalid endpoint]]

The same run on v5.1.0 initialises no REST provider at all. Note the gRPC provider is still constructed (endpoints=[beacon:4000]) — the validator does not discard it, but the health check runs over REST, fails to build a request, and the fallback layer concludes no beacon is responsive. Net effect: zero duties.

Fix

- --beacon-rest-api-provider=${EXTERNAL_IP}:${RPC_PORT:-4000}
+ --beacon-rest-api-provider=http://beacon:${GRPC_GATEWAY_PORT:-3500}

Verified on v7.1.8 with isolated datadirs per container, so no bolt-lock contention between runs:

Config REST endpoint REST feature parse errors in 40s
0.0.0.0:4000 (current main) [invalid endpoint] on 4
http://beacon:3500 (this PR) http://beacon:3500 on 0
flag dropped entirely http://127.0.0.1:3500 (default, unused) off 0

Note on the approach

Dropping the flag was the other option: it restores v5.1.0's behaviour exactly, since the feature flag then stays off and gRPC carries the duties. Repointing it instead puts the validator on the standardised Beacon API, which is where Prysm is heading and which works against any consensus client — but it is a transport change, not only a syntax fix, and deserves a validator-in-the-loop test before the fleet moves.

What is verified here is that the endpoint parses and initialises correctly. What is not verified is that duties actually flow over REST end to end — that needs a live beacon and real keys.

Scope

docker-compose.yml only. The Caddyfile is untouched; public endpoint exposure was handled separately in #27.

🤖 Generated with Claude Code

--beacon-rest-api-provider was set to ${EXTERNAL_IP}:${RPC_PORT}, which renders as
"0.0.0.0:4000". That is wrong three ways: a REST endpoint is a URL and needs a scheme,
the gRPC dial target must not have one, and 4000 is the gRPC port while the HTTP API is
on 3500.

It did not matter until now. On Prysm v5.1.0 the REST API is enabled only by the
explicit --enable-beacon-rest-api boolean, so this flag was inert and the validator
used gRPC:

    if ctx.Bool(EnableBeaconRESTApi.Name) {

v7.1.8 also enables it implicitly whenever the provider flag is set:

    if ctx.Bool(EnableBeaconRESTApi.Name) || ctx.IsSet(BeaconRESTApiProviderFlag.Name) {

so the value starts being used. Reproduced against validator:v7.1.8, where it fails
every 12 seconds and the validator performs no duties:

    INFO  rest: Initialized REST connection provider endpoints=[[invalid endpoint]]
    ERROR beacon-api: failed to get health of node ...
          parse "0.0.0.0:4000/eth/v1/node/health": first path segment in URL cannot
          contain colon
    WARN  fallback: No responsive beacon node found tried=[[invalid endpoint]]

The same run on v5.1.0 initialises no REST provider at all.

This matters here because the commit that moves PRYSM_VERSION to v7.1.8 is what makes
the flag live, so without this fix that bump costs every operator their attestations.
Verified with isolated datadirs on v7.1.8: http://beacon:3500 initialises as
endpoints=[http://beacon:3500] with no parse errors, against four with the old value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@inf-1oop
inf-1oop merged commit 47db176 into main Sep 14, 2026
3 checks passed
@inf-1oop
inf-1oop deleted the fix/validator-rest-endpoint branch September 14, 2026 10:50
maciejwitowski added a commit that referenced this pull request Sep 14, 2026
`npx semantic-release@25.0.9` pinned only the top package. semantic-release
loads its plugins from its own node_modules tree, and every dependency
below it (release-notes-generator ^14.1.0, conventional-changelog-writer
^8.0.0, the GitHub plugin's Octokit stack) resolved fresh on each run with
no lockfile. Any new npm release in that tree reached a job that holds a
contents:write token, unreviewed.

Move the toolchain into .github/release/package.json with a committed
package-lock.json: semantic-release 25.0.9, commit-analyzer 13.0.1,
release-notes-generator 14.1.1, github 12.0.9, conventionalcommits preset
9.3.1. The job installs it with `npm ci --ignore-scripts` (no package in the
tree has an install script) and runs that binary from the repo root. It
lives under .github/ so the operator checkout gets no root package.json.
Node is pinned to 24, the version this was verified on.

Verified on Node 24 against main at 47db176, running the two workflow steps
as written with the full .releaserc and a GitHub token, `--dry-run --no-ci`,
no root node_modules: next version 2.3.2, notes render Bug Fixes with #26,
#27 and #28.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dqz4KWjLzxNEJuea75Avb7
inf-1oop added a commit that referenced this pull request Sep 16, 2026
## `main` currently ships a validator that cannot start

#26 set `PRYSM_VERSION=v7.1.8`, but the validator command still passes
`--db-backup-output-dir`, which Prysm **removed** in v7:

```
Incorrect Usage: flag provided but not defined: -db-backup-output-dir
```

The container then crash-loops. This is not latent — it affects anyone
who pulls `main` and starts a validator today. I hit it for real
upgrading `contango-validator-1`, where it cost ~2.5 minutes of missed
duties.

Reproduced against `gcr.io/prysmaticlabs/prysm/validator:v7.1.8` with
the exact flag set `main` renders:

| flag set | result on v7.1.8 |
|---|---|
| `main` today | `Incorrect Usage: flag provided but not defined:
-db-backup-output-dir` |
| this PR | `INFO main: Prysm Validator started
version=Prysm/v7.1.8/51b5a75...` |

## Three fixes, all in `docker-compose.yml`

### 1. Drop the two dead backup flags

`--db-backup-output-dir` is gone in v7. `--enable-db-backup-webhook`
goes with it — v7 keeps it only in `config/features/deprecated_flags.go`
as a hidden no-op, so it logs `is deprecated and has no effect` on every
start and does nothing.

### 2. `validator-backup` actually backs something up

The service POSTs to the validator's `/db/backup` webhook — which is
precisely what the deprecated flag used to enable. **The call returns
success while backing up nothing.**

That matters more than a normal dead feature: the README tells operators
to run it before upgrading, and **slashing protection history lives in
that database**. A backup step that silently no-ops is worse than no
backup step, because it buys false confidence before the one operation
where you need the file.

It now runs `slashing-protection-history export`, which writes the
standard **EIP-3076** file that any client can import.

One constraint, spelled out in a comment in the file: it must run with
the validator **stopped**, because `validator.db` is a bolt database
with a single-writer lock.

```bash
docker compose stop validator
docker compose --profile backup run --rm validator-backup
docker compose --profile node --profile validator --profile init up -d validator
```

### 3. Restart policy for `geth`, `beacon`, `validator`

Only `stats` and `caddy` used the `x-restart` anchor. Without it, an OOM
kill or crash stops attesting until a human notices — an operator fleet
has already hit this, with a geth OOM-killed during an archive sync
sitting dead for two days while the node still reported "up".

## Verification

- `main`'s rendered flag set fails immediately on v7.1.8 with the error
above
- this PR's flag set reaches `Prysm Validator started`
- `validator slashing-protection-history export` exists in v7.1.8 and
accepts `--slashing-protection-export-dir`
- `docker compose config` clean across the `node`, `validator`, `init`
and `backup` profiles
- the running upgrade on `contango-validator-1` (canonical validator
v7.1.8 + `v7.1.8-vega.1` beacon) has been performing all duties —
attestations, aggregates, sync committee messages and contributions, and
block proposals — for 4+ hours

## Deliberately not in this PR

These came out of the same upgrade but need testing rather than review,
so they belong to the validator rollout session:

- **`--enable-doppelganger`** is absent from the repo. It is a real
anti-double-signing control and operators are hand-rolling it in
overrides. Shipping it costs ~2 epochs of missed attestations per start,
which deserves a deliberate decision.
- **The override trap.** Compose *replaces* `command:` rather than
merging, so an operator override restating the validator command
silently bypasses every future fix here — #28's REST fix was in the repo
and inert on the node for exactly this reason. Fixes 3 (and
doppelganger) reduce the *reason* to keep an override at all; the README
should also state the rule.
- **v5→v7 renamed log strings** (`Submitted new sync message` →
`Submitted sync committee messages`). Any alerting that greps validator
logs goes quiet and looks healthy. This fooled my own checks twice.
- **No rollback path.** A v7→v5 validator downgrade is unsupported and
untested.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
maciejwitowski added a commit that referenced this pull request Sep 24, 2026
## Problem

The Release workflow fails at `generateNotes` on every push that should
cut a release. No GitHub release has been published since v2.3.1
(2026-08-03), so #27, #26 and #28 are on `main` with no release. And
v2.3.1 itself shipped with an empty body.

| Run | Commit | Result |
|---|---|---|
| 2026-08-03 | ffd802d #21 | success, but v2.3.1 notes are the version
header only |
| 2026-08-12 → 08-19 | chore/ci commits | success (no release due, notes
step never ran) |
| 2026-09-04 | a4cb188 #27 | failure |
| 2026-09-14 | a409872 #26 | failure |
| 2026-09-14 | 47db176 #28 | failure |

```
Failed step "generateNotes" of plugin "@semantic-release/release-notes-generator"
Error: Missing helper: "conventional-changelog-conventionalcommits requires conventional-changelog-writer@9 or newer ..."
```

## Cause

The workflow installed the release tooling unpinned, with no lockfile:

- `npx semantic-release` resolves to 25.0.9. Its
`@semantic-release/release-notes-generator` 14.1.1 (the latest) depends
on `conventional-changelog-writer ^8`.
- `npm install --no-save conventional-changelog-conventionalcommits`
resolves to the latest preset. Preset **v10** targets writer 9:
- 10.0.0–10.3.0 (from 2026-06-26): no error, but writer 8 renders only
the version header. That is why the [v2.3.1
release](https://github.com/vana-com/vana/releases/tag/v2.3.1) has no
notes, while v2.3.0 (January, preset 9.x) does.
- 10.4.0 (2026-08-18): added an explicit guard, which turned the silent
empty output into the hard failure above.

Pinning only the top-level packages does not close this.
semantic-release loads its plugins from its own `node_modules` tree, and
every range below it (`release-notes-generator ^14.1.0`,
`conventional-changelog-writer ^8.0.0`, the GitHub plugin's Octokit
stack) resolves fresh on each run. The job holds a `contents: write`
token, so any new npm release in that tree runs with it, unreviewed.

## Fix

- New `.github/release/package.json` + committed `package-lock.json` for
the toolchain:

  | Package | Version |
  |---|---|
  | `semantic-release` | 25.0.9 |
  | `@semantic-release/commit-analyzer` | 13.0.1 |
  | `@semantic-release/release-notes-generator` | 14.1.1 |
  | `@semantic-release/github` | 12.0.9 |
| `conventional-changelog-conventionalcommits` | 9.3.1, the last major
built for writer 8 |

- The job installs it with `npm ci --ignore-scripts --prefix
.github/release` (no package in the tree has an install script) and runs
`.github/release/node_modules/.bin/semantic-release` from the repo root,
so `.releaserc.yaml` is still the config.
- It lives under `.github/` so the operator checkout gets no root
`package.json`. `.gitignore` covers its `node_modules`.
- Node pinned to `24`, the version this was verified on.
- A comment in the workflow says when to bump the preset: once the notes
generator ships writer 9.

## Verification

Ran the two workflow steps as written, on Node 24 / npm 11, on a
checkout of `main` at 47db176 with no root `node_modules`, the full
`.releaserc.yaml` and a GitHub token, `--dry-run --no-ci`:

- `npm ci` installs one deduped copy of each package; every plugin and
the preset resolve from `.github/release/node_modules`.
- GitHub `verifyConditions` passes, next version **2.3.2**, notes render
`### Bug Fixes` with #26, #27, #28, same format as v2.3.0.

Preset comparison (same command pair, earlier run):

| Preset | Result |
|---|---|
| 9.3.1 | notes render Bug Fixes |
| 10.3.0 | notes are the header only |
| 10.4.0 | fails with the same "Missing helper" error as CI |

Cosmetic: #26's commit body contains the word "closes" before
"compatibility", so its line gets a bogus `closes #compatibility` link.
Harmless; editing the release body after publish fixes it if wanted.

## Heads-up on merging

Merging this triggers the workflow on `main`, which will **publish
v2.3.2** as a public GitHub release containing #26, #27 and #28 (Prysm
v7 config keys, the deposit contract switch config, the validator REST
fix). The GitHub plugin also comments "included in version 2.3.2" on
those PRs and labels them `released`, as it did on #21 for v2.3.1.
Anyone watching this repo's releases gets notified, so merge when that
announcement is fine to go out.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Dqz4KWjLzxNEJuea75Avb7

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 2.4.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant