Skip to content

translator: drain, survive fallbacks, hold SV1 miners, measurement and jemalloc (mara-0.9 3/3) - #36

Open
average-gary wants to merge 19 commits into
m09/poolfrom
m09/translator
Open

average-gary wants to merge 19 commits into
m09/poolfrom
m09/translator

Conversation

@average-gary

Copy link
Copy Markdown
Collaborator

translator: drain, survive fallbacks, hold SV1 miners, measurement and jemalloc (mara-0.9 3/3)

This is the third PR of the mara-0.9 port, stacked on #35 (pool), which is stacked on #34 (base). It carries every Mara translator patch onto upstream v0.9.2:

Each patch was rewritten against 0.9.2 from its essence: behaviour, invariants, config keys, metrics, log lines and tests. They were not rebased. Upstream reworked the translator in 0.9: +3328/−1333 across 26 files, a new max_target.rs, reconnect on ChannelEndpointChanged/Reconnect, FallbackCoordinator and paced accept(). Tagging this stack after approval gives v0.9.2-mara.1, which is "mara.11".

Commits

Commit From mara
C1 drain SV1 miners gradually on SIGTERM/SIGINT cbde41a0…de0c37bd
C2–C6 keep used upstreams eligible; upstream state in monitoring and health; SetupConnection timeout; TCP keepalive; monitoring up across fallbacks #26 4eafea25
C7 keep the SV1 server and its miners across upstreams (generation epochs, hold off) #28 cfa2f98a
C8 raw-TCP SV1 mock miner (IT helper) #28
C9 hold SV1 miners across an upstream fallback #28
C10 ITs for held miners, including credit by the next pool #28
M1–M2 heap gauges and heap::use_allocator, SIGUSR1 trim #30, #33 (6e769f25)
M3 Block Found attribution, translator and JDC (cherry-pick) 33be4b3f
M4–M5 SV1 share-outcome and block-found reporting #30
M6 optional jemalloc build #33
M7–M8 held miners' outcomes across fallbacks; upstream-gone drops #30
CI clippy/test --features jemalloc #33

Behaviour operators and infra will see

  • SIGTERM with drain_seconds = 0 is a graceful shutdown that exits 0 (it was 143). Upstream stratum-apps: refine shutdown signal handling (support SIGTERM) stratum-mining/sv2-apps#915 always handles SIGTERM. With drain_seconds > 0, the first signal drains miners gradually and a second signal abandons the drain.
  • Planned drains and shutdowns neither record a fallback nor start a hold. sv2_upstream_fallbacks_total doesn't move during a rolling restart.
  • A pool-requested Reconnect (upstream harden tProxy channel state, difficulty targets and upstream endpoint changes stratum-mining/sv2-apps#909) counts toward max_fallbacks and sv2_upstream_fallbacks_total, and reports under the configured entry it came from. It is bounded by upstream's 5-minute minimum interval.
  • The SV1 listener stays bound during upstream walks. Downstream ids and the .minerN suffix are monotonic across fallbacks. With no generation, sv1_clients_total reads 0 rather than failing the scrape.
  • Health while holding is 200 with sv2_upstream_connected 0 everywhere. pool-v4's TranslatorUpstreamDisconnected already alerts on that.
  • Unchanged metrics:
    • sv2_upstream_fallbacks_total stays a Gauge, as shipped.
    • Duplicate upstream addresses share one sv2_upstream_connected series, as on mara.
  • New keys are all #[serde(default)], with the hold off by default. hold_keepalive_notify_secs ≥ 90 logs a warning.
  • Unchanged from mara: every metric name, type, label and help string, and the Block Found!!! 💰{hash} log lines.
  • glibc builds need glibc ≥ 2.33 to start (mallinfo2, for the heap gauges). This applies to pool and JDC too, because the gauges come from stratum-apps. pool-v4's debian:12-slim runtime has 2.36; a bullseye-based image would not start.

Interactions with upstream 0.9 and stratum-core 0.7.0

  • Reconnect / ChannelEndpointChanged, the SetTarget > max_target fallback and FallbackCoordinator all end a generation the same way a fallback does: epoch bump, then queue drain, then acknowledgement. A generation end that is never acknowledged fails the fallback, which fails closed.
  • Exactly one place sends from the SV1 side to the channel manager: send_to_channel_manager_in, which is generation-tagged. The epoch and the cancellation token are checked under its lock.
  • Every nominal_hash_rate sent is finite and ≥ 0, as #2436 requires (finite_hashrate).
  • Nothing in the hold or re-attach path sets a channel target the channel would refuse (#2456). The channel manager's own bounds check still falls back on a violation.

Re-attach

  • A held miner keeps its socket.
  • It is told a new extranonce only if the one it was actually told differs. That fixes a re-hold-before-first-job case found in review, where it would otherwise have hashed on a stale prefix.
  • It gets a forced clean job.
  • Its difficulty is the easier of its vardiff target and the new upstream target.
  • Shares on jobs whose prefix changed get error 21 rather than a false true.
  • held_miner_is_credited_by_the_next_pool_{aggregated,non_aggregated} proves, through a sniffer, that pool B accepts a share mined after the re-attach.

Measurement

  • The SV1 server owns the share totals, the drop counters and the block list. Every channel manager is built with them (ChannelManager::new requires them), and monitoring reads them through CurrentGeneration. So they are process-lifetime and survive fallbacks, which held_miner_is_credited_by_the_next_pool_{aggregated,non_aggregated} proves: sv1_shares_accepted_total is ≥ 1 before the fallback and ≥ 2 after it.
  • upstream-gone counts shares that were in flight when their upstream went away. They are counted at both the epoch gate and the Ended drain.
  • jemalloc: --features jemalloc, with a baked background_thread:true,thp:never on Linux. Under it the sv2_heap_* gauges are absent, not 0, and SIGUSR1 only logs. It is the same code translator: optional jemalloc build for mara.11 (minimal) #33 was approved with.
  • pool-v4-infra must build with TRANSLATOR_FEATURES=jemalloc in the same commit as the SV2_REF bump, or the image ships glibc.

Public API changes

  • UpstreamEntry.tried_or_flagged removed.
  • Upstream::new gains UpstreamConnectOptions.
  • Sv1Server::start / start_with_drain lose the coordinator argument.
  • TproxyErrorKind gains SetupConnectionTimeout, FallbackLoop and GenerationEndUnacknowledged.
  • PrometheusMetrics::new takes 4 bools.
  • GlobalInfo and MonitoringSnapshot gain upstream.
  • ChannelManager::new is pub(crate) and requires the share-accounting handles.
  • Sv1ClientsMonitoring gains four default methods.
  • Sv1ClientInfo.shares and Sv1ClientsSummary.{shares,dropped} are added, with #[serde(default)] (openapi.json regenerated).
  • MonitoringSnapshot gains sv1_blocks_found and sv1_blocks_found_total.
  • New monitoring::heap module.
  • The SV1→channel-manager message is now (MiningOwned, Option<ShareOrigin>).

Known limits

  • The re-attach-done UpdateChannel can be starved by a steady stream of new connects. It is bounded by the 60 s vardiff tick.
  • The epoch-gate and ack-timeout invariants are pinned by unit tests only (mutation-checked). The ITs can't stall the server loop deterministically.
  • A share the channel manager has taken off the queue and is still sending upstream when a fallback cancels its loop is counted nowhere. The window is one send, the same as mara.
  • upstream-gone has no race IT by design: a share racing a fallback would make it flaky. Both counting sites are unit-tested, and monitoring_integration checks that the series exists.
  • The *_group_channels ITs need --test-threads 1, which predates this port.

Tests

Local only.

  • Every commit builds. The final tree passes fmt and clippy -D warnings for the translator (default, --no-default-features, --features jemalloc), stratum-apps --all-features and jd-client, plus the unit tests: translator lib 238, stratum-apps 183 (--all-features) plus tests/heap_allocator.rs.
  • Builds: a --locked release build, including --features jemalloc.
  • Full IT suite on the tip (8f8849f1), run with 4 threads and 1 retry: 156 of 186 passed (2 flaky). The other 30 failed:
    • 27 pass when re-run one at a time. Every one of them failed in fund_wallet() (bitcoind's generate_to_address(101) returning an RPC error while several regtest nodes start at once). They are spread across pool, JD, solo-mining, IPC, sv1 and translator tests, most of which this stack doesn't touch.
    • 3 are pre-existing: the bitcoin_core_ipc_block_weight tests (v30x/v31x/v32x) time out at 120 s on this machine, the same way on bare v0.9.2 (origin/mara-0.9).
    • No failure is attributable to this stack. The held, fallback, reconnect, drain and monitoring ITs pass, including held_miner_is_credited_by_the_next_pool_* and tproxy_api_endpoints_with_miner.
  • Mutations: every add CI for docs.rs stratum-mining/sv2-apps#524/chore(deps): update stratum-core to 083b217 stratum-mining/sv2-apps#556 fix, the drain, setup timeout, I/O cancel, planned-shutdown counting, the extranonce re-hold fix and every measurement wire (10 unit and 2 IT mutations for measurement alone) were checked by reverting the fix and confirming the named test fails.
  • Rig storm A/B: same commit (75e47651, behaviour-identical to the tip), glibc vs jemalloc, 500 miners, 3 drop-and-reconnect storms, local aarch64, THP=madvise. The upstream is the rig's 0.8-based pool harness; only the translator is under test. Pass = RSS 30 s after the reconnect settles ≤ pre-storm + max(5 MiB, 5%).
    • glibc: 1/3 storms passed. RSS stepped 57.4 → 100.4 MiB, and at idle it returned nothing (100.4 → 100.4 MiB). glibc's free-but-held heap grew from 34 to 58 MiB across the storms.
    • jemalloc: 2/3 storms passed. It converged by storm 3 (pre 82.1, +30 s 86.1 MiB) and returned memory at idle: 87.2 → 74.8 MiB in 2 min.
    • This is the same shape as translator: optional jemalloc build for mara.11 (minimal) #33's evidence on 0.8 (glibc 0/3, 52.9 → 94.8 MiB, nothing returned; jemalloc 2/3, 80.3 → 66.9 MiB at idle). Absolute RSS is a few MiB higher on 0.9.2.
    • Raw data: prof-rig/runs/storm-m09-{glibc,je}-accepted-20261113-*.
  • Reviews: Opus adversarial review per group, with blockers found and fixed: the drain wait, the I/O-cancel test, planned drains counting fallbacks, the extranonce re-hold, and stale aggregated contexts. Fable final review per group.

Merge method

Rebase-merge after #35, not squash.

@gimballock gimballock left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial pass at 8f8849f1, in three parts: hold and generations (C7–C10), drain and fallback (C1–C6), and measurement and jemalloc (M1–M8 and CI). Each part compares the 0.9.2 behaviour with what was approved on mara (060cea58) and on #33 (b3c109dc). No must-fix. The rewrite preserves the approved invariants. There are three should-fixes, one process gap, and a few small items.

Process: CI has never run on this stack. ci.yaml and tests.yaml trigger only on mara-0.9, and this PR targets m09/pool, so there are 0 checks here, including the new jemalloc jobs. Everything below is local evidence. Please let CI go green once the stack retargets, before tagging v0.9.2-mara.1.

Should-fix

  1. The re-attach UpdateChannel isn't bounded by the vardiff tick, as the Known limits say.
    • The "done" refresh needs request_id_to_downstream_id to be empty (sv1_server/mod.rs:901), and the per-open aggregate refresh is suppressed while reattaching (:1913-1915).
    • The vardiff tick returns early when no estimate changed (difficulty_manager.rs:164). That's the normal state for re-attached miners, which keep their learned targets.
    • New connects keep the map non-empty. Under a steady connect stream, for example non-subscribers reconnecting after the re-attach, the pool keeps seeing the aggregate channel at the first re-opened miner's nominal hashrate until a quiet 10 ms tick or a disconnect.
    • Suggested fix: on the vardiff tick, while aggregated and reattaching, refresh the aggregate unconditionally. Or count only held re-opens toward "done".
  2. "At most 256 opens in flight" holds only for the held queue. New connects open directly (:1494), bypassing the cap while counting toward it (:878). So new connects can starve held re-opens, and they themselves are uncapped. Removing the >= MAX_CHANNEL_OPENS_IN_FLIGHT check leaves all 238 tests passing.
  3. A drain doesn't count as a planned shutdown for the fallback guards.
    • failure_outcome (lib/mod.rs:177-185) and cleanup_for_fallback (translator_runtime.rs:803) check only cancellation_token. A drain cancels drain_token, and is_draining() exists but isn't consulted.
    • So an upstream drop during the 20 s drain records a fallback, starts a hold if one is configured, and if the walk fails, exits 1 and drops the miners not yet drained.
    • 0.8 behaves the same, so this isn't a regression. But it contradicts "planned drains and shutdowns neither record a fallback nor start a hold".
    • Suggested fix: treat is_draining() like cancellation in both places, or narrow the claim.

Tests that would pin the fail-closed details

Three of these survive mutation, despite the "every stratum-mining#524/stratum-mining#556 fix mutation-checked" line:

  • moving drained.send(()) before the queue drains (mod.rs:730-736); production is masked only by the 1 s sleep in connect_with_retries;
  • deleting stop_holding() on GenerationEndUnacknowledged (translator_runtime.rs:834);
  • removing the 256 cap (should-fix 2).

Small items, for a follow-up

  • The default hold-expiry release is a burst. It's paced over drain_span(drain_window()), and drain_seconds defaults to 0. Same as 0.8.
  • A walk can outlast the hold window, which is checked only between walks (translator_runtime.rs:489-497). Health then goes 503 while miners are still held. Same as 0.8.
  • Mid-stack builds export flat share counters. At M5 and M6, CurrentGeneration doesn't forward the share totals, drops or blocks. That only matters if someone builds or bisects mid-stack, since this is rebase-merged.
  • The glibc ≥ 2.33 wording. mallinfo2 is linked through BUILT_IN on every linux-gnu build, the jemalloc translator included. So an older glibc fails at load, not when called, for pool, JDC and the translator alike. The release builds use musl and are unaffected.
  • mallinfo2 still runs on a tokio worker (snapshot_cache.rs:544), #30's follow-up 4.

What held up

  • Hold:
    • one send site (mod.rs:691), with every caller epoch-tagged;
    • every 0.9 generation-ending path cancels the fallback token, then drains and acks with fail-closed handling: Reconnect and ChannelEndpointChanged (5-minute gate), SetTarget > max_target, channel-manager fallbacks, and shutdown during a walk;
    • error 21 on a changed prefix is real;
    • "easier" difficulty can't violate max_target;
    • no std mutex is held across an await;
    • the killed mutations: the epoch check, cancellation check, remove_if, the "told" comparison, channel_epoch tagging, held-share rejection, both context clears, and the queue discard.
  • Drain and fallback:
    • a single signal listener; the second signal isn't lost; drain_seconds = 0 exits 0;
    • health stays 200 while draining;
    • the walk includes the last-used entry, can't spin, and doesn't exit while another entry is healthy;
    • Reconnect is counted and reported under the configured entry it came from;
    • the SetupConnection timeout leaks no tasks.
  • Measurement and jemalloc:
    • prometheus_metrics.rs and sv1.rs match mara, and so do all 32 metric name/help pairs;
    • jemalloc.rs, heap.rs and heap_allocator.rs are byte-identical to b3c109dc;
    • upstream-gone can't double count, since the gate returns before try_send and the drain sees only what's still queued;
    • held shares are counted once under job-not-found;
    • openapi.json regenerates identically;
    • no per-miner series.

Run locally

  • translator lib: 238/238, plus 3 with --features jemalloc;
  • stratum-apps --all-features: 183, plus heap_allocator;
  • clippy -D warnings clean for the translator (default, --no-default-features, jemalloc), JDC and the stratum-apps lib;
  • integration tests, one at a time: held_ 7/7, fallback 7/7, reconnect 7/7, drain 2/2, health, setup_connection, budget and only_upstream all pass.

Storm data (0.9.2): the same shape as #33, jemalloc converging and returning memory at idle while glibc doesn't. The amd64 run on dev closes the last gap.

average-gary and others added 19 commits November 16, 2026 17:03
Cancelling the single global token on a stop signal drops every miner
at the same instant. Behind a load balancer that turns each restart
into a thundering-herd reconnect.

Add an opt-in drain on top of the ShutdownListener signal handling:

- `drain_seconds` (default 0, env TPROXY__DRAIN_SECONDS): on the first
  stop signal the SV1 listener is closed, then the connected miners are
  disconnected one at a time, one per equal slot of the window with a
  random offset inside the slot. The upstream SV2 connection stays up so
  miners not yet drained keep mining. Shutdown starts once the last
  miner is gone, or at the end of the window, when the remaining miners
  are cut by the normal shutdown path. The window is capped at one day
  so the deadline cannot overflow `Instant`.
- `drain_send_reconnect` (default false): send `client.reconnect` with
  no parameters before closing each miner's connection.

Disconnects are spread over the window less a tenth (at most 1s) so the
last miner is drained before the shutdown deadline instead of racing
it. The drain ends as soon as every downstream is gone, rather than
sleeping out the remaining slots. Drained miners go through the normal
downstream cleanup, so the SV1 client counts in the monitoring API fall
one by one. The drain task is not tracked by the fallback coordinator,
so it never holds up fallback cleanup; a fallback clears the downstreams
and ends the drain.

The first signal now always runs `drain_and_shutdown()`. With
`drain_seconds = 0` that is exactly `shutdown()`, so SIGTERM is a
graceful shutdown that exits 0, as ShutdownListener defines it. A second
signal still abandons the graceful path. `Sv1Server::start` keeps its
signature and delegates to `start_with_drain`.

`shutdown()` now creates its Notified future before checking is_alive,
so it cannot miss a teardown that finishes concurrently, which the
drain makes routine.

Refs: MaraPool/pool-v4-infra#301

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`initialize_upstream` flagged an upstream as `tried_or_flagged` as soon
as it connected, and nothing ever cleared the flag. Every upstream could
therefore serve miners once per process: after the last configured entry
had been used, the next fallback found nothing to try and the translator
exited. With a single `[[upstreams]]` entry that meant any pool restart
or network blip killed the process and every SV1 connection with it.

Replace the flag with an `UpstreamWalk`:

- Each walk visits every entry at most once (3 attempts each, as before)
  and starts at the entry after the one that was last connected. A
  fallback moves to the next upstream first and only returns to the one
  that just failed after all others were tried. A walk that finds no
  upstream still ends in `CouldNotInitiateSystem` and exits.
- Because a used upstream is picked again, one that accepts connections
  and then keeps failing (for example a payout mismatch) would loop
  forever. More than `max_fallbacks` (default 5) fallbacks within
  `fallback_window_secs` (default 600) now exits with `FallbackLoop`,
  handing restarts back to the process supervisor. `max_fallbacks = 0`
  exits on the first fallback.

A reconnect the upstream asks for (Reconnect, ChannelEndpointChanged)
also goes through the fallback path, so it counts toward the budget; the
minimum interval between protocol reconnects keeps that to a couple per
default window. The redirected endpoint is not in the configured list,
so it is recorded as the configured entry it was redirected from.

No entry is flagged as bad. An upstream rejected during setup
(SetupConnectionError, a bad authority key) costs its 3 attempts on
every walk. One that accepts the connection and then fails (a payout
mismatch) is only bounded by the fallback budget: with
`[A (bad payout), B]`, every B outage costs two fallbacks and adds A's
connect time to the outage.

This changes default behaviour: a translator now survives fallbacks it
used to exit on, and a single-upstream translator retries its upstream
3 times before exiting instead of exiting at once.

`UpstreamEntry::tried_or_flagged` is removed.

Refs: MaraPool/pool-v4-infra#305
Refs: stratum-mining#673

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Operators have no way to see whether an app is connected to its
upstream: the server monitoring types describe channels opened on an
upstream, not whether one is reachable, and `/api/v1/health` answers 200
as long as the process is up. An app that keeps its monitoring server
running while it has no upstream (before the first connection, or
between fallbacks) is therefore indistinguishable from a healthy one.

Add an opt-in `UpstreamMonitoring` source, enabled with
`MonitoringServer::with_upstream_monitoring`. It reports every
configured upstream endpoint with its connection state, plus the number
of fallbacks since the process started. Following the design in
stratum-mining#463, the values
are read into the snapshot cache on refresh and exported as gauges, so
scrapes never call into business logic:

- `sv2_upstream_connected{pool_address}`: 1 for the connected endpoint
  and 0 for every other configured one, so each configured upstream
  always has a series and an alert can tell "disconnected" from "no
  data". Series of endpoints that are no longer reported are removed.
- `sv2_upstream_fallbacks_total`, a gauge set from the snapshot.

The same data is returned as `upstream` by `/api/v1/global`, which is
`null` for apps that do not opt in, like the other optional sections.

For apps that opt in, `/api/v1/health` now answers 503 with status
`upstream_disconnected` while no upstream is connected, so a load
balancer stops routing to an instance that cannot serve work. The check
calls `UpstreamMonitoring::is_connected` on every request instead of
reading the snapshot, which can be a whole refresh interval old; that
method is therefore required to be cheap and lock-free. Apps that do not
opt in keep answering 200 "ok" unchanged, which pool and JDC rely on.

`with_upstream_monitoring` and `with_sv1_monitoring` both rebuild the
Prometheus metrics from the sources the cache already has, so
`PrometheusMetrics::new` gains an `enable_upstream_metrics` flag and
each builder preserves the other's gauges, in either call order.

`SnapshotCache::snapshot_timestamp` reads when the snapshot was taken
without cloning it, for callers that only need the age. The OpenAPI
schema is regenerated.

Refs: MaraPool/pool-v4-infra#305

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
After the Noise handshake the translator sends SetupConnection and
waits for the upstream's answer on the io channel with no deadline. A
pool that completes Noise but never answers, because it is wedged or
because a middlebox swallows the reply, keeps the translator stuck on
that connection forever: it never retries and never moves on to the next
configured upstream.

Bound the wait with `setup_connection_timeout_secs` (default 10s, the
same as the Noise handshake timeout; 0 selects the default since a zero
timeout would fail every attempt). Only the wait for the response is
timed. On expiry the attempt fails with a new
`TproxyErrorKind::SetupConnectionTimeout` carrying the fallback action,
so the runtime treats it like any other setup failure and retries or
advances to the next upstream.

The reader and writer tasks of an abandoned connection are not stopped
by a setup failure on their own: the writer exits when the upstream is
dropped, but the reader keeps waiting on a socket the peer may never
close, and both are tracked by the fallback coordinator, which then
holds up the next fallback. Give each connection a child of the
shutdown token and cancel it when setup fails, so those tasks exit with
the attempt. The channel-manager channels held by `UpstreamIo` are
shared across attempts and are deliberately left open.

The per-connection settings travel in `UpstreamConnectOptions`, built
from the `TranslatorConfig`, so later connection options do not keep
growing the `Upstream::new` signature.

Refs: MaraPool/pool-v4-infra#305
Refs: stratum-mining#667

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A pool that disappears without closing the connection (a dropped route,
a NAT entry that expired, a crashed host) leaves the translator waiting
on a healthy-looking socket until the kernel gives up retransmitting,
which on Linux takes around 15 minutes of retransmission timeouts once
there is unacknowledged data, and never if the connection is idle.
Application-level detection would need protocol support; TCP keepalive
gets the same result from the kernel for free.

Add three optional settings, all off by default so behaviour does not
change for existing deployments:

    upstream_tcp_keepalive_idle_secs
    upstream_tcp_keepalive_interval_secs
    upstream_tcp_keepalive_probes

The idle time is the switch. The interval and probe count only override
the operating system defaults when it is set; if they are present
without it the translator logs a warning at startup, since there is no
established config validation step to reject them. A value of 0 is
treated as unset.

Keepalive is applied right after the TCP connection is established,
before the Noise handshake, so it also covers a peer that goes silent
mid-handshake. Failing to set the option is logged and the connection
proceeds without keepalive. The interval and probe count are set on
the platforms where socket2 supports them and ignored, with a warning,
elsewhere.

socket2 0.6.3 is already in Cargo.lock through tokio, so no new version
is introduced.

Refs: MaraPool/pool-v4-infra#305
Refs: stratum-mining#667

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The monitoring server started only after an upstream connected and was
stopped and restarted with every fallback, so metrics and health were
unreachable for the whole upstream walk, which is exactly when an
operator needs them.

Start it once, before the first upstream connection, and keep it
running until shutdown:

- It reads the ChannelManager and Sv1Server through `CurrentGeneration`,
  which is set when an upstream generation starts serving and cleared on
  fallback, so it never reports a generation that is being torn down.
  The SV1 miner telemetry loop still runs per generation.
- It reports `UpstreamWalk`'s state through `UpstreamMonitoring`:
  `sv2_upstream_connected{pool_address}` and
  `sv2_upstream_fallbacks_total`, and `/api/v1/health` answers 503
  `upstream_disconnected` whenever no upstream is connected.

The health check keeps its meaning for load balancers and readiness
probes: before this change the endpoint was unreachable without an
upstream, now it answers 503, so a translator with no upstream still
receives no miners. "Connected" means the upstream accepted
SetupConnection, which happens shortly before the SV1 listener is up;
health stays 200 while draining.

While no generation is serving, `sv1_clients_total` and the server
gauges report 0 instead of the scrape failing.

`CurrentGeneration` holds the ChannelManager and the SV1 server
separately, so each can be set and cleared on its own.

Refs: MaraPool/pool-v4-infra#305
Refs: stratum-mining#463

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Every fallback rebuilt the SV1 server together with the channel manager
and the upstream: its listener was dropped, every miner task ended on
the fallback token, and all per-miner state went with them. Holding
miners across a fallback needs the SV1 side to outlive the upstream
side first.

The SV1 server, its listener, its downstream tasks and the two channels
between it and the channel manager are now created once. Only the
upstream, its io tasks and the channel manager are rebuilt per upstream
generation, and only they are tracked by the FallbackCoordinator. The
runtime tells the SV1 server about the boundary in band, on a dedicated
channel it polls before channel-manager messages:

- `Ended` is sent after the old channel manager has exited, so it lands
  behind everything that manager sent. The server discards whatever is
  still queued in either direction, clears every channel, job, prevhash
  and pending request of that generation, and disconnects every miner,
  which is what miners saw before.
- `Connected` is sent before the next channel manager starts, carrying
  that generation's fallback token. Channel opens requested while no
  generation was connected are sent then, paced and bounded.

Because the SV1 server now outlives channel managers, a message it reads
from one generation must never reach the next one, where channel ids
are handed out again and usually belong to another miner. A stale
`CloseChannel` there would silently drop that miner's channel while the
SV1 server kept accepting its shares, uncredited. Every message for the
channel manager (shares, channel opens and closes, `UpdateChannel`) is
therefore tagged with the generation epoch its contents were read under
and queued, under the generation lock, only while that epoch is current
and connected and its fallback has not started. `Ended` advances the
epoch under the same lock before discarding the queue, and the runtime
waits for it to do so before starting the next channel manager. If the
SV1 server does not acknowledge `Ended` within 5 seconds, the fallback
fails and the translator exits rather than hand the queue over
undrained.

A miner's own task, which handles its disconnect, reads the epoch before
anything else, and its `CloseChannel` carries the epoch the channel was
assigned under. It unmaps a channel only while the channel still
belongs to it. A fallback requested by the SV1 server, by the channel
manager (including a `SetTarget` above the channel's `max_target`) or by
the upstream (a requested reconnect) ends the generation the same way.

Any channel open unanswered for 10 seconds sends its miner
`client.reconnect`, so a lost answer cannot keep it waiting. Vardiff is
skipped while no generation serves.

Two things differ for an operator: the SV1 listener stays bound while
the translator looks for an upstream, so miners that connect then are
queued (and dropped if no upstream is found) instead of refused; and
downstream ids and the `.minerN` user identity suffix keep counting
across fallbacks instead of restarting at 1.

`Sv1Server::start` and `start_with_drain` no longer take a
`FallbackCoordinator`.

A drain has nothing to wait for while no SV1 miner is connected
or held, which includes the first walk, so `drain_and_shutdown` then
shuts down at once instead of waiting out `drain_seconds`.

The epoch gate and the acknowledgement timeout are pinned by unit
tests only: the integration tests cannot stall the SV1 server, so none
of them can make a message cross a boundary or an `Ended` go
unacknowledged.

Refs: MaraPool/pool-v4-infra#305
Refs: MaraPool/pool-v4-infra#524
Refs: MaraPool/pool-v4-infra#556

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Tests that need to show an SV1 connection stays open across an upstream
fallback of translator_sv2 have to know which TCP connection they are
looking at and what the translator sent on it. The existing SV1 sniffer
sits between two roles and the ad-hoc TcpStream miners in the translator
tests do not read anything, so neither can answer that.

Sv1MockMiner is the SV1 client itself. It performs the handshake
(optional mining.configure and mining.extranonce.subscribe, then
subscribe and authorize), records every server message in order, tracks
the extranonce from the subscribe response and later set_extranonce, and
reports when the server closes the connection. Its local address is
captured at connect, so a test can assert the connection is the same one
before and after an event. wait_for_method keeps a cursor, so messages
that arrive before the call are not missed and consumed ones are not
returned twice. submit sends a dummy share for a notified job with the
correct extranonce2 size and the job's ntime.

It is exercised against a small in-process fake SV1 server, so it needs
no translator.

start_pool_with_server_id lets a test pick the pool's server_id, so two
pools hand out different extranonce prefixes. start_pool keeps its
signature and uses server_id 1.

Ported from mara's cfa2f98 (#28).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Losing the upstream disconnected every SV1 miner, so a pool restart or
a switch to the next `[[upstreams]]` entry meant a reconnect storm of
the whole fleet, plus whatever time each firmware takes to come back or
fail over to its own backup pool.

With `upstream_loss_hold_secs > 0` the SV1 server keeps every miner
that had a channel when the generation ends:

- Held miners stay connected and are sent a keep-alive job every
  `hold_keepalive_notify_secs` (default 30) until their channel is
  re-opened: the last job each one was sent, with nTime rolled forward
  (up to Bitcoin Core's future-time limit, then only the id changes)
  and a new `#n` job id. Each miner keeps that job across generations,
  so a second fallback before the next job still has one. cgminer-based
  firmware gives up on a pool that sends nothing for 90 seconds, and
  cpuminer stops hashing after 120 seconds unless the job id changes.
  These jobs are never valid for submission; shares from held miners
  are answered with error 21 (job not found) at once and never reach a
  channel manager. Vardiff is paused between generations so the gap is
  not read as lost hashrate, and a difficulty change still waiting for
  its job is dropped, so the miner keeps the difficulty it works at.
- The runtime keeps walking the upstream list, pausing 1 s doubling to
  16 s with jitter between walks, until one connects or the window
  would end before the next walk. `/api/v1/health` stays 200 while the
  window lasts, so a load balancer does not deregister the translator
  and cut the connections it is keeping. `sv2_upstream_connected` still
  reports 0 for every endpoint, so a hold can be alerted on.
- When an upstream connects, each held miner gets a channel through the
  normal open path, with at most 256 opens outstanding, keeping its
  `.minerN` identity and the hashrate vardiff learned for it. In
  aggregated mode that is one upstream open, the rest served locally,
  and one `UpdateChannel` when the whole re-attach is done. The miner
  keeps the target vardiff wants for it and is advertised the easier of
  that and the new upstream channel's target, as for any channel. Its
  own task compares the new extranonce with the one it has, behind
  everything still queued for it: an unchanged one needs nothing, a
  changed one is sent as `mining.set_extranonce` to a miner that sent
  `mining.extranonce.subscribe`, and any other miner is sent
  `client.reconnect`. The first job after that has `clean_jobs = true`.
- When the window ends without an upstream, every miner is sent
  `client.reconnect` across the drain window and the translator exits,
  as it did before.

A hashrate that goes out as a `nominal_hash_rate` must be finite and
not negative, or the message carrying it cannot be serialized. The
hashrate a held miner is re-opened with, and the ones summed into the
aggregate, are clamped to that; an unusable learned hashrate falls back
to `min_individual_miner_hashrate`.

A miner whose extranonce changes mid-session without
`mining.extranonce.subscribe` support is now sent `client.reconnect`
before its connection is closed, instead of being closed silently.

`upstream_loss_hold_secs` defaults to 0, which disconnects miners as
soon as the upstream is lost.

A shutdown or drain that overlaps a fallback or an upstream walk ends
`start` with `Ok`, and a fallback requested after the translator was
cancelled is not recorded. A planned stop therefore exits 0 and neither
bumps `sv2_upstream_fallbacks_total` nor spends the fallback budget.

Refs: MaraPool/pool-v4-infra#305

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two pools behind one template provider, with different `server_id`s so
they hand out different extranonce prefixes, and a raw-TCP SV1 miner
that can tell whether its own connection survived:

- after the first pool stops, a miner that sent
  `mining.extranonce.subscribe` keeps the same connection, receives
  `mining.set_extranonce` and a clean job from the second pool
  (aggregated and non-aggregated);
- a miner without that support is sent `client.reconnect`;
- with no pool to go to, a held miner receives keep-alive jobs, its
  share on one is rejected with error 21, health stays 200, and it is
  sent `client.reconnect` when the hold window ends, after which the
  translator exits;
- an upstream `Reconnect` ends the generation like a fallback does: the
  miner is held and moved in place onto the requested endpoint, which
  is scripted with mock pools.

They run on a multi-threaded runtime, like the translator binary: when
a pool stops while frames from it are still queued, the reader task
that owned the connection drops its buffer pool, whose `Drop` spins
until the frames are released, which a current-thread runtime never
gets to do.

Refs: MaraPool/pool-v4-infra#305

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A process's resident memory also counts memory the allocator has freed
but not returned to the OS, so a translator whose memory grows cannot
tell from it whether it is still holding that memory or only the
allocator is. The two call for different fixes: a leak in the code, or
allocator tuning.

Export `sv2_heap_in_use_bytes`, `sv2_heap_free_bytes` and
`sv2_heap_mapped_bytes`, set on every snapshot refresh from the
allocator's statistics. glibc reports them through `mallinfo2`, so a
glibc build needs glibc 2.33 or later to start. `mallinfo2` walks the
free lists under each arena's lock, which is why it runs with the
periodic refresh and not on every scrape.

The gauges exist only if the active allocator reports statistics when
the monitoring server starts, so they are absent on targets without
glibc. An application using another allocator registers it with
`heap::use_allocator`, before starting the monitoring server, so that
the glibc figures do not describe a heap it no longer uses. Only the
first registration takes effect. `heap::resident_bytes` and
`heap::release_free_heap` serve the SIGUSR1 heap release that follows.

The gauges come from the shared crate, so the pool and the JD client
export them as well.

Refs: MaraPool/pool-v4-infra#464

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The heap gauges show how much memory the allocator holds free, but not
whether it can give that memory back. `release_heap_on_sigusr1 = true`
makes SIGUSR1 run glibc's `malloc_trim(0)` off the async runtime,
logging heap usage and resident memory before and after. Resident
memory is the figure to read: the trim returns free pages from the
middle of the heap without shrinking it, so the heap figures can stay
put while resident memory drops. Memory that drops was retained by the
allocator; memory that stays is in use, or fragmented below page size,
which the in-use figure tells apart.

It is opt-in because SIGUSR1 otherwise keeps its default action and
terminates the process, and because the trim holds the allocator's
locks while it runs. The handler is installed after logging starts and
before the translator starts, and the setting is read once at startup.
Without glibc, or without the monitoring feature, the handler logs that
it cannot report or release heap memory, or that the setting needs a
build that can.

Refs: MaraPool/pool-v4-infra#464

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…worker

Same tracing fields as the pool's Block Found lines (downstream_id,
channel_id, user_identity, share_user_identity, template_id), message text
unchanged so existing `Block Found` greps keep matching.

- jd-client: the four downstream/forwarding-upstream Block Found lines get
  downstream_id, channel_id, user_identity, template_id; the forwarding lines
  add upstream_channel_id; extended lines add share_user_identity from the
  worker TLV (previously parsed into `_user_identity` and dropped). The cached
  share line gets upstream_channel_id (no downstream context there).
- translator: a block found against the upstream channel was folded into the
  valid-share arm and never logged. It now logs
  `SubmitSharesExtended: 💰 Block Found!!! 💰<hash>` with channel_id,
  upstream_channel_id (aggregated mode), user_identity and
  share_user_identity (the SV1 worker name sent upstream as the TLV).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 33be4b3)
Nothing counted the translator's answers to `mining.submit`. In
aggregated mode the per-channel share metrics cover only the one
upstream channel, so neither a miner's stale rate nor the fleet's was
visible. Nor did any metric name the miner of a block candidate:
`sv2_server_blocks_found_total` is a single count per upstream channel,
so attributing a block meant searching logs.

Give `Sv1ClientsMonitoring` the data to report both, and export it:

- `sv1_shares_accepted_total` and `sv1_shares_rejected_total{reason}`
  count every answer since the process started, by the error code sent
  to the miner. `job-not-found` (21) includes stale shares, as well as
  job ids that are evicted or unknown. Every reason is exported from
  the start, so a rate never starts mid-way.
- `/api/v1/sv1/clients` reports each connection's counts under `shares`.
  There is no per-miner Prometheus series, so the cardinality stays
  fixed.
- "Accepted" means answered `true` to the miner. The translator can
  still drop such a share before it goes upstream, and those are
  counted too: `sv1_shares_below_upstream_target_total` for shares that
  miss the upstream channel's target, and
  `sv1_shares_dropped_total{reason}` for the rest. Dropped shares are a
  subset of the accepted ones, so a stale rate adds them to its
  numerator only. Shares the upstream itself rejects are counted by the
  per-channel metrics.
- `sv1_block_found_info{sv1_username,worker,user_identity,share_hash,
  height}` has one series per recent block candidate, valued at the
  Unix time it was submitted. A candidate is a share that met the
  network target as the translator validated it; the pool may still
  reject it or lose the race. Series drop out as the source's bounded
  list moves on. `sv1_blocks_found_total` counts candidates since
  start, for the alerts and rates the bounded list cannot serve.

The sources default to reporting nothing, so existing implementations
keep working. The snapshot builds the SV1 summary from the clients it
has just read instead of walking them a second time. The translator
reports through these in the commits that follow; its client list
carries empty counts for now.

Refs: MaraPool/pool-v4-infra#508
Refs: MaraPool/pool-v4-infra#549

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Report what the translator does with the shares it receives, through
the SV1 monitoring added in the previous commit.

Each answer to `mining.submit` is counted once, at the point it becomes
final, by the error code sent to the miner. sv1_api rejects some
submits (an unauthorized worker, a wrong extranonce2 size) before the
server's own checks, and an unsubscribed miner is answered directly, so
counting inside `handle_submit` would miss them. The count goes to the
connection's own totals and to the SV1 server's totals, which outlive
any single miner. Answers to requests that are not submits, such as
`mining.extranonce.subscribe`, are not shares and are not counted.

The channel manager counts the shares it accepted from SV1 miners but
does not send upstream, in every path that returns without forwarding:
below the upstream channel's target, each share validation error, and
an unknown upstream channel.

To name the miner of a block candidate, the SV1 server sends the
channel manager each share's origin (downstream id, SV1 username and
worker name) instead of the worker name alone. When a share meets the
network target, in either mode, the channel manager logs it and records
it with the miner's names, its channel's user identity and the BIP34
height of the share's job coinbase, empty if that cannot be read. The
record keeps the last 32. The
"Block Found" log line keeps its message and existing fields as they
were formatted, and adds `downstream_id`, `sv1_username` and `height`.

The per-share lines logged for each `mining.submit` and for each share
forwarded upstream move to debug. At info level every share wrote a
line from the single SV1 server task, about a thousand a second at 10k
miners, and the counters now report the outcomes.

The SV1 server owns the dropped-share counts and the block list and
every channel manager is built with the handles, so the counts outlive
any one channel manager and a channel manager cannot be built without
them.

Refs: MaraPool/pool-v4-infra#508
Refs: MaraPool/pool-v4-infra#549

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resident memory of a glibc translator stays high after a reconnect
storm: freed memory is retained by the allocator rather than returned
to the OS. Add a `jemalloc` cargo feature that installs
tikv-jemallocator as the global allocator, so freed pages go back to
the OS in the background.

The build bakes in a `_rjem_malloc_conf` of
`background_thread:true,thp:never` on Linux, and nothing elsewhere.
`_RJEM_MALLOC_CONF` still overrides it. Only the options that change
behaviour are set. The vendored jemalloc (tikv-jemalloc-sys 0.7.1,
jemalloc 5.3.1) already defaults `muzzy_decay_ms` to 0 and
`dirty_decay_ms` to 10000 (DIRTY_DECAY_MS_DEFAULT and
MUZZY_DECAY_MS_DEFAULT in arena_types.h). background_thread defaults to
false, and thp to `default`, which leaves the kernel setting alone, so
those two are set explicitly.

The glibc heap gauges would describe an allocator the Rust heap no
longer uses, so the translator registers one whose stats return None
via heap::use_allocator, as the first thing it does: the sv2_heap_*
gauges are not exported, and SIGUSR1 logs that the allocator cannot
report or release. The documentation of `release_heap_on_sigusr1`, the
example configs and the monitoring README say so.

hotpath-alloc installs its own allocator, so it takes precedence
over jemalloc, with a single startup warning. The feature is
unavailable on MSVC. A test reads opt.background_thread and opt.thp
through mallctl to check that the baked configuration is applied; the
thp check is skipped where the kernel exposes no THP setting.

This brings tikv-jemallocator 0.7.0 and tikv-jemalloc-sys 0.7.1 into
Cargo.lock and nothing else.

(adapted from commits a727f36 and b3c109d)

Refs: MaraPool/pool-v4-infra#464

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The SV1 server outlives every upstream generation, so the share totals,
dropped-share counts and block list it owns are the process-lifetime
figures monitoring is meant to export. Two paths did not reach them:

- A share from a miner held between generations is answered with
  error 21 before it gets to the normal submit handling. It is now
  counted as `job-not-found` like any other rejection, so held shares
  show in `sv1_shares_rejected_total`.
- The monitoring server reads SV1 data through `CurrentGeneration`,
  which only forwarded the client list. The trait defaults for the
  share totals, dropped shares and block list return zeros, so the
  exported counters would have stayed flat. All four are forwarded to
  the SV1 server.

The tests pin both: every channel manager built for a new generation
shares the SV1 server's handles, so nothing is counted on a copy that
dies with the generation, and `CurrentGeneration` reports what the SV1
server holds. The held-miner unit test asserts its rejected share is
counted, and the integration test for a held miner credited by the
next pool checks `sv1_shares_accepted_total` keeps growing across the
fallback.

Refs: MaraPool/pool-v4-infra#508

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An accepted SV1 share that the channel manager never sees is lost
without a trace: it is neither forwarded nor rejected, so neither the
pool's counters nor the dropped-share metrics account for it. That
happens when the upstream generation it was meant for ends, in two
places in the SV1 server:

- `send_to_channel_manager_in` refuses a message once its generation
  has ended or its fallback has started.
- Ending a generation discards what is still queued for its channel
  manager.

Both now count each `SubmitSharesExtended` they discard as dropped with
reason `upstream-gone`, in the totals the SV1 server keeps across
generations. Other messages (channel closes, difficulty updates) are
not shares and are not counted. The count is added after the
generation lock is released.

Refs: MaraPool/pool-v4-infra#508

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Nothing else builds the code that the `jemalloc` feature turns on, nor
runs its test that the baked-in allocator configuration is applied.
Add one clippy run over all targets and one test run with the feature.

Refs: MaraPool/pool-v4-infra#464

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@average-gary

Copy link
Copy Markdown
Collaborator Author

Thanks, @gimballock.

Restack: #34 was rebase-merged into mara-0.9, and #35 now targets mara-0.9. This PR is rebased onto the new #35 head with its content unchanged from 8f8849f1, which you reviewed. It now carries #35's CI-trigger commit underneath it.

Process gap (CI): you're right that CI has never run on the stack. The cause is wider than the triggers: GitHub Actions is disabled for this repository (workflow_dispatch returns HTTP 422), and no run has been created since 2026-10-06. I've asked for it to be re-enabled. #35 now triggers on m09/** as well, so once Actions is back the checks run here on the next push. No tag until they're green.

Should-fixes, as follow-up commits on top (in progress, with Opus and Fable review):

  1. Re-attach UpdateChannel:
    • Only held re-opens count toward "re-attach done", so new connects can't keep reattaching true.
    • In aggregated mode, while re-attaching, the vardiff tick refreshes the aggregate unconditionally, so it's really bounded by the tick.
    • A test runs a steady stream of new connects after a re-attach. Both halves are mutation-checked.
  2. 256-opens cap: the cap now counts and gates held re-opens only. New connects are already paced by accept() (fix: pace accept() retries after a failed accept stratum-mining/sv2-apps#955) and no longer count toward it. A new test with more than 256 held miners fails if the cap check is removed.
  3. Upstream loss during a drain:
    • It's still handled as a real fallback, so the miners not yet drained keep mining on the fallback for the rest of the window.
    • If the walk fails or the budget runs out while draining, the translator now exits 0 through the normal shutdown instead of 1.
    • The PR body is updated to say "planned drains don't cause fallbacks". You were right that the old wording overclaimed.
  4. Tests for the surviving mutations: drained.send(()) must come after the queue drains, and stop_holding() must be called on GenerationEndUnacknowledged. Both get tests that are mutation-checked. The 256 cap is covered by item 2.

Small items:

  • The release burst at hold expiry, and a walk outlasting the hold window: same as 0.8. I'll list them under Known limits.
  • Mid-stack builds exporting flat counters: noted in the body. It doesn't matter for a rebase-merge, but it does for a bisect.
  • glibc ≥ 2.33: I'll correct the wording. It's a load-time requirement on every linux-gnu build, including the jemalloc translator, and the musl release builds are unaffected.
  • mallinfo2 running on a tokio worker: still translator: heap, SV1 share-outcome and block-found measurement (mara.11) #30's follow-up 4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants