Skip to content

ci, deps: fork workflows, utoipa 6, RUSTSEC fixes and TaskManager fix (mara-0.9 1/3) - #34

Merged
based64god merged 8 commits into
mara-0.9from
m09/base
Nov 16, 2026
Merged

based64god merged 8 commits into
mara-0.9from
m09/base

Conversation

@average-gary

Copy link
Copy Markdown
Collaborator

ci, deps: fork workflows, utoipa 6, RUSTSEC fixes and the TaskManager fix (upstream stratum-mining#714)

This is the first PR of the mara-0.9 port, which carries the Mara patch set onto upstream tag v0.9.2. Gary decided on 2026-11-12 to build the next image (v0.9.2-mara.1, "mara.11") on 0.9.2. The plan is in pool-v4-dashboard docs/sv2-0.9-upgrade-plan.md.

The PRs are stacked: this one (base) → pool → translator (lifecycle + measurement, coming next). mara stays the 0.8 hotfix line.

What's here

  1. ci: the fork's workflows and Docker build now target mara-0.9.
    • docker-release.yaml builds upstream's docker/Dockerfile. Upstream moved the Dockerfiles there; the fork's old root Dockerfiles are gone in 0.9.2.
    • It pushes ECR pool, jdc and tproxy on mara-0.9 and on v*-mara.N tags, amd64 only, with no :latest (ECR tags are immutable) and cargo caching through BuildKit cache mounts.
    • There is no bitcoind image and there are no envsubst templates any more; the 0.9 images are configured with POOL__*-style env vars. pool-v4 builds its own images from SV2_REF, so it isn't affected.
    • version-bump-check is dropped: Mara tags keep upstream's crate versions.
    • stratum-core-sync, notify-*, release-apps and auto-rebase only run on stratum-mining/sv2-apps. On the fork, auto-rebase would force-push every labelled PR onto mara-0.9, whatever its base.
  2. utoipa 6 / utoipa-swagger-ui 10.0.2, cherry-picked from upstream main (build(stratum-apps): bump utoipa to 6 and utoipa-swagger-ui to 10.0.2 stratum-mining/sv2-apps#995, plebhash), with its Swagger/OpenAPI route tests. This is the zip 3 → 9.4.1 fix that isn't in a release yet. upstream main's lockfile was not taken, because it pins stratum-core from git. The lock was regenerated on v0.9.2 instead.
  3. fix(deps): cargo audit on the single root Cargo.lock goes from 23 to 0 vulnerabilities. Four allowed unmaintained/yanked warnings remain, all via criterion in integration-tests.
    • The bumps: rustls 0.23.46, rustls-webpki 0.103.16, webpki-root-certs 1.0.12, quinn-proto 0.11.19, chacha20 0.10.2, anyhow 1.0.104.
    • tokio is pinned at 1.53.3, the minimum that clears RUSTSEC-2026-0386. 1.54 is avoided because of its runtime changes (accept readiness, timers).
    • prometheus is built without its default protobuf feature; only TextEncoder is used.
    • integration-tests uses minreq 3, which uses rustls 0.23. minreq 3's status code is a u16, hence the one-line change in integration-tests/lib/utils.rs.
    • This is what turns pool-v4-infra's sv2 rust gate green for this ref.
  4. upstream perf(stratum-apps): make TaskManager::spawn tracking constant-time stratum-mining/sv2-apps#714 (TaskManager O(N²) spawn): Eric's three commits, cherry-picked verbatim, so they can be dropped if perf(stratum-apps): make TaskManager::spawn tracking constant-time stratum-mining/sv2-apps#714 merges upstream.
  5. fix(stratum-apps): keep tasks reachable when join_all is cancelled is a fork-only fix on top of perf(stratum-apps): make TaskManager::spawn tracking constant-time stratum-mining/sv2-apps#714, kept separate for the same reason.

Known, not in this PR

  • After abort_all, the second join_all in the three runtimes has no time limit. v0.9.2 had the same exposure, and those tasks also watch the cancellation token. Follow-up.

Tests

Local only.

  • Every commit: fmt (3 generated bitcoin-core-sv2 files differ on bare v0.9.2 under nightly fmt and are left alone), the per-crate clippy from ci.yaml, cargo test --workspace --exclude integration_tests_sv2, a --locked release build of translator_sv2 and pool_sv2, and cargo audit.
  • docker build -f docker/Dockerfile --build-arg APP=translator_sv2 . succeeds.
  • Integration tests: they ran in parallel with other builds. The fund_wallet()/bitcoind RPC timeouts there reproduce on bare v0.9.2 under load and pass alone. A serialized full IT run on the integrated branch will be posted here.
  • Reviews: an Opus adversarial review (changes required, then fixed and approved) and a Fable final review (approve).

Merge method

Please rebase-merge (or fast-forward), not squash, so the upstream authorship and the (cherry picked from …) lines survive.

average-gary and others added 8 commits November 12, 2026 13:12
The fork publishes container images to ECR from this branch, so the
workflows follow it instead of upstream's main, and the image build
does what the fork's pipeline needed on 0.8.

- Every push/pull_request trigger and the auto-rebase script target
  mara-0.9 rather than main.
- docker-release builds docker/Dockerfile once per app and pushes
  pool, jdc and tproxy to ECR on this branch and on the fork's
  vX.Y.Z-mara.N tags. Version tags publish only the version tag: the
  ECR repositories are tag-immutable, so a floating :latest can never
  be moved. Branch pushes publish the commit SHA. The builds are
  amd64 only, and cargo's registry, git and target dirs persist
  across runs through cache mounts in docker/Dockerfile.
- Upstream's version-bump-check job is dropped. It requires every
  touched crate to exceed its crates.io version, which an upstream
  release merge never satisfies and a container-image fork has no use
  for.
- The images are built from upstream's docker/Dockerfile, so there is
  no bitcoind image and no envsubst config templates; the apps are
  configured through POOL__*-style environment variables. pool-v4
  builds its own images.
- stratum-core-sync, the two notify-* workflows and release-apps act
  on upstream (a lockfile PR for upstream's stratum pin, issues in
  stratum-mining/sv2-ui, crates.io publishing), so they run only on
  stratum-mining/sv2-apps.
- auto-rebase is guarded the same way: elsewhere it would force-push
  every labelled PR onto mara-0.9, whatever its base.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
utoipa-swagger-ui 9.0.2 unpacks the Swagger UI archive in its build
script with zip 3.0.0, a release covered by published security
advisories whose fixes only exist in zip 9. utoipa-swagger-ui 10.0.2
builds with zip 9 and requires utoipa 6.

utoipa 6 lists the null variant of a nullable reference last in its
`oneOf`, so openapi.json is regenerated. The schema is otherwise
unchanged.

(cherry picked from commit 2973e3c)
Nothing exercised `/swagger-ui` or `/api-docs/openapi.json`, so a
`utoipa-swagger-ui` bump that broke how the Swagger UI assets are
embedded would have passed every test.

The test router now mounts Swagger UI the way `MonitoringServer::run`
does, and three tests request the page with two of its embedded
assets, the initializer script that carries the spec route, and the
spec itself.

(cherry picked from commit 07f81b9)
cargo audit reported 23 vulnerabilities at v0.9.2; utoipa-swagger-ui 10
(zip 9) takes it to 19, and this commit clears the remaining 19.

- rustls 0.23.40 -> 0.23.46 (RUSTSEC-2026-0285, -0346, -0348, -0349)
- rustls-webpki 0.103.13 -> 0.103.16 (RUSTSEC-2026-0382)
- webpki-root-certs 1.0.7 -> 1.0.12 (RUSTSEC-2026-0383)
- quinn-proto 0.11.14 -> 0.11.19 (RUSTSEC-2026-0185, -0338, -0339,
  -0340, -0342)
- tokio 1.52.3 -> 1.53.3 (RUSTSEC-2026-0386), the minimum that
  patches it, which avoids the runtime changes in 1.54
- anyhow 1.0.102 -> 1.0.104 (RUSTSEC-2026-0190)
- event-listener 5.4.1 -> 5.4.2 (RUSTSEC-2026-0221)
- rustix 1.1.4 -> 1.1.6 (RUSTSEC-2026-0486)
- chacha20 0.10.0 -> 0.10.2, which was yanked.
- stratum-apps no longer enables prometheus's default `protobuf`
  feature (RUSTSEC-2024-0437). Metrics are only rendered as text, so
  protobuf is dropped from the lockfile.
- integration-tests moves to minreq 3.0.0. minreq 2.14.1 pulled in
  rustls 0.21, rustls-webpki 0.101 and webpki-roots 0.25
  (RUSTSEC-2026-0098, -0099, -0104, -0346, -0349, -0381 and -0382),
  whereas 3.0.0 uses rustls 0.23. minreq 2.14.1 stays in the lockfile
  for jsonrpc, which enables no TLS feature. minreq 3 reports the
  status code as u16.

cargo audit now reports no vulnerabilities, only the unmaintained
async-std, instant and pin-utils and the yanked async-executor, all of
them reached through criterion in integration-tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`TaskManager::spawn` pruned its `Vec<JoinHandle<()>>` with
`retain(|h| !h.is_finished())` on every spawn, a linear scan of every live
handle per spawn, taken under the tracking mutex. Callers spawn several tasks
per downstream connection, so reaching N concurrent connections cost O(N^2)
and serialized that scan on one lock. Under load the connection-accept rate
plateaus at low total CPU, because a single core saturates on the scan while
the rest idle.

Track tasks in a `tokio::task::JoinSet` instead. Insertion is constant time.
A `JoinSet` releases an entry only when that entry is joined, and `join_all`
runs only at shutdown, so `spawn` also drains already-finished tasks with the
non-blocking `try_join_next`. That drain touches only tasks the runtime has
already marked complete, costing O(tasks finished since the previous spawn) --
amortized constant per task -- and keeps the set bounded to roughly the live
task count. Without the drain the set would retain one task control-block per
task ever spawned, growing with cumulative rather than concurrent tasks.

Measured on a distributed scale test with the pool isolated on its own host,
this raised the connection ceiling from 34,543 to 61,259 while pool CPU went
from 25.3% to 99.6% -- the plateau-at-low-CPU signature disappears.

Two behavioural notes, neither of which any caller in this repo relies on:
`join_all` now awaits in completion order rather than reverse-spawn order, and
`abort_all` leaves aborted entries in the set until they are joined, so callers
wanting them reaped should follow it with `join_all`. The tracking mutex is
retained; what this removes is the per-spawn scan, not the lock.

Adds three tests, since `TaskManager` had none: `join_all` drains every
tracked entry, `spawn` releases finished entries so the set stays bounded
under churn, and a panicking task is still reaped.

(cherry picked from commit 1a4eff9)
Both the previous `Vec<JoinHandle<()>>` implementation and the `JoinSet` that
replaced it discard join results, so a managed task that panicked disappeared
without a trace: the entry was released and nothing was logged. Operators had
no signal that a task died, only its absent effects.

Log a warning for join errors that are not cancellations. Cancellations are the
expected outcome of `abort_all` during shutdown, so they stay quiet.

In `spawn` the errors are collected under the tracking lock and reported after
it is released. Reporting inline would put subscriber work on the contended
path, and would deadlock if a subscriber ever spawned through this manager,
because the lock is not reentrant. `Vec::new` does not allocate, so the common
case of no failures costs nothing beyond a discriminant check per drained task.
`join_all` already owns its set outside the lock, so it reports inline.

This is separate from the preceding commit because the scale measurement quoted
there predates it: that run exercised the tracking change alone.

(cherry picked from commit a5b6459)
The `# Tracking cost` section restated the PR description almost verbatim — the
`Vec` + `retain` critique, the O(N²), the `try_join_next` amortization and the
retained-mutex note are all already in the PR, which additionally has the
benchmark table the comment did not.

That argument is read once, at review. The rustdoc is read every time someone
opens the file. What a maintainer needs there is the invariant: tasks live in a
`JoinSet`, and `spawn` drains finished entries because `join_all` only runs at
shutdown.

Also dropped the note that `JoinSet::spawn` behaves as the prior `tokio::spawn`
did — migration equivalence stops being useful once the migration lands — and
the `Vec::new` non-allocation aside.

Kept both hazards, which are invisible from the code: the lock is not reentrant,
so reporting under it would deadlock on a subscriber that spawns through this
manager; and a `std::sync::Mutex` guard must not cross an `.await`.

34 comment lines removed, no code changed. `cargo fmt --check` and `cargo check`
clean; the 9 `uninlined_format_args` clippy errors in `coinbase_output_constraints.rs`
and `key_utils/mod.rs` are pre-existing and reproduce with this commit reverted.

(cherry picked from commit c217a90)
join_all moved the whole JoinSet into a local before draining it. When
a caller's shutdown timeout cancelled join_all, that local was dropped,
which aborts the tasks without reaping them. The follow-up abort_all
and join_all in the runtimes then acted on an empty set, so a late
panic was never logged.

join_all now joins entries in place by polling the set under the lock,
taking the lock per poll and never across an await. A cancelled
join_all leaves the unfinished tasks tracked, and the next abort_all
and join_all reap them and report abnormal exits.

Side effect: tasks spawned while join_all is waiting are awaited too,
where before they landed in a fresh set. spawn is only briefly
contended by a poll of the set.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@average-gary

Copy link
Copy Markdown
Collaborator Author

Full integration-test run, on the top of the stack (#36, 8f8849f1), 4 threads with 1 retry: 156 of 186 passed. Of the 30 failures, 27 pass when re-run one at a time; all of them failed in fund_wallet() while several regtest bitcoinds were starting at once. The other 3 (bitcoin_core_ipc_block_weight v30x/v31x/v32x) time out the same way on bare v0.9.2 (origin/mara-0.9). None of the failures is caused by this stack. Logs are kept locally; ask if you want them.

@based64god
based64god merged commit 2ce137b into mara-0.9 Nov 16, 2026

@gimballock gimballock left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial pass at 17692585. The code is fine: the TaskManager cherry-pick is byte-identical to upstream stratum-mining#714's head, and the cancellation fix is correct and mutation-pinned. Every writer workflow is gated to stratum-mining/sv2-apps. Swagger UI builds from the checksum-pinned vendored zip with no network. Two must-fixes, neither in the Rust.

Must-fix

  1. GitHub Actions hasn't run on this repository since 2026-10-06 00:42Z (the v0.8.0-mara.7 docker build).
    • Every workflow shows as active, yet the v0.8.0-mara.8, .9 and .10 tags never started docker-release.
    • validate-openapi.yaml, which has no branch filter, didn't run on this PR, which touches stratum-apps/src/monitoring/**.
    • So none of this PR's CI or docker changes has ever executed, and the 0.9 stack has 0 checks. That's more than the trigger filter.
    • actions/permissions returns 403 for us, so someone with admin access needs to check: Actions disabled, billing, or an org policy.
  2. cargo audit reports 1 vulnerability: RUSTSEC-2026-0538 in quinn-udp 0.5.14, published 11-13, a day after 128efd6.
    • pool-v4-infra's sv2-upstream-rust-gate audits this ref, so it goes red there, against the PR body's "0".
    • cargo update -p quinn-udp --precise 0.5.16 resolves cleanly.

Should-fix

  • The image push uses static AWS keys, not OIDC (docker-release.yaml:40-45), the same as on 0.8. The tag trigger has no branch restriction, and workflow_dispatch runs on any ref. So anyone with write access can push v0.9.2-mara.N on an unmerged commit and burn an immutable ECR tag. An OIDC role trusting only refs/heads/mara-0.9 and refs/tags/v*-mara.*, plus a tag ruleset, closes it.
  • Stacked PRs get no CI: pull_request.branches: [mara-0.9] in ci, tests, lockfiles, msrv, integration-tests and semver-check. Adding 'm09/**' fixes that once Actions runs at all.
  • "pool-v4 isn't affected" is incomplete: pool-v4-infra's scratch and testnet4 lanes pin these same pool/tproxy repos by digest (apps-scratch/pool-scratch.yaml:239, apps-testnet4/pool-testnet4.yaml:184, tproxy-testnet4.yaml:193). Their configs rely on the 0.8 image's envsubst template. A 0.9 image (POOL__* env only) repinned there would start on defaults. Worth a line in RB-09 or the PR.

Nits: the abnormal-exit warn isn't tested; tokio is pinned only in the lockfile (Cargo.toml still says 1.44.1); the Swagger tests copy the production router merge rather than test it.

Run: stratum-apps --all-features 153/153, and the CI clippy is clean. Reverting join_all to mem::take + join_next is caught by cancelled_join_all_leaves_tasks_reachable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants