Repository navigation
ci: trim CI artifacts and caches - #7942
Conversation
The guix output directory holds .tar.gz, .zip and .dmg files, so the default zip compression of upload-artifact only burns CPU.
Entries were written at ccache's default level 1 and then every build, including pull request builds that never save the cache, ran ccache -X 9 to recompress them. Writing at level 9 from the start removes that pass and also shrinks what gets uploaded to the remote ccache, which only ever saw the level 1 bytes. On 1 GB of object files level 9 is 7% smaller than level 1, and level 5 gains nothing over level 1.
Test logs are highly repetitive text, and on a 160 MB sample level 9 is 13% smaller than level 5 while taking 0.3 s instead of 0.2 s. The highest Level 19 would be 32% smaller but more than a hundred times slower, which does not scale to the multi-gigabyte sanitizer logs. Level 9 seems as a good balance of CPU / storage
slim-workspace.sh removes test_dash, bench_dash, fuzz, test_dash-qt and the generated qrc_*.cpp right after extracting the build bundle, so the functional test job pays to download them and thte build job pays to compress and upload them for nothing. In a local build these files are about two thirds of the binaries in the bundle. The unit tests run before bundling, so leaving them out of the archive changes nothing else. dash-qt and dash-tx stay in the bundle: the win64 and mac bundles have no test job and exist so people can download the product binaries.
The mac depends and build jobs fetched the SDK at job time: a cache lookup in check-cache, a download from bitcoincore.org on a miss, an artifact handoff for runs whose cache token is read-only, and a restore-or-download-or-fail step triplet in each consuming job. That is nine workflow steps and a workflow output/input pair for a 200 MB input that only changes when depends/hosts/darwin.mk bumps the Xcode version, and it took a slot in the 10 GB repository cache that the ccache generations already fill. Download, verify and extract the SDK while building ci.Dockerfile instead, the way bitcoin#27028 moved the SDK into its CI images, and hand its location to depends through SDK_PATH, since depends/Makefile defaults that to depends/SDKs inside the checkout. The depends and ccache cache keys already hash ci.Dockerfile, so an SDK bump there invalidates them as before. contrib/containers/guix/scripts/setup-sdk stays for the Guix container.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Guix Automation has started building this PR as |
|
✅ Final review complete — no blockers (commit 289c7f8) · triage: normal |
|
No actionable comments were generated in the recent review.
|
x
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 1 + Phase 2
The artifact-trimming changes correctly remove the listed test binaries for Unix-like targets, but the exclusions do not cover the .exe names produced by the Windows build. Consequently, the win64 bundle still contains the test executables that the PR claims to remove. The CI snapshot also includes a failed aarch64 build, but no failure details are available to establish a separate issue.
🟡 1 suggestion(s)
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: dash-core-commit-history); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — This is a cross-cutting CI and build-infrastructure change affecting artifact contents, compression settings, caching behavior, and the macOS SDK provisioning workflow, but it does not modify a critical code surface. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— dash-core-commit-history (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `ci/dash/bundle-artifacts.sh`:
- [SUGGESTION] ci/dash/bundle-artifacts.sh:45-52: Exclude Windows .exe test binaries from bundles
The exclusion entries use extensionless names, while Automake appends `$(EXEEXT)` for the Windows target. The win64 build therefore produces `src/bench/bench_dash.exe`, `src/qt/test/test_dash-qt.exe`, `src/test/fuzz/fuzz.exe`, and `src/test/test_dash.exe`. Because `build-src.yml` runs the unit tests and then bundles the build for this target, these files exist when the script runs, and the extensionless patterns do not match them. The win64 artifact consequently still contains the test binaries that this PR says are no longer bundled.
| # Test binaries and generated sources that only the unit tests, run | ||
| # before bundling, need | ||
| "src/bench/bench_dash" | ||
| "src/qt/qrc_bitcoin.cpp" | ||
| "src/qt/qrc_dash_locale.cpp" | ||
| "src/qt/test/test_dash-qt" | ||
| "src/test/fuzz/fuzz" | ||
| "src/test/test_dash" |
There was a problem hiding this comment.
🟡 Suggestion: Exclude Windows .exe test binaries from bundles
The exclusion entries use extensionless names, while Automake appends $(EXEEXT) for the Windows target. The win64 build therefore produces src/bench/bench_dash.exe, src/qt/test/test_dash-qt.exe, src/test/fuzz/fuzz.exe, and src/test/test_dash.exe. Because build-src.yml runs the unit tests and then bundles the build for this target, these files exist when the script runs, and the extensionless patterns do not match them. The win64 artifact consequently still contains the test binaries that this PR says are no longer bundled.
| # Test binaries and generated sources that only the unit tests, run | |
| # before bundling, need | |
| "src/bench/bench_dash" | |
| "src/qt/qrc_bitcoin.cpp" | |
| "src/qt/qrc_dash_locale.cpp" | |
| "src/qt/test/test_dash-qt" | |
| "src/test/fuzz/fuzz" | |
| "src/test/test_dash" | |
| # Test binaries and generated sources that only the unit tests, run | |
| # before bundling, need | |
| "src/bench/bench_dash" | |
| "src/bench/bench_dash.exe" | |
| "src/qt/qrc_bitcoin.cpp" | |
| "src/qt/qrc_dash_locale.cpp" | |
| "src/qt/test/test_dash-qt" | |
| "src/qt/test/test_dash-qt.exe" | |
| "src/test/fuzz/fuzz" | |
| "src/test/fuzz/fuzz.exe" | |
| "src/test/test_dash" | |
| "src/test/test_dash.exe" |
source: muse-spark-1.3-contributor (phase1-reviewer: general); gpt-6.1-sol (phase2-reviewer: general, dash-core-commit-history)
|
Guix Automation build succeeded: https://github.com/dashpay/dash-dev-branches/releases/tag/v24.1.0-devpr7942.289c7f8a |
…the hosted queue backs up 29cbd9d ci: run linux64 build and functional tests on the fast path runner (pasta) 50f1b75 ci: pick a self-hosted runner for a linux64 fast path (pasta) Pull request description: ## Issue being fixed or feature implemented Most of the time a Core PR spends in CI goes to waiting for a GitHub-hosted runner, not to building or testing. Over the last week of `build.yml` runs, the linux64 critical path looked like this: | job | queue p50 | queue p90 | run p50 | |---|---|---|---| | `x86_64-pc-linux-gnu / Build depends` (ran in 14 of 24 runs) | 5.5 min | 84 min | 17 min | | `linux64-build` | 7.4 min | 43 min | 6 min | | `linux64-test` | 2.7 min | 28 min | 16 min | On average that path spends about 41 min queued against about 32 min of work. The pre-lint jobs already move to our own hardware when the hosted queue backs up (#7901, #7902, #7942). This PR does the same for the linux64 build and its functional tests. ## What was done? - `select_dynamic_runner.py` gains a `runner_fast` output. When the repository variable `RUNNER_SELFHOSTED_FAST` names a label, it picks that label using the pre-lint rung's rules: - some hosted job is queued; - no `blacksmith-ci` label overrides the choice; - the label isn't saturated: at most 2 jobs queued for it (`SELFHOSTED_FAST_MAX_QUEUED`). These jobs hold a runner for minutes rather than seconds, so the cap is lower than pre-lint's 6. Otherwise it falls back to the amd64 runner. Trust is the existing `SELFHOSTED_LINT_AUTHORS` allowlist, plus every push to `develop` or `master`: those carry merged code and already require write access. Fork PRs from anyone else stay on the hosted ladder. - The pre-lint selection now takes a precomputed "allowed" flag, so both rungs share one function. Its decisions and reason strings don't change. - `build.yml` sends the linux64 depends build job (cache misses only), `linux64-build` and `linux64-test` to `runner-fast`. They replace their hosted runs rather than run alongside them, so required check names stay the same and no hosted capacity is used twice. - `build-depends.yml` gains `runs-on-build` for the build job only. `runs-on` still forms the depends cache key, so a fast-path build saves the same entry the hosted asan, fuzz and sqlite builds restore. - `ci/README.md` documents the gate, the fallback and what a runner for the label must provide. Nothing changes until `RUNNER_SELFHOSTED_FAST` is set; unsetting it turns the path off again. Runner setup (a dedicated single-use, Docker-capable label) is handled outside this repository. `linux64-test` is the test job that moves. It consumes the same `linux64` bundle, so the fast path needs no extra build, and it is the broadest functional run (previous releases, `--extended`). A trimmed subset would have meant a new job and new required-check names. ## How Has This Been Tested? - `python3 .github/workflows/test_select_dynamic_runner.py`: 44 tests pass, 9 of them new. They cover the variable being unset, trusted and untrusted PRs (author, actor and cross-fork head), develop and master pushes by a non-allowlisted actor, other pushes and tags, no backlog, a measurement error, the saturation cap at 2 and 3 queued jobs, the Blacksmith label and amd64 inheritance, and fast-label jobs not counting as hosted backlog. The existing pre-lint tests pass unchanged. - `actionlint` 1.7.12 reports nothing new against `develop`. `test/lint/lint-github-actions.py` (zizmor), `lint-whitespace.py` and `lint-files.py` are clean. - `pull_request_target` runs the base branch's workflows, so this PR's own CI does not exercise the change. A push of this branch to a fork ran it with the variable unset: `Check skip conditions` succeeded, and the linux64 depends build and `linux64-build` resolved through `runner-fast` to `ubuntu-24.04`, the same runner as before. The self-hosted path itself can only be tested once `RUNNER_SELFHOSTED_FAST` is set. ## Breaking Changes None. Behaviour is unchanged unless `RUNNER_SELFHOSTED_FAST` is set. ## Checklist: - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Top commit has no ACKs. Tree-SHA512: e71bcf9fb1bb4febe33af9e07478949d225d865a39a47e69a79d2302e92e31138990edeb33571d04cbb807873f3555e092840dc7ae327cd83cb30bf2e30225c8
…image) 7312925 ci: bake the macOS SDK into the CI runner image (pasta) Pull request description: ## Issue being fixed or feature implemented Pull requests into `v24.0.x` run develop's workflow files (`pull_request_target`). Since #7942 (289c7f8, "ci: bake the macOS SDK into the CI runner image"), those workflows no longer fetch the macOS SDK at job time. They expect it in the CI image built from the PR's own `contrib/containers/ci/ci.Dockerfile`, found through `SDK_PATH`. `v24.0.x`'s Dockerfile does not do this, so on every depends cache miss `x86_64-apple-darwin / Build depends` fails with `configure: error: could not find a working compiler`: `depends/SDKs` is empty. This affects every PR into `v24.0.x` (seen on #7967), independent of what the PR changes. ## What was done? Partial backport of #7942: only the `contrib/containers/ci/ci.Dockerfile` hunk of 289c7f8, picked with `-x`. It is identical to develop's. The image downloads the Xcode 26.1.1 (17B100) SDK from bitcoincore.org, verifies its SHA256 and extracts it to `/opt/SDKs`, with `ENV SDK_PATH=/opt/SDKs`. Left out: - 289c7f8's workflow hunks (`build.yml`, `build-depends.yml`, `build-src.yml`), so this branch's own push workflows keep downloading the SDK as before. With `SDK_PATH` set in the image, depends uses the baked copy either way. - The other four #7942 commits (artifact and cache trimming), which are not needed here. The Xcode version matches this branch's `depends/hosts/darwin.mk` (`XCODE_VERSION=26.1.1`, `XCODE_BUILD_ID=17B100`), and `depends/Makefile` here already has `SDK_PATH ?= $(BASEDIR)/SDKs`. ## How Has This Been Tested? - Downloaded `Xcode-26.1.1-17B100-extracted-SDK-with-libcxx-headers.tar` and checked it against the SHA256 in the Dockerfile: OK. - The CI image itself is built by this PR's CI (`Build container`), and the mac depends job is the test. ## Breaking Changes None. CI only. ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have assigned this pull request to a milestone 🤖 Generated with [Claude Code](https://claude.com/claude-code) Top commit has no ACKs. Tree-SHA512: 47adbaef6388bf076e66b286c28e4de2ad8870334d6cb29d8a66fd6bf52b15330d9c2f595f27d36cfa8ec5519209b026a8632ca24e7453e0e115984137778799
Issue being fixed or feature implemented
A develop run uploads about 4.1 GB of artifacts and the repository cache sits at 7.9 GB of its 10 GB limit (run https://github.com/dashpay/dash/actions/runs/39637558287). Much of that is data nobody reads:
ccache -X 9. The remote ccache only ever receives the level 1 bytes.What was done?
One commit per item:
ci: bake the macOS SDK into the CI runner image: download, verify and extract the SDK while building ci.Dockerfile, the way ci: Cache more stuff in the ci images: msan, iwyu, pip, sdks bitcoin/bitcoin#27028 does, and point depends at it through SDK_PATH. Removes the cache lookup, download, artifact handoff and restore steps from build-depends.yml and build-src.yml.ci: stop bundling the test binaries that the test job deletes on arrival: bundle-artifacts.sh excludes the test binaries and generated qrc sources, and slim-workspace.sh no longer lists them. dash-qt and dash-tx stay in the bundle because the win64 and mac bundles have no test job and exist for people to download product binaries.ci: compress functional test logs with zstd level 9: on a 160 MB log sample level 9 is 13% smaller than level 5 at 0.3 s instead of 0.2 s. Level 19 would be 32% smaller but over a hundred times slower, which does not scale to the multi-gigabyte sanitizer logs.ci: write ccache entries at zstd level 9 instead of recompressing later: CCACHE_COMPRESSLEVEL=9 in ci/test/00_setup_env.sh and theccache -X 9pass dropped from build-src.yml. On 1 GB of object files level 9 is 7% smaller than level 1; level 5 gains nothing over level 1.ci: upload guix outputs without recompressing them: compression-level 0 on the guix artifact upload.How Has This Been Tested?
Note that pull_request_target runs the workflow YAML from develop, the removal of the SDK workflow steps, the removal of
ccache -X 9from build-src.yml had been tested by PR in the fork: knst#11CI run of this PR: https://github.com/dashpay/dash/actions/runs/40204419499 (all jobs green), compared against the develop run https://github.com/dashpay/dash/actions/runs/39637558287.
This PR reduces artifact volume by 38 percent and simplifies the mac SDK handling; it does not change billed minutes.
Artifact sizes, develop vs this PR:
Step timings, baseline vs the two warm runs
SDK baked into the image: the image sets SDK_PATH and depends/Makefile only defaults it when unset. The mac job went from 5.3 min to 4.1 and 3.6 min; most of that is a 44 s ccache
restore in the baseline against 1-2 s here.
Breaking Changes
None for users. The build-* artifacts no longer contain test_dash, bench_dash, fuzz or test_dash-qt; anyone who downloaded those from CI must build them locally.
Checklist: