Skip to content

ci: build attestation-verifier test fixtures once instead of per test - #5297

Open
zackees wants to merge 2 commits into
masterfrom
infra/4896-attest-test-speed
Open

zackees wants to merge 2 commits into
masterfrom
infra/4896-attest-test-speed

Conversation

@zackees

@zackees zackees commented Nov 19, 2026

Copy link
Copy Markdown
Member

The admission-verifier suite launches 2,154 Git subprocesses, which can exhaust ./push's client-check budget. The expanded suite now launches 191 (11.3x fewer).

Build one fixture template and give each test independent copies of its files, objects and mutable Python state. Write additional real Git objects in process, with object IDs checked against Git plumbing and fsck. Reuse successful immutable reads only while their object bytes and configuration match; shallow history, grafts/replacements, missing/corrupt objects and failures reach real Git. Production commit metadata is read in one Git query instead of four.

All 45 original test methods and assert_runs are AST-identical, preserving 118 static assertion/helper call sites. Five additional regressions cover cost, isolation, Git-object parity and cache invalidation.

Measurement Before: 45 tests After: 50 tests
Git subprocesses 2,154 191
Serial unittest time 186.506 s 30.539 s
Same eight-worker runner batch 522.1 s 47.7 s

The parallel baseline used frozen original sources and a measurement-only 1800 s deadline matching #5283. No repository deadline changed. In the full 399-file runner, the original suite timed out at 600 s; GREEN passed in 101.0 s, and in 50.1 s inside ./push. The original 45-test subset alone uses 173 Git calls (12.5x fewer); an additional concurrent stress run took 175.032 s. Shared-host load makes wall time variable.

RED 4ff698d: setup used 9 subprocesses instead of 0, and metadata used 4 instead of 1. GREEN 4c16a44 passes both and the additional regressions.

Validation: Ruff and clud-review clean, including a simulated Windows CRLF/autocrlf reproduction. Full ./lint ran once: verifier passed in 146.2 s, but unrelated MCP-launch (15 s) and apt-cache (5 s) timeout tests failed. Subsequent ./push passed all 399 self-test modules, all six-target clippy/Dylint checks, 9,264 main host tests (6 skipped), doctests, 920 utilities feature tests and Windows GNU test linking. It published 19 Ci-Attestation records plus Client-Checks, with a verified final signature. Optional Swift proof was unavailable because swift-format is absent; these files do not select Swift.

These ci/ trust-surface edits retain real hosted checks. Rust/C/API, differential and protocol implementations, public headers, C tests, workflows and packaging are unchanged.

Refs #4896.

RED: original 45 tests pass but launch 2,154 Git subprocesses in 186.506 s locally. Fixture isolation/no-spawn and single metadata read regressions fail before optimization. Refs #4896.
Construct isolated copies of one real-object template; write fixture objects in process and reuse successful immutable Git reads. Batch production commit metadata into one Git query. Preserve all 45 original test bodies and their assertions. Refs #4896.

Ci-Attestation: {"at":1795126715,"gate":"data/all/compatibility","host":"linux-x86_64","key":"","lane":"client-checks","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":339,"stamp":"94670f9b5e08d2fb7ddc2c28eebe58fc","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126715,"gate":"data/all/flatpak-manifest","host":"linux-x86_64","key":"","lane":"client-checks","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":339,"stamp":"65da1b88d8bc87f04bbe32752e2a3560","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126715,"gate":"data/all/qt-xml","host":"linux-x86_64","key":"","lane":"client-checks","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":339,"stamp":"64a58b4c32b3f3e62dd90a847e90f9b5","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126715,"gate":"data/all/services-schema","host":"linux-x86_64","key":"","lane":"client-checks","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":339,"stamp":"4b979e159f45825e2972b99782dc6237","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126352,"gate":"general/all/client-lint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"671957504840b0c53ee88c0f6260dc6e","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126715,"gate":"python/all/self-test","host":"linux-x86_64","key":"","lane":"client-checks","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":339,"stamp":"633fae30b2ff88de5625fd6b95a8819e","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126358,"gate":"rust/aarch64-apple-darwin/client-clippy","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"1b4b804e509b70482241f755891d5605","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126360,"gate":"rust/aarch64-apple-darwin/client-dylint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"22e0790913ba40688ce90372e144ce25","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126355,"gate":"rust/aarch64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"e01cd62afe179c6d1b04ef59f4f85b52","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126355,"gate":"rust/aarch64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"cdf09131a9ab802c0d8ba2dcaffc2674","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126353,"gate":"rust/aarch64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"186f949148b5b21bb4eb3ef8967ef63f","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126353,"gate":"rust/aarch64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"59b43d883b141f34955a79d849113157","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126357,"gate":"rust/x86_64-apple-darwin/client-clippy","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"f8fe050c0f7b0ccbd9d3eaab971f501b","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126357,"gate":"rust/x86_64-apple-darwin/client-dylint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"9fdfbb20a5ffb10b91eeea577d1e21fc","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126354,"gate":"rust/x86_64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"a436d1515f05a1be77ce4278d98c0293","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126355,"gate":"rust/x86_64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"ddb109d1e399d3a102c4607f44c575b0","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126352,"gate":"rust/x86_64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"8ebd0bdacfc2d7f916032b08247970a1","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795126352,"gate":"rust/x86_64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"c59997cca264ff0efd43cd9cbeb49a5be661f27f44b47921a7f42b64312a92bc","lane":"client-lint","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":133,"stamp":"ed9857e87c2e1df5b09edc995b5dae47","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}
Ci-Attestation: {"at":1795128338,"gate":"rust/x86_64-unknown-linux-gnu/client-test","host":"linux-x86_64","key":"a08bf3f165984e0525be4c0ff929849a7b945a6200fd7f328a99d146eb362f25","lane":"client-test-x86_64-unknown-linux-gnu","parents":["4ff698d238e05c6e19fe439e8bd7c5947a7fe7f7"],"secs":1617,"stamp":"931917ed6ba1789d4c1da2979fc6487c","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958","v":1,"via":"run"}

Client-Checks: {"checks":{"base":"fb7d018d535fabef94fc62230556ce620132f294","coverage":"all-tracked-applicable-files","diff":["ci/pr_attestation.py","ci/test_pr_attestation.py"],"head":"e968ff17d7ac3b0d89df4d0672a36fde0caee457","tools":{"orchestrator-python":"3.14.7 (main, Aug 14 2026, 15:35:28) [Clang 22.1.3 ]","schema-dependencies":"3.14.7 (main, Aug 14 2026, 15:35:28) [Clang 22.1.3 ]\n{'jsonschema': '4.26.0', 'json_source_map': '1.0.5'}","uv":"uv 0.13.5 (x86_64-unknown-linux-gnu)","xmllint":"xmllint: using libxml version 21504\n   compiled with: Threads Tree Output Push Reader Patterns Writer SAXv1 DTDValid HTML C14N Catalog XPath XPointer XInclude Iconv ISO8859X Regexps Automata RelaxNG Schemas Modules Debug"},"tracked-digest":"7a3e43b61f87b99d235bacd040858566f2d5218e6b72c05115dadbf8a39cfa33","tree":"3a8bb83e0ca02c902c1b542cfa75548e05b0d958"}}
@zackees
zackees enabled auto-merge November 19, 2026 22:48

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant