Skip to content

ci: keep libobs module cache keys stable across PCH builds - #5176

Merged
zackees merged 2 commits into
masterfrom
infra/tier2-module-pch-keys
Nov 19, 2026
Merged

zackees merged 2 commits into
masterfrom
infra/tier2-module-pch-keys

Conversation

@zackees

@zackees zackees commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The libobs module cache's GCC dependency scan changes its key after a PCH is built. Hosted Tier 2 enables -fpch-preprocess for its ccache front layer; this lets -M -MG load the PCH and omit its header closure. Master then saves keys that a fresh hosted tree cannot reproduce, and the Dell additionally differs because its empty launcher does not add this flag. Omit this preprocessing-output flag from module key/probe arguments so both paths hash the full textual headers. Compilation is unchanged.

E640 check 16 (.worker-pool/e640-measure/report-16.md, 2026-11-18) reported a median 372 s Build Tier 2 step and 0/7 modules restored. Six inspected runs show that archive transport and master saves work: master jobs 135187090875 and 135200900974 restored 4.8 MB master caches, reported 0/7 hits and saved new master caches. Dell jobs 135176110873 and 135179638652 likewise restored master caches and restored 0/7. The first master job's artifact contains -fpch-preprocess in all 26 module group commands. Save conditions, restore prefixes and GitHub ref scoping need no change. The separate draft configure optimization changes kept-tree configuration, not this key computation.

The saving is limited by what actually compiles. In master job 135187090875's .ninja_log, the six C modules' 38 PCH/object/archive edges span 6.393 s and sum to 25.557 s with the compiler cache warm. Archive hits remove that C compilation (approximately 0 s module compilation), suggesting a seconds-scale saving in that observed hosted case; unrelated Rust rebuilds and the whole build step remain. The Dell cold restore-to-store regions were 171.1 s and 167.1 s, including reconfigure and Rust/tests, so they are only upper bounds, not estimates of C-module savings. No post-change CI or matched Dell warm/cold speedup is claimed. A master build must first seed the corrected keys. Genuine source/toolchain changes still invalidate modules.

Local RED commit dfe9dfac2: uv run --no-project python -m unittest ci.test_libobs_modules_cache.KeyTest ran six tests with three expected failures: PCH creation changed the key, a built PCH hid a header edit, and the preprocessing flag changed the key. The published GREEN head d1db0e6b933bc77236ee2695a89eb6d8aba1449d passes the same regressions (the original local GREEN commit was amended only by publication attestations and signing). RED and GREEN are pushed together under the focused CI exception.

Validation: 43 focused CI self-tests pass; the single read-only clud-review found no issues. timeout 1500 env SOLDR_PORCELAIN=0 ./lint passed in 265.9 s (its self-tests passed and soldr replayed matching Rust analysis verdicts). An additional 20 module-cache/workflow tests passed in the existing Ubuntu validation image. ./push completed six-target Clippy/Dylint, full client self-tests/data checks and the host proof (9,098 workspace tests passed with six skips; another 920 feature tests passed; client-test lane completed in 441.2 s). The published head is signed; Swift returned its expected not-applicable exit 75 on Linux. These trust-surface edits still require ordinary PR CI. This is a Python CI helper change; existing Rust/C/protocol test tiers are unchanged, and no runtime behavior, workflows, public headers, C tests, WebSocket code, Dell host configuration or repo variables are changed.

Refs #5165

Ci-Attestation: {"at":1795048655,"gate":"data/all/compatibility","host":"linux-x86_64","key":"","lane":"client-checks","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":258,"stamp":"3d39937beb764431b16f2d8873b498f9","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048655,"gate":"data/all/flatpak-manifest","host":"linux-x86_64","key":"","lane":"client-checks","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":258,"stamp":"fe182a3935492b5fed75e1803705f1ff","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048655,"gate":"data/all/qt-xml","host":"linux-x86_64","key":"","lane":"client-checks","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":258,"stamp":"c118d0e18be122da99a7fa5eef4c74f5","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048655,"gate":"data/all/services-schema","host":"linux-x86_64","key":"","lane":"client-checks","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":258,"stamp":"61239f9c9d845a02cc6f10ff1c3d12d9","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048394,"gate":"general/all/client-lint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"fded6c9c04db2938b7b0fcf68a89596c","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048655,"gate":"python/all/self-test","host":"linux-x86_64","key":"","lane":"client-checks","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":258,"stamp":"16d16509c4da0e318341c2710cb5f881","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048396,"gate":"rust/aarch64-apple-darwin/client-clippy","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"162e91c018d04d92e9567832752aac8c","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048396,"gate":"rust/aarch64-apple-darwin/client-dylint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"89d71be8b70688da7d1c46673ff644ef","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/aarch64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"94f401528848eacdb3efc493a5e530e0","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/aarch64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"7ee8c8e6e29d78a4e58cb95e47b17246","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/aarch64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"10f0da03cd5c0bd270d4645685947013","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/aarch64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"e8b16debc7a8f557103abc17ebb48afd","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/x86_64-apple-darwin/client-clippy","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"1468d27790717abfd4fb6063a5f8d8e7","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/x86_64-apple-darwin/client-dylint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"7c95216187c868f2ffd73c556565909e","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/x86_64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"350a382996ac898e17d522df7054d63a","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/x86_64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"15071aa363f83bb4d78394bc5f041c12","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/x86_64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"150c59e0baf410f522850319c65f2ae9","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795048395,"gate":"rust/x86_64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"7dadac353cc1a5aa2ebe2eb97be131e006cd54be2299aa1055d35e6ef4233725","lane":"client-lint","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":22,"stamp":"1344f99c2119c3c03fb97643f17cdf39","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}
Ci-Attestation: {"at":1795049097,"gate":"rust/x86_64-unknown-linux-gnu/client-test","host":"linux-x86_64","key":"cd7f345db38e1a95831552b5879e0760802b67a363852f8068cc5ef1b34b7294","lane":"client-test-x86_64-unknown-linux-gnu","parents":["dfe9dfac225facbeb8ae2d219f5ba350b33bbba5"],"secs":441,"stamp":"62726ccda774cca9258d02e5a72e424f","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144","v":1,"via":"run"}

Client-Checks: {"checks":{"base":"a62bfe717d5d9610a5136ca211273e66a1ae1f84","coverage":"all-tracked-applicable-files","diff":["ci/libobs_modules.py","ci/test_libobs_modules_cache.py"],"head":"95dea74c77b381786fb672a0ee0a49fb4f053e37","tools":{"orchestrator-python":"3.13.15 (main, Aug 14 2026, 15:34:59) [Clang 22.1.3 ]","schema-dependencies":"3.13.15 (main, Aug 14 2026, 15:34:59) [Clang 22.1.3 ]\n{'jsonschema': '4.26.0', 'json_source_map': '1.0.5'}","uv":"uv 0.13.5 (x86_64-unknown-linux-gnu)","xmllint":"xmllint: using libxml version 21504\n   compiled with: Threads Tree Output Push Reader Patterns Writer SAXv1 DTDValid HTML C14N Catalog XPath XPointer XInclude Iconv ISO8859X Regexps Automata RelaxNG Schemas Modules Debug"},"tracked-digest":"25bca37e28e14ac712462d671ac47ce85ca6f99de2c0b0b06e79a4915c7b86f3","tree":"33d3da3f6ccc8b8296871d492ea91c11bc42a144"}}
@zackees zackees added the pool-active Pool worker owns this PR; automated rebases must skip it label Nov 19, 2026
@zackees zackees changed the title ci: scan PCH headers consistently for module cache keys (Refs #5165) ci: keep libobs module cache keys stable across PCH builds Nov 19, 2026
@zackees
zackees enabled auto-merge November 19, 2026 00:46
@zackees zackees removed the pool-active Pool worker owns this PR; automated rebases must skip it label Nov 19, 2026
@zackees
zackees added this pull request to the merge queue Nov 19, 2026
Merged via the queue into master with commit d0b9810 Nov 19, 2026
44 checks passed
@zackees
zackees deleted the infra/tier2-module-pch-keys branch November 19, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant