Skip to content

ci: parallelize full Tier 2 cross-link checks (Refs #5224) - #5231

Merged
zackees merged 8 commits into
masterfrom
infra/5224-linkcheck-split
Nov 19, 2026
Merged

zackees merged 8 commits into
masterfrom
infra/5224-linkcheck-split

Conversation

@zackees

@zackees zackees commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The Windows link-check job became the merge-group critical path after #5195 serialized two full Tier 2 cross-link passes behind the Rust category links. Split the categories and Windows MSVC/Darwin arm64 fixture passes into independent jobs. Each alias still links its complete CMake-discovered *_rust inventory and hybrid libobs, and CI Gate explicitly requires both matrix legs even on attested PRs. Fail-fast is disabled.

A fresh 512M checksum-pinned ccache shares identical support compiles within each alias. The verified launcher has a stable per-build path, while cache objects/configuration are ephemeral, remote/sloppy inherited policy is removed, and final links are unchanged. Pinned CMake 3.31.10/Ninja 1.13.0 uv wheels replace the apt step that cost 165 s hosted. Existing runner selection, SDKs, registrations, support fragments, public headers and runtime tests retain their behavior.

Refs #5224.

Baseline job-log measurements, seconds. Hosted run 44495240642 / job 135460997206; Dell run 44500429449 / job 135478767653.

Runner Total Setup Rust categories Windows prep/config Windows native build/links Darwin prep/config Darwin native build/links Teardown
hosted 1234.0 194.0 264.2 81.4 303.1 99.5 285.5 6.3
dell 1021.0 40.3 160.8 165.0 247.2 163.9 233.4 10.4

Setup includes native tools: hosted 165 s, Dell 5 s. Rust category timings include target setup and SDK provisioning. Each native phase builds hybrid libobs once and compiles/links all 55 *_rust fixtures, with four Ninja workers already in use. Final fixture-link log emissions span hosted Windows 0.89 s, Darwin 2.87 s; Dell Windows 1.84 s, Darwin 3.03 s. Ninja console buffering means those spans are not isolated linker wall-time measurements. Most native time is support-file compilation, not serial final linking.

CMake configure/generate reports hosted Windows 56.8/1.7 s, Darwin 63.6/1.7 s; Dell Windows 142.8/3.2 s, Darwin 144.4/3.2 s. The prep/config columns also include the explicit source-fresh Rust archive build, SDK/dependency resolution and target setup.

Hybrid and fixture compilation overlap in the original single Ninja invocation, so their wall times are not additive. Darwin hybrid dylib completion is logged 28.02 s into the hosted native phase and 16.15 s into Dell's, followed by 257.52/217.26 s of remaining fixture work. Windows logs do not expose a separate obs.dll completion timestamp; its object/Rust prefix finishes within the first 28.03 s hosted / 17.15 s Dell, while fixture work continues for about 275.04/230.08 s. These prefix timings are bounds, not independent libobs timing measurements. Raw and ANSI-stripped timestamped logs and exact per-alias fixture inventories are retained beside this file.

No hybrid build is duplicated between categories and cmocka: categories are standalone Rust binaries. The two alias hybrid libraries cannot be shared across PE x64/Mach-O arm64. Each alias's existing CMake graph shares its hybrid library with all its fixtures. The Rust release archive is built both explicitly for hybrid freshness and in Corrosion's own tree for fixtures; that existing correctness behavior is retained.

Local after measurements (fresh build trees, four workers on the shared Linux host; SDK/dependency and Rust cache warmth differ from hosted):

Pass Rust archive/target prep CMake configure/generate Native build and all links Whole checker
Windows MSVC 10.47 s 128.47 s 199.30 s 346.46 s

Windows links exactly the baseline 55 fixture names plus hybrid libobs. All 56 artifacts have verified PE x64 headers. Cache: 1,296/2,322 hits (55.81%), 1,026 misses, three uncacheable calls. A controlled precursor with CMake 4.3.4 linked the same artifacts but all 2,325 calls were uncacheable; its native phase took 248.54 s. CMake 4.3's -clang:-M* flags are unsupported by the pinned ccache. CMake 3.31.10's /showIncludes fixes that without weakening dependency tracking. The native phase improved 19.8%; overall precursor/after wall times are not an unbiased cache comparison because Rust warmed between passes.

Darwin: 18.80 s Rust prep, 251.37 s configure/generate, 181.23 s native build/links, 451.71 s whole checker. All baseline 55 fixture names plus hybrid libobs match and all 56 outputs have verified arm64 Mach-O headers. Cache: 1,843/2,552 hits (72.22%), 709 misses, two uncacheable calls. The unusually slow shared-host configure is reported separately; these local results do not establish the hosted critical path.

Full SOLDR_PORCELAIN=0 ./lint passed in 377.5 s (local-test-8bZaqU.log). An initial invocation stopped at C901 before the remaining lanes; the common alias manifest was extracted and the complete lint run then passed. Hosted after evidence, PR run 44520546103, exact signed head 0993a99:

Hosted job Setup/wheels/cache helper Rust archive + SDK prep CMake configure/generate Tool queries/inventory Native build + all links Post Total
Windows MSVC 33.87 s 23.50 s 42.94 s 2.07 s 111.36 s 4.26 s 218 s
Darwin arm64 21.77 s 40.43 s 51.78 s 0.09 s 109.24 s 4.69 s 228 s

Both completed successfully on GitHub-hosted runners. Parsed final-link names exactly match all 55 baseline fixtures per alias. Hybrid libobs is built by the same target graph. Windows records 1,296/2,322 cache hits (55.81%); Darwin 1,843/2,551 (72.25%). Native build/link time falls from 303.07 to 111.36 s on Windows and 285.54 to 109.24 s on Darwin. Darwin hybrid completion is logged 23.38 s into its native phase; fixture work overlaps that prefix and finishes 85.85 s later. Windows does not emit a separate hybrid DLL completion timestamp, so an isolated hybrid build duration cannot be recovered from its log.

The Rust category job also passed: 243 s total, 27.82 s setup, 211.25 s linking/target preparation, 3.93 s post. Its log contains all eight category executables and no serial cmocka pass. All three hosted executions are below six minutes; the slowest is now 243 s (4:03), versus 1,234 s (20:34) before, an 80.3% reduction in the longest job. Both alias target sets exactly match the baseline. The three jobs consumed 689 runner-seconds in total versus 1,234 for the old job. GitHub allocation delayed Windows by 113 s and categories by 308 s after scope/CI-mode readiness; this is separate from the execution durations above. Because of these allocation waits, the observed span from the first link job starting to the last finishing was 548 s (9:08), so this is not a claim that the full PR workflow or capacity-inclusive elapsed span is below six minutes. Dell after timings have not been measured, and its routing/host configuration are unchanged.

./push passed complete six-target Clippy/Dylint (282.2 s), CI self-tests/validators, 9,170 workspace tests with six skipped, 920 feature tests, supported doctests, and all eight Windows GNU category links. Publication completed through ./push. The normal local windows_link_check.py command retains its categories-plus-both-alias preflight; only its CI job sets OBS_CMOCKA_LINK=false to run categories independently.

RED commit ad6872d pins independent jobs, both aliases in PR/queue gate manifests, full target forwarding, and cache isolation. The focused command failed locally with missing jobs/helpers and the old serialized dispatch (local-test-Rh3fgG.log). GREEN is fcc0529 plus the cacheable-CMake, stable-launcher, local-preflight and manifest-refactoring follow-ups. Focused CI changes use local RED evidence; no remote RED run was needed.

Validation: 115 focused selector/gate/script/cache self-tests and 78 associated workflow/cache/admission tests passed; real Windows MSVC category linking; fresh full Windows MSVC and Darwin arm64 fixture links; full ./lint. Runtime Tiers 2/3/4 do not change in this CI orchestration update. No Rust test executable is added. One clud-review primary reviewer found no correctness blockers; its stable-launcher advisory was addressed.

Ci-Attestation: {"at":1795089086,"gate":"data/all/compatibility","host":"linux-x86_64","key":"","lane":"client-checks","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":182,"stamp":"39629d0f3866d2a427ffb9c28603bb59","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795089086,"gate":"data/all/flatpak-manifest","host":"linux-x86_64","key":"","lane":"client-checks","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":182,"stamp":"ecf22b9bdace6c9048c0357d843e7af9","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795089086,"gate":"data/all/qt-xml","host":"linux-x86_64","key":"","lane":"client-checks","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":182,"stamp":"6b8ab9cc5405dd09233c9ecc0934d1a8","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795089086,"gate":"data/all/services-schema","host":"linux-x86_64","key":"","lane":"client-checks","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":182,"stamp":"b91898b4feb0a425a48ef4a519566eec","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088900,"gate":"general/all/client-lint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"8b3e98f4fdac17ea1a93f35878314841","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795089086,"gate":"python/all/self-test","host":"linux-x86_64","key":"","lane":"client-checks","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":182,"stamp":"e4b9712789d7725f8341ac89b0e89901","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088902,"gate":"rust/aarch64-apple-darwin/client-clippy","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"a188d8bf8072102221e608601b11f6e2","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088902,"gate":"rust/aarch64-apple-darwin/client-dylint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"fb990c34fd44224710a64f22b1d000bc","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/aarch64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"dcc4458dc0046768b49efb5c8fe226ad","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/aarch64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"254a47ac647bf954be2a7f8e1b99727f","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/aarch64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"5a01359f1a04d79d5a854d8f12c11a24","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/aarch64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"757a8517f2681a8e47c4a1894ffbf863","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088902,"gate":"rust/x86_64-apple-darwin/client-clippy","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"f25e2c536089fec1e4c73cbae0c212e5","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088902,"gate":"rust/x86_64-apple-darwin/client-dylint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"246d37e9b11c590b1a22b952ecd1e573","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/x86_64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"dfb51d1a27a66d8b22536fd193ea326e","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/x86_64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"b5a89f076446f2f70da638223b7bdf28","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/x86_64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"d249f8bef8c7f698323c96704dc11d6b","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795088901,"gate":"rust/x86_64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"a04109cd97370655fc7486151e4bd358d61fc6ec0a50501a751a6d9e23f58036","lane":"client-lint","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":282,"stamp":"68ef39277f69fe1bda8d9ba91a4e5d9f","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}
Ci-Attestation: {"at":1795089869,"gate":"rust/x86_64-unknown-linux-gnu/client-test","host":"linux-x86_64","key":"5d845c9b23bc03a0e42397787689b4eed693f623e5931dc92110c521181ef4c3","lane":"client-test-x86_64-unknown-linux-gnu","parents":["e245584ce0f2ade1f754ec81f6de1dc15c7a0b34"],"secs":782,"stamp":"5d8f932b44911b6d47d31e26d8972e1a","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738","v":1,"via":"run"}

Client-Checks: {"checks":{"base":"7ff57244c874a81d1812a3d57d5253ee7d8c5e1d","coverage":"all-tracked-applicable-files","diff":[".github/workflows/build-project.yaml","ci/ci_scope.py","ci/cmocka_link_check.py","ci/gate_jobs.py","ci/test_ci_scope.py","ci/test_cmocka_link_check.py","ci/test_cook_profiles.py","ci/test_gate_jobs.py","ci/test_pr_attestation.py","ci/test_rust_cache_policy.py","ci/test_windows_link_check.py","ci/windows_link_check.py","docs/ci-tooling.md","libobs/build/libobs.core+.h.c","libobs/build/libobs.ported-type-catalog-enum+.h.c","libobs/build/ported-source-sets/type-catalog-enum.json","libobs/cmake/rust-exports.d/obs-core-type_catalog_enum.txt","libobs/obs-type-catalog-enum-fields.c.h","rust/README.md","rust/obs-build/tests/native.rs","rust/obs-build/tests/native/registration/canonical_type_catalog_enum_swap_is_registered.rs","rust/obs-c-oracle/oracle/type_catalog_enum.c","rust/obs-core/src/ffi/mod.rs","rust/obs-core/src/ffi/type_catalog_enum.rs","rust/obs-core/src/lib.rs","rust/obs-core/src/type_catalog_enum.rs","rust/obs-core/tests/obs_core/main.rs","rust/obs-core/tests/obs_core/type_catalog_enum.rs","rust/obs-core/tests/obs_core/type_catalog_enum_parity.rs","test/cmocka/video-info-query-link-support.c"],"head":"f5906ef1be96f8f90500ae7aabbc3ebaabf29efe","tools":{"orchestrator-python":"3.13.15 (main, Aug 14 2026, 15:34:59) [Clang 22.1.3 ]","schema-dependencies":"3.13.15 (main, Aug 14 2026, 15:34:59) [Clang 22.1.3 ]\n{'jsonschema': '4.26.0', 'json_source_map': '1.0.5'}","uv":"uv 0.13.5 (x86_64-unknown-linux-gnu)","xmllint":"xmllint: using libxml version 21504\n   compiled with: Threads Tree Output Push Reader Patterns Writer SAXv1 DTDValid HTML C14N Catalog XPath XPointer XInclude Iconv ISO8859X Regexps Automata RelaxNG Schemas Modules Debug"},"tracked-digest":"f6a291b69ef7bda8b401269feca09e01d209bd731a7e08b83abf046faa696308","tree":"52b335b50a5006a7280acd353ec0cbd0ea256738"}}
@zackees zackees added the pool-active Pool worker owns this PR; automated rebases must skip it label Nov 19, 2026
@zackees zackees changed the title ci: share the required alias manifest within the complexity budget ci: parallelize full Tier 2 cross-link checks (Refs #5224) Nov 19, 2026
@zackees
zackees enabled auto-merge November 19, 2026 12:18
@zackees zackees removed the pool-active Pool worker owns this PR; automated rebases must skip it label Nov 19, 2026
@zackees
zackees added this pull request to the merge queue Nov 19, 2026
Merged via the queue into master with commit 1c936eb Nov 19, 2026
88 of 91 checks passed
@zackees
zackees deleted the infra/5224-linkcheck-split branch November 19, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant