Skip to content

ci: overlap macOS Tier 2 Cargo preparation with CTest (Refs #5282) - #5292

Queued
zackees wants to merge 2 commits into
masterfrom
infra/5282-macos-tier2-tail
Queued

zackees wants to merge 2 commits into
masterfrom
infra/5282-macos-tier2-tail

Conversation

@zackees

@zackees zackees commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Refs #5282

The macOS Tier 2 validator currently waits for the CMake OFF/ON suites before preparing the Cargo native graph, then runs the two native suites. Start the first two suites and Cargo preparation together on the same runner, wait for both children, and then run the native suites. Every phase status contributes to the final job result; a failed CMake suite still permits independent preparation and runnable native suites to finish. Completed phase logs are printed in separate groups and retained under build_rust/tail-logs with elapsed times and exit codes.

Keep all four CTest passes serial: they share Testing/Temporary, working directories, fixtures, fixed ports/temp paths and process-local RUN_SERIAL/RESOURCE_LOCK scheduling. Cargo preparation only compiles and inspects artifacts, reads completed CMake/dependency inputs, and writes its existing cargo-native-rust, cargo-native-tests and cargo-native-core directories. Cargo commands remain sequential in their original target directories, avoiding lock serialization and duplicate compilation. Runtime graphics/capture probes start after the join. Framework rediscovery is restricted to CMake's libobs directory so reentry cannot select a Cargo framework. All existing CTest commands, loader checks, native negative controls, repeat options, and crash diagnostics are preserved.

The original job log (135830424089) confirms 725 cases in each pass: CMake OFF took 72 seconds and ON took 54 seconds, followed by about 208 seconds of Cargo preparation and about 134 seconds of native probes/suites. Hiding the first 126 seconds behind preparation should reduce the approximately 468-second tail to 342 seconds before CPU contention, saving about 125–130 seconds. This is an estimate, not a measured macOS result; the p50-under-1000-second acceptance still needs hosted measurements. No runner or workflow job is added. ci.yml delegates the actual job to build-project.yaml; its existing validate.sh entry point adopts this change without a workflow edit. The preparation/runtime boundary can also be reused by #5009's Linux cross-build/run-only work.

Local RED commit 08b709f records uv run --no-project python -m unittest ci/test_tier2_macos_tail.py exiting 1 against the orchestrator stub (four tests, six errors). GREEN: six orchestrator tests cover subprocess and CLI failures, completion of independent phases, the concurrency barrier, launch errors, argument forwarding, retained/grouped logs, and framework rediscovery. The final focused Python 3.14 run passed all 18 tests, including existing Tier 2 scheduling/cache/build-contract and CI import-policy checks. Shell syntax and git diff checks passed. The pre-push review is clean.

The single requested full SOLDR_PORCELAIN=0 ./lint run exited 1 during CI self-tests: test_pr_attestation.py exceeded 600 seconds, then the existing timeout handler in ci/local_gate.py:297 raised TypeError: can only concatenate str (not "bytes") to str. Several other broad self-tests also reported failures before that runner error. Earlier import-policy findings in the new test were corrected and both import-policy checks pass in the final focused run. The new orchestrator and existing macOS scheduling/cache tests also passed inside full lint. Full lint was not repeated; Rust lint stages after the failed self-test stage were not reached. The private full output is retained as local-test-nThnLp.log in the worker handoff.

The publication script's automatic client-lint lane subsequently passed (380.8 seconds), including Clippy and Dylint on all six supported targets. Its automatic client-checks bundle also passed all 398 CI self-test files against the final source, including PR attestation (410.9 seconds) and the checks that failed in the earlier lint run. The host nextest suite passed all 9,257 executed Rust tests (six existing skips); the concurrent-pipe stress case completed successfully after 947 seconds. The separate obs-util feature run passed all 920 tests, including the same stress case after 603 seconds; workspace and feature doctests passed too. The final Windows GNU workspace test-binary link also passed; the complete client-test lane finished successfully (3176.6 seconds). ./push signed and published the verified final head 77e2488.

Actual macOS execution of the unchanged 725-case suites in all four variants and a hosted before/after timing comparison were not available on this Linux worker. Rust port tiers and product behavior are unchanged; this adds only CI self-tests. The ci/ and rust/tools/ edits are trust-surface changes and retain real selected PR and merge-queue checks.

Ci-Attestation: {"at":1795124182,"gate":"data/all/compatibility","host":"linux-x86_64","key":"","lane":"client-checks","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":580,"stamp":"6630af8dfd70b433e0f67b778880d485","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795124182,"gate":"data/all/flatpak-manifest","host":"linux-x86_64","key":"","lane":"client-checks","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":580,"stamp":"fb189ac0f4c92e84b8cf79436a9b7148","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795124182,"gate":"data/all/qt-xml","host":"linux-x86_64","key":"","lane":"client-checks","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":580,"stamp":"4a4eed87dfac57de50bdaa362acf6da6","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795124182,"gate":"data/all/services-schema","host":"linux-x86_64","key":"","lane":"client-checks","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":580,"stamp":"ee93c7a54c0a21d4233b47299618ce98","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123577,"gate":"general/all/client-lint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"0c83cc2cfa2d9ef1c060c53f6de1658b","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795124182,"gate":"python/all/self-test","host":"linux-x86_64","key":"","lane":"client-checks","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":580,"stamp":"d40b9e9f90b50e64628748bf26506846","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123591,"gate":"rust/aarch64-apple-darwin/client-clippy","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"b3dac566ef01deb99647abd4c333d3f5","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123592,"gate":"rust/aarch64-apple-darwin/client-dylint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"635876ed611f3309a67f2162700ed822","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123585,"gate":"rust/aarch64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"9d1f2b7f60aa17b6ad19114ff55adda7","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123587,"gate":"rust/aarch64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"7e3a6316bf650346af15d7fd60f40e0b","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123580,"gate":"rust/aarch64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"2ed0516bf06941ebae8b498447de294a","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123581,"gate":"rust/aarch64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"4843e6d0eb97e7f4b5a0a51b2cd562fb","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123588,"gate":"rust/x86_64-apple-darwin/client-clippy","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"a5c73fc2ac8526669c73b87b93fe64e7","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123591,"gate":"rust/x86_64-apple-darwin/client-dylint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"03cbea2b8cf4904103392d39b7c6828e","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123583,"gate":"rust/x86_64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"a70c21ad418f03fa2fe779fc33c299b1","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123583,"gate":"rust/x86_64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"04ecd7ec99ac2c7c9bbfa512be0ba31b","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123578,"gate":"rust/x86_64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"9b71b0de3a352b0e9dd072097ea886aa","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795123580,"gate":"rust/x86_64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"1d9532b8e6888c9a8812dbed3983271935a2cef712e90166c3ce374f3c28443f","lane":"client-lint","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":383,"stamp":"66ffec0b7ac655194b06b2ba050d129a","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}
Ci-Attestation: {"at":1795127363,"gate":"rust/x86_64-unknown-linux-gnu/client-test","host":"linux-x86_64","key":"b44772078e4aa6ab6488a257274405fca18de151c42e8bc50b34acd70abbb56c","lane":"client-test-x86_64-unknown-linux-gnu","parents":["08b709f0c5867fd5732e18e001f743934e7b3e03"],"secs":3177,"stamp":"c274c56457aa5a1be4b385ed9bb64a38","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11","v":1,"via":"run"}

Client-Checks: {"checks":{"base":"c24fc60f859b57fa944ab59705a95747f8577153","coverage":"all-tracked-applicable-files","diff":["ci/test_tier2_macos_tail.py","ci/test_tier2_native_target.py","ci/tier2_macos_tail.py","rust/tools/tier2-validate/validate.sh","test/cmocka/known-flaky.txt"],"head":"7c2dffb93653159c213962a711f7aacf32294b40","tools":{"orchestrator-python":"3.14.7 (main, Aug 14 2026, 15:35:28) [Clang 22.1.3 ]","schema-dependencies":"3.14.7 (main, Aug 14 2026, 15:35:28) [Clang 22.1.3 ]\n{'jsonschema': '4.26.0', 'json_source_map': '1.0.5'}","uv":"uv 0.13.5 (x86_64-unknown-linux-gnu)","xmllint":"xmllint: using libxml version 21504\n   compiled with: Threads Tree Output Push Reader Patterns Writer SAXv1 DTDValid HTML C14N Catalog XPath XPointer XInclude Iconv ISO8859X Regexps Automata RelaxNG Schemas Modules Debug"},"tracked-digest":"60bbc79a374fc8729a0e51df1971cd04befbe6c3c09bc0e80169440f98b80729","tree":"d63caf9ecfad9fec8fcd762c9a92084e529d8c11"}}
@zackees zackees added pool-active Pool worker owns this PR; automated rebases must skip it and removed pool-active Pool worker owns this PR; automated rebases must skip it labels Nov 19, 2026
@zackees
zackees enabled auto-merge November 19, 2026 22:32
@zackees
zackees added this pull request to the merge queue Nov 19, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant