Skip to content

ci: keep local-gate failure reporting from crashing on subprocess timeouts - #5283

Open
zackees wants to merge 2 commits into
masterfrom
fix/4896-local-gate-timeout
Open

zackees wants to merge 2 commits into
masterfrom
fix/4896-local-gate-timeout

Conversation

@zackees

@zackees zackees commented Nov 19, 2026

Copy link
Copy Markdown
Member

Refs #4896

A client self-test timeout could crash its reporter before the gate printed the other parallel failures. On #5279 this hid the sanitizer-harness diagnostic and prevented complete client-checks evidence. Normalize each stdout/stderr stream independently (bytes with replacement decoding, strings, and missing output), and collect worker/launch exceptions as failed modules so aggregation completes. The lane still fails whenever any module fails.

Give only test_pr_attestation.py an 1800-second deadline; all other files retain 600 seconds and the eight-file pool stays unchanged. The verifier is serial and Git-heavy, with no nested executor. The incident suite passed alone in 159.835 s but exceeded 600 s alongside the other checks. This host exposed 16 CPUs with load averages 162/151/156 and CPU pressure around 64%; twenty serial git --version calls alone took 10.885 s (0.036–1.398 s each) through the preserved clud shim. Reducing this pool cannot bound other slots, and splitting the suite would not reduce its serial Git work. A fresh serial-suite profile passed all 45 tests in 889.037 s: 2,154 Git calls took 876.106 s (98.55% of wall time). This shared-host run overlapped lint during its latter portion. The scoped 1800 s deadline is about twice that measured completion time and is a contention mitigation; no execution speedup is claimed.

RED → GREEN:

  • RED 5bd55d718: the focused regression produced four timeout-reporting errors plus a launch-error aggregation failure; the verifier deadline regression separately failed with 600 instead of 1800. Both local failure logs were retained before implementing the fix.
  • GREEN 926bc28c8 (before proof stamping): all 38 test_local_gate.py tests pass in 4.356 s, and Ruff passes. Coverage includes invalid UTF-8 bytes, strings, missing streams, mixed stream types, a second parallel failure, launch errors, and per-suite deadlines/report text. The pre-push clud-review is clean and its independent focused tests pass.

./lint ran once: all 397 files finished, and the parallel verifier passed in 947.6 s under the extended deadline. Aggregation printed all three unrelated module failures: browser startup (child 2 s timeout), apt-cache concurrency (mock shell 10 s timeout/barrier), and shared-link support (CMake build 60 s timeout). Apt-cache passed 18 tests alone; shared-link passed two alone in 130.576 s; browser alone hit its outer 60 s timeout. The full lint therefore failed; none of those tests was edited. ./push published RED and GREEN together, but produced no client evidence (no Ci-Attestation, Client-Checks or legacy trailers). Linux x64/arm64 clippy and Dylint plus Windows x64 clippy passed; Windows Dylint stopped when the existing driver version probe exceeded its 2-second deadline. The tool reported binary-or-exit driver acquisition failure; client-checks and host tests were consequently not reached. Warnings: "client proof failed or unavailable; partial evidence only, ordinary PR CI" and "pushed without complete signed client evidence; ordinary PR CI remains required." Ordinary hosted CI remains the gate. The stock ./push proof/sign/publication code ran through an external adapter that deferred only automatic PR creation, keeping publication and PR creation in separate GitHub mutations. This changes Python CI tooling only; Rust/C API tiers 1–4, packaged artifacts and native implementations are unaffected. No workflow, C test, public header, websocket or native-coverage opt-in changes. This is a CI trust-surface edit, so real selected hosted checks remain required even with complete client evidence.

@zackees
zackees enabled auto-merge November 19, 2026 21:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant