CI gate freshness alert: 6 gate(s) have no recent successful main run #7966

Activity

1. proggeramlug commented on Aug 12, 2026

   

   proggeramlug

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Diagnosed. The eleven do not share one cause, and the issue's own hypothesis (starvation, #7856) is half of it.

   Nine are starved and hit a #7205 relapse on the arm #7856 created. The concurrency groups key on ${{ github.event_name == 'push' && github.sha || github.ref }} — the github.sha arm is guarded on the event being push, which was correct while the main-line arm was push: branches: [main]. #7856 moved that arm to schedule:, the guard stopped matching, and the group fell through to github.ref (constant). Observed identically on all ten scheduled gates:

   13:37Z schedule pending      <- blocked on the group
   07:46Z schedule cancelled    <- jobs: 0
   02:37Z schedule cancelled    <- jobs: 0
   19:15Z schedule queued       <- holds the group, 20h in the runner queue
   

   jobs: 0 is #7205's exact zero-execution signature. gate-freshness itself was cancelled the same way at 13:38Z.

   Two are not starved at all — they are broken. gc-native-roots has never had a successful run on any branch (#7970). llvm-inprocess fails when it runs and reports green on PRs while skipping its only real job (#7971).

   Queue depth is 1,529, not 100 (the snippet in this issue body pages at 100 — use -q '.total_count'). 794 are pull_request runs across 63 head branches, of which 61 no longer exist: ~790 runs, 51% of the queue, are work for already-merged PRs holding slots in front of the main gates.

   Fix in #7969: groups keyed per github.run_id for non-PR events across 22 workflows; a new check_schedule_group in scripts/gc_gate_wiring_check.py (required lint) so this cannot relapse a fourth time — it found ten more workflows already carrying the same latent bug, including test.yml's nightly safety net; plus a queue reaper for the dead PR runs.

   Two things #7969 deliberately does not close:

   • parity and compile-smoke are required contexts whose jobs never run on a pull request (test.yml's push: is tags-only). A required context that never reports blocks the merge button forever, so every merge needs an admin bypass — which bypasses the required contexts that do work. That is upstream of this whole incident: bypass is the normal path, so a pending gate stopped looking unusual. Branch-protection state, admin only.
   • gc-ratchet would not have caught perf(gc): main regressed the retain cluster 2.2-4.8x — retain now runs 2 full collections where it ran none (suspect #7901/#7902) #7965 even had it run. No gate in this repo ratchets a collection-kind count; the counter that found it (collection_kind: "full" 0 → 2) is not a gc-ratchet metric, and the two dimensions that did move (wall time, RSS) are explicitly "gating": false in the shared_ci profile CI uses. perf(gc): main regressed the retain cluster 2.2-4.8x — retain now runs 2 full collections where it ran none (suspect #7901/#7902) #7965's third ask stays open, and a fully-green freshness dashboard would not have caught it either.

   Leaving this issue open for gate-freshness.yml to close itself once the gates are actually fresh.
2. proggeramlug

   added a commit that references this issue on Aug 12, 2026

   ci: fix the #7205 relapse on the scheduled gate arm, and reap dead PR…

   ...

   e316ecb
3. github-actions

   changed the title [-]CI gate freshness alert: 11 gate(s) have no recent successful `main` run[/-] [+]CI gate freshness alert: 10 gate(s) have no recent successful `main` run[/+] on Aug 13, 2026
4. github-actions

   changed the title [-]CI gate freshness alert: 10 gate(s) have no recent successful `main` run[/-] [+]CI gate freshness alert: 7 gate(s) have no recent successful `main` run[/+] on Aug 13, 2026
5. github-actions

   changed the title [-]CI gate freshness alert: 7 gate(s) have no recent successful `main` run[/-] [+]CI gate freshness alert: 6 gate(s) have no recent successful `main` run[/+] on Aug 13, 2026
6. proggeramlug commented on Aug 14, 2026

   

   proggeramlug

   on Aug 14, 2026

   ContributorAuthor

   More actions

   Interim exact-main local acceptance after the merged #8054/#8056/#8062/#8063/#8066/#8068 fixes. This is local product evidence, not a CI result; no mini workflow was used. The issue remains open because #8051 ratchet attribution is still open.

   Exact source and pinned artifacts

   • origin/main: 0da668c95150d78ba5aa2b8eff7b78c04cc381c1
   • tree: 86fa39aa89fa6eac71fd993218fe69d9cfbee442
   • clean detached audit worktree; primary dirty worktree unchanged
   • reused the GC ratchet has broad deterministic fingerprint shifts that need attribution #8051 worker's exact-main release artifacts only after its inventory exited, it released the target read-only, and I independently verified no target writer
   • perry: 104,946,752 bytes, mtime 1786674954, SHA-256 a9074f78d3368b908bce1f8b514236583ecb66dd942dc3330d448f68cf690829
   • libperry_runtime.a: 31,142,648 bytes, mtime 1786675824, SHA-256 80a98f3e5227c4927ab0de3d500124ca48e9bf2fc650d34c0b5c78caeb07160b
   • libperry_stdlib.a: 80,871,000 bytes, mtime 1786675919, SHA-256 191a26f29af113b94c7cf58769742cffcf6653098026755db9564dd05f87e018
   • all three identities were unchanged after the product runs; PERRY_RUNTIME_DIR was absolute and PERRY_NO_AUTO_OPTIMIZE=1

   Compiler-free checks ran first on that exact tree:

   • gc_gate_wiring_check --self-test: 18/18; live repository check: 7 gates and 31 workflows, including the ci: fix the #7205 relapse on the scheduled gate arm, and reap dead PR runs (#7966) #7969 unique github.run_id scheduled-run groups
   • current copy-minor parser self-test: legacy/current/promotion/in-place-negative directions pass
   • parse-churn checker: all 8 clean/sabotage directions pass
   • TLS verdict checker: all 8 rejection modes pass; pre-attach and sampler-status negative directions pass
   • gc-ratchet harness: 93/93 tests pass, including its reject/sabotage paths

   Original six rows

   1. gc-ratchet.yml — not yet accepted. The gate is schedulable and its checker remains able to fail, but GC ratchet has broad deterministic fingerprint shifts that need attribution #8051 is still open and no baseline was changed here. Its broad deterministic fingerprint shift still needs the recorded per-row attribution/reviewed repin. This alone keeps CI gate freshness alert: 6 gate(s) have no recent successful main run #7966 open.

   2. tls-budget.yml — locally green and non-vacuous. tls-budget asyncpipe exits before Darwin sampling and yields zero-root profile #8050 was fixed by fix(ci): keep TLS asyncpipe live for sampling #8066 without weakening the 2,000-root, direct-TSD, claimed-site, correctness, or percentage requirements. Exact-main Darwin run:

      • asyncpipe: 5,564 root samples; 20 _tlv_get_addr (0.4%, budget 5.0%); claimed=152, direct_tsd=1; completed profiled pipeline and stdout checks passed
      • interp: 5,637 root samples; 1 _tlv_get_addr (0.0%, budget 3.0%); claimed=137, direct_tsd=1; stdout check passed

   3. gc-ptr-shape-off-witness.yml — locally green and non-vacuous. Moving-GC witness gates misparse current copying-minor diagnostics as zero relocation #8049 was fixed by fix(gc-gates): parse current copy-minor relocation evidence #8056: the keyed parser counts copied+promoted relocation only for in_place=false and preserves requires=move. Exact-main rep_ptr_shape_off matrix: 23/23 PASS, 0 UNVER/XFAIL/FAIL, byte-exact 23/23; collected/reclaimed/moved/copy-minor 23/23; relocation counter 987,800.

   4. gc-parse-churn-gate.yml — locally green and non-vacuous. Typed-parse rodata globals collide across specialised and boxed function bodies #8048/fix(codegen): keep emitted rodata symbols module-unique #8054 made the specialised and boxed typed-parse rodata names module-unique; fix(gc-gates): parse current copy-minor relocation evidence #8056 repaired its current diagnostic parser. The exact-main wrapper compiled the real fixture and produced BLOB_BYTES 245781, PARSED_LENGTH 4000, CHURN_TOUCH 240000, MISMATCHES 0, PARSE_CHURN_LAYOUT_GATE_OK, exit 0. It observed seven copying minors/eight loop polls and a current-format non-in-place minor with copied_objects=183.

   5. gc-moving-witnesses.yml — locally green and non-vacuous. Same Moving-GC witness gates misparse current copying-minor diagnostics as zero relocation #8049/fix(gc-gates): parse current copy-minor relocation evidence #8056 parser repair, with requires=move unchanged. Exact-main loop-polls matrix: 41/41 PASS, 0 UNVER/XFAIL/FAIL, byte-exact 41/41; collected/reclaimed/moved/copy-minor 41/41; relocation counter 1,419,470. The workflow-equivalent separate every-cell and on-disk registration assertion also passed.

   6. llvm-inprocess.yml — exact-current evidence is green and non-vacuous. llvm-inprocess diff has deterministic object-byte mismatch after behavior parity #8052/fix(codegen): lower precise roots in native LLVM construction #8062 added precise-root lowering to direct and split native construction; llvm-inprocess liveness grep false-fails under pipefail with SIGPIPE #8053/fix(ci): avoid pipefail SIGPIPE in LLVM liveness checks #8063 removed the SIGPIPE false negative while retaining all four liveness assertions; follow-on llvm-inprocess RS4GC runs before text/native constant-folding converges, changing code and GC maps #8065/fix(codegen): canonicalize constants before RS4GC #8068 canonicalized constant folding before RS4GC without weakening the exact byte oracle. I did not repeat this expensive matrix: the closing llvm-inprocess reports green on PRs while skipping its only real job (hazard 4), and fails when it does run #7971 local audit records the complete 991-test/spike/batch/EH matrix against this exact SHA and tree, and freshly fetched origin/main plus all gate input paths remain that identical tree. The explicit inference is therefore that the recorded exact-main result still applies because neither revision nor inputs changed—not that a result from an earlier tree was carried forward.

   Disposition: five non-ratchet rows are locally proven on exact current main. Keep #7966 open and wait for #8051 to land; after that, rerun the remaining current-main ratchet acceptance before considering closure. If that run exposes a distinct failure, split it into a dedicated issue and leave this open.
7. proggeramlug commented on Aug 14, 2026

   

   proggeramlug

   on Aug 14, 2026

   ContributorAuthor

   More actions

   Clarification on disposition: #7966 is also the automation-owned successful main-run freshness alert. Local acceptance establishes correctness but does not make that stated freshness predicate true. Therefore it remains open both because #8051 is not merged and because the six real main workflow executions have not yet satisfied the freshness bookkeeping. I will not manually close it; any later workflow dispatches are bookkeeping evidence only, not substitutes for the local product proof above.
8. proggeramlug commented on Aug 14, 2026

   

   proggeramlug

   on Aug 14, 2026

   ContributorAuthor

   More actions

   Final local acceptance after #8069 merged; no CI status, mini workflow, product rebuild, or new measurement is used as correctness proof. I am leaving #7966 open for its successful-main freshness predicate and will not manually close it.

   Exact merged main

   • origin/main: f9475fb9ae0e3e1fc21d7e9c7146d963f859bdec
   • tree: 52fd3d3b402193ae45f6b352b67eae034075edac
   • relative to the fully product-tested 0da668c95150d78ba5aa2b8eff7b78c04cc381c1, exactly five intended files changed:
     • benchmarks/gc_ratchet/README.md
     • benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json
     • benchmarks/gc_ratchet/gc_ratchet.py
     • changelog.d/8069-gc-ratchet-attribution.md
     • tests/test_gc_ratchet.py
   • no compiler, runtime, fixture, non-ratchet checker, or workflow blob changed. Therefore the explicit carry-forward inference for TLS, Ptr-OFF, parse-churn, moving witnesses, and llvm-inprocess is limited to byte-unchanged product/input paths; their itemized exact-product evidence in the preceding comment remains applicable to this main tree.

   Merged ratchet acceptance

   • GC ratchet has broad deterministic fingerprint shifts that need attribution #8051 is closed by fix(gc): attribute deterministic ratchet shifts #8069.
   • Saved measurement: generated 2026-08-14T02:54:24+00:00 on exact product source 0da668c95, Darwin arm64/M1 Max, 14 probes, 7 normal repeats + 2 traced runs; file SHA-256 4dca775dc54d5888e9f8e7c37d7420eff3992a1b1ff550364544f9b23537b3d8.
   • This is an exact-current product measurement for f9475fb9a because fix(gc): attribute deterministic ratchet shifts #8069 changes only the ratchet acceptance/checker files listed above; rebuilding or remeasuring would add no product coverage.
   • merged checker tests: 97/97 pass, including malformed/stale receipt rejection and future-regression sabotage
   • merged baseline structural validation: pass
   • saved measurement through the merged baseline/checker with --profile shared_ci: gc-ratchet: OK
   • the checker reports exactly 21 attributed deterministic cells, tied to perf(runtime): right-size small objects — INLINE_SLOT_FLOOR 4 -> 2 (#7916, #7714) #7928/fix(gc): denominate the nursery constant band in objects (#7929) #7961/perf(gc): let the first copying minor decide its promotion from its own trace (#7937) #7960; thresholds are unchanged, and informational RSS/timing rows remain non-gating rather than being used to hide a deterministic regression

   Local disposition: every one of the original six rows is now green and non-vacuous on the exact current product tree:

   • gc-ratchet: merged checker/baseline accepts the saved exact-current 7+2 measurement
   • tls-budget: exact Darwin sample/root/runtime-symbol/claimed-site budgets pass
   • gc-ptr-shape-off-witness: 23/23 byte-exact and moved
   • gc-parse-churn-gate: real 4,000-record fixture passes with seven copying minors and relocation
   • gc-moving-witnesses: 41/41 byte-exact and moved; separate registration/every-cell assertion passes
   • llvm-inprocess: exact-current 991-test plus spike/batch/EH behavior and object-byte matrix passes

   Freshness bookkeeping is still separate. All six real workflow files are active on GitHub and expose workflow_dispatch, so they can be launched on main. Those executions can satisfy the automation-owned freshness predicate; they are not substitutes for the local acceptance above. If any dispatch produces a distinct substantive failure, split it into a dedicated issue and leave #7966 open.
9. proggeramlug commented on Aug 14, 2026

   

   proggeramlug

   on Aug 14, 2026

   ContributorAuthor

   More actions

   Freshness-bookkeeping dispatches are now in flight on exact main f9475fb9ae0e3e1fc21d7e9c7146d963f859bdec: 31766259491, 31766260952, 31766262469, 31766263782, 31766265058, 31766266548. I am monitoring only these six real workflow runs. Their outcomes serve the automation-owned freshness predicate; they are not being used as correctness proof, which is the local evidence recorded above.
10. proggeramlug commented on Aug 14, 2026

    

    proggeramlug

    on Aug 14, 2026

    ContributorAuthor

    More actions

    Freshness dispatch stopped at the llvm-inprocess boundary; I did not dispatch gate-freshness and did not rerun anything.

    Five exact-main dispatches completed successfully, but llvm-inprocess run 31766266548 failed its unit gate: 990/991 passed; native smoke was skipped. The sole failure was native_emit::tests::split_native_construction_lowers_precise_roots_before_rs4gc. The native object omitted __PERRY_GCMAP,__perry_gcmap while the text reference carried it. Independent scheduled run 31766084651 produced the same boundary and byte shapes.

    This is now reproduced locally and source-causal, not accepted from CI as proof. On exact f9475fb with LLVM 22.1.4, forcing PERRY_CODEGEN_UNIT_JOBS=1 makes the focused test fail deterministically. The text arm stays on the producer and honors NativeRootsPin; compile_module_units_native still spawns a worker even at one job, and that worker reads fresh thread-local defaults. Both native_plan_args and optimize_and_emit_module therefore disable RS4GC on the native arm. On a higher-core default, both arms spawn workers and lose the decision, making byte equality pass vacuously while both objects are unmapped.

    Dedicated product issue: #8070. #7966 must remain open; its successful-main-run freshness predicate is not met and the local llvm acceptance was proven vacuous at this boundary.
11. github-actions commented on Aug 14, 2026

    

    github-actionsbot

    on Aug 14, 2026 – with GitHub Actions

    More actions

    All post-merge gates are fresh again; closing automatically.
12. github-actions

    closed this as completedon Aug 14, 2026
13. proggeramlug

    added 2 commits that reference this issue on Aug 14, 2026

    ci(release): detect when the published npm package falls behind main (P…

    ...

    03e9a01

    ci(release): detect when the published npm package falls behind main (#…

    ...

    32b2082