Skip to content

CI resilience (per-PR): detect Actions-policy startup_failure instead of mis-attributing it #15

Closed

Description

@hyperpolymath

Context — the estate-wide "Actions-policy CI outage"

Two failure modes recur across the estate: mode-1 (allowed_actions=selected + empty patterns_allowed → external uses: rejected → startup_failure) and mode-2 (sha_pinning_required=true + tag-pinned workflow files → github-owned actions fail the pin rule). The squabbler is the per-PR / per-repo reactive lane for a single stuck gate — but today it mis-attributes this class:

  • a governance-reusable startup_failure → AssignGateOwner{OwnedUpstream:standards} (wrong: the reusable is fine; the local Actions policy blocks it);
  • a tag-pinned codeql.yml startup_failure → EscalateToExpert{Security} (wrong: a scan won't fix a tag-vs-SHA rejection).

The symptom already lands in the Gate (fetch.rs::parse_rollup maps STARTUP_FAILURE → CheckRun::Failed); what's missing is the why-probe. This is a real added rule inside the existing diagnose→classify→move pipeline. Fleet-scale remediation stays out of scope (escalate to ExpertGroup::HypatiaFleet → gitbot-fleet).

Decision of record

Estate default = allowed_actions=all + sha_pinning_required=true. The squabbler's settings-move should propose that posture (or an explicit patterns_allowed superset) while keeping sha_pinning_required=true — never disable pinning. squabble ≠ bypass still holds: these moves only let a required check start; they drop no required context and weaken no check.

Changes

  1. crates/squabble-cli/src/fetch.rs — add an Actions-policy why-probe. When any required context resolves to STARTUP_FAILURE (already handled at parse_rollup, ~L171–184), additionally call gh api repos/{o}/{r}/actions/permissions + .../selected-actions and thread {allowed_actions, sha_pinning_required, patterns_allowed} into the fight as classification input (it already fetches the ruleset via gh api .../rules/branches/{b}).

  2. crates/squabble-fight/src/workflows.rs — extend WorkflowInfo with tag_pinned_uses: Vec<String> and external_uses: Vec<String> (parse in parse_workflow alongside reusable_repo()). In classify(), when the emitting workflow uses an external reusable/action absent from a selected allowlist (mode-1) or a tag-pinned uses: under sha_pinning_required (mode-2), return the NEW move instead of the current OwnedUpstream / Escalate-Security mis-attribution.

  3. crates/squabble-core/src/moves.rs — add Move variants: PinWorkflowActions { workflow, refs } (self-win, SHA-pin at source; twin of the existing RepinReusableWorkflow) and ReconcileActionsPolicy { blocked_ref, add_patterns } / SetActionsAllowedAll (a legitimate non-bypass settings move). Add them to every_move_is_legitimate_by_construction and describe() coverage so the SPARK invariant still holds.

  4. crates/squabble-core/src/lib.rs — no structural change; diagnose_with_hints is the existing plug seam — feed it the new hints from the host.

  5. crates/squabble-fight/src/lib.rs::structural_scan() — add a probe surfacing "allowed_actions=selected with empty/insufficient patterns_allowed while an external reusable/action is used" and "tag-pinned uses: under sha_pinning_required" even before the check has (yet) startup_failure'd, mirroring the existing well-known/path-filter/AGPL probes.

  6. examples/actions-policy-deadlock.json — new offline gate fixture (mirroring examples/gate-deadlock.json) with a required check as missing/startup_failure for regression coverage. The squabbler's own repo currently shows a Secret Scanner startup_failure — an in-repo dogfood target for .github/workflows/dogfood-gate.yml.

  7. docs/CHARTER.adoc — add the "Actions-policy startup_failure deadlock" class to the v0.1 fight playbook, stating the tradeoff explicitly (prefer allowed_actions=all while KEEPING sha_pinning_required=true, plus SHA-pin the workflow refs).

DO-NOT pointer: .github/settings.yml (probot/settings) manages branch protection/labels/merge only and cannot set Actions permissions — the fix goes via the REST actions/permissions + .../selected-actions endpoints (single-repo from the squabbler, fleet-wide from gitbot-fleet), never via settings.yml.

Acceptance criteria

  • A required-context STARTUP_FAILURE triggers the why-probe; the fight receives {allowed_actions, sha_pinning_required, patterns_allowed}.
  • classify() returns PinWorkflowActions for a tag-pinned-under-sha-pinning gate and ReconcileActionsPolicy/SetActionsAllowedAll for an empty-allowlist gate — not OwnedUpstream{standards} / Escalate{Security}.
  • New Move variants pass every_move_is_legitimate_by_construction (SPARK invariant intact) and have describe() text.
  • examples/actions-policy-deadlock.json is diagnosed to the new move in a unit test.

Model tier

Opus — Rust across three crates plus the SPARK-legitimacy invariant; the classification changes are subtle (distinguishing "policy blocks the check from starting" from genuine upstream/out-of-lane reds).

Related

Fleet-scale detection/remediation belong to hyperpolymath/hypatia + hyperpolymath/gitbot-fleet (the squabbler escalates the fleet dimension to ExpertGroup::HypatiaFleet); source-of-truth allowlist in hyperpolymath/standards.

Activity

  1. hyperpolymath commented on Jul 11, 2026

    @hyperpolymath
    OwnerAuthor

    Part of the estate-wide CI-resilience remediation set (Actions-policy startup_failure, root-caused by the 2026-07-11 estate scan). One owner per responsibility:

    Decision of record: estate default = allowed_actions=all + sha_pinning_required=true. The one-off sweep of the 91 broken hyperpolymath repos is intentionally deferred to gitbot-fleet#362 (self-healing behind the kill-switch), not a manual PUT.

  2. added
    bugSomething is broken or behaves incorrectly
    on Aug 24, 2026
  3. added a commit that references this issue on Sep 27, 2026
    65c3d21
  4. added
    scope:estateAffects many or all repos across the estate
    status:readyFully specified and ready to be picked up
    on Sep 30, 2026
  5. hyperpolymath commented on Oct 8, 2026

    @hyperpolymath
    OwnerAuthor

    Closing as implemented and verified — the work landed in #110 (merged 2026-09-27), which referenced this issue but did not use a closing keyword, so it stayed open.

    #110 was authored without a Rust toolchain ("compilation + tests delegated to CI"), so I re-verified locally rather than trusting the PR body.

    Verified on main @ a4cd088 (worktree, rustc 1.101.0-nightly via the repo's mise.toml):

    Gate Result
    cargo test --workspace --all-targets exit 0, all suites pass
    cargo clippy --workspace --all-targets -- -D warnings exit 0 (one manifest note: unused thiserror in squabble-core, not a lint failure)
    cargo fmt --all -- --check exit 0

    Acceptance criteria, checked against code and named tests:

    1. Why-probe on a required-context STARTUP_FAILURE. fetch::fetch_actions_policy (fetch.rs:429) reads actions/permissions and .../selected-actions. It is threaded into the fight as ActionsPolicyFacts (fight.rs:58). Tests: only_required_contexts_with_startup_failure_trigger_the_probe and startup_failure_still_classifies_failed_in_the_gate.
    2. classify() returns the new moves, not the mis-attributions. workflows.rs:244–259. Tests: empty_selected_allowlist_classifies_to_set_allowed_all, selected_allowlist_gap_classifies_to_reconcile_with_the_missing_pattern and tag_pinned_under_sha_pinning_classifies_to_pin_not_escalate_security.
    3. Legitimacy invariant holds. The PinWorkflowActions, ReconcileActionsPolicy and SetActionsAllowedAll variants are in every_move_is_legitimate_by_construction, have describe() arms, and pass actions_policy_moves_serialise_round_trip.
    4. Fixture diagnosed in a unit test. the_offline_fixture_diagnoses_to_the_new_moves loads examples/actions-policy-deadlock.json and asserts SetActionsAllowedAll and PinWorkflowActions{codeql.yml}. It also asserts the absence of OwnedUpstream{standards} and Escalate{Security}. The negative control the_same_fixture_without_the_probe_shows_the_old_misattribution proves the probe inputs are what carries the redirect.

    Changes 5 and 7 are also present: the structural-scan probe (latent_policy_deadlocks_surface_before_any_startup_failure) and the CHARTER playbook entry (docs/CHARTER.adoc:112–130).

    Horizon of this verification: unit and fixture level on main. Not exercised live against a real deadlocked PR. The estate's mode-1 outage has largely been repaired since (see gitbot-fleet#362), so no live specimen was at hand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaves incorrectlypriority:p2Normal - queue itscope:estateAffects many or all repos across the estatestatus:readyFully specified and ready to be picked up

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions