Skip to content

fix(workflow-pins): fail loud on a ref: expression the step-output reader cannot parse (BE-8253) - #203

Merged
mattmillerai merged 9 commits into
mainfrom
matt/be-8253-workflow-pins-unparsed-ref
Aug 21, 2026
Merged

mattmillerai merged 9 commits into
mainfrom
matt/be-8253-workflow-pins-unparsed-ref

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Aug 21, 2026 •

Copy link
Copy Markdown
Member

Update: #202 (this PR's former base) has merged to main; GitHub retargeted this PR to main directly, so it is no longer stacked — the banner that used to be here is stale and removed. This branch was then merged with the current main tip to pick up #202/#197/#200/#196; see the Provenance section below for what that merge verified.

ELI-5

The workflow-pins lint follows a checkout's ref: back to the step that produced it, so a job that must never fail still can't silently check out mutable scripts. It read that ref: with one regex and gave one answer per line — so any spelling it couldn't chew recorded no site at all and the file passed. Now it asks two questions: "does this ref: mention a step output at all?" and "how many of those could I actually read?" If the first number is bigger than the second, the lint says so out loud instead of shrugging. And where a ref: names several step outputs, every one of them is judged, not just the first the regex happened to land on.

What was wrong

ref_checkouts's else-arm dropped any line _steps_output_match couldn't match. Five shapes escaped, all confirmed fail-open through the real CLI (see Evidence):

  1. ref: ${{ steps.x.outputs.ref || format('refs/heads/{0}', 'main') }} — the } in {0} breaks the [^}]*? fallback stretch.
  2. ref: ${{ (steps.x.outputs.ref) || 'main' }} — ) after the output name matches neither || nor }}.
  3. ref: ${{ steps.x.outputs.ref && 'main' }} — only || was accepted. Runtime is wrong in both arms: truthy checks out at the mutable literal, falsey resolves to '' and actions/checkout reads '' as the default branch.
  4. ref: "${{ steps.a.outputs.ref }}${{ steps.b.outputs.ref }}" — the block pattern's greedy .* landed on the LAST interpolation; a was never judged even though both feed the value.
  5. ref: ${{ steps.a.outputs.ref || steps.b.outputs.ref }} — the fallback stretch swallowed b; a covered a passed the whole site.

What changed

Two-tier reader. LOOSE counts steps.<id>.outputs.<out> mentions in the comment-stripped ref: value (so ref: main # was ${{ steps.x.outputs.ref }} cannot fire it). STRICT runs the existing patterns over every interpolation and every top-level || operand, yielding a list rather than one match. Loose > strict records the site with via_step_output = 'unparsed' — truthy, so unguarded_ref_checkouts needs no change, the same trick 'dangling'/'non-leading' already use — and check_dir prints a message naming the two spellings the lint can judge. Widening the strict reader to evaluate format() / && / parenthesized expressions is deliberately not the fix: an unknown spelling asks its author for a supported one rather than growing an expression parser inside the lint.

All operands judged, strongest requirement wins. _record_steps_output takes the list; each (id, out) must independently pass (declared as an earlier step, covered by a fail-closed resolver or the exact non-empty if:, reachable by operand order). A covered sibling excuses nothing. 'non-leading' outranks 'dangling' for the reason it is judged first: no change to the id or the step order fixes operand order.

_lead_reaches_output now falls through a leading step output. An unresolved output is '', which is falsey, so A || B really can resolve to B. That second operand only became visible with the exhaustive reader, and it is judged in its own right — so nothing is excused by falling through. (Chesterton's-fence note: the check was written in #202 when only one step output per expression was ever visible, so this case could not arise; the falsey-literal behaviour and every _FALSEY_OPERANDS answer are unchanged, pinned by test_a_falsey_leading_operand_still_reaches_the_output.)

Gated by _is_ref_input, exactly as the other states are — a job-level outputs: mapping or a run: heredoc holding an unparseable expression is not a checkout. That is the false-CI-failure channel BE-8215 already had to close once; mutation-checking confirms reordering the gate breaks a named test.

Flow entry boundary is interpolation-aware. The naive [^,}] bound cut a value mid-expression and hid the rest from both tiers. ${{ … }} is atomic now; outside one, the first ,/} still ends the entry, so a sibling entry still cannot be misread as the ref (pinned by the pre-existing decoy tests — reverting it fails four of them).

Applied to all three spellings (block, flow, continuation under ref: >- / ref: |).

Evidence — the refusal path is empirically justified, not assumed

Before shipping a diff whose user-facing outcome is "this lint refuses your expression", I ran each shape through both CLIs (base = #202 head b68df60, new = this branch), against a fixture workflow with a real resolver and checkout:

ref: value base exit this branch
${{ steps.resolve_ref.outputs.ref }} (bare, uncovered) 1 (correctly fails) 1
${{ steps.resolve_ref.outputs.ref || format('refs/heads/{0}', 'main') }} 0 — passed 1
${{ (steps.resolve_ref.outputs.ref) || 'main' }} 0 — passed 1
${{ steps.resolve_ref.outputs.ref && 'main' }} 0 — passed 1
"${{ steps.soft.outputs.ref }}${{ steps.resolve_ref.outputs.ref }}" (first operand uncovered) 0 — passed 1
${{ steps.resolve_ref.outputs.ref || steps.soft.outputs.ref }} (second operand uncovered) 0 — passed 1

So the capability being refused genuinely does not exist on any path: these expressions were never judged and approved, they were never read. There is no second lint in this repo covering ref: guards — pinact/zizmor check uses: pins, not this. The refusal redirects to the two spellings that are judged, both of which I verified still pass unchanged.

I also read the two review threads the work is built on (r3826102157, r3826102163) — public and fetchable — and they name exactly these shapes.

Tests

15 new tests (154 total in this suite, up from 139), each mutation-checked: revert the branch, a named test fails.

Mutation Tests that fail
never return 'unparsed' 7, incl. the three refused spellings + the error text
strict tier stops at first match 4, incl. both concatenated and both || operands
step output does not fall through a lead 3
'unparsed' recorded before the _is_ref_input gate 1 (…outside_a_with_input_is_not_a_checkout)
loose tier on the un-stripped line 1 (…commented_out_step_output_is_not_a_ref_use)
any(verdicts) instead of all(...) 2
name the first operand, not the offending one 1
naive [^,}] flow entry bound 4 pre-existing tests
no dedicated 'unparsed' message 1

Judgment calls

  • Error-message naming with several operands. 'non-leading' is answerable from the line alone, so it still names the exact operand. 'dangling' is a property of the job, not the line, so a multi-output value falls back to the <id>/<out> placeholders rather than pointing the author at a sibling that is fine — the same fallback the ref: >- spelling already takes. Pinned by test_a_multi_operand_error_does_not_name_the_wrong_step. Threading the offending operand through the found tuple would have changed its shape and rippled through fix(workflow-pins): report a step-output ref: whose id cannot resolve, and read the || fallback spelling (BE-8220) #202's tests; not worth it for a shape that has no instance in the tree.
  • Two deliberate behaviour changes beyond the five shapes, both in the fail-loud direction, both tested: (a) a flow-form fallback containing a comma (with: {ref: "${{ … || 'a,b' }}"}) is now 'unparsed' rather than silently dropped — [^,}] is the flow form's real entry boundary, so that spelling genuinely cannot be read there (the block form reads it, unchanged); (b) a continuation value with a literal prefix (ref: >- then refs/heads/${{ … }}) is now a judged site, matching what the block form always did. Neither occurs in this repo's tree.
  • steps_output_ref is no longer called inside the module — check_dir goes through the new _reported_step_output. It stays as the module's line-level accessor and is still exercised by fix(workflow-pins): report a step-output ref: whose id cannot resolve, and read the || fallback spelling (BE-8220) #202's decoy tests; deleting it would lose that coverage.

Verification

  • python3 -m unittest discover -s .github/workflow-pins/tests -p 'test_*.py' — 154 passed, 0 failed (139 before).
  • python3 .github/workflow-pins/check_workflow_pins.py — exit 0, OK — 10 workflow(s) declare workflows_ref, none with a default, every ref checkout guarded (0 exempt). Identical to the baseline on the fix(workflow-pins): report a step-output ref: whose id cannot resolve, and read the || fallback spelling (BE-8220) #202 head. The tree's only real step-output site is cursor-review.yml:456 (bare spelling); zero new failures there.
  • Unaffected suites re-run for safety: cursor-review 194 passed, agents-md-integrity 46 passed, groom 365 passed (1 skipped), bump-callers 424 passed + shellcheck -x clean.

Unexercised artifacts

  • The spike ticket this work descends from, and its findings comment, are on Linear and unreachable from this environment. The design was reconstructed from the two public GitHub review threads above, which I did read and which carry the same five shapes — but I could not confirm the spike's comment adds nothing beyond them.

Out of scope

Teaching the strict reader to actually evaluate format() / && / parenthesized expressions. That is the point of the two-tier design, not a gap in it.

Refs rather than a closing keyword: the driving ticket's own evidence artifact went unexercised (above), so the ticket stays open for a human to confirm nothing in it was missed.

Provenance

…llings (BE-8130)

Cursor-review panel findings on the BE-8130 detector. Each of these was a
site the reader could not see, and each mirrors a precedent already in this
file rather than inventing a rule.

- `_skips_on_empty_output` scanned only lines at the step's key column, so
  `- if: steps.x.outputs.ref != ''` — the condition on the list-marker line —
  was never seen and a correctly guarded checkout was reported unguarded.
  Normalize the marker exactly as `_binding_step_id` already does to read an
  `id:` off it.

- The same reader demanded spaces around `!=`. Actions accepts
  `…outputs.ref!=''`, so the lint was failing the remedy it had just asked
  for. Spacing is now normalized to one space either side; nothing else is,
  so an OR-widened condition and an `if:` on a different output stay refused
  however they are spaced (both pinned by tests).

- `steps_output_ref`'s flow matcher took the LEFTMOST `[{,]` boundary, which
  a comma inside a quoted sibling can meet — planting a decoy `ref:` that is
  string content, resolving to a step no `resolvers` entry holds, and so
  dropping the real checkout out of coverage entirely. It now keeps looking
  past a match that is inside quotes, using the `_outside_quotes` helper
  `_pins_to_job_workflow_sha` already applies to its own flow match. Stopping
  at the decoy the way that function does would lose the site, so this one
  iterates.

- README: the prose claimed coverage whenever the producing step's `env:`
  "binds workflows_ref", which is wider than what the code detects — say
  "in one of the two literal spellings below", which the preconditions block
  then gives.

Two tests that were passing vacuously now assert what they name:
`test_a_resolver_below_its_consumer_is_never_credited` filtered on `guarded`,
which an empty site list satisfies, so it could not tell "reported" from
"dropped" — it now pins the site list and documents that today it is dropped,
though the runtime value is guaranteed ''. And `_binding_step_id`'s own
list-marker normalization had no test at all; deleting it kept the suite
green.
…` fallback spelling (BE-8220)

A step-output `ref:` whose id matches no step declared before the
consuming one used to be silently dropped — at runtime the expression is
'' and checkout takes the default branch unconditionally. A per-job
step-id pre-scan now separates that dangling case (reported, with its
own BE-8215 message) from the legitimately out-of-scope earlier step.
`_STEPS_OUTPUT_BODY` also reads `||` continuations now, judged by
leading-operand order exactly as the input side is.
…dge any leading operand (BE-8220)

Review follow-ups on the BE-8220 dangling/`||` reader.

- **The dangling verdict no longer fires on lines that are not checkouts.**
  The walk asks EVERY line of the job about step-output refs, so a job-level
  `outputs:` mapping — which conventionally sits above `steps:`, and therefore
  above every step id — and a `ref:` line inside a `run:` heredoc both reached
  `_record_steps_output` looking exactly like a dangling consumer. Verified:
  a compliant workflow whose job declares `outputs: {ref: ${{ steps.pick.outputs.ref }}}`
  HARD-FAILED with an error naming a mutable checkout ref where there is no
  checkout at all. `_is_ref_input` gates on the one thing every real site has
  and neither of those does — the `ref:` is an entry of its step's `with:`
  mapping — restoring the old harmless drop for the rest.

- **A non-leading step output gets its own message.** `${{ 'main' ||
  steps.x.outputs.ref }}` was recorded as an ordinary step-output site, so
  `check_dir` printed the two BE-8130 remedies, neither of which can ever make
  it pass; the author applies the printed advice and CI stays red with the
  identical error. It is now a distinct state whose message names operand
  order. Worded for any leading operator, not just `||`, because `${{ A &&
  steps.x.outputs.ref }}` reaches the same state. Judged before step existence:
  when the leading operand wins, the output is never consulted, so "no such
  step" is a red herring.

- **A leading operand `||` falls THROUGH still reaches the output.** `||`
  returns the first TRUTHY operand, so `${{ false || steps.x.outputs.ref }}`
  and `${{ '' || steps.x.outputs.ref }}` resolve to exactly the guarded output
  — yet were reported unguarded unconditionally, with a message no edit can
  satisfy. `_lead_reaches_output` reads the chain instead of testing emptiness.

- **`[^,}]` is the FLOW form's entry boundary and nothing else's.** Carrying it
  into the block and continuation bodies left `|| 'a,b'` matching nothing at
  all — the same fail-open the `||` reader exists to close. Those two bodies
  now use `[^}]`, which still cannot cross a `}}`; the flow form keeps `[^,}]`,
  so a sibling entry is still not swallowed into the ref's value.

- **The dangling error names the actual id and output** rather than the `<id>`
  / `<out>` placeholders, since a typo'd id is the first cause it lists.

- **README**: the deleted "an unrecognized resolver is silent" warning is
  narrowed back rather than dropped — it still holds for the BINDING
  precondition, which the pre-scan did not touch — and the spellings this
  reader still does not recognize (a brace inside the fallback, a parenthesized
  operand, `&&`) are documented as recording no site.

Tests: 139 passed, 0 failed (133 before). Six mutation checks — reverting the
`with:` gate, the `[^}]` widening, the falsey-lead judgment, the non-leading
state, the id interpolation, and the operator-agnostic wording — each fails
exactly the tests that name it and nothing else. `check_workflow_pins.py` is
green on the repo's real tree, 16 sites, 0 exempt.
…ead a `with:` the YAML nests differently (BE-8220)

Round-2 review of #202. Six findings, all reproduced before they were fixed.

Two were false FAILURES — a compliant workflow hard-failing, the worst
direction for a lint people can route around:

- Hoisting the `not leading` verdict above the `step_id not in resolvers`
  block also hoisted it above the OUT-OF-SCOPE early return, so
  `ref: ${{ inputs.pr_sha || steps.detect.outputs.sha }}` resolved from a real
  earlier step this lint has no claim on was reported `'non-leading'` — an
  error demanding an operand reorder that changes runtime semantics. Scope is
  asked first now; it holds in every operand position. The dangling/non-leading
  ordering the hoist existed for is unchanged: a dangling id behind a leading
  operand still gets the operand-order message.
- `_is_ref_input` answered "which key encloses this `ref:`" from indentation
  alone, so a `run: |` heredoc emitting a whole STEP put a `with:` line at
  exactly the indent the backward walk stops on and its script output was
  judged a checkout. `_in_block_scalar` tracks the open `|`/`>` scalar, which
  closes the flow spelling's copy of the same hole.

Two were fail-OPEN drops of the shape the lint exists to close:

- `_WITH_KEY_RE` could not match a `with:` riding the list-item line
  (`- with:` / `ref:` / `uses: checkout` is valid Actions YAML — mapping keys
  are unordered), so the site was dropped entirely. `_dedash` hoists the `- `
  normalization the sibling readers already make into one helper. That exposed
  a second gap it would otherwise have introduced: `_step_bounds` read the
  MARKER's column as the step's key column, so a correctly guarded `- with:`
  checkout could not resolve its own `if:` — now normalized too.
- A `ref: >-` whose expression is split across physical lines folds to one
  value at runtime, so a single newline hid the `|| <literal>` spelling
  BE-8215 just closed. The pending scalar's lines are joined and re-asked,
  only after the single line has failed, so the join can only ever add a site.

Two were README claims that did not match the code, both verified against it:
a LEADING `&&` matches and fails `non-leading` (only a TRAILING one is
silent), and the flow spelling of a fallback whose literal contains a comma
records no site — now named in the unrecognized list beside the brace and
parenthesized operands.

Five regression tests, one per fix.
…reader cannot parse (BE-8253)

The step-output reader gave ONE verdict per line from ONE regex family, and
`ref_checkouts`'s else-arm silently skipped anything it could not match. Five
shapes escaped it, confirmed fail-open through the real CLI: a fixture whose
checkout reads `ref: ${{ steps.x.outputs.ref || format('refs/heads/{0}',
'main') }}` passed the lint (exit 0, "every ref checkout guarded") while the
identical workflow spelled bare correctly failed.

The reader is now two-tier. A LOOSE tier counts `steps.<id>.outputs.<out>`
mentions in the comment-stripped `ref:` VALUE; the STRICT tier reads every
interpolation and every top-level `||` operand in that value. Loose > strict
records the site as `'unparsed'` — truthy, so the unguarded projection needs no
change, same trick as `'dangling'`/`'non-leading'` — and `check_dir` prints a
message naming the two spellings the lint CAN judge. Widening the strict reader
to evaluate `format()`, `&&` or parenthesized operands is deliberately not the
fix: an unknown spelling asks its author for a supported one.

The exhaustive strict tier also closes two partial parses where the operand
nobody judged was a live checkout ref: a value concatenating two interpolations
(the block pattern's greedy `.*` judged only the last) and a `||` chain of two
step outputs (the fallback stretch swallowed the second). Every operand is now
judged independently and the strongest requirement wins — a covered sibling
excuses nothing. A step output ahead of another operand no longer makes it
non-leading, because an unresolved output is `''` and `||` really falls through
it; that leading output is judged in its own right.

Gated by `_is_ref_input` exactly as the other states are: a job-level `outputs:`
mapping or a `run:` heredoc holding an unparseable expression is not a checkout,
and hard-failing one is the false-CI-failure channel BE-8215 already closed once.

The flow entry boundary is now interpolation-aware, since the naive `[^,}]`
bound cut a value mid-expression and hid the rest from both tiers.

Refs BE-8253
@mattmillerai mattmillerai added the agent-coded Authored by the agent-work loop label Aug 21, 2026
@mattmillerai
mattmillerai marked this pull request as ready for review August 21, 2026 00:23
@coderabbitai

coderabbitai Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your current included review allowance is based on your included PR review attempts over the past 7 days.

Next review available in: 8 minutes

Limit details: You’ve used the included review currently available. Your 93 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8a76b1ca-f5dd-474a-b99a-0a177212b08d

📥 Commits

Reviewing files that changed from the base of the PR and between cf308dc and 9c9cf9e.

📒 Files selected for processing (3)
  • .github/workflow-pins/README.md
  • .github/workflow-pins/check_workflow_pins.py
  • .github/workflow-pins/tests/test_check_workflow_pins.py

Comment @coderabbitai help to get the list of available commands.

@mattmillerai mattmillerai added the cursor-review Multi-model cursor review label Aug 21, 2026
…mantic conflict

Both branches touched _record_steps_output and the ref: >- continuation
walk. Combined:
- BE-8220's scope gate (out-of-scope step ids judged before operand
  order, per operand) with BE-8253's multi-operand verdict loop.
- BE-8220's fold-across-physical-lines join for a step output split
  under `ref: >-` with BE-8253's UNPARSED state, so a genuinely
  unparseable expression split across lines is still refused instead of
  being finalized early off one line or silently dropped.
- README: kept BE-8253's two-tier-reader prose, folded in BE-8220's
  leading-vs-trailing && clarification, and corrected the flow-comma
  fallback line to 'unparsed' (BE-8253 changed it from 'no site').

Added a regression test for the fold + UNPARSED interaction, which had
no existing coverage on either branch.
Base automatically changed from matt/be-8220-dangling-step-id-and-or-fallback to main August 21, 2026 03:54
This branch already carried a prior merge (f438b1b) combining BE-8253's
two-tier step-output reader with BE-8220's scope-gate/with-key fixes,
resolved against main's pre-squash commit 1c66f7f. Main has since
squash-merged that same work as #202 (b92f1df) — byte-identical content
under .github/workflow-pins/, different commit hash — so this merge's
conflicts were git failing to recognize already-incorporated content
across the squash, not new upstream changes. Resolved by keeping HEAD's
already-combined version of the three workflow-pins files (verified: its
tree for that directory is unchanged from before this merge, and both
BE-8220's and BE-8253's added symbols are present). Also picked up two
unrelated, cleanly-auto-merged main changes: the cursor-review SHA bump
(#200) and the pr-size rename-discount fix (#196).

Verified: full workflow-pins test suite (160 tests) and the lint itself
pass; pr-size go tests pass except one pre-existing, pre-merge failure
(TestResolveMergeBaseWithAdvancedBaseBranch, environment-dependent,
reproduces identically on an unmodified origin/main checkout — unrelated
to this merge).
Port BE-8146's steps.<id>/outputs.<out> bracket-accessor support
(id_idx/out_idx) into BE-8253's rewritten two-tier LOOSE/STRICT
step-output reader, which otherwise would have silently dropped
bracket-spelled step outputs from lint coverage entirely.
Resolves a semantic conflict in _record_steps_output between this
branch's multi-operand `sites`/UNPARSED reader (BE-8253) and main's
BE-8221, which dropped the fail-closed-resolver exemption so coverage
comes ONLY from the consuming step's own exact `if:`. Kept BE-8253's
per-operand verdict loop but removed the `resolvers[step_id] or` shortcut
so every operand — not just the first — needs its own covering `if:`.

Updated the two multi-operand tests whose HARD_RESOLVER fixtures relied
on the now-removed exemption to instead assert the checkout is still
reported when no `if:` covers every operand, and added `EXACT_IF`-backed
cases proving one covered operand does not excuse an uncovered sibling.

Tests: all 216 workflow-pins tests pass; `check_workflow_pins.py` runs
clean over the repo's own workflows.
@mattmillerai
mattmillerai merged commit 6a374b7 into main Aug 21, 2026
5 checks passed
@mattmillerai
mattmillerai deleted the matt/be-8253-workflow-pins-unparsed-ref branch August 21, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant