Repository navigation
fix(workflow-pins): fail loud on a ref: expression the step-output reader cannot parse (BE-8253) - #203
Conversation
…llings (BE-8130)
Cursor-review panel findings on the BE-8130 detector. Each of these was a
site the reader could not see, and each mirrors a precedent already in this
file rather than inventing a rule.
- `_skips_on_empty_output` scanned only lines at the step's key column, so
`- if: steps.x.outputs.ref != ''` — the condition on the list-marker line —
was never seen and a correctly guarded checkout was reported unguarded.
Normalize the marker exactly as `_binding_step_id` already does to read an
`id:` off it.
- The same reader demanded spaces around `!=`. Actions accepts
`…outputs.ref!=''`, so the lint was failing the remedy it had just asked
for. Spacing is now normalized to one space either side; nothing else is,
so an OR-widened condition and an `if:` on a different output stay refused
however they are spaced (both pinned by tests).
- `steps_output_ref`'s flow matcher took the LEFTMOST `[{,]` boundary, which
a comma inside a quoted sibling can meet — planting a decoy `ref:` that is
string content, resolving to a step no `resolvers` entry holds, and so
dropping the real checkout out of coverage entirely. It now keeps looking
past a match that is inside quotes, using the `_outside_quotes` helper
`_pins_to_job_workflow_sha` already applies to its own flow match. Stopping
at the decoy the way that function does would lose the site, so this one
iterates.
- README: the prose claimed coverage whenever the producing step's `env:`
"binds workflows_ref", which is wider than what the code detects — say
"in one of the two literal spellings below", which the preconditions block
then gives.
Two tests that were passing vacuously now assert what they name:
`test_a_resolver_below_its_consumer_is_never_credited` filtered on `guarded`,
which an empty site list satisfies, so it could not tell "reported" from
"dropped" — it now pins the site list and documents that today it is dropped,
though the runtime value is guaranteed ''. And `_binding_step_id`'s own
list-marker normalization had no test at all; deleting it kept the suite
green.
…` fallback spelling (BE-8220) A step-output `ref:` whose id matches no step declared before the consuming one used to be silently dropped — at runtime the expression is '' and checkout takes the default branch unconditionally. A per-job step-id pre-scan now separates that dangling case (reported, with its own BE-8215 message) from the legitimately out-of-scope earlier step. `_STEPS_OUTPUT_BODY` also reads `||` continuations now, judged by leading-operand order exactly as the input side is.
…dge any leading operand (BE-8220)
Review follow-ups on the BE-8220 dangling/`||` reader.
- **The dangling verdict no longer fires on lines that are not checkouts.**
The walk asks EVERY line of the job about step-output refs, so a job-level
`outputs:` mapping — which conventionally sits above `steps:`, and therefore
above every step id — and a `ref:` line inside a `run:` heredoc both reached
`_record_steps_output` looking exactly like a dangling consumer. Verified:
a compliant workflow whose job declares `outputs: {ref: ${{ steps.pick.outputs.ref }}}`
HARD-FAILED with an error naming a mutable checkout ref where there is no
checkout at all. `_is_ref_input` gates on the one thing every real site has
and neither of those does — the `ref:` is an entry of its step's `with:`
mapping — restoring the old harmless drop for the rest.
- **A non-leading step output gets its own message.** `${{ 'main' ||
steps.x.outputs.ref }}` was recorded as an ordinary step-output site, so
`check_dir` printed the two BE-8130 remedies, neither of which can ever make
it pass; the author applies the printed advice and CI stays red with the
identical error. It is now a distinct state whose message names operand
order. Worded for any leading operator, not just `||`, because `${{ A &&
steps.x.outputs.ref }}` reaches the same state. Judged before step existence:
when the leading operand wins, the output is never consulted, so "no such
step" is a red herring.
- **A leading operand `||` falls THROUGH still reaches the output.** `||`
returns the first TRUTHY operand, so `${{ false || steps.x.outputs.ref }}`
and `${{ '' || steps.x.outputs.ref }}` resolve to exactly the guarded output
— yet were reported unguarded unconditionally, with a message no edit can
satisfy. `_lead_reaches_output` reads the chain instead of testing emptiness.
- **`[^,}]` is the FLOW form's entry boundary and nothing else's.** Carrying it
into the block and continuation bodies left `|| 'a,b'` matching nothing at
all — the same fail-open the `||` reader exists to close. Those two bodies
now use `[^}]`, which still cannot cross a `}}`; the flow form keeps `[^,}]`,
so a sibling entry is still not swallowed into the ref's value.
- **The dangling error names the actual id and output** rather than the `<id>`
/ `<out>` placeholders, since a typo'd id is the first cause it lists.
- **README**: the deleted "an unrecognized resolver is silent" warning is
narrowed back rather than dropped — it still holds for the BINDING
precondition, which the pre-scan did not touch — and the spellings this
reader still does not recognize (a brace inside the fallback, a parenthesized
operand, `&&`) are documented as recording no site.
Tests: 139 passed, 0 failed (133 before). Six mutation checks — reverting the
`with:` gate, the `[^}]` widening, the falsey-lead judgment, the non-leading
state, the id interpolation, and the operator-agnostic wording — each fails
exactly the tests that name it and nothing else. `check_workflow_pins.py` is
green on the repo's real tree, 16 sites, 0 exempt.
…ead a `with:` the YAML nests differently (BE-8220) Round-2 review of #202. Six findings, all reproduced before they were fixed. Two were false FAILURES — a compliant workflow hard-failing, the worst direction for a lint people can route around: - Hoisting the `not leading` verdict above the `step_id not in resolvers` block also hoisted it above the OUT-OF-SCOPE early return, so `ref: ${{ inputs.pr_sha || steps.detect.outputs.sha }}` resolved from a real earlier step this lint has no claim on was reported `'non-leading'` — an error demanding an operand reorder that changes runtime semantics. Scope is asked first now; it holds in every operand position. The dangling/non-leading ordering the hoist existed for is unchanged: a dangling id behind a leading operand still gets the operand-order message. - `_is_ref_input` answered "which key encloses this `ref:`" from indentation alone, so a `run: |` heredoc emitting a whole STEP put a `with:` line at exactly the indent the backward walk stops on and its script output was judged a checkout. `_in_block_scalar` tracks the open `|`/`>` scalar, which closes the flow spelling's copy of the same hole. Two were fail-OPEN drops of the shape the lint exists to close: - `_WITH_KEY_RE` could not match a `with:` riding the list-item line (`- with:` / `ref:` / `uses: checkout` is valid Actions YAML — mapping keys are unordered), so the site was dropped entirely. `_dedash` hoists the `- ` normalization the sibling readers already make into one helper. That exposed a second gap it would otherwise have introduced: `_step_bounds` read the MARKER's column as the step's key column, so a correctly guarded `- with:` checkout could not resolve its own `if:` — now normalized too. - A `ref: >-` whose expression is split across physical lines folds to one value at runtime, so a single newline hid the `|| <literal>` spelling BE-8215 just closed. The pending scalar's lines are joined and re-asked, only after the single line has failed, so the join can only ever add a site. Two were README claims that did not match the code, both verified against it: a LEADING `&&` matches and fails `non-leading` (only a TRAILING one is silent), and the flow spelling of a fallback whose literal contains a comma records no site — now named in the unrecognized list beside the brace and parenthesized operands. Five regression tests, one per fix.
…reader cannot parse (BE-8253)
The step-output reader gave ONE verdict per line from ONE regex family, and
`ref_checkouts`'s else-arm silently skipped anything it could not match. Five
shapes escaped it, confirmed fail-open through the real CLI: a fixture whose
checkout reads `ref: ${{ steps.x.outputs.ref || format('refs/heads/{0}',
'main') }}` passed the lint (exit 0, "every ref checkout guarded") while the
identical workflow spelled bare correctly failed.
The reader is now two-tier. A LOOSE tier counts `steps.<id>.outputs.<out>`
mentions in the comment-stripped `ref:` VALUE; the STRICT tier reads every
interpolation and every top-level `||` operand in that value. Loose > strict
records the site as `'unparsed'` — truthy, so the unguarded projection needs no
change, same trick as `'dangling'`/`'non-leading'` — and `check_dir` prints a
message naming the two spellings the lint CAN judge. Widening the strict reader
to evaluate `format()`, `&&` or parenthesized operands is deliberately not the
fix: an unknown spelling asks its author for a supported one.
The exhaustive strict tier also closes two partial parses where the operand
nobody judged was a live checkout ref: a value concatenating two interpolations
(the block pattern's greedy `.*` judged only the last) and a `||` chain of two
step outputs (the fallback stretch swallowed the second). Every operand is now
judged independently and the strongest requirement wins — a covered sibling
excuses nothing. A step output ahead of another operand no longer makes it
non-leading, because an unresolved output is `''` and `||` really falls through
it; that leading output is judged in its own right.
Gated by `_is_ref_input` exactly as the other states are: a job-level `outputs:`
mapping or a `run:` heredoc holding an unparseable expression is not a checkout,
and hard-failing one is the false-CI-failure channel BE-8215 already closed once.
The flow entry boundary is now interpolation-aware, since the naive `[^,}]`
bound cut a value mid-expression and hid the rest from both tiers.
Refs BE-8253
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 8 minutes Limit details: You’ve used the included review currently available. Your 93 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Comment |
…mantic conflict Both branches touched _record_steps_output and the ref: >- continuation walk. Combined: - BE-8220's scope gate (out-of-scope step ids judged before operand order, per operand) with BE-8253's multi-operand verdict loop. - BE-8220's fold-across-physical-lines join for a step output split under `ref: >-` with BE-8253's UNPARSED state, so a genuinely unparseable expression split across lines is still refused instead of being finalized early off one line or silently dropped. - README: kept BE-8253's two-tier-reader prose, folded in BE-8220's leading-vs-trailing && clarification, and corrected the flow-comma fallback line to 'unparsed' (BE-8253 changed it from 'no site'). Added a regression test for the fold + UNPARSED interaction, which had no existing coverage on either branch.
This branch already carried a prior merge (f438b1b) combining BE-8253's two-tier step-output reader with BE-8220's scope-gate/with-key fixes, resolved against main's pre-squash commit 1c66f7f. Main has since squash-merged that same work as #202 (b92f1df) — byte-identical content under .github/workflow-pins/, different commit hash — so this merge's conflicts were git failing to recognize already-incorporated content across the squash, not new upstream changes. Resolved by keeping HEAD's already-combined version of the three workflow-pins files (verified: its tree for that directory is unchanged from before this merge, and both BE-8220's and BE-8253's added symbols are present). Also picked up two unrelated, cleanly-auto-merged main changes: the cursor-review SHA bump (#200) and the pr-size rename-discount fix (#196). Verified: full workflow-pins test suite (160 tests) and the lint itself pass; pr-size go tests pass except one pre-existing, pre-merge failure (TestResolveMergeBaseWithAdvancedBaseBranch, environment-dependent, reproduces identically on an unmodified origin/main checkout — unrelated to this merge).
Port BE-8146's steps.<id>/outputs.<out> bracket-accessor support (id_idx/out_idx) into BE-8253's rewritten two-tier LOOSE/STRICT step-output reader, which otherwise would have silently dropped bracket-spelled step outputs from lint coverage entirely.
Resolves a semantic conflict in _record_steps_output between this branch's multi-operand `sites`/UNPARSED reader (BE-8253) and main's BE-8221, which dropped the fail-closed-resolver exemption so coverage comes ONLY from the consuming step's own exact `if:`. Kept BE-8253's per-operand verdict loop but removed the `resolvers[step_id] or` shortcut so every operand — not just the first — needs its own covering `if:`. Updated the two multi-operand tests whose HARD_RESOLVER fixtures relied on the now-removed exemption to instead assert the checkout is still reported when no `if:` covers every operand, and added `EXACT_IF`-backed cases proving one covered operand does not excuse an uncovered sibling. Tests: all 216 workflow-pins tests pass; `check_workflow_pins.py` runs clean over the repo's own workflows.
Update: #202 (this PR's former base) has merged to
main; GitHub retargeted this PR tomaindirectly, so it is no longer stacked — the banner that used to be here is stale and removed. This branch was then merged with the currentmaintip to pick up #202/#197/#200/#196; see the Provenance section below for what that merge verified.ELI-5
The workflow-pins lint follows a checkout's
ref:back to the step that produced it, so a job that must never fail still can't silently check out mutable scripts. It read thatref:with one regex and gave one answer per line — so any spelling it couldn't chew recorded no site at all and the file passed. Now it asks two questions: "does thisref:mention a step output at all?" and "how many of those could I actually read?" If the first number is bigger than the second, the lint says so out loud instead of shrugging. And where aref:names several step outputs, every one of them is judged, not just the first the regex happened to land on.What was wrong
ref_checkouts's else-arm dropped any line_steps_output_matchcouldn't match. Five shapes escaped, all confirmed fail-open through the real CLI (see Evidence):ref: ${{ steps.x.outputs.ref || format('refs/heads/{0}', 'main') }}— the}in{0}breaks the[^}]*?fallback stretch.ref: ${{ (steps.x.outputs.ref) || 'main' }}—)after the output name matches neither||nor}}.ref: ${{ steps.x.outputs.ref && 'main' }}— only||was accepted. Runtime is wrong in both arms: truthy checks out at the mutable literal, falsey resolves to''andactions/checkoutreads''as the default branch.ref: "${{ steps.a.outputs.ref }}${{ steps.b.outputs.ref }}"— the block pattern's greedy.*landed on the LAST interpolation;awas never judged even though both feed the value.ref: ${{ steps.a.outputs.ref || steps.b.outputs.ref }}— the fallback stretch swallowedb; a coveredapassed the whole site.What changed
Two-tier reader. LOOSE counts
steps.<id>.outputs.<out>mentions in the comment-strippedref:value (soref: main # was ${{ steps.x.outputs.ref }}cannot fire it). STRICT runs the existing patterns over every interpolation and every top-level||operand, yielding a list rather than one match. Loose > strict records the site withvia_step_output = 'unparsed'— truthy, sounguarded_ref_checkoutsneeds no change, the same trick'dangling'/'non-leading'already use — andcheck_dirprints a message naming the two spellings the lint can judge. Widening the strict reader to evaluateformat()/&&/ parenthesized expressions is deliberately not the fix: an unknown spelling asks its author for a supported one rather than growing an expression parser inside the lint.All operands judged, strongest requirement wins.
_record_steps_outputtakes the list; each(id, out)must independently pass (declared as an earlier step, covered by a fail-closed resolver or the exact non-emptyif:, reachable by operand order). A covered sibling excuses nothing.'non-leading'outranks'dangling'for the reason it is judged first: no change to the id or the step order fixes operand order._lead_reaches_outputnow falls through a leading step output. An unresolved output is'', which is falsey, soA || Breally can resolve toB. That second operand only became visible with the exhaustive reader, and it is judged in its own right — so nothing is excused by falling through. (Chesterton's-fence note: the check was written in #202 when only one step output per expression was ever visible, so this case could not arise; the falsey-literal behaviour and every_FALSEY_OPERANDSanswer are unchanged, pinned bytest_a_falsey_leading_operand_still_reaches_the_output.)Gated by
_is_ref_input, exactly as the other states are — a job-leveloutputs:mapping or arun:heredoc holding an unparseable expression is not a checkout. That is the false-CI-failure channel BE-8215 already had to close once; mutation-checking confirms reordering the gate breaks a named test.Flow entry boundary is interpolation-aware. The naive
[^,}]bound cut a value mid-expression and hid the rest from both tiers.${{ … }}is atomic now; outside one, the first,/}still ends the entry, so a sibling entry still cannot be misread as the ref (pinned by the pre-existing decoy tests — reverting it fails four of them).Applied to all three spellings (block, flow, continuation under
ref: >-/ref: |).Evidence — the refusal path is empirically justified, not assumed
Before shipping a diff whose user-facing outcome is "this lint refuses your expression", I ran each shape through both CLIs (base = #202 head
b68df60, new = this branch), against a fixture workflow with a real resolver and checkout:ref:value${{ steps.resolve_ref.outputs.ref }}(bare, uncovered)${{ steps.resolve_ref.outputs.ref || format('refs/heads/{0}', 'main') }}${{ (steps.resolve_ref.outputs.ref) || 'main' }}${{ steps.resolve_ref.outputs.ref && 'main' }}"${{ steps.soft.outputs.ref }}${{ steps.resolve_ref.outputs.ref }}"(first operand uncovered)${{ steps.resolve_ref.outputs.ref || steps.soft.outputs.ref }}(second operand uncovered)So the capability being refused genuinely does not exist on any path: these expressions were never judged and approved, they were never read. There is no second lint in this repo covering
ref:guards —pinact/zizmorcheckuses:pins, not this. The refusal redirects to the two spellings that are judged, both of which I verified still pass unchanged.I also read the two review threads the work is built on (r3826102157, r3826102163) — public and fetchable — and they name exactly these shapes.
Tests
15 new tests (154 total in this suite, up from 139), each mutation-checked: revert the branch, a named test fails.
'unparsed'||operands'unparsed'recorded before the_is_ref_inputgate…outside_a_with_input_is_not_a_checkout)…commented_out_step_output_is_not_a_ref_use)any(verdicts)instead ofall(...)[^,}]flow entry bound'unparsed'messageJudgment calls
'non-leading'is answerable from the line alone, so it still names the exact operand.'dangling'is a property of the job, not the line, so a multi-output value falls back to the<id>/<out>placeholders rather than pointing the author at a sibling that is fine — the same fallback theref: >-spelling already takes. Pinned bytest_a_multi_operand_error_does_not_name_the_wrong_step. Threading the offending operand through thefoundtuple would have changed its shape and rippled through fix(workflow-pins): report a step-outputref:whose id cannot resolve, and read the||fallback spelling (BE-8220) #202's tests; not worth it for a shape that has no instance in the tree.with: {ref: "${{ … || 'a,b' }}"}) is now'unparsed'rather than silently dropped —[^,}]is the flow form's real entry boundary, so that spelling genuinely cannot be read there (the block form reads it, unchanged); (b) a continuation value with a literal prefix (ref: >-thenrefs/heads/${{ … }}) is now a judged site, matching what the block form always did. Neither occurs in this repo's tree.steps_output_refis no longer called inside the module —check_dirgoes through the new_reported_step_output. It stays as the module's line-level accessor and is still exercised by fix(workflow-pins): report a step-outputref:whose id cannot resolve, and read the||fallback spelling (BE-8220) #202's decoy tests; deleting it would lose that coverage.Verification
python3 -m unittest discover -s .github/workflow-pins/tests -p 'test_*.py'— 154 passed, 0 failed (139 before).python3 .github/workflow-pins/check_workflow_pins.py— exit 0,OK — 10 workflow(s) declare workflows_ref, none with a default, every ref checkout guarded (0 exempt). Identical to the baseline on the fix(workflow-pins): report a step-outputref:whose id cannot resolve, and read the||fallback spelling (BE-8220) #202 head. The tree's only real step-output site iscursor-review.yml:456(bare spelling); zero new failures there.shellcheck -xclean.Unexercised artifacts
Out of scope
Teaching the strict reader to actually evaluate
format()/&&/ parenthesized expressions. That is the point of the two-tier design, not a gap in it.Refsrather than a closing keyword: the driving ticket's own evidence artifact went unexercised (above), so the ticket stays open for a human to confirm nothing in it was missed.Provenance
main(b92f1df, which squash-landed fix(workflow-pins): report a step-outputref:whose id cannot resolve, and read the||fallback spelling (BE-8220) #202/fix(workflow-pins): stop the step-output reader failing compliantif:spellings (BE-8130) #197/ci: bump cursor-review to github-workflows@00b0eef #200/fix(pr-size): require a rename's SOURCE to be linguist-generated too (BE-8137) #196) into this branch to clear aCONFLICTING→CLEANmerge state; the conflicts were git failing to recognize content already combined by a prior merge across a squash-merge (identical tree, different commit hash) — resolved by keeping that already-combined content, verified both BE-8220's and BE-8253's added symbols are present. Post-merge: workflow-pins unittest 160 passed / 0 failed;check_workflow_pins.pyon the real tree exit 0, 10 workflows, 0 unguarded;pr-sizego tests pass except one pre-existing, environment-dependent failure (TestResolveMergeBaseWithAdvancedBaseBranch) reproduced identically on an unmodifiedmaincheckout, unrelated to this change; base-vs-branch CLI comparison on 6 fixtureref:spellings (table above)