Repository navigation
feat(linear-ticket): make the warn-only pilot fail the check without blocking the merge - #186
Conversation
…blocking the merge The warn-only pilot publishes a green `linear-ticket` status on every outcome, so its verdict lives only in a marker comment among a dozen other bot comments. On Comfy-Org/cloud #6690 that is exactly what happened: the gate correctly found three referenced-but-unlinked identifiers and nobody noticed, because a passing check is a check nobody reads. A pilot's whole purpose is observation, and it was observing quietly. Add `soft-fail` (default true), read only when `enforce: false`: warn-only now publishes a red `failure` status, so the check shows as failing in the PR's check list exactly as enforcing would — while blocking nothing, because blocking is branch protection requiring the `linear-ticket` context and neither live caller does. The marker comment carries an explicit "this does not block the merge" note so a red X is never read as a merge block, and `soft-fail: false` restores the previous silent always-green behaviour. The job itself stays green in warn-only. This validator runs on the default branch off workflow_run, so its run is not attached to the PR — the commit status is the only PR-visible signal, and a red default-branch run for a deliberately non-gating check would read as "main is broken" in the Actions tab: louder, in the wrong place. The mode decision moves out of validate.py into a pure `lib.failure_outcome()` (verdict, status state, description, exit code, advisory flag), matching the file's contract that every decision is testable without a network call. New tests pin the invariants that matter: only enforce ever exits nonzero, the "does not block" note never rides along with a gating verdict, and every mode's status description fits GitHub's 140-char limit. Verified `linear-ticket` is absent from the required-status contexts of both live callers' rulesets, so the red status is advisory on both today.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 31 minutes Limit details: You’ve used the included review currently available. Your 94 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
Included review availability: 0 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour. 📝 WalkthroughWalkthroughThe workflow now supports configurable warn-only reporting. The validator centralizes failure outcomes, publishes advisory or enforcing statuses, and preserves terminal status-write failures. Documentation and tests cover the new modes. ChangesLinear ticket reporting
Sequence Diagram(s)sequenceDiagram
participant Workflow
participant Validator
participant OutcomeLogic
participant GitHubStatus
Workflow->>Validator: pass SOFT_FAIL
Validator->>OutcomeLogic: classify failure category
OutcomeLogic-->>Validator: return reporting outcome
Validator->>GitHubStatus: publish terminal status
Validator-->>Workflow: return computed exit code
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 7 finding(s).
| Severity | Count |
|---|---|
| 🟠 High | 1 |
| 🟡 Medium | 3 |
| 🟢 Low | 3 |
Panel: 8/8 reviewers contributed findings.
…iguration Addresses the cursor-review panel on #186. - The advisory note no longer asserts "This does not block the merge". The validator never reads the caller's ruleset, so that claim is false in exactly the configuration the docs call a footgun (warn-only + the `linear-ticket` context already required) — where the red check IS blocking the author, and the note sent them to debug the wrong thing. It now states what is knowable (this workflow is not what blocks; your ruleset is) and names the misconfiguration to report. The soft-fail verdict string drops "not blocking" for the factual "warn-only". - Status descriptions are category-aware. `infra_error` means Linear could not be queried, so "no linked Linear issue (infra_error)" was a diagnosis the run never made; the advisory note likewise no longer tells an author to link a ticket to fix a broken credential. - An ABSENT `SOFT_FAIL` env now means the SILENT variant, not the loud one. linear-ticket.yml always passes it, so absent can only mean the `uses:` and `workflows_ref` pins have skewed apart — and a caller that cannot express the input must not be silently upgraded from a green status to a red one. - "the job never exits nonzero" narrowed to "a failing VERDICT never exits the job nonzero" in the workflow input, both READMEs, the caller doc and the validate.py contract: warn-only still exits 1 on a failed terminal status write, a missing token/repo, malformed team-keys, or a non-pull_request event. - New "Upgrading an existing `enforce: false` caller" section so the default `soft-fail: true` flip is documented at the callers it changes, with the one-line way to keep the old always-green behaviour. Tests cover the hedged note, the category-aware copy, the env parsing, and a keyset guard so _STATUS_HEADLINE cannot drift from _GUIDANCE.
Why
On Comfy-Org/cloud#6690 the gate worked perfectly — it found three referenced-but-unlinked identifiers (
PILOT-449, BE-7081, BE-3033) and explained exactly how to fix them. Nobody noticed. In warn-only the status publishes green, so the verdict lives only in a marker comment sitting among a dozen other bot comments on a busy PR.A pilot exists to be observed. A passing check is a check nobody reads.
What changes
New
soft-failinput (boolean, defaulttrue), read only whenenforce: false:enforce: truefailureenforce: false,soft-fail: true(new default)failureenforce: false,soft-fail: falsesuccessNothing becomes a blocker. Blocking is branch protection requiring the
linear-ticketstatus context — never this workflow's call. I checked both live callers before writing this: neitherComfy-Org/cloud's "Requirements to Merge to Main" ruleset norComfy-Org/comfy-infra'smainruleset listslinear-ticketamong its required contexts, so the red status is advisory on both today. A non-required failing check leaves the merge button live.The marker comment now says so, so a red X is never misread as a merge block. The wording is
deliberately hedged — the validator never reads the caller's ruleset, so it cannot assert
that nothing is blocked (see the review round below):
…plus one category-aware sentence: link the ticket, or — for
infra_error— there is nothing tofix on the PR at all. The headline becomes
❌ fail (exists_not_linked) — warn-only.Two deliberate calls
The job stays green in warn-only. This validator runs on the default branch off
workflow_run, so its run is never attached to the PR — the commit status is the only PR-visible signal. A red default-branch run for a check that is deliberately not gating would read as "main is broken" in the Actions tab: louder, in the wrong place. Onlyenforce: trueexits nonzero.Default
true, so the SHA bump alone delivers it.bump-callers.shmoves pins, not inputs; defaulting off would mean hand-editing every caller. The one thing this makes possible is a footgun, documented at the input, in the docs table, and as a gotcha:enforce: false+soft-fail: true+ the context already required would block merges. Don't require the context before flippingenforce: true— which was already the documented rollout order (step 6, after step 5).Rollout ladder, restated
enforce: true, still not required → PR-visible result unchanged; the validator's own run now also goes red, so systemic breakage shows in the Actions tab.linear-ticketto the ruleset. Only now does anything block.Implementation
The mode decision moves out of
validate.pyinto a purelib.failure_outcome()returning(verdict, state, description, exit_code, advisory)—lib.py's stated contract is that every decision is testable with no network call, and this was the last one inline in the orchestrator.New tests pin the invariants worth pinning: only enforce ever exits nonzero, the advisory note never rides along with a gating verdict, enforce ignores
soft-failentirely, and every mode's status description fits GitHub's 140-char truncation (a description that only survives truncation would silently lose the category).One behaviour deliberately unchanged: a failed terminal status write is still exit 1 even in warn-only. Soft-fail's whole point is that the red status lands; a missing write leaves the previous status standing as this commit's answer.
Tests
Review round (cursor-review panel, all 7 threads)
Four fixes landed in
feca179; three findings answered on the thread without a code change.claim about the caller's ruleset, which this validator never reads — and it is false in
exactly the footgun this PR documents. Rewritten to state only what is knowable, and to name
the misconfiguration to report if the merge button really is blocked.
infra_errormeans Linear could not bequeried, so
no linked Linear issue (infra_error)was a diagnosis the run never made; thenote likewise no longer tells an author to link a ticket to fix a broken credential. (This
string predates the PR — soft-fail just promoted it to a louder surface.)
SOFT_FAILenv now means the silent variant. The workflow always passes it,so absent can only mean the
uses:andworkflows_refpins skewed apart — and a caller thatcannot even express the input must not be silently upgraded from green to red. Pin skew is
now inert in the safe direction, matching
ENFORCE's fail-closed default.nonzero" across the input description, both READMEs, the caller doc and the
validate.pycontract — warn-only still exits 1 on a failed terminal status write, a missing token/repo,
malformed
team-keys, or a non-pull_requestevent.enforce: falsecaller" doc section, plus a BEHAVIOUR CHANGEparagraph on the input itself, so the green→red flip is documented at the callers it changes
along with the one-line
soft-fail: falseopt-out. The default staystrue— see below.Answered without a change: the
soft-failname (the polarity matches Buildkite'ssoft_fail— a real failure that does not fail the build; happy to rename while no caller setsit yet), the comment-before-status write order (reversing it trades an unexplained red check
for an unlanded one — pre-existing since #183), and the default
trueitself.Note
One open call for the merger.
soft-faildefaults totrue, so taking a SHA bump flipsan existing
enforce: falsecaller from a green status to a red (still non-required) one.That is the intent — a green pilot is a pilot nobody reads — and it is now documented at the
input and in the caller doc rather than shipping silently. Flipping the default to
falseandmaking the loud mode opt-in is a one-line change if you would rather ship it additive.
Follow-up (not in this PR)
The auto-bump PR into
Comfy-Org/cloudwill move the SHA only, leavinglinear-ticket-validate.yml's header comment saying "WARN-ONLY pilot (enforce: false): every outcome is green" — true before this change, stale after. Happy to open a one-line comment fix there; say the word.Provenance
Authored by: agent-work loop
Verified:
python3 -m unittest discover -s scripts/linear-ticket/tests -p 'test_*.py'— 63 passed;python3 -m py_compile scripts/linear-ticket/{lib,validate}.py— ok. All PR checks green on the pre-review-round commit; the review-round commit re-runs them.Deviations: Three of the seven panel findings were answered on the thread rather than fixed — the
soft-failinput name, the comment-before-status write order, and thedefault: truebehaviour change (mitigated by documentation rather than by flipping the default). Rationale on each thread.AGENTS.mdfails the repo's own 200-line integrity ceiling at 273 lines, but that is pre-existing onmain, is not a check on this PR, and was left untouched.🤖 Generated with Claude Code