Skip to content

feat(linear-ticket): make the warn-only pilot fail the check without blocking the merge - #186

Merged
mattmillerai merged 2 commits into
mainfrom
matt/linear-ticket-soft-fail
Aug 20, 2026
Merged

mattmillerai merged 2 commits into
mainfrom
matt/linear-ticket-soft-fail

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Aug 20, 2026 •

Copy link
Copy Markdown
Member

Why

On Comfy-Org/cloud#6690 the gate worked perfectly — it found three referenced-but-unlinked identifiers (PILOT-449, BE-7081, BE-3033) and explained exactly how to fix them. Nobody noticed. In warn-only the status publishes green, so the verdict lives only in a marker comment sitting among a dozen other bot comments on a busy PR.

A pilot exists to be observed. A passing check is a check nobody reads.

What changes

New soft-fail input (boolean, default true), read only when enforce: false:

mode status job exit on the PR
enforce: true 🔴 failure 1 red check (blocks once the context is required)
enforce: false, soft-fail: true (new default) 🔴 failure 0 red check + comment — blocks nothing
enforce: false, soft-fail: false 🟢 success 0 green; verdict only in summary + comment (previous behaviour)

Nothing becomes a blocker. Blocking is branch protection requiring the linear-ticket status context — never this workflow's call. I checked both live callers before writing this: neither Comfy-Org/cloud's "Requirements to Merge to Main" ruleset nor Comfy-Org/comfy-infra's main ruleset lists linear-ticket among its required contexts, so the red status is advisory on both today. A non-required failing check leaves the merge button live.

The marker comment now says so, so a red X is never misread as a merge block. The wording is
deliberately hedged — the validator never reads the caller's ruleset, so it cannot assert
that nothing is blocked (see the review round below):

⚠️ Advisory — this check is warn-only. It is reported as failing so it shows up in the PR's check list, but nothing this workflow does blocks a merge: what blocks is your repository's ruleset requiring the linear-ticket status context, and a warn-only pilot is meant to run before that context is required. If your merge button is actually blocked on linear-ticket, then the context is already required while the pilot is still warn-only — tell the repository owners, because that combination is a misconfiguration.

…plus one category-aware sentence: link the ticket, or — for infra_error — there is nothing to
fix on the PR at all. The headline becomes ❌ fail (exists_not_linked) — warn-only.

Two deliberate calls

The job stays green in warn-only. This validator runs on the default branch off workflow_run, so its run is never attached to the PR — the commit status is the only PR-visible signal. A red default-branch run for a check that is deliberately not gating would read as "main is broken" in the Actions tab: louder, in the wrong place. Only enforce: true exits nonzero.

Default true, so the SHA bump alone delivers it. bump-callers.sh moves pins, not inputs; defaulting off would mean hand-editing every caller. The one thing this makes possible is a footgun, documented at the input, in the docs table, and as a gotcha: enforce: false + soft-fail: true + the context already required would block merges. Don't require the context before flipping enforce: true — which was already the documented rollout order (step 6, after step 5).

Rollout ladder, restated

  1. Land warn-only, context not required → red-but-advisory check. (← the pilot now lives here)
  2. Observe.
  3. enforce: true, still not required → PR-visible result unchanged; the validator's own run now also goes red, so systemic breakage shows in the Actions tab.
  4. Add linear-ticket to the ruleset. Only now does anything block.

Implementation

The mode decision moves out of validate.py into a pure lib.failure_outcome() returning (verdict, state, description, exit_code, advisory) — lib.py's stated contract is that every decision is testable with no network call, and this was the last one inline in the orchestrator.

New tests pin the invariants worth pinning: only enforce ever exits nonzero, the advisory note never rides along with a gating verdict, enforce ignores soft-fail entirely, and every mode's status description fits GitHub's 140-char truncation (a description that only survives truncation would silently lose the category).

One behaviour deliberately unchanged: a failed terminal status write is still exit 1 even in warn-only. Soft-fail's whole point is that the red status lands; a missing write leaves the previous status standing as this commit's answer.

Tests

python3 -m unittest discover -s scripts/linear-ticket/tests -p 'test_*.py'   # 63 passed (was 46)
python3 -m py_compile scripts/linear-ticket/{lib,validate}.py                # ok
python3 .github/workflow-pins/check_workflow_pins.py                         # 10 workflows, 0 defaults
bash .github/bump-callers/tests/test_paths_contract.sh                       # 66 passed

Review round (cursor-review panel, all 7 threads)

Four fixes landed in feca179; three findings answered on the thread without a code change.

  • The advisory note asserted something it cannot know. "This does not block the merge" is a
    claim about the caller's ruleset, which this validator never reads — and it is false in
    exactly the footgun this PR documents. Rewritten to state only what is knowable, and to name
    the misconfiguration to report if the merge button really is blocked.
  • Status descriptions are now category-aware. infra_error means Linear could not be
    queried, so no linked Linear issue (infra_error) was a diagnosis the run never made; the
    note likewise no longer tells an author to link a ticket to fix a broken credential. (This
    string predates the PR — soft-fail just promoted it to a louder surface.)
  • An absent SOFT_FAIL env now means the silent variant. The workflow always passes it,
    so absent can only mean the uses: and workflows_ref pins skewed apart — and a caller that
    cannot even express the input must not be silently upgraded from green to red. Pin skew is
    now inert in the safe direction, matching ENFORCE's fail-closed default.
  • "the job never exits nonzero" narrowed to "a failing verdict never exits the job
    nonzero"
    across the input description, both READMEs, the caller doc and the validate.py
    contract — warn-only still exits 1 on a failed terminal status write, a missing token/repo,
    malformed team-keys, or a non-pull_request event.
  • New "Upgrading an existing enforce: false caller" doc section, plus a BEHAVIOUR CHANGE
    paragraph on the input itself, so the green→red flip is documented at the callers it changes
    along with the one-line soft-fail: false opt-out. The default stays true — see below.

Answered without a change: the soft-fail name (the polarity matches Buildkite's
soft_fail — a real failure that does not fail the build; happy to rename while no caller sets
it yet), the comment-before-status write order (reversing it trades an unexplained red check
for an unlanded one — pre-existing since #183), and the default true itself.

Note

One open call for the merger. soft-fail defaults to true, so taking a SHA bump flips
an existing enforce: false caller from a green status to a red (still non-required) one.
That is the intent — a green pilot is a pilot nobody reads — and it is now documented at the
input and in the caller doc rather than shipping silently. Flipping the default to false and
making the loud mode opt-in is a one-line change if you would rather ship it additive.

Follow-up (not in this PR)

The auto-bump PR into Comfy-Org/cloud will move the SHA only, leaving linear-ticket-validate.yml's header comment saying "WARN-ONLY pilot (enforce: false): every outcome is green" — true before this change, stale after. Happy to open a one-line comment fix there; say the word.

Provenance

Authored by: agent-work loop
Verified: python3 -m unittest discover -s scripts/linear-ticket/tests -p 'test_*.py' — 63 passed; python3 -m py_compile scripts/linear-ticket/{lib,validate}.py — ok. All PR checks green on the pre-review-round commit; the review-round commit re-runs them.
Deviations: Three of the seven panel findings were answered on the thread rather than fixed — the soft-fail input name, the comment-before-status write order, and the default: true behaviour change (mitigated by documentation rather than by flipping the default). Rationale on each thread. AGENTS.md fails the repo's own 200-line integrity ceiling at 273 lines, but that is pre-existing on main, is not a check on this PR, and was left untouched.

🤖 Generated with Claude Code

…blocking the merge

The warn-only pilot publishes a green `linear-ticket` status on every outcome, so its
verdict lives only in a marker comment among a dozen other bot comments. On Comfy-Org/cloud
#6690 that is exactly what happened: the gate correctly found three referenced-but-unlinked
identifiers and nobody noticed, because a passing check is a check nobody reads. A pilot's
whole purpose is observation, and it was observing quietly.

Add `soft-fail` (default true), read only when `enforce: false`: warn-only now publishes a
red `failure` status, so the check shows as failing in the PR's check list exactly as
enforcing would — while blocking nothing, because blocking is branch protection requiring
the `linear-ticket` context and neither live caller does. The marker comment carries an
explicit "this does not block the merge" note so a red X is never read as a merge block, and
`soft-fail: false` restores the previous silent always-green behaviour.

The job itself stays green in warn-only. This validator runs on the default branch off
workflow_run, so its run is not attached to the PR — the commit status is the only
PR-visible signal, and a red default-branch run for a deliberately non-gating check would
read as "main is broken" in the Actions tab: louder, in the wrong place.

The mode decision moves out of validate.py into a pure `lib.failure_outcome()` (verdict,
status state, description, exit code, advisory flag), matching the file's contract that
every decision is testable without a network call. New tests pin the invariants that matter:
only enforce ever exits nonzero, the "does not block" note never rides along with a gating
verdict, and every mode's status description fits GitHub's 140-char limit.

Verified `linear-ticket` is absent from the required-status contexts of both live callers'
rulesets, so the red status is advisory on both today.
@coderabbitai

coderabbitai Bot commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your current included review allowance is based on your included PR review attempts over the past 7 days.

Next review available in: 31 minutes

Limit details: You’ve used the included review currently available. Your 94 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1a590f22-384e-4631-883d-64e0965c49e8

📥 Commits

Reviewing files that changed from the base of the PR and between 26339cd and feca179.

📒 Files selected for processing (7)
  • .github/workflows/linear-ticket.yml
  • README.md
  • docs/callers/linear-ticket.md
  • scripts/linear-ticket/README.md
  • scripts/linear-ticket/lib.py
  • scripts/linear-ticket/tests/test_lib.py
  • scripts/linear-ticket/validate.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: df6d68c0-8b71-4287-807a-e9bae22ee485

📥 Commits

Reviewing files that changed from the base of the PR and between 1952e13 and 26339cd.

📒 Files selected for processing (7)
  • .github/workflows/linear-ticket.yml
  • README.md
  • docs/callers/linear-ticket.md
  • scripts/linear-ticket/README.md
  • scripts/linear-ticket/lib.py
  • scripts/linear-ticket/tests/test_lib.py
  • scripts/linear-ticket/validate.py

Included review availability: 0 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.


📝 Walkthrough

Walkthrough

The workflow now supports configurable warn-only reporting. The validator centralizes failure outcomes, publishes advisory or enforcing statuses, and preserves terminal status-write failures. Documentation and tests cover the new modes.

Changes

Linear ticket reporting

Layer / File(s) Summary
Failure outcome contract and tests
scripts/linear-ticket/lib.py, scripts/linear-ticket/tests/test_lib.py
Added FailureOutcome, failure_outcome, and ADVISORY_NOTE. Tests cover enforcement, warn-only modes, metadata, exit codes, and invalid categories.
Validator reporting integration
scripts/linear-ticket/validate.py
Added SOFT_FAIL parsing and validator configuration. Failure comments, summaries, statuses, and exit codes now use centralized outcome data.
Workflow wiring and reporting guidance
.github/workflows/linear-ticket.yml, README.md, docs/callers/linear-ticket.md, scripts/linear-ticket/README.md
Added the soft-fail input and documented red advisory statuses, silent green warn-only behavior, rollout steps, and required-check constraints.

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant Validator
  participant OutcomeLogic
  participant GitHubStatus
  Workflow->>Validator: pass SOFT_FAIL
  Validator->>OutcomeLogic: classify failure category
  OutcomeLogic-->>Validator: return reporting outcome
  Validator->>GitHubStatus: publish terminal status
  Validator-->>Workflow: return computed exit code
Loading

Possibly related PRs

Suggested reviewers: luke-mino-altherr

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch matt/linear-ticket-soft-fail
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch matt/linear-ticket-soft-fail

Comment @coderabbitai help to get the list of available commands.

@mattmillerai mattmillerai added the cursor-review Multi-model cursor review label Aug 20, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 7 finding(s).

Severity Count
🟠 High 1
🟡 Medium 3
🟢 Low 3

Panel: 8/8 reviewers contributed findings.

Comment thread .github/workflows/linear-ticket.yml Resolved
Comment thread scripts/linear-ticket/lib.py Resolved
Comment thread .github/workflows/linear-ticket.yml Resolved
Comment thread scripts/linear-ticket/validate.py Resolved
Comment thread scripts/linear-ticket/lib.py Resolved
Comment thread scripts/linear-ticket/validate.py Resolved
Comment thread .github/workflows/linear-ticket.yml Resolved
…iguration

Addresses the cursor-review panel on #186.

- The advisory note no longer asserts "This does not block the merge". The
  validator never reads the caller's ruleset, so that claim is false in exactly
  the configuration the docs call a footgun (warn-only + the `linear-ticket`
  context already required) — where the red check IS blocking the author, and
  the note sent them to debug the wrong thing. It now states what is knowable
  (this workflow is not what blocks; your ruleset is) and names the
  misconfiguration to report. The soft-fail verdict string drops "not blocking"
  for the factual "warn-only".
- Status descriptions are category-aware. `infra_error` means Linear could not
  be queried, so "no linked Linear issue (infra_error)" was a diagnosis the run
  never made; the advisory note likewise no longer tells an author to link a
  ticket to fix a broken credential.
- An ABSENT `SOFT_FAIL` env now means the SILENT variant, not the loud one.
  linear-ticket.yml always passes it, so absent can only mean the `uses:` and
  `workflows_ref` pins have skewed apart — and a caller that cannot express the
  input must not be silently upgraded from a green status to a red one.
- "the job never exits nonzero" narrowed to "a failing VERDICT never exits the
  job nonzero" in the workflow input, both READMEs, the caller doc and the
  validate.py contract: warn-only still exits 1 on a failed terminal status
  write, a missing token/repo, malformed team-keys, or a non-pull_request event.
- New "Upgrading an existing `enforce: false` caller" section so the default
  `soft-fail: true` flip is documented at the callers it changes, with the
  one-line way to keep the old always-green behaviour.

Tests cover the hedged note, the category-aware copy, the env parsing, and a
keyset guard so _STATUS_HEADLINE cannot drift from _GUIDANCE.
@mattmillerai
mattmillerai merged commit 77b951c into main Aug 20, 2026
5 checks passed
@mattmillerai
mattmillerai deleted the matt/linear-ticket-soft-fail branch August 20, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants