Skip to content

feat(coderabbit-config): validate .coderabbit.yaml so a broken config fails its own PR (BE-7757) - #179

Merged
mattmillerai merged 2 commits into
mainfrom
matt/be-7757-coderabbit-config-validate
Aug 18, 2026
Merged

mattmillerai merged 2 commits into
mainfrom
matt/be-7757-coderabbit-config-validate

Conversation

@mattmillerai

Copy link
Copy Markdown
Member

ELI-5

CodeRabbit reads a .coderabbit.yaml in your repo to know how to review your PRs. If that file breaks the rules of its own format, CodeRabbit throws away the whole file and reviews with generic defaults instead — every instruction you wrote silently stops mattering, and nothing looks different. Worse, it checks the file on the base branch, so the PR that broke it goes green and the damage shows up on the next person's PR, blamed on them.

This adds a shared CI check that validates the file on the PR that changes it, so the breakage lands on the right diff, with an annotation on the right line.

What's here

File
.github/workflows/coderabbit-config-validate.yml The reusable workflow_call check. workflows_ref REQUIRED, no default; checker loaded from this repo at the pinned ref, never from the caller's checkout.
.github/coderabbit-config/ check_coderabbit_config.py, the vendored schema.v2.json, schema_drift.py, hash-pinned requirements.txt, tests, README.
.github/workflows/test-coderabbit-config-validate.yml 35 unit tests + 6 end-to-end CLI smoke tests.
.github/workflows/bump-coderabbit-config-callers.yml The SHA-bump fleet, on the shared preflight.sh + bump-callers.sh. Roster is a secret (CODERABBIT_CONFIG_CALLERS), seeded empty.
.github/workflows/refresh-coderabbit-schema.yml Weekly: fetch upstream, compare semantically, open a reviewable PR on drift.
docs docs/callers/coderabbit-config-validate.md, README catalog row, AGENTS.md, dependabot pip entry.

Severity is split to mirror what CodeRabbit itself does, not what a schema library calls invalid: a YAML parse error, a maxLength violation or a type/enum error fails (those are what it rejects the whole file for); an unknown/additional property warns (it strips those, so the config loads while everything under the key does nothing) unless strict_unknown_keys: true. A repo with no config passes, and says so — "no config here" must never read the same as "config validated clean" in a log.

Verification

Repo suites — all green locally: the new coderabbit-config suite (35 tests), the 6 CLI smoke steps extracted from the test workflow and executed verbatim, check_workflow_pins.py (8 reusables, no workflows_ref default, every ref checkout guarded) + its 78 tests, test_paths_contract.sh (58 checks — the guard that this fleet's paths: filter exactly matches the WATCHED/WATCHED_ASSETS it hands preflight.sh), test_bump_callers.sh, shellcheck, and the agents-md / cursor-review / groom / refresh-reviewers / workflow-pins suites.

Against the artifacts the ticket names. I swept 18 org repos read-only for a .coderabbit.yaml; 11 carry one. Running this checker over all 11 reproduces the ticket's Evidence table exactly:

Default strict_unknown_keys: true
The 3 configs the ticket reports as invalid warn (exit 0) fail (exit 1)
The other 8 clean clean

Each of the three is reported with the offending root key named, the correct source line, and a Did you mean \reviews.tools`?hint. ZeromaxLength` violations org-wide — which is the second half of the ticket's point, and why a file-size check would have gated nothing. Repo names are redacted here: most are private and this repo is public.

The half I am not fixing: the 3 invalid configs stay invalid — they are out of scope by the ticket's own design (one child ticket per repo, so each lands as its own PR in its own repo). And the check is currently wired to 0 of 11 repos: this PR ships the machinery, the fleet makes the rollout a follow-up.

The install path is exercised for real: pip install --require-hashes --only-binary=:all: resolved cleanly for linux/cp312 in a dry run, including the typing-extensions transitive that a 3.13+ resolver silently drops. The same install runs in the test workflow, so a wrong hash set fails on the PR that broke it rather than in every consumer's CI.

Judgment calls

The vendored schema, and the one dependency. This repo is stdlib-only on purpose, and this directory breaks that. Neither half has a stdlib answer: there is no YAML parser in the standard library, and the point of the check is to reproduce the verdict CodeRabbit's own Draft 2020-12 validator reaches — a hand-written subset validator would agree with the real one until it didn't, and the day it diverged the check would either gate nothing or fail a valid config. So the dependency is accepted and pinned hard: exact versions + sha256, --require-hashes, and a Dependabot pip entry so the pins don't rot the way an unwatched inline pip install x==1.2.3 does. AGENTS.md now records the exception rather than still claiming stdlib-only.

Not extending agents-md-integrity.yml — a sibling workflow, per the ticket. Folding this in would push a Python dependency and a new failure mode onto every one of its callers, for a check about a different class of file.

Failure modes fail loudly, in the right direction. Exit 2 (could-not-run) is distinct from exit 1 (config invalid), because "I could not check" must never look like "it's fine". Both the checker and the drift detector refuse a file that isn't a JSON Schema object rather than reporting "no drift" / "all valid" — a check that cannot fail is worse than no check. That is the concrete redirect-stub case the ticket names, and it's covered by a smoke test.

Path containment + injection. config_file is guarded against an absolute or ../-escaping path and against an empty value (which would otherwise join to the repo root and report a clean pass over a repo nobody looked at). Every annotation is workflow-command-escaped — the strings come from a PR-author-controlled file, and YAML permits a newline in a quoted key.

Unmet criteria and unexercised artifacts

  • The comfy-conventions caller is NOT in this PR. That acceptance criterion needs a commit in a different repository; I am scoped to one repo and one PR. It is a two-line caller plus a roster entry — docs/callers/coderabbit-config-validate.md is the copy-pasteable version. The fleet entrypoint also needs one workflow_dispatch run from main after merge (landing a fleet touches no path its own filter watches, so it fires no run of its own).
  • The severity split rests on a premise I could not re-verify from CI. That CodeRabbit strips an unknown key rather than rejecting the file comes from the ticket's runtime observation. The circumstantial evidence is consistent — three repos with a root-level tools: block are reviewed normally rather than defaulted — but I did not observe CodeRabbit's loader directly. If it turns out to be file-rejecting, strict_unknown_keys is the switch, and the default should flip.
  • Sub-issues BE-7759 and BE-7761 were named to me but not fetched, and their bodies are unreachable from this sandbox. I worked from the parent ticket's description of them (the per-repo cleanups, deliberately out of scope here).
  • Not covered: duplicate YAML keys. PyYAML silently keeps the last one; whether CodeRabbit's parser errors instead is untested. Worth its own look if someone hits it.

Refs BE-7757 — the ticket stays open: the caller wiring criterion is unmet above, and the three invalid configs it reports are still invalid, tracked on its children.

🤖 Generated with Claude Code

… fails its own PR (BE-7757)

CodeRabbit rejects an invalid .coderabbit.yaml WHOLE — it discards the file and
reviews with org-wide UI defaults, so every reviewer instruction, path filter,
path instruction and WIP-skip rule in it goes silently inert. And it validates
the BASE branch, not the PR head, so the PR that breaks the file goes green and
the breakage first surfaces on the NEXT PR, blamed on a change that did not
cause it. Nothing in CI caught this; the same field on the same repo was fixed
twice in six days by two independent tickets.

Adds coderabbit-config-validate.yml (reusable, workflows_ref REQUIRED, checker
loaded from this repo at the pinned ref), its checker + VENDORED schema under
.github/coderabbit-config/, the test workflow, the bump fleet entrypoint, and a
weekly refresh job that opens a PR when the vendored schema drifts upstream.

Severity mirrors CodeRabbit's own behaviour rather than a schema library's:
parse / maxLength / type / enum errors FAIL; an unknown key WARNS (CodeRabbit
strips those, so the config loads while everything under the key does nothing)
unless strict_unknown_keys: true. A repo with no config passes, and says so.

The schema is vendored, never fetched on the validation path: a live fetch makes
every consumer's CI depend on a third party, and an upstream tightening would
redden the fleet with no change here.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Aug 18, 2026
@coderabbitai

coderabbitai Bot commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 20 minutes

Limit details: You’ve used all 1 included review currently available under your plan. You completed 99 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a77cd11c-4d76-4dbc-becc-ac65bab03b03

📥 Commits

Reviewing files that changed from the base of the PR and between 6446399 and 959c374.

📒 Files selected for processing (17)
  • .github/bump-callers/README.md
  • .github/coderabbit-config/README.md
  • .github/coderabbit-config/check_coderabbit_config.py
  • .github/coderabbit-config/requirements.txt
  • .github/coderabbit-config/schema.v2.json
  • .github/coderabbit-config/schema_drift.py
  • .github/coderabbit-config/tests/test_check_coderabbit_config.py
  • .github/coderabbit-config/tests/test_schema_drift.py
  • .github/dependabot.yml
  • .github/workflows/bump-coderabbit-config-callers.yml
  • .github/workflows/coderabbit-config-validate.yml
  • .github/workflows/refresh-coderabbit-schema.yml
  • .github/workflows/test-coderabbit-config-validate.yml
  • AGENTS.md
  • README.md
  • docs/callers/README.md
  • docs/callers/coderabbit-config-validate.md

Comment @coderabbitai help to get the list of available commands.

@socket-security

socket-security Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​jsonschema@​4.25.199100100100100
Addedpypi/​pyyaml@​6.0.3100100100100100
Addedpypi/​typing-extensions@​4.16.0100100100100100
Addedpypi/​attrs@​26.1.0100100100100100
Addedpypi/​rpds-py@​2026.6.3100100100100100
Addedpypi/​jsonschema-specifications@​2025.9.1100100100100100
Addedpypi/​referencing@​0.37.0100100100100100

View full report

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 10 finding(s).

Severity Count
🟠 High 4
🟡 Medium 3
🟢 Low 3

Panel: 8/8 reviewers contributed findings.

…dator (BE-7757)

Ten findings from the cursor-review panel, addressed in place.

Checker:
- Containment now resolves symlinks on BOTH sides (`realpath`, not `abspath`).
  A config committed as a symlink out of the tree cleared the old textual guard
  and was read anyway, and a YAML error quotes the target into a public log.
- A path that exists but is not a regular file (directory, dangling symlink) is
  exit 2, not a silent pass.
- `.coderabbit.yml` under the default name is validated with a warning instead
  of reported "absent — pass". CodeRabbit honours both spellings, so that was a
  permanently green check over a config nobody looked at.
- `config_file` and `--root` are escaped in the banner and result lines like
  every other print; a newline in that workflow input could emit
  `::stop-commands::` and suppress the annotations of a failing check.
- Bound the PR-controlled input (512 KiB) and the reported findings (100).

Schema drift:
- Diff the UNION of old and new caps. The old walk skipped a cap upstream ADDS
  to a previously uncapped property — the single most config-invalidating drift
  there is — and never looked at one it removes; both then printed "No property
  or `maxLength` changes", which is false.
- Walk `anyOf`/`oneOf`/`allOf`, `patternProperties`, schema-valued
  `additionalProperties`, list-form `items` and `$defs`. The vendored schema
  already uses two of those shapes, so drift inside them was invisible.
- An unexpected exception exits 2, not Python's default 1 — 1 means DRIFTED to
  the caller, which force-resets a shared branch on it.

Workflows:
- `python3 -P -m pip`: that step's working directory is the untrusted caller
  checkout, and `-m` puts it first on `sys.path`, so a PR adding `pip/__main__.py`
  ran its own code before the pinned checker and could rewrite it.
- The refresh's invariant suite no longer aborts the run before the PR exists.
  It asserts exact facts (the 14-cap set, tone_instructions == 250), so any real
  upstream cap change failed it — and the later steps' implicit `success()` then
  opened no PR at all, silently dropping the drift class this prioritizes. The
  verdict now rides in the PR body and reddens the run at the end.
- Verify drift.md before the force-push, so the shared branch never moves with
  nothing to explain it.

Docs scoped to reality: unknown-key detection only fires where the schema CLOSES
an object, so a typo inside `reviews` is not caught. Widening it is follow-up work.
@mattmillerai

Copy link
Copy Markdown
Member Author

Resolved the review panel in 959c374 — all 10 threads addressed and closed. Nine fixed in code, one deferred with the docs corrected to match reality.

The two that mattered most

  • refresh-coderabbit-schema.yml could not propose the drift class it exists to catch first. The invariant suite asserts exact facts (len(caps) == 14, tone_instructions == 250), so any real upstream cap change failed that step — and because the later steps carry an implicit success(), no PR was opened at all. The suite now runs captured; the PR opens either way with the failure in its body, and a final step reddens the run.
  • schema_drift.py skipped a maxLength upstream ADDS to a previously uncapped property — the single most config-invalidating drift there is — and never looked at one it removes. Both then printed "No property or maxLength changes". It now diffs the union of both cap sets.

Also fixed: symlink escape of the containment guard (realpath both sides); python3 -P -m pip so the untrusted caller checkout is not first on sys.path; exit 2 for a config path that is not a regular file; .coderabbit.yml under the default name is now validated with a warning instead of a green "absent — pass"; workflow-command escaping for config_file/--root; bounded input (512 KiB) and findings (100); combinator/patternProperties/additionalProperties coverage in both schema walkers; unexpected drift exceptions exit 2 rather than being read as "drifted".

Deferred: unknown-key detection only fires where the schema CLOSES an object, so reviews.profil is not caught. The docs in all three places now say that rather than over-claiming; widening it needs its own diff and tests (false positives there would fail valid configs fleet-wide). Follow-up ticket proposed.

Verification: 45 unit tests (up from 34) plus two new workflow smoke tests, all green in CI. The refresh workflow cannot be exercised by CI, so I ran its real extracted step bodies under bash against a deliberately tampered schema with stubbed git/gh — confirming both the failing-invariants path (PR opens, body carries the warning and the suite output) and the pass path, and that the drift.md guard aborts before any force-push.

Note for whoever merges: the cursor-review panel graded 3f86cee, not this commit.

@mattmillerai

Copy link
Copy Markdown
Member Author

🤖 The reviews loop filed Linear follow-up ticket(s) for review thread(s) deferred as out of scope for this PR:

  • BE-7779 — coderabbit-config: detect unknown keys inside objects the schema leaves open — filed as agent-spike (premise unverified)

The following carry agent-spike instead of agent-ok because their reachability claim was not backed by evidence (BE-5378) — the claim is investigated before any code is written, and "the premise does not hold" is a valid, successful outcome:

  • coderabbit-config: detect unknown keys inside objects the schema leaves open — no reachability block in the proposal

@mattmillerai
mattmillerai merged commit a98de8b into main Aug 18, 2026
8 checks passed
@mattmillerai
mattmillerai deleted the matt/be-7757-coderabbit-config-validate branch August 18, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants