Repository navigation
feat(coderabbit-config): validate .coderabbit.yaml so a broken config fails its own PR (BE-7757) - #179
Conversation
… fails its own PR (BE-7757) CodeRabbit rejects an invalid .coderabbit.yaml WHOLE — it discards the file and reviews with org-wide UI defaults, so every reviewer instruction, path filter, path instruction and WIP-skip rule in it goes silently inert. And it validates the BASE branch, not the PR head, so the PR that breaks the file goes green and the breakage first surfaces on the NEXT PR, blamed on a change that did not cause it. Nothing in CI caught this; the same field on the same repo was fixed twice in six days by two independent tickets. Adds coderabbit-config-validate.yml (reusable, workflows_ref REQUIRED, checker loaded from this repo at the pinned ref), its checker + VENDORED schema under .github/coderabbit-config/, the test workflow, the bump fleet entrypoint, and a weekly refresh job that opens a PR when the vendored schema drifts upstream. Severity mirrors CodeRabbit's own behaviour rather than a schema library's: parse / maxLength / type / enum errors FAIL; an unknown key WARNS (CodeRabbit strips those, so the config loads while everything under the key does nothing) unless strict_unknown_keys: true. A repo with no config passes, and says so. The schema is vendored, never fetched on the validation path: a live fetch makes every consumer's CI depend on a third party, and an upstream tightening would redden the fleet with no change here.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 20 minutes Limit details: You’ve used all 1 included review currently available under your plan. You completed 99 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (17)
Comment |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 10 finding(s).
| Severity | Count |
|---|---|
| 🟠 High | 4 |
| 🟡 Medium | 3 |
| 🟢 Low | 3 |
Panel: 8/8 reviewers contributed findings.
…dator (BE-7757) Ten findings from the cursor-review panel, addressed in place. Checker: - Containment now resolves symlinks on BOTH sides (`realpath`, not `abspath`). A config committed as a symlink out of the tree cleared the old textual guard and was read anyway, and a YAML error quotes the target into a public log. - A path that exists but is not a regular file (directory, dangling symlink) is exit 2, not a silent pass. - `.coderabbit.yml` under the default name is validated with a warning instead of reported "absent — pass". CodeRabbit honours both spellings, so that was a permanently green check over a config nobody looked at. - `config_file` and `--root` are escaped in the banner and result lines like every other print; a newline in that workflow input could emit `::stop-commands::` and suppress the annotations of a failing check. - Bound the PR-controlled input (512 KiB) and the reported findings (100). Schema drift: - Diff the UNION of old and new caps. The old walk skipped a cap upstream ADDS to a previously uncapped property — the single most config-invalidating drift there is — and never looked at one it removes; both then printed "No property or `maxLength` changes", which is false. - Walk `anyOf`/`oneOf`/`allOf`, `patternProperties`, schema-valued `additionalProperties`, list-form `items` and `$defs`. The vendored schema already uses two of those shapes, so drift inside them was invisible. - An unexpected exception exits 2, not Python's default 1 — 1 means DRIFTED to the caller, which force-resets a shared branch on it. Workflows: - `python3 -P -m pip`: that step's working directory is the untrusted caller checkout, and `-m` puts it first on `sys.path`, so a PR adding `pip/__main__.py` ran its own code before the pinned checker and could rewrite it. - The refresh's invariant suite no longer aborts the run before the PR exists. It asserts exact facts (the 14-cap set, tone_instructions == 250), so any real upstream cap change failed it — and the later steps' implicit `success()` then opened no PR at all, silently dropping the drift class this prioritizes. The verdict now rides in the PR body and reddens the run at the end. - Verify drift.md before the force-push, so the shared branch never moves with nothing to explain it. Docs scoped to reality: unknown-key detection only fires where the schema CLOSES an object, so a typo inside `reviews` is not caught. Widening it is follow-up work.
|
Resolved the review panel in 959c374 — all 10 threads addressed and closed. Nine fixed in code, one deferred with the docs corrected to match reality. The two that mattered most
Also fixed: symlink escape of the containment guard (realpath both sides); Deferred: unknown-key detection only fires where the schema CLOSES an object, so Verification: 45 unit tests (up from 34) plus two new workflow smoke tests, all green in CI. The refresh workflow cannot be exercised by CI, so I ran its real extracted step bodies under bash against a deliberately tampered schema with stubbed Note for whoever merges: the cursor-review panel graded 3f86cee, not this commit. |
|
🤖 The reviews loop filed Linear follow-up ticket(s) for review thread(s) deferred as out of scope for this PR:
The following carry
|
ELI-5
CodeRabbit reads a
.coderabbit.yamlin your repo to know how to review your PRs. If that file breaks the rules of its own format, CodeRabbit throws away the whole file and reviews with generic defaults instead — every instruction you wrote silently stops mattering, and nothing looks different. Worse, it checks the file on the base branch, so the PR that broke it goes green and the damage shows up on the next person's PR, blamed on them.This adds a shared CI check that validates the file on the PR that changes it, so the breakage lands on the right diff, with an annotation on the right line.
What's here
.github/workflows/coderabbit-config-validate.ymlworkflow_callcheck.workflows_refREQUIRED, no default; checker loaded from this repo at the pinned ref, never from the caller's checkout..github/coderabbit-config/check_coderabbit_config.py, the vendoredschema.v2.json,schema_drift.py, hash-pinnedrequirements.txt, tests, README..github/workflows/test-coderabbit-config-validate.yml.github/workflows/bump-coderabbit-config-callers.ymlpreflight.sh+bump-callers.sh. Roster is a secret (CODERABBIT_CONFIG_CALLERS), seeded empty..github/workflows/refresh-coderabbit-schema.ymldocs/callers/coderabbit-config-validate.md, README catalog row, AGENTS.md, dependabotpipentry.Severity is split to mirror what CodeRabbit itself does, not what a schema library calls invalid: a YAML parse error, a
maxLengthviolation or a type/enum error fails (those are what it rejects the whole file for); an unknown/additional property warns (it strips those, so the config loads while everything under the key does nothing) unlessstrict_unknown_keys: true. A repo with no config passes, and says so — "no config here" must never read the same as "config validated clean" in a log.Verification
Repo suites — all green locally: the new
coderabbit-configsuite (35 tests), the 6 CLI smoke steps extracted from the test workflow and executed verbatim,check_workflow_pins.py(8 reusables, noworkflows_refdefault, every ref checkout guarded) + its 78 tests,test_paths_contract.sh(58 checks — the guard that this fleet'spaths:filter exactly matches theWATCHED/WATCHED_ASSETSit handspreflight.sh),test_bump_callers.sh,shellcheck, and the agents-md / cursor-review / groom / refresh-reviewers / workflow-pins suites.Against the artifacts the ticket names. I swept 18 org repos read-only for a
.coderabbit.yaml; 11 carry one. Running this checker over all 11 reproduces the ticket's Evidence table exactly:strict_unknown_keys: trueEach of the three is reported with the offending root key named, the correct source line, and a
Did you mean \reviews.tools`?hint. ZeromaxLength` violations org-wide — which is the second half of the ticket's point, and why a file-size check would have gated nothing. Repo names are redacted here: most are private and this repo is public.The half I am not fixing: the 3 invalid configs stay invalid — they are out of scope by the ticket's own design (one child ticket per repo, so each lands as its own PR in its own repo). And the check is currently wired to 0 of 11 repos: this PR ships the machinery, the fleet makes the rollout a follow-up.
The install path is exercised for real:
pip install --require-hashes --only-binary=:all:resolved cleanly for linux/cp312 in a dry run, including thetyping-extensionstransitive that a 3.13+ resolver silently drops. The same install runs in the test workflow, so a wrong hash set fails on the PR that broke it rather than in every consumer's CI.Judgment calls
The vendored schema, and the one dependency. This repo is stdlib-only on purpose, and this directory breaks that. Neither half has a stdlib answer: there is no YAML parser in the standard library, and the point of the check is to reproduce the verdict CodeRabbit's own Draft 2020-12 validator reaches — a hand-written subset validator would agree with the real one until it didn't, and the day it diverged the check would either gate nothing or fail a valid config. So the dependency is accepted and pinned hard: exact versions + sha256,
--require-hashes, and a Dependabotpipentry so the pins don't rot the way an unwatched inlinepip install x==1.2.3does. AGENTS.md now records the exception rather than still claiming stdlib-only.Not extending
agents-md-integrity.yml— a sibling workflow, per the ticket. Folding this in would push a Python dependency and a new failure mode onto every one of its callers, for a check about a different class of file.Failure modes fail loudly, in the right direction. Exit 2 (could-not-run) is distinct from exit 1 (config invalid), because "I could not check" must never look like "it's fine". Both the checker and the drift detector refuse a file that isn't a JSON Schema object rather than reporting "no drift" / "all valid" — a check that cannot fail is worse than no check. That is the concrete redirect-stub case the ticket names, and it's covered by a smoke test.
Path containment + injection.
config_fileis guarded against an absolute or../-escaping path and against an empty value (which would otherwise join to the repo root and report a clean pass over a repo nobody looked at). Every annotation is workflow-command-escaped — the strings come from a PR-author-controlled file, and YAML permits a newline in a quoted key.Unmet criteria and unexercised artifacts
comfy-conventionscaller is NOT in this PR. That acceptance criterion needs a commit in a different repository; I am scoped to one repo and one PR. It is a two-line caller plus a roster entry —docs/callers/coderabbit-config-validate.mdis the copy-pasteable version. The fleet entrypoint also needs oneworkflow_dispatchrun frommainafter merge (landing a fleet touches no path its own filter watches, so it fires no run of its own).tools:block are reviewed normally rather than defaulted — but I did not observe CodeRabbit's loader directly. If it turns out to be file-rejecting,strict_unknown_keysis the switch, and the default should flip.Refs BE-7757 — the ticket stays open: the caller wiring criterion is unmet above, and the three invalid configs it reports are still invalid, tracked on its children.
🤖 Generated with Claude Code