Skip to content

feat(coderabbit-config): flag unknown keys inside schema-open objects (BE-7786) - #188

Merged
mattmillerai merged 3 commits into
mainfrom
matt/be-7786-coderabbit-config-unknown-keys
Aug 20, 2026
Merged

mattmillerai merged 3 commits into
mainfrom
matt/be-7786-coderabbit-config-unknown-keys

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Aug 20, 2026 •

Copy link
Copy Markdown
Member

ELI-5

CodeRabbit throws away any key in .coderabbit.yaml it doesn't recognize. The checker only told you about that in five places — the handful of spots where the schema explicitly says "nothing else allowed here". Everywhere else it said nothing, because the JSON Schema library it leans on had no rule to trip. So reviews: {profil: chill} — one letter off, and every setting under it silently inert — came back clean, in both normal and strict mode.

This makes the check look at the other 103 objects too. It reports the same thing, the same way, in the same words; it just no longer waits for the schema to spell out a rule it can restate.

The rule, and why it is drawn this way

Walking the parsed config alongside the schema, an object is treated as closed by omission only when it declares a properties block and none of additionalProperties, unevaluatedProperties, patternProperties, anyOf, oneOf, allOf, $ref, $dynamicRef, if, dependentSchemas, propertyNames, not. Each of those is a way an object can legitimately hold a name its properties does not list, so reading any of them as "closed" would invent a warning on a valid config — the failure mode that matters most here, because this check gates other repos' CI. Today's schema uses exactly one of the twelve, so eleven entries are pure future-proofing against a vendored-schema bump.

The walk descends only through matched properties and through items; never into a combinator branch, an additionalProperties schema, or the value under a key it just reported (that key has no subschema, so nothing beneath it is judgeable). Where the document and the schema disagree about shape it stays silent — that's a type error and jsonschema already describes it, in better words than a second complaint would.

No double-reporting, by construction rather than by de-duplication. jsonschema fires only where the keyword is present; the walk only where it is absent. Complements, not overlapping sets. test_the_two_halves_never_report_the_same_key exercises both at once on one config so a future refactor that merged them fails here instead of double-annotating someone's PR.

That last claim is stated as a fact about this vendored schema, not as a property of the two conditions — they are complements at a single node, but jsonschema evaluates a document path against every applicable subschema, so a node this walk judges closed whose $ref target or combinator branch carried additionalProperties: false would be reported by both halves. OpennessTest pins the two premises that rule it out today: the schema carries no $ref, and no node declaring properties also carries a combinator (so such a node counts as OPEN here and is never judged closed).

Findings from both halves go through one builder, so severity (warning, or error under strict_unknown_keys), message text, path, YAML line and the "did you mean" hint are identical by construction. They share MAX_FINDINGS, and the truncation note fires on the combined total — including the case where a single additionalProperties: false error fans out into more findings than the cap, which the islice on iter_errors does not bound. The walk carries its own budget, charged per key inspected and per node visited: the document is PR-controlled, and while PyYAML builds an anchored mapping once, an aliased one is walkable as a far larger tree, including through lists, which name no key at all. A walk that stops early — on the budget or the depth cap — reports that as a finding at the unknown-key severity, so it is a warning by default and a failure under strict_unknown_keys: a scan that could not rule out unknown keys must not hand back a green check to a repo that asked for unknown keys to fail.

Verification, against the artifacts the ticket names

The reported miss, reproduced first. On origin/main, reviews: {profil: chill}, reviews.tools.golangci-lint.enabld, and reviews.tools.github-checks.timeout_ms each returned zero findings in default and strict mode. After the change each returns exactly one, at the right path and the right line.

Acceptance, verbatim. A config containing only reviews.tools.github-checks.timeout_ms warns and exits 0 by default; exits 1 under --strict-unknown-keys.

Suites. coderabbit-config 72 tests pass (45 pre-existing, unchanged — none asserted the old miss — plus 27 added across the original change and the review round). All 8 existing CLI smoke steps from the test workflow, executed verbatim, pass; a 9th is added for this class. Also green: workflow-pins (78 + the lint itself, 10 workflows clean), agents-md-integrity (46), cursor-review (194), groom (365), refresh-reviewers (48), shellcheck on the bumper scripts.

Sized against the real corpus, not a fixture. I swept 194 non-archived org repos read-only; 20 carry a .coderabbit.yaml. Across all 20, findings go from 1 → 15. Every one of the 14 new findings is the same key, reviews.tools.github-checks.timeout_ms — i.e. zero false positives on real-world configs, which is the number that matters for a check other repos' CI depends on. All 20 still exit 0 in default mode, so enrolling this turns nothing red; 15 would exit 1 under strict_unknown_keys, which is opt-in and is exactly what the new rollout note in the caller doc warns about. Repo names are withheld — most are private and this repo is public; of the six public ones, ComfyUI, comfy-python-sdk, comfy-api-proxy, comfy-typescript-sdk and ComfyUI_prev carry the stale key.

The "stale key" claim, falsified rather than assumed. I fetched the live upstream schema and checked it directly: reviews.tools.github-checks names only enabled, declares no additionalProperties, and upstream today has the same 5-explicit / 103-by-omission split as the vendored copy. The repo's own schema_drift.py against live upstream reports no property or maxLength changes — the drift is cosmetic and touches nothing this walk's behaviour depends on. (Moving the vendored pin is refresh-coderabbit-schema.yml's job, not this PR's.)

Judgment calls

The shared finding builder changes one pre-existing behaviour, deliberately. A non-string YAML key is now named by its source spelling, read off the composed key node, on both paths. Previously such a key rendered through _format_path as a list index ([1] for 1: x) and the line lookup compared an int against the composed node's string value and returned no line. An intermediate str() coercion fixed the list-index half but not the rest: safe_load resolves plain true/on/yes to True, null to None and 0x10 to 16, so those keys were annotated reviews.True / reviews.None / reviews.16 — paths naming a key that appears nowhere in the file — and still lost the line, because the lookup then compared "True" against the node's "true". Matching key nodes tag-aware (constructing them the way safe_load did, which also keeps a quoted "true" from matching the boolean) fixes the name and the line together. str() survives only as the fallback for a key whose node cannot be found, e.g. an aliased or merge-keyed mapping.

The "did you mean" hint stays silent rather than guessing. It was written for the specific-name case, where the one other place the schema uses a name is almost certainly the intended home — the root-level tools: block. Keys found by the by-omission walk are dominated by generic names (enabled sits at ~72 places in the schema, plus scope, path, instructions), for which the alphabetically-first three are three wrong homes. The shortlist now prefers candidates under the offending key's own parent prefix and suppresses the hint past three remaining candidates; the motivating tools: → reviews.tools case is pinned by a test so the suppression cannot swallow it.

Two new tests pin schema facts (5 open / 103 closed-by-omission, and that only additionalProperties is used today). They will fail on a schema refresh that changes either. That is intentional and matches the existing SchemaTest convention in this file — a refresh that redefines what this check means should say so out loud in its own PR.

Bound by key-inspections and node-visits, not one or the other. A node-only budget is the obvious choice and the wrong one: one visit to a mapping with 100k keys costs 100k, so a node cap alone leaves the work superlinear in an adversarial document. But a key-only budget leaves list traversal outside the bound entirely — elements recurse without naming a key, and an alias-amplified nested list can charge 1 for an outer entry while thousands of element visits below it are free. Charging both is what makes the bound cover the aliased-list shape. The "did not see the whole document" flag is set where the walk actually bails rather than derived from the leftover budget afterwards, so an exact landing on the budget is a complete walk and a depth cutoff is a partial one; stopping on the findings cap is neither, and is reported as truncation instead. The depth cap remains belt-and-braces — descent follows the schema, which is finite and shallow, and $ref is an opener this walk never resolves.

Residual, and what stays open

  • The 14 org repos carrying reviews.tools.github-checks.timeout_ms are not cleaned by this PR. Each is a one-line edit in a different repository; this PR is scoped to one repo. They warn (exit 0) once enrolled, and the caller doc now tells an operator to clear them before switching on strict_unknown_keys.
  • How many enrolled callers this affects, I cannot state. The CODERABBIT_CONFIG_CALLERS roster is a repo secret by design and is not readable from here. The org-wide numbers above are the upper bound on impact, not the enrolled count.
  • AGENTS.md is 273 lines, over the 200-line ceiling its own standard enforces. Pre-existing on main (273 there too) and untouched by this change — my edit is deliberately net-zero on line count. This repo does not self-enroll agents-md-integrity.yml, so nothing catches it in CI. Out of scope to fix here, but it is a real finding on a file this change touches.
  • Inherited, still-unverified premise. That CodeRabbit strips an unknown key rather than rejecting the file is the severity split's foundation and comes from the original runtime observation, not from anything I observed. Same disclosure the check shipped with; if it turns out to be file-rejecting, strict_unknown_keys is the switch and the default should flip. This change widens where that premise applies, so it widens the blast radius of it being wrong.
  • Unexercised artifacts. The ticket references a prior read-only investigation and its findings comment; neither is reachable from this sandbox, so I worked from the description reproduced in my brief and re-derived its numbers independently (the 5/103 split and the stale-key corpus above are my own measurements, and they agree with it — except on repo count, where the ticket says six and I measure fourteen today).

Refs BE-7786 — the ticket stays open: the stale keys it names as evidence remain in the org's repos, unfixed by this PR, and the enrolled-caller count is unverifiable from here.

Provenance

  • Authored by: agent-work loop
  • Verified: re-run on the review-round commit — coderabbit-config unittest 72 passed / 0 failed, and 72 again on the tree merged with main (so no semantic conflict with the base); the test workflow's CLI smoke steps executed verbatim, all pass; agents-md-integrity 46 passed; cursor-review 194 passed; groom 365 passed (1 skipped); each of the eight panel findings reproduced against the pre-fix checker before being fixed and re-run after; worst-case key-lookup timing measured against the pre-fix build (no regression). Carried from the authoring run and not re-run here, since this round touches no workflow or shell file: workflow-pins 78 passed + check_workflow_pins.py clean (10 workflows); refresh-reviewers 48 passed; shellcheck -x on bump-callers.sh + preflight.sh clean; checker run read-only over 20 real org .coderabbit.yaml files (all exit 0 default; 1 → 15 findings, all 14 new ones the same genuine stale key)
  • Deviations: none on the ticket's acceptance criteria; the residuals above are out-of-scope items, not skipped criteria. The review round changed three behaviours the original body described differently — the bounded-walk report escalates under strict mode instead of staying a note, non-string keys are named by source spelling instead of str(), and the walk budget covers node visits as well as keys — and the prose above has been corrected to match rather than left describing the first round.

… (BE-7786)

The unknown-key check only reported keys jsonschema complained about, which
means only the five objects upstream's schema closes with an explicit
`additionalProperties: false`. 103 more objects declare `properties` and say
nothing about the rest — `reviews`, `chat`, `knowledge_base`,
`code_generation` and every individual tool config among them — so a typo
inside one (`reviews.profil`, `golangci-lint.enabld`) produced no finding in
either mode, even though CodeRabbit strips those keys exactly like a
root-level one.

Adds a conservative schema-walk that closes the gap. A node counts as closed
BY OMISSION only when it declares a `properties` block and none of the twelve
keywords that let an object accept a name it does not list; the walk descends
only through matched properties and `items`, never into a combinator branch,
an `additionalProperties` schema, or the value under a key it just reported,
and it stays silent where the document and the schema disagree about shape
(that is a type error, and jsonschema already says so). The two halves are
disjoint by construction — one needs the keyword present, the other needs it
absent — so nothing is reported twice.

Findings share one builder with the jsonschema path, so severity, wording,
path, line resolution and the "did you mean" hint are identical by
construction; non-string YAML keys are coerced for display, which also stops
an integer key rendering as a list index. The walk's findings count toward
MAX_FINDINGS with the jsonschema ones, and the walk is itself bounded by a
key-inspection budget (the document is PR-controlled and aliased YAML is
walkable as a much larger tree) that reports a partial scan rather than
letting it read as a clean one.
@mattmillerai mattmillerai added agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review labels Aug 20, 2026
@mattmillerai
mattmillerai marked this pull request as ready for review August 20, 2026 03:31
@coderabbitai

coderabbitai Bot commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your current included review allowance is based on your included PR review attempts over the past 7 days.

Next review available in: 10 minutes

Limit details: You’ve used the included review currently available. Your 95 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 308096fd-65da-49a5-a45b-ba137edf8122

📥 Commits

Reviewing files that changed from the base of the PR and between d97ac24 and 350dd93.

📒 Files selected for processing (6)
  • .github/coderabbit-config/README.md
  • .github/coderabbit-config/check_coderabbit_config.py
  • .github/coderabbit-config/tests/test_check_coderabbit_config.py
  • .github/workflows/test-coderabbit-config-validate.yml
  • AGENTS.md
  • docs/callers/coderabbit-config-validate.md

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 8 finding(s).

Severity Count
🟡 Medium 4
🟢 Low 3
⚪ Nit 1

Panel: 8/8 reviewers contributed findings.

…ce spelling, bound list traversal (BE-7786)

Addresses the cursor-review panel findings on #188.

* The `islice` on `iter_errors` bounds the ERROR count, not the finding
  count: `additionalProperties: false` reports a whole mapping in ONE
  error and `_extra_keys` splits it into one finding per key, so 150
  unknown root keys emitted 150 annotations into a public run log with
  `truncated` still False — and drove `remaining` to 0, silently
  disabling the by-omission walk on top. Re-apply the cap to the
  findings themselves before anything is derived from their number.

* A walk that gave up early was only a note, and a note does not reach
  the exit code — so `strict_unknown_keys: true` exited 0 over keys
  nothing ever looked at. It is now a finding at the unknown-key
  severity: warn-only mode is unchanged, strict mode fails.

* List elements recursed without charging the walk budget, leaving array
  traversal outside the bound its own comment claimed. Charge per node
  visit as well as per key, so an alias-amplified nested list is covered.

* `bounded` is set where the walk actually bails rather than derived from
  the leftover budget: an exact landing on MAX_WALK_KEYS was a full walk
  reported as partial, and a MAX_WALK_DEPTH cutoff was a partial walk
  reported as clean.

* `str()` on a loaded key is not its source spelling — `true:`/`on:`/`yes:`
  all annotated `reviews.True`, `0x10:` `reviews.16`, `null:` `reviews.None`,
  each naming a key that appears nowhere in the file and each losing its
  line number, because the lookup then compared "True" against the node's
  "true". Resolve key nodes tag-aware and read both name and line off the
  matched node.

* The "did you mean" hint was written for the specific-name case; nested
  unknown keys are dominated by generic ones (`enabled` sits at ~72 places
  in the schema), so `reviews.tools.enabled` was answered with three wrong
  `issue_enrichment.*` homes. Prefer candidates under the offending key's
  own parent, and stay silent past three.

* Docstring/README: the two unknown-key halves being duplicate-free is a
  fact about THIS vendored schema, not a property of the two conditions —
  now stated as such, with `OpennessTest` pinning the premises ($ref
  absent, no properties node carrying a combinator).

* Document the by-omission staleness window: a property upstream ADDS
  reads as unknown until the schema refresh and each caller's pin land,
  which under strict mode is a hard failure on a key the repo was right
  to add.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review and removed cursor-review Multi-model cursor review labels Aug 20, 2026
@mattmillerai

Copy link
Copy Markdown
Member Author

Review round addressed in 72aa76d — all eight panel findings fixed, each reproduced against the pre-fix checker first. Summary:

Medium

  • Findings cap overflowed silently. islice bounds the error count, not the finding count; one additionalProperties: false error fans out through _extra_keys into one finding per key, so 150 unknown root keys emitted 150 annotations with truncated still False — and drove remaining to 0, disabling the by-omission walk on top. Cap is now re-applied to findings before remaining is derived from it.
  • A partial walk passed under strict mode. The bounded-walk report was a note, and a note does not reach the exit code. It is now a finding at the unknown-key severity: unchanged warning by default, failure under strict_unknown_keys.
  • List traversal sat outside the walk budget. Elements recurse without naming a key. The budget is now charged per node visit as well as per key, so an alias-amplified nested list is covered.
  • Vendored-schema staleness window. Documented rather than made advisory — by-omission is 103 of the 108 closed objects, so exempting it from strict mode would gate almost nothing. The caller doc now states which direction it fails, that the five explicitly-closed objects have no such window, and that the escape hatch is flipping strict_unknown_keys off for the one PR while the refresh merges.

Low / Nit

  • Keys are named by source spelling. true:/on:/yes: annotated reviews.True, 0x10: reviews.16, null: reviews.None — each naming a key that appears nowhere in the file, each without a line. Key nodes are now matched tag-aware and both name and line come off the matched node.
  • bounded is set where the walk bails, not derived from the leftover budget: an exact landing on MAX_WALK_KEYS was a full walk reported as partial, and a depth cutoff was a partial walk reported as clean.
  • "Did you mean" no longer guesses on generic names. reviews.tools.enabled was answered with three wrong issue_enrichment.* homes. Candidates under the offending key's own parent now win, and the hint is suppressed past three; the motivating tools: → reviews.tools case is pinned by a test.
  • Duplicate-freedom reworded as a fact about this vendored schema rather than a property of the two conditions, with OpennessTest gaining test_the_schema_carries_no_ref to pin the premise that was unasserted.

Nothing was deferred. 72 tests pass (up from 63), and 72 again against the tree merged with main, so there is no semantic conflict with the base. The workflow's CLI smoke steps were re-run verbatim. Worst-case key-lookup timing was measured against the pre-fix build — no regression, since reported keys always sit near the front of their mapping.

The PR body has been corrected where the first round's prose no longer describes the code.

…ect a stale test comment (BE-7786)

The caller doc's severity table did not carry the new behaviour: an
unknown-key scan that stops early warns by default and fails under
`strict_unknown_keys`, which is the same "I could not check must never
look like a pass" rule the exit-2 row above it states.

The comment on the non-string-key test still named `_key_line`, which the
previous commit replaced with `_key_node`, and still described `str()`
coercion as the fix when it is now only the fallback.
@mattmillerai
mattmillerai merged commit b09780e into main Aug 20, 2026
5 checks passed
@mattmillerai
mattmillerai deleted the matt/be-7786-coderabbit-config-unknown-keys branch August 20, 2026 04:15

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Round 2 — ledger: 8 prior finding(s) across 1 round(s) (0 never answered).

Found 8 finding(s).

Severity Count
🟠 High 2
🟢 Low 5
⚪ Nit 1

Panel: 8/8 reviewers contributed findings.

@mattmillerai

Copy link
Copy Markdown
Member Author

⚠️ This PR merged (b09780e) while its round-2 review was still running, so the eight findings that round raised could not be pushed here — the branch was deleted on merge. They are fixed in #192, which is rebased on current main.

Two of them are regressions in this PR's own review-round fix, and are live on main now:

  1. 🟠 A file-rejecting error can be capped away, flipping the exit code to 0. The cap re-applied here slices the finding list in error-sort order, and the root additionalProperties error has an empty absolute_path — so it sorts first and fans out to one finding per unknown key. ~150 unknown root keys plus one maxLength violation slice the file-rejecting error away, leaving 100 warnings and exit 0 on a config CodeRabbit rejects whole. Reproduced end to end.
  2. 🟠 The cap bounds the output but not the work. Each finding rescans the whole mapping constructing candidate key nodes; ~50k unknown root keys fit inside MAX_CONFIG_BYTES at ~1e9 constructions, so the job burns to its timeout instead of printing 100 annotations. This PR made it materially worse by replacing a string compare with a SafeConstructor per candidate.

Both affect every consumer repo running coderabbit-config-validate.yml, so #192 is worth landing promptly rather than batching.

The other six are lower severity: the incomplete-scan report exceeding the cap, the [0]/[] path-vocabulary mismatch in the "did you mean" hint, two key shapes (.nan: and merge keys) still falling back to str(), a fully-walked document reported as partial, and two wording/doc corrections — including one that told callers strict mode was safe against upstream root-level additions when it is not.

All eight threads above are answered and resolved against #192. 87 tests there, up from 72; every case is pinned.

I did not merge anything, and I have not touched main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants