diff --git a/.github/coderabbit-config/README.md b/.github/coderabbit-config/README.md index 8ab1e4e4..33ab206f 100644 --- a/.github/coderabbit-config/README.md +++ b/.github/coderabbit-config/README.md @@ -30,22 +30,56 @@ a schema library calls "invalid": | YAML parse error | error | CodeRabbit cannot read the file at all. | | `maxLength` violation | error | File-rejecting. The schema carries **14** per-field caps, 50 → 20,000 chars. | | type / enum error | error | File-rejecting. | -| unknown / additional property | **warning** | CodeRabbit *strips* an unrecognized key rather than rejecting the file. Only where the schema closes the object — see below. | +| unknown / additional property | **warning** | CodeRabbit *strips* an unrecognized key rather than rejecting the file. Reported wherever the schema object accepts only the names it lists — see below. | | config path that is not a regular file, or > 512 KiB | exit 2 | "I could not check" must never look like a pass. A path resolving outside the repo root (symlinks resolved) is refused the same way. | `strict_unknown_keys: true` escalates the last row to an error for a repo that has cleaned up and wants to stay clean. -**How far the unknown-key row reaches.** It fires where the schema CLOSES an -object — the document root, `knowledge_base.mcp`, and the two tool objects that -declare `additionalProperties: false`. Most nested objects (`reviews`, `chat`, -`knowledge_base`, `code_generation`, the individual tool configs) are open in -upstream's schema, so a typo *inside* one — `reviews.profil`, or -`golangci-lint.enabld` — produces no finding, with or without -`strict_unknown_keys`. That is a real gap and not a claim this checker makes: -what it catches is a key in the wrong PLACE, which is the failure that actually -recurred here. Widening it means cross-checking parsed keys against the schema -index rather than waiting for jsonschema to complain. +**How far the unknown-key row reaches.** It fires wherever a schema object accepts +only the property names it lists — which upstream expresses two different ways, +and both are checked: + +* **explicitly**, with `additionalProperties: false`. Five objects: the document + root, `knowledge_base.mcp`, `knowledge_base.linked_repositories[]`, and the + `htmlhint` / `stylelint` tool configs. jsonschema reports these. +* **by omission** — a `properties` block and no statement at all about anything + else. **103** objects, including `reviews`, `chat`, `knowledge_base`, + `code_generation` and every individual tool config. jsonschema has no keyword to + fire on here, so a schema-walk in `_walk_unknown_keys` reports them. This is + what catches a typo *inside* an object — `reviews.profil`, + `golangci-lint.enabld` — and a key upstream has since REMOVED, such as + `reviews.tools.github-checks.timeout_ms`. + +The rule the walk applies is conservative: a node is closed by omission only when +it declares a `properties` block and NONE of `additionalProperties`, +`unevaluatedProperties`, `patternProperties`, `anyOf`, `oneOf`, `allOf`, `$ref`, +`$dynamicRef`, `if`, `dependentSchemas`, `propertyNames` or `not`. Each of those +is a way the object can legitimately accept a name its `properties` does not list, +so reading any of them as "closed" would invent a finding — `additionalProperties` +given a SCHEMA rather than `false` (`reviews.mutually_exclusive_groups`, where the +group names are the user's to choose) is the live example. The walk descends only +through matched properties and through `items`, never into a combinator branch or +under a key it just reported, and where the document and the schema disagree about +shape it stays silent, because that is a type error and jsonschema owns it. + +The two halves are disjoint at a single node — one needs the keyword present, the +other needs it absent — and against **this** vendored schema that is enough to +make the combined output duplicate-free. It is a fact about the schema rather than +about the two conditions: jsonschema evaluates a document path against every +applicable subschema, so a node the walk judges closed whose `$ref` target carried +`additionalProperties: false` would be reported by both halves. `OpennessTest` +pins the two premises that rule it out today — the schema carries no `$ref`, and +no node declaring `properties` also carries a combinator — so a schema refresh +that breaks either fails there rather than double-annotating a real PR. + +**The by-omission half is only as current as the vendored schema.** It reads those +103 objects as closed against the copy committed here, so a property upstream ADDS +is an unknown key until `refresh-coderabbit-schema.yml` lands the bump and each +caller's pin moves. Warn-only mode absorbs that as a warning; a caller running +`strict_unknown_keys: true` gets a hard failure on a key it was right to add. Same +freshness, opposite sign, as catching a key upstream removed — see the caller doc's +rollout section. A stripped key is not a cosmetic problem. The recurring instance is a `tools:` block written at the document root instead of under `reviews:` — the schema root diff --git a/.github/coderabbit-config/check_coderabbit_config.py b/.github/coderabbit-config/check_coderabbit_config.py index 681ecfbd..3efe0e02 100644 --- a/.github/coderabbit-config/check_coderabbit_config.py +++ b/.github/coderabbit-config/check_coderabbit_config.py @@ -27,6 +27,12 @@ class of problem rather than what a schema library calls "invalid": just not a file-rejecting one. `--strict-unknown-keys` escalates it to a failure for a repo that has cleaned up and wants to stay clean. +An unknown key is reported wherever the schema object it sits in accepts only the +names it lists — whether it says so with `additionalProperties: false` (five +objects, which jsonschema reports) or simply by naming `properties` and nothing +else (103 more, which `_walk_unknown_keys` reports). A node that opts OUT of that +— any of `_OPENER_KEYWORDS` — is left alone in both halves. + A repo with no `.coderabbit.yaml` passes, and says so in the log — not every consumer of the reusable workflow has one. @@ -90,11 +96,66 @@ class of problem rather than what a schema library calls "invalid": # rather than rejecting the document. Everything else is file-rejecting. STRIPPED_KEYWORDS = frozenset({"additionalProperties", "unevaluatedProperties"}) +# Keywords that leave a schema object OPEN to keys its own `properties` block does +# not name. Upstream's schema closes only five objects with an explicit +# `additionalProperties: false` (the document root, the `htmlhint` and `stylelint` +# tool objects, `knowledge_base.mcp`, and `knowledge_base.linked_repositories[]`), +# while 103 more declare `properties` and say nothing at all about the rest. A +# typo inside one of those — `reviews.profil`, `golangci-lint.enabld` — is stripped +# by CodeRabbit exactly like a root-level one, but jsonschema has nothing to +# complain about, so `iter_errors` alone reports it nowhere. `_walk_unknown_keys` +# closes that gap by treating "declares `properties`, declares none of these" as +# closed BY OMISSION. +# +# The list is deliberately wider than this schema needs (it currently carries one +# `anyOf` and one `propertyNames`, and none of the rest). Every entry names a way +# a node can legitimately accept a key its `properties` does not list — through a +# combinator branch, a `$ref`, a pattern, a conditional. Reading any of them as +# "closed" would invent a finding, so a future vendored-schema bump that starts +# using one produces silence here rather than a false positive. +_OPENER_KEYWORDS = ( + "additionalProperties", + "unevaluatedProperties", + "patternProperties", + "anyOf", + "oneOf", + "allOf", + "$ref", + "$dynamicRef", + "if", + "dependentSchemas", + "propertyNames", + "not", +) + +# The walk's own bounds, for the same reason `iter_errors` is collected through +# `islice`: the document is PR-controlled, and while PyYAML constructs an anchored +# mapping once, an aliased document is walkable as a much larger tree — the same +# billion-laughs shape in a few hundred bytes. The budget is charged BOTH per key +# inspected and per node visited, and it needs both halves: per-key is what keeps +# the bound linear in real work (one visit to a mapping with 100k keys costs +# 100k, not 1), while per-visit is what covers list traversal, which names no key +# at all — an aliased nested list charges 1 for the outer entry and would +# otherwise descend through thousands of elements for free. +# +# The depth cap is belt-and-braces rather than the working bound: descent follows +# the SCHEMA (matched properties and `items`), whose nesting is finite and shallow, +# and `$ref` is an opener this walk never resolves — so a self-referential future +# schema cannot make it recurse forever. 64 is far past any real config and far +# short of the interpreter's recursion limit. +MAX_WALK_KEYS = 200_000 +MAX_WALK_DEPTH = 64 + # Guards the "did you mean" suggestion below: an unknown key is only worth # proposing a home for when the name is specific enough that a same-named # property elsewhere in the schema is likely the intended one. _SUGGESTABLE_KEY_RE = re.compile(r"^[A-Za-z][A-Za-z0-9_.-]{2,}$") +# How many homes may be named before the hint stops being a hint. A name with +# more plausible homes than this has no single likely one, so listing the +# alphabetically-first few is guessing out loud — see `_suggest_home`. +_MAX_SUGGESTIONS = 3 + class ConfigError(Exception): """A problem with how the checker was invoked, not with the config.""" @@ -163,13 +224,34 @@ def walk(node, path): def _suggest_home(key, schema_index, offending_path): - """A one-line 'did you mean' for an unknown key, or '' when there is none.""" + """A one-line 'did you mean' for an unknown key, or '' when there is none. + + Written for the root-level `tools:` case: a name specific enough that the one + other place the schema uses it is almost certainly where it belonged. Keys + found by the closed-by-omission walk break that assumption, because nested + unknown keys are dominated by GENERIC names — `enabled` sits at ~72 places in + this schema, plus `scope`, `path`, `instructions` — and for one of those the + three alphabetically-first paths are three homes that are all probably wrong + (`reviews.tools.enabled` used to be answered with `issue_enrichment.*`). + + So the shortlist prefers candidates under the offending key's OWN parent — + that is the "written one level too high" shape the hint exists for — and a + name with more homes than `_MAX_SUGGESTIONS` to choose between gets no hint + at all. No hint is strictly better than a confident wrong one. + """ if not isinstance(key, str) or not _SUGGESTABLE_KEY_RE.match(key): return "" candidates = [p for p in schema_index.get(key, []) if p != offending_path] if not candidates: return "" - shown = ", ".join(f"`{c}`" for c in sorted(candidates)[:3]) + parent = offending_path.rsplit(".", 1)[0] if "." in offending_path else "" + if parent: + nearer = [c for c in candidates if c.startswith(f"{parent}.")] + if nearer: + candidates = nearer + if len(candidates) > _MAX_SUGGESTIONS: + return "" + shown = ", ".join(f"`{c}`" for c in sorted(candidates)) return f" Did you mean {shown}?" @@ -217,22 +299,42 @@ def _line_for_path(node, path_parts): return current.start_mark.line + 1 -def _key_line(node, parent_parts, key): - """1-based YAML line of the KEY `key` inside the mapping at `parent_parts`. +def _resolved_key(key_node): + """The Python value a composed KEY node loads to — tag-aware, like `safe_load`. + + A raw text compare is not enough: `safe_load` resolves plain `true`/`on`/`yes` + to True, `null`/`~` to None and `0x10` to 16, so the key sitting in the loaded + dict and the key node's source text are routinely different objects. Going the + other way (compare `str(loaded)` to the text) is what dropped the line number + for exactly those keys. Constructing the node the way `safe_load` constructed + it makes the two comparable again, and gets the mirror case right too: a + quoted `"true"` keeps its string tag and must NOT match the boolean. + """ + try: + return yaml.constructor.SafeConstructor().construct_object(key_node, deep=True) + except Exception: # pragma: no cover - a key safe_load accepted constructs here + return getattr(key_node, "value", None) + + +def _key_node(node, parent_parts, key): + """The composed KEY node for `key` inside the mapping at `parent_parts`, or None. - An unknown key has no value the schema knows about, so the useful annotation - points at the key itself. + An unknown key has no value the schema knows about, so both the useful + annotation line and the name worth printing come from the key itself. """ parent = node for part in parent_parts: - parent = _descend(parent, part) if parent is None: return None + parent = _descend(parent, part) if not isinstance(parent, yaml.MappingNode): return None for key_node, _ in parent.value: - if getattr(key_node, "value", None) == key: - return key_node.start_mark.line + 1 + try: + if _resolved_key(key_node) == key: + return key_node + except Exception: # pragma: no cover - an exotic key that will not compare + continue return None @@ -249,6 +351,143 @@ def _descend(node, part): return None +def _unknown_key_finding(severity, node, parent_parts, key, schema_index): + """One finding for `key` not being recognized inside the mapping at `parent_parts`. + + Shared by BOTH unknown-key paths — the `additionalProperties: false` errors + jsonschema raises, and the closed-by-omission walk below — so the two are + identical in severity, wording, path, line resolution and "did you mean" by + construction rather than by two copies staying in step. + + YAML keys need not be strings (`1:`, `true:`, `2026-01-01:` all parse to + non-strings), and the display name must be the SOURCE SPELLING, taken from + the composed key node. `str()` on the loaded value is not that spelling and + not even always a key present in the file: `true:`/`on:`/`yes:` all load as + True and rendered `reviews.True`, `0x10:` rendered `reviews.16`, `null:` + rendered `reviews.None` — each naming a key that appears nowhere, with no line + number to correct the impression, because the lookup then compared "True" + against the node's "true" and found nothing. Reading name and line off the one + node fixes both together. + + The `str()` coercion survives as the fallback for a key whose node cannot be + found (an aliased or merge-keyed mapping the composed tree will not follow), + where it is still needed: without it `_format_path` would render an integer + key as a LIST INDEX (`reviews[1]` for `reviews: {1: x}`). `_suggest_home` + needs no coercion guard of its own — `_SUGGESTABLE_KEY_RE` already rejects + anything that is not a plausible property name. + """ + key_node = _key_node(node, parent_parts, key) + text = getattr(key_node, "value", None) + if isinstance(text, str): + name = text + line = key_node.start_mark.line + 1 + else: + name = key if isinstance(key, str) else str(key) + line = None + full = _format_path(parent_parts + [name]) + where = ( + "at the document root" + if not parent_parts + else f"under `{_format_path(parent_parts)}`" + ) + return ( + severity, + full, + line, + f"unknown key `{name}` {where}. CodeRabbit STRIPS keys it " + f"does not recognize, so the config still loads but " + f"everything under this key silently does nothing." + + _suggest_home(name, schema_index, full), + ) + + +def _walk_unknown_keys(data, schema, node, schema_index, severity, limit): + """Unknown keys inside objects the schema closes BY OMISSION. + + jsonschema reports an unknown key only where the schema says `additionalProperties` + / `unevaluatedProperties` — five objects in upstream's schema. This walks the + parsed document alongside the schema and reports the other 103: a node that + declares a dict-valued `properties` and NONE of `_OPENER_KEYWORDS` accepts + exactly the names it lists, so any other key in the document is stripped. + + Conservative on purpose, in three ways: + + * it descends ONLY through matched `properties` values and through `items` + when the document has a list and the schema node's `items` is a dict. + Never into an `additionalProperties` schema, a combinator branch, or the + value under a key it just reported — a key the schema does not name has no + subschema, so anything below it is unjudgeable, not more findings. + * where the document and the schema disagree about shape (a mapping where + the schema wants a scalar, and so on) it stays SILENT. That is a type + error, and reporting it is `iter_errors`' job — saying it twice, in two + different vocabularies, is worse than saying it once. + * against THIS vendored schema it never fires where jsonschema does, which + is what makes the combined output duplicate-free by construction rather + than by a de-duplication pass that could drift. Note that is a fact about + the schema, not a property of the two conditions: they are complements at + a SINGLE node — jsonschema needs the keyword present, this needs it absent + — but jsonschema evaluates a document path against every applicable + subschema, so a node reached through `properties` with no opener of its + own (which this walk judges closed) whose `$ref` target or a combinator + branch carried `additionalProperties: false` would be reported by both + halves. It holds today because the schema has no `$ref`, and because a + node that declares `properties` alongside a combinator counts as OPEN + here — both asserted in `OpennessTest`. + + `limit` caps the findings (the caller sizes it from what is left of + MAX_FINDINGS). Returns (findings, bounded) — the flag says the walk hit one of + its OWN limits (`MAX_WALK_KEYS` or `MAX_WALK_DEPTH`) and therefore did NOT see + the whole document, which the caller reports rather than passing off as a + clean bill of health. It is set at the sites that actually bail, not derived + from the leftover budget afterwards: a document whose keys land exactly on + `MAX_WALK_KEYS` was walked in full and must not be reported as partial, and a + depth cutoff is just as partial as an exhausted budget. Stopping on `limit` is + NOT bounded — that is truncation, which the caller reports separately. + """ + findings = [] + remaining_keys = [MAX_WALK_KEYS] + bounded = [False] + + def walk(value, subschema, path, depth): + if len(findings) >= limit: + return + if remaining_keys[0] <= 0 or depth > MAX_WALK_DEPTH: + bounded[0] = True + return + remaining_keys[0] -= 1 + if isinstance(value, list): + items = subschema.get("items") + if isinstance(items, dict): + for index, element in enumerate(value): + walk(element, items, path + [index], depth + 1) + return + if not isinstance(value, dict): + return + properties = subschema.get("properties") + if not isinstance(properties, dict): + return + closed = not any(keyword in subschema for keyword in _OPENER_KEYWORDS) + for key, child in value.items(): + if len(findings) >= limit: + return + if remaining_keys[0] <= 0: + bounded[0] = True + return + remaining_keys[0] -= 1 + if key in properties: + child_schema = properties[key] + if isinstance(child_schema, dict): + walk(child, child_schema, path + [key], depth + 1) + elif closed: + findings.append( + _unknown_key_finding(severity, node, path, key, schema_index) + ) + + if isinstance(schema, dict): + walk(data, schema, [], 0) + return findings, bounded[0] + + def _describe(error, path_str): """A human sentence for one file-rejecting schema error.""" if error.validator == "maxLength": @@ -354,21 +593,9 @@ def _order(error): ) continue for key in extras: - full = _format_path(parent_parts + [key]) - where = ( - "at the document root" - if not parent_parts - else f"under `{_format_path(parent_parts)}`" - ) findings.append( - ( - unknown_severity, - full, - _key_line(node, parent_parts, key), - f"unknown key `{key}` {where}. CodeRabbit STRIPS keys it " - f"does not recognize, so the config still loads but " - f"everything under this key silently does nothing." - + _suggest_home(key, schema_index, full), + _unknown_key_finding( + unknown_severity, node, parent_parts, key, schema_index ) ) continue @@ -378,6 +605,56 @@ def _order(error): ("error", path_str, _line_for_path(node, parent_parts), _describe(error, path_str)) ) + # `islice` above bounds the ERROR count, not the finding count, and one error + # fans out to many findings: `additionalProperties: false` reports a whole + # mapping in a SINGLE error and `_extra_keys` splits it into one finding per + # unknown key, so 150 unknown root keys arrive here as 150 findings from one + # error. Re-apply the cap to the findings themselves before anything is + # derived from their number — unfixed, that overflow printed 150 annotations + # into a public run log with `truncated` still False, and drove `remaining` + # to 0, silently disabling the by-omission walk on top. + if len(findings) > MAX_FINDINGS: + del findings[MAX_FINDINGS:] + truncated = True + + # The other 103 objects: closed by OMISSION, so jsonschema said nothing about + # them. Appended after the jsonschema findings rather than merged into their + # sort, which keeps both halves in document order within themselves and the + # whole list stable run to run. One past `remaining` so an overflow is + # detectable; `remaining` can be 0, in which case a single walk finding is + # enough to prove the combined total passed the cap. + remaining = MAX_FINDINGS - len(findings) + walked, walk_bounded = _walk_unknown_keys( + data, schema, node, schema_index, unknown_severity, remaining + 1 + ) + if len(walked) > remaining: + del walked[remaining:] + truncated = True + findings.extend(walked) + + if walk_bounded: + # Say so rather than let a partial walk read as a clean one — the same + # reason an absent config is reported instead of passing silently. + # + # A FINDING, not a note, and carrying `unknown_severity`: a note does not + # reach the exit code, so a walk that gave up early still exited 0 even + # under `strict_unknown_keys: true` — a repo that asked for unknown keys + # to FAIL got a green check over keys nothing ever looked at. That is the + # "I could not check" reading as a pass that an oversized config exits 2 + # for. Warn-only mode is unchanged: it stays a warning, exactly like the + # unknown keys it could not go looking for. + findings.append( + ( + unknown_severity, + "(document root)", + None, + f"the unknown-key scan stopped early — after {MAX_WALK_KEYS} keys " + f"or at depth {MAX_WALK_DEPTH} (a very large or heavily aliased " + f"document) — so nested keys past that point were NOT checked. " + f"This run cannot vouch for them.", + ) + ) + if truncated: notes.append( f"more than {MAX_FINDINGS} schema violations in this file — only the " diff --git a/.github/coderabbit-config/tests/test_check_coderabbit_config.py b/.github/coderabbit-config/tests/test_check_coderabbit_config.py index 35e191a5..b202483f 100644 --- a/.github/coderabbit-config/tests/test_check_coderabbit_config.py +++ b/.github/coderabbit-config/tests/test_check_coderabbit_config.py @@ -8,7 +8,11 @@ tickets because nothing detected it; * a top-level `tools:` block that belongs under `reviews:` — three org repos carried this at the time of writing, and on one of them it silently inverted - a `golangci-lint: enabled: false` into the schema default of `true`. + a `golangci-lint: enabled: false` into the schema default of `true`; + * a `reviews.tools.github-checks.timeout_ms` that upstream has since removed — + live in six org repos at the time of writing, and the whole reason the + unknown-key check cannot stop at the five objects the schema closes + explicitly (see `ClosedByOmissionTest`). Every test validates against the REAL vendored schema rather than a toy one, so a schema refresh that would change a verdict fails here first. @@ -19,7 +23,9 @@ import os import sys import unittest -from contextlib import redirect_stdout +from contextlib import contextmanager, redirect_stdout + +import yaml sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) @@ -59,6 +65,18 @@ OVERLONG_TONE = "tone_instructions: " + ("x" * 300) + "\n" +# A key upstream REMOVED from `reviews.tools.github-checks`, still carried by six +# org repos. The object names only `enabled` and says nothing about the rest, so +# jsonschema has no keyword to fire on — this is the exact class the walk exists +# for, and it is a stale key doing nothing, not a hypothetical typo. +STALE_GITHUB_CHECKS = """\ +reviews: + tools: + github-checks: + enabled: true + timeout_ms: 90000 +""" + MALFORMED = """\ reviews: profile: chill @@ -67,6 +85,20 @@ """ +@contextmanager +def budget(keys=None, depth=None): + """Shrink the walk's own limits so a bail-out is testable without a huge fixture.""" + saved = (checker.MAX_WALK_KEYS, checker.MAX_WALK_DEPTH) + if keys is not None: + checker.MAX_WALK_KEYS = keys + if depth is not None: + checker.MAX_WALK_DEPTH = depth + try: + yield + finally: + checker.MAX_WALK_KEYS, checker.MAX_WALK_DEPTH = saved + + def severities(findings): return [f[0] for f in findings] @@ -394,5 +426,366 @@ def test_unreadable_schema_exits_two(self): self.assertIn("::error::", out) +class ClosedByOmissionTest(unittest.TestCase): + """Unknown keys inside objects the schema closes by saying nothing. + + Upstream declares `additionalProperties: false` on five objects and leaves 103 + more with a `properties` block and no statement about anything else. CodeRabbit + strips an unrecognized key in either kind, so both must be reported — and the + two halves must stay disjoint, since one fires only where the keyword is + present and the other only where it is absent. + """ + + def test_a_typo_inside_reviews_is_flagged_at_its_own_line(self): + findings, _ = checker.validate("reviews:\n profil: chill\n", SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + severity, path, line, message = findings[0] + self.assertEqual(severity, "warning") + self.assertEqual(path, "reviews.profil") + self.assertEqual(line, 2) + self.assertIn("unknown key `profil`", message) + self.assertIn("under `reviews`", message) + + def test_a_typo_inside_a_nested_tool_is_flagged_at_its_full_path(self): + text = "reviews:\n tools:\n golangci-lint:\n enabld: true\n" + findings, _ = checker.validate(text, SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + _severity, path, line, _message = findings[0] + self.assertEqual(path, "reviews.tools.golangci-lint.enabld") + self.assertEqual(line, 4) + + def test_the_valid_fixture_stays_silent(self): + # The walk's false-positive canary: a config using nothing but real keys + # must produce nothing, or every enrolled repo gets a warning it cannot act on. + findings, notes = checker.validate(VALID, SCHEMA) + self.assertEqual(findings, [], msg=f"unexpected findings: {messages(findings)}") + self.assertEqual(notes, []) + + def test_a_schema_valued_additional_properties_object_is_open(self): + # `reviews.mutually_exclusive_groups` declares `additionalProperties` as a + # SCHEMA (arrays of ≥2 strings), i.e. user-chosen group names are the + # point. Only the literal `false` closes an object, and that case never + # reaches the walk because jsonschema already owns it. + text = "reviews:\n mutually_exclusive_groups:\n risk: [high, low]\n" + findings, _ = checker.validate(text, SCHEMA) + self.assertEqual(findings, [], msg=f"{messages(findings)}") + + def test_an_anyof_item_shape_is_open(self): + # `knowledge_base.code_guidelines.filePatterns[]` is `anyOf: [string, + # object]`. The item node names no `properties` of its own and the walk + # does not descend into combinator branches, so it judges nothing here. + text = ( + "knowledge_base:\n" + " code_guidelines:\n" + " filePatterns:\n" + ' - files: "**/A.md"\n' + ' applyTo: "**"\n' + ) + findings, _ = checker.validate(text, SCHEMA) + self.assertEqual(findings, [], msg=f"{messages(findings)}") + + def test_severity_follows_strict_unknown_keys(self): + text = "reviews:\n profil: chill\n" + self.assertEqual(severities(checker.validate(text, SCHEMA)[0]), ["warning"]) + self.assertEqual( + severities(checker.validate(text, SCHEMA, strict_unknown_keys=True)[0]), + ["error"], + ) + + def test_a_stale_upstream_removed_key_is_flagged(self): + # The regression anchor: `timeout_ms` was live in six org repos and + # reported nowhere, in either mode, before this walk existed. + findings, _ = checker.validate(STALE_GITHUB_CHECKS, SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + _severity, path, line, _message = findings[0] + self.assertEqual(path, "reviews.tools.github-checks.timeout_ms") + self.assertEqual(line, 5) + + def test_the_two_halves_never_report_the_same_key(self): + # Disjoint by construction (keyword present vs absent), asserted on a + # config that exercises both halves at once so a future refactor that + # merged them would fail here rather than double-annotate a real PR. + text = ( + "tools:\n" + " golangci-lint:\n" + " enabled: false\n" + "reviews:\n" + " profil: chill\n" + " tools:\n" + " github-checks:\n" + " timeout_ms: 1\n" + ) + findings, _ = checker.validate(text, SCHEMA) + paths = [f[1] for f in findings] + self.assertEqual(sorted(paths), sorted(set(paths)), msg=f"duplicated: {paths}") + self.assertEqual( + sorted(paths), + [ + "reviews.profil", + "reviews.tools.github-checks.timeout_ms", + "tools", + ], + ) + + def test_a_non_string_key_is_not_rendered_as_an_index(self): + # YAML keys need not be strings. Left as the loaded int, `_format_path` + # would render this as `reviews[1]` — a list index into an object. The + # sibling test below covers the keys `str()` alone still got wrong. + findings, _ = checker.validate("reviews:\n 1: x\n", SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + _severity, path, line, _message = findings[0] + self.assertEqual(path, "reviews.1") + self.assertEqual(line, 2) + + def test_a_key_is_named_by_its_source_spelling_not_its_loaded_value(self): + # `safe_load` resolves plain `true`/`on`/`yes` to True, `null` to None and + # `0x10` to 16, so `str()` on the loaded key produced `reviews.True` / + # `reviews.None` / `reviews.16` — paths naming a key that appears NOWHERE + # in the file — and then found no line to correct the impression, because + # the lookup compared "True" against the node's "true". + for source, expected in ( + ("true", "reviews.true"), + ("on", "reviews.on"), + ("null", "reviews.null"), + ("0x10", "reviews.0x10"), + ('"true"', "reviews.true"), + ): + with self.subTest(source=source): + findings, _ = checker.validate(f"reviews:\n {source}: x\n", SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + _severity, path, line, _message = findings[0] + self.assertEqual(path, expected) + self.assertEqual(line, 2) + + def test_a_generic_key_gets_no_did_you_mean_rather_than_three_wrong_ones(self): + # `enabled` sits at ~72 places in the schema. The hint was written for the + # root-level `tools:` case, where the one other path is almost certainly + # the intended home; for a generic name the alphabetically-first three + # were three wrong homes (`issue_enrichment.*` for `reviews.tools`). + findings, _ = checker.validate("reviews:\n tools:\n enabled: true\n", SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + self.assertNotIn("Did you mean", findings[0][3]) + + def test_the_motivating_misplaced_block_still_gets_its_hint(self): + # The regression guard on the test above: suppressing generic names must + # not suppress the case the hint exists for. + findings, _ = checker.validate(MISPLACED_TOOLS, SCHEMA) + hinted = [f for f in findings if "Did you mean `reviews.tools`?" in f[3]] + self.assertEqual(len(hinted), 1, msg=f"{messages(findings)}") + + def test_a_shape_disagreement_is_left_to_jsonschema(self): + # `reviews.profile` is a string enum. A mapping there is a type error and + # jsonschema says so; the walk must not add a second, differently-worded + # complaint about the keys inside it. + findings, _ = checker.validate("reviews:\n profile:\n nope: 1\n", SCHEMA) + self.assertTrue(findings) + # Every finding is jsonschema's own (its `type` message quotes the + # instance, so `nope` legitimately appears in the text); what must NOT + # exist is a second finding AT the inner key's path. + self.assertEqual(set(severities(findings)), {"error"}) + self.assertNotIn("reviews.profile.nope", [f[1] for f in findings]) + + def test_the_value_under_an_unknown_key_is_not_descended_into(self): + # One finding for the unknown key, not one per leaf beneath it: the + # schema names no subschema for it, so nothing under it is judgeable. + text = "reviews:\n profil:\n a: 1\n b: 2\n" + findings, _ = checker.validate(text, SCHEMA) + self.assertEqual([f[1] for f in findings], ["reviews.profil"]) + + def test_findings_count_toward_the_shared_cap_and_note_the_truncation(self): + # The cap is over BOTH halves. Half the unknown keys here are root-level + # (jsonschema's) and half are nested (the walk's). + root = "".join(f"unknown_root_{i}: 1\n" for i in range(60)) + nested = "reviews:\n" + "".join(f" unknown_nested_{i}: 1\n" for i in range(60)) + findings, notes = checker.validate(root + nested, SCHEMA) + self.assertEqual(len(findings), checker.MAX_FINDINGS) + self.assertTrue(any("only the first" in n for n in notes), msg=f"{notes}") + + def test_a_bounded_walk_says_so_rather_than_reporting_clean(self): + # A partial scan must never read like a completed one. Forced via the + # budget rather than a megabyte fixture. + with budget(keys=1): + findings, _notes = checker.validate(STALE_GITHUB_CHECKS, SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + severity, path, _line, message = findings[0] + self.assertEqual(severity, "warning") + self.assertEqual(path, "(document root)") + self.assertIn("were NOT checked", message) + + def test_a_bounded_walk_fails_under_strict_rather_than_exiting_zero(self): + # A note does not reach the exit code, so a walk that gave up early used + # to exit 0 even for a repo that asked for unknown keys to FAIL — a green + # check over keys nothing looked at. Warn-only mode is unchanged. + with budget(keys=1): + warn, _ = checker.validate(STALE_GITHUB_CHECKS, SCHEMA) + strict, _ = checker.validate( + STALE_GITHUB_CHECKS, SCHEMA, strict_unknown_keys=True + ) + self.assertEqual(severities(warn), ["warning"]) + self.assertEqual(severities(strict), ["error"]) + + def test_a_depth_cutoff_is_reported_like_an_exhausted_budget(self): + # Both are "I did not see the whole document"; only one used to be said. + with budget(depth=0): + findings, _ = checker.validate(STALE_GITHUB_CHECKS, SCHEMA) + self.assertEqual(len(findings), 1, msg=f"{messages(findings)}") + self.assertIn("were NOT checked", findings[0][3]) + + def test_a_budget_landing_exactly_on_the_last_key_is_not_partial(self): + # The flag is set where the walk actually bails, not derived from the + # leftover budget: a document walked in FULL on its last unit of budget + # is complete, and saying otherwise cries wolf on every exact fit. + text = "reviews:\n profil: chill\n" + with budget(keys=4): # two node visits + two keys + exact, _ = checker.validate(text, SCHEMA) + with budget(keys=3): + short, _ = checker.validate(text, SCHEMA) + self.assertEqual([f[1] for f in exact], ["reviews.profil"]) + self.assertEqual([f[1] for f in short], ["(document root)"]) + + def test_list_traversal_is_charged_against_the_walk_budget(self): + # Elements recurse without naming a key, so a key-only budget left array + # traversal outside the bound entirely — and a YAML-aliased document far + # under the 512 KiB cap can alias one inner list into thousands of outer + # entries, each charging 1 while everything below it is visited free. + text = ( + "a: &a [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n" + "b: &b [*a, *a, *a, *a, *a, *a, *a, *a, *a, *a]\n" + "c: [*b, *b, *b, *b, *b, *b, *b, *b, *b, *b]\n" + ) + nested = {"type": "array", "items": {"type": "array", "items": { + "type": "array", "items": {"type": "object", "properties": {}}}}} + data = yaml.safe_load(text) + with budget(keys=50): # far below the 1000+ element visits below `c` + _findings, bounded = checker._walk_unknown_keys( + {"c": data["c"]}, + {"type": "object", "properties": {"c": nested}}, + None, + {}, + "warning", + 100, + ) + self.assertTrue(bounded) + + def test_one_error_fanning_out_still_respects_the_findings_cap(self): + # `islice` bounds the ERROR count, not the finding count: a closed + # mapping reports every extra key in ONE error, which `_extra_keys` + # splits into one finding each. 150 unknown root keys used to emit 150 + # annotations into a public run log with no truncation note at all. + text = "".join(f"unknown_root_{i}: 1\n" for i in range(checker.MAX_FINDINGS + 50)) + findings, notes = checker.validate(text, SCHEMA) + self.assertEqual(len(findings), checker.MAX_FINDINGS) + self.assertTrue(any("only the first" in n for n in notes), msg=f"{notes}") + + def test_main_exits_zero_by_default_and_one_under_strict(self): + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + with open(os.path.join(tmp, ".coderabbit.yaml"), "w", encoding="utf-8") as f: + f.write(STALE_GITHUB_CHECKS) + buf = io.StringIO() + with redirect_stdout(buf): + code = checker.main(["--root", tmp, "--schema", SCHEMA_PATH]) + out = buf.getvalue() + self.assertEqual(code, 0, msg=out) + self.assertIn("::warning ", out) + self.assertIn("timeout_ms", out) + + buf = io.StringIO() + with redirect_stdout(buf): + code = checker.main( + ["--root", tmp, "--schema", SCHEMA_PATH, "--strict-unknown-keys"] + ) + out = buf.getvalue() + self.assertEqual(code, 1, msg=out) + self.assertIn("::error ", out) + + +class OpennessTest(unittest.TestCase): + """Facts about the VENDORED schema that decide what the walk reports. + + The walk's whole premise is that "declares `properties`, declares no opener" + means closed. If a schema refresh starts using a keyword this file does not + know, the refresh PR should say so out loud rather than quietly changing what + the check means. + """ + + @staticmethod + def _nodes(schema): + """Every subschema reachable through properties / items / combinators.""" + seen = [] + + def walk(node): + if isinstance(node, list): + for item in node: + walk(item) + return + if not isinstance(node, dict): + return + seen.append(node) + for key in ( + "properties", + "patternProperties", + "$defs", + "definitions", + "dependentSchemas", + ): + for sub in (node.get(key) or {}).values(): + walk(sub) + for key in ("items", "additionalProperties", "propertyNames", "not", + "if", "then", "else", "unevaluatedProperties"): + walk(node.get(key)) + for key in ("anyOf", "oneOf", "allOf"): + walk(node.get(key)) + + walk(schema) + return seen + + def test_the_two_halves_partition_the_objects_that_declare_properties(self): + explicit, omitted = [], [] + for node in self._nodes(SCHEMA): + if not isinstance(node.get("properties"), dict): + continue + if any(k in node for k in checker._OPENER_KEYWORDS): + explicit.append(node) + else: + omitted.append(node) + # Not an arbitrary snapshot: this is the size of the gap being closed. + self.assertEqual(len(explicit), 5) + self.assertEqual(len(omitted), 103) + + def test_every_explicitly_open_object_uses_additional_properties(self): + # The audit behind `_OPENER_KEYWORDS`: the schema uses one opener today, + # and the wider list is future-proofing, not a description of it. + for node in self._nodes(SCHEMA): + if not isinstance(node.get("properties"), dict): + continue + used = [k for k in checker._OPENER_KEYWORDS if k in node] + self.assertIn(used, ([], ["additionalProperties"]), msg=f"{used}") + + def test_the_schema_carries_no_ref(self): + # `_walk_unknown_keys` claims the two unknown-key halves never report the + # same key. That is a fact about THIS schema, not about the conditions: + # jsonschema evaluates a document path against every applicable subschema, + # so a node this walk judges closed (no opener of its OWN) whose `$ref` + # target carried `additionalProperties: false` would be reported twice. + # A combinator sibling cannot do it — the test above pins that a node + # declaring `properties` never carries one — but a `$ref` could, so a + # refresh that introduces one should fail HERE, where the docstring's + # premise is written down, rather than start double-annotating real PRs. + raw = json.dumps(SCHEMA) + self.assertNotIn('"$ref"', raw) + self.assertNotIn('"$dynamicRef"', raw) + + def test_the_document_root_is_the_explicitly_closed_case(self): + self.assertIs(SCHEMA.get("additionalProperties"), False) + + def test_github_checks_names_only_enabled(self): + # If upstream ever adds `timeout_ms` back, the regression anchor above + # stops being a finding and this fails first, naming why. + tools = SCHEMA["properties"]["reviews"]["properties"]["tools"]["properties"] + self.assertEqual(list(tools["github-checks"]["properties"]), ["enabled"]) + + if __name__ == "__main__": unittest.main() diff --git a/.github/workflows/test-coderabbit-config-validate.yml b/.github/workflows/test-coderabbit-config-validate.yml index 2f19a7f4..acc78c27 100644 --- a/.github/workflows/test-coderabbit-config-validate.yml +++ b/.github/workflows/test-coderabbit-config-validate.yml @@ -158,6 +158,36 @@ jobs: echo "$out" echo "$out" | grep -q '::error ' + - name: Smoke test — a typo INSIDE an object warns, then fails under strict + run: | + # The closed-by-omission half. `reviews.tools.github-checks` names only + # `enabled` and says nothing about the rest, so jsonschema has no + # keyword to fire on — `timeout_ms` is a key upstream removed that is + # still live in several org repos, doing nothing. A regression that + # loses this path is invisible from the default-mode exit code alone, + # which is why the strict direction is asserted too. + root="$(mktemp -d)" + cat > "$root/.coderabbit.yaml" <<'YAML' + reviews: + tools: + github-checks: + enabled: true + timeout_ms: 90000 + YAML + out="$(python3 "$CHECK" --root "$root")" # must NOT fail by default + echo "$out" + echo "$out" | grep -q '::warning ' + echo "$out" | grep -q 'unknown key `timeout_ms`' + echo "$out" | grep -q 'reviews.tools.github-checks' + + if out="$(python3 "$CHECK" --root "$root" --strict-unknown-keys)"; then + echo "::error::strict_unknown_keys did not fail a stale key inside a tool object" + echo "$out" + exit 1 + fi + echo "$out" + echo "$out" | grep -q '::error ' + - name: Smoke test — malformed YAML fails run: | root="$(mktemp -d)" diff --git a/AGENTS.md b/AGENTS.md index 2a853b32..553a42fe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -144,8 +144,8 @@ tests — run the matching command above for whatever you touched. against the vendored CodeRabbit schema, on the PR that breaks it. CodeRabbit rejects an invalid config WHOLE (everything in it goes inert) and validates the BASE branch, so the breakage otherwise surfaces one PR late on the wrong diff. - Parse/`maxLength`/type errors FAIL; an unknown key WARNS (only where the schema - CLOSES the object) unless `strict_unknown_keys: true`; an absent file passes, + Parse/`maxLength`/type errors FAIL; an unknown key WARNS wherever its object is + closed (`additionalProperties: false`, 5; or by omission, 103) unless `strict_unknown_keys: true`; an absent file passes, though the other extension spelling is validated, not called absent. A SIBLING of agents-md-integrity, deliberately not an extension of it — that workflow is scoped to agent-instruction files, and folding this in would push a Python diff --git a/docs/callers/coderabbit-config-validate.md b/docs/callers/coderabbit-config-validate.md index f4483e5d..72a3137c 100644 --- a/docs/callers/coderabbit-config-validate.md +++ b/docs/callers/coderabbit-config-validate.md @@ -29,6 +29,7 @@ Severity is split, mirroring what CodeRabbit itself does with each problem: | No `.coderabbit.yaml` at all | **passes** | Reported in the log, so "no config here" never reads the same as "config validated clean". | | `.coderabbit.yml` present under the default name | **validated, with a warning** | CodeRabbit honours both spellings, so the file that exists is the config in effect. Validating it is the point; the warning tells you to set `config_file` explicitly. | | Config path outside the repo, not a regular file, or over 512 KiB | **errors (exit 2)** | "I could not check" must never look like a pass. Symlinks are resolved before the containment test. | +| Unknown-key scan stopped early (a huge or heavily aliased document) | **warns; fails under `strict_unknown_keys`** | Same rule as the row above, applied to a *partial* scan: keys past the cutoff were never inspected, so a repo that asked for unknown keys to fail must not get a green check over them. | The warning class is not cosmetic. The most common instance is a `tools:` block written at the document root instead of under `reviews:`; the root is closed, so @@ -38,12 +39,56 @@ defaults to `true`, so a root-level `enabled: false` runs the linter it meant to disable). The annotation names the offending key path and suggests where it belongs. -One limit worth knowing: the unknown-key check fires where the schema CLOSES an -object, which upstream does at the document root and for a handful of nested -objects. `reviews`, `chat`, `knowledge_base`, `code_generation` and the individual -tool configs are open, so a typo *inside* one (`reviews.profil`) is not flagged -even under `strict_unknown_keys`. What is caught is a key in the wrong PLACE — -the failure that actually recurs. +The unknown-key check fires wherever a schema object accepts only the property +names it lists. Upstream says that two ways and both are checked: **explicitly**, +with `additionalProperties: false` (five objects, including the document root), +and **by omission** — a `properties` block and nothing said about anything else +(103 objects, including `reviews`, `chat`, `knowledge_base`, `code_generation` and +every individual tool config). So a key in the wrong PLACE is caught, and so is a +typo *inside* an object (`reviews.profil`, `golangci-lint.enabld`) and a key +upstream has since removed. + +The by-omission rule is deliberately conservative: an object counts as closed only +if it declares `properties` and none of `additionalProperties`, +`unevaluatedProperties`, `patternProperties`, `anyOf`, `oneOf`, `allOf`, `$ref`, +`$dynamicRef`, `if`, `dependentSchemas`, `propertyNames` or `not` — each of which +is a way the schema legitimately accepts a name it does not list. Nothing is +reported under `reviews.mutually_exclusive_groups` (whose group names are yours to +choose) or inside `knowledge_base.code_guidelines.filePatterns[]` (an `anyOf` +shape), and a document/schema shape disagreement is left to the type error that +already describes it. + +**Rolling this out to a repo that already has a config.** A repo carrying +`reviews.tools.github-checks.timeout_ms` — a key upstream removed, still present +in several org repos — will see a NEW warning it did not see before. That is the +point of the check: the key does nothing today. Default mode still exits 0, so +nothing turns red on enrollment. If you plan to set `strict_unknown_keys: true`, +run the checker locally first and remove the stale keys it names — otherwise the +strict switch turns them into a hard failure on whatever PR happens to be open: + +```bash +python3 -m pip install --require-hashes --only-binary=:all: \ + -r .github/coderabbit-config/requirements.txt # from a checkout of Comfy-Org/github-workflows +python3 .github/coderabbit-config/check_coderabbit_config.py --root /path/to/your/repo +``` + +**The by-omission half has a staleness window — mind it before turning strict on.** +The 103 by-omission objects are read as closed against the schema *vendored at +your pinned `workflows_ref`*, so a property upstream ADDS — a new linter under +`reviews.tools`, a new knob on an existing one — is an unknown key to this +checker until two things land: the weekly `refresh-coderabbit-schema.yml` PR in +`Comfy-Org/github-workflows`, **and** your repo's SHA bump on top of it. In +default warn-only mode that window costs you a warning on a config CodeRabbit is +perfectly happy with. Under `strict_unknown_keys: true` it is a hard CI failure, +on whatever unrelated PR happens to be open, over a key you were right to add. +The five *explicitly* closed objects (the document root among them) have no such +window — jsonschema reports those straight from the keyword. + +That is the trade `strict_unknown_keys` makes, not a bug: the same freshness that +lets the checker catch a key upstream REMOVED is what makes it briefly wrong about +a key upstream ADDED. If you hit it, the escape hatch is to flip +`strict_unknown_keys` back to `false` for the one PR and merge the schema refresh +— not to delete the key. The checker and the schema it validates against live in [`.github/coderabbit-config/`](../../.github/coderabbit-config) and are loaded