Skip to content

feat(review): opt-in test-gap check and focused AI batches; verifier reads every changed file - #1772

Open
RaghwendraSingh wants to merge 8 commits into
feat/review-ai-input-coveragefrom
feat/review-ai-focused-batches
Open

RaghwendraSingh wants to merge 8 commits into
feat/review-ai-input-coveragefrom
feat/review-ai-focused-batches

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Changes since review: cf6d583 (review comments): the verifier's read allowlist drops excluded and build/vendored paths, and doc() names from installed packages count as defined. 7b5bb4b (review feedback): credential files are never readable by the verifier, focused batches honour batchChars, a deletion-only YAML break anchors on the parser's error line, and focus settings require positive integers (ALTIMATE_REVIEW_AI_FOCUS_MAX_BATCHES).

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Stacked on #1771. Adds two opt-in AI-lane options, lets the verifier read every changed file, and fixes the rule gaps the planted-bug suite found. Defaults are unchanged.

What changed

  • ALTIMATE_REVIEW_AI_TEST_GAP=1: when a changed macro, materialization or generic test changes runtime behaviour, the AI says whether the PR adds a test that runs it, as one test_coverage comment. Each batch also gets the PR's other changed paths.
  • ALTIMATE_REVIEW_AI_FOCUSED=1: on PRs with 7 or more files, related files are reviewed together in batches of up to 5. Related means a macro with its dispatch variants, a test with the model it refs, and a yml with the models it declares. Grounding is filtered per batch, findings are deduplicated, and there are at most 4 batches. No paid measurement yet, so it stays opt-in.
  • The verifier can read every file the PR changes, so it can confirm "no test was added".
  • Rule fixes from the planted-bug suite:
    • three recall rules;
    • the join chain across a join with no ON clause;
    • parse-breaking findings are kept in dev, sandbox and scratch folders.

Results

Test-gap check

  • A private 5-project monorepo PR that changed a materialization with no test: the AI raised the missing test in 10 of 10 runs, against 2 of 21 before.
  • 10 tuning-repo PRs: catches unchanged (5 of 16), useful comments 51% → 57%, wrong 3% → 0% (blind Claude judge), tokens 1.02×.

Planted-bug suite on repos never used for tuning: 236 of 238 caught, 1 false alarm on 237 no-defect twins.

How did you verify your code works?

  • bun test test/altimate/review: 748 pass.
  • bunx tsc --noEmit: clean.
  • New: test/altimate/review-ai-test-gap.test.ts, plus focused-batch cases in review-ai-inputs.test.ts.

Cost side effects

None by default; both options are opt-in. ALTIMATE_REVIEW_AI_TEST_GAP=1 measured at about 1.02x AI-lane tokens. ALTIMATE_REVIEW_AI_FOCUSED=1 splits PRs with 7 or more files into at most 4 batches, so up to 4 AI calls per review instead of one (not yet measured, hence opt-in). Letting the verifier read every changed file only applies when the verifier is enabled and stays within its 12k-token cap. No warehouse queries or cloud resources.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c5a7cfa5-4a44-4207-af66-4fff05aa616b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

const others = [...new Set([...newContent.matchAll(DOC_REF)].map((m) => m[1]))].filter((n) => !missing.includes(n)).slice(0, 5)
let resolves = false
for (const n of others) if ((resolves = await docExists(n))) break
if (!others.length && projectHasDocs) resolves = await projectHasDocs()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: projectHasDocs() cannot tell a project docs block from one that resolves in an installed package

docExists only greps the dbt project's own paths (grepProject pathspecs come from the dbt_project.yml model/macro/seed/snapshot/test dirs, run.ts:431-438), so a doc('name') defined by an installed package already counts as "missing". This new fallback then treats any docs block anywhere in the project as proof that the name resolves, so for a file whose only doc() is the new one (typically a newly added schema yml) a package-provided doc is reported as "dbt fails to parse" — exactly the package case the guard above was added to avoid. The finding body also says "the other doc() references in this file resolve there", which is false on this branch (there are none). Consider also searching installed-package .md files before concluding the name is missing, or keep the previous "another doc() in this file resolves" requirement for names that could come from a package.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 31cb5c4: doc() names defined in the .md files of installed packages (dbt_packages/) now count as defined. The no-other-doc() fallback also requires the packages' blocks to be known (no packages declared, or packages installed), so a package-provided doc() stays silent when packages aren't installed. The body now says why the finding is trusted ("the other doc() references resolve" or "the project defines its own docs blocks") and no longer claims the first on the fallback path.

// The project defines docs blocks of its own (any `{% docs x %}` in its .md files).
let docsAny: Promise<boolean> | undefined
const projectHasDocs = () =>
(docsAny ??= (input.grepProject?.("docs", "*.md") ?? Promise.resolve([]))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: the generic docs pattern can exhaust the search cap before reaching a docs block

grepFixedAt stops after 50 total hits (git.ts:194-210; -m 50 per file), and docs shows up in ordinary markdown prose, so those hits can be used up before the file holding {% docs ... %} is scanned. projectHasDocs then returns false and the new doc-block-missing finding stays silent — the recall gap this change targets. A block-specific fixed string (e.g. separate searches for {% docs and {%- docs) would keep unrelated prose from diluting the result.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Hitting the cap makes the rule go silent (a missed finding, not a wrong one); grepping for {% docs / {%- docs fixes it.

const TEST_GAP_FILES_MAX = 40

/** The PR's other changed paths, appended to a batch message for the test-gap instruction. */
export function testGapFileList(message: string, batch: AiReviewFile[], prFiles: string[] = []): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: three copies of the same "other PR paths" renderer

This function, the focused-batch ## OTHER FILES CHANGED IN THIS PR list in buildBatchMessage (lines 777-782), and buildVerifyMessage's "Files this PR changes" list (ai-verify.ts:386-402) each implement slice(0, 40) + "… and N more" with their own 40 constant (TEST_GAP_FILES_MAX, FOCUS_OTHER_FILES, PR_FILES_MAX). With ALTIMATE_REVIEW_AI_FOCUSED=1 and ALTIMATE_REVIEW_AI_TEST_GAP=1, the same batch message carries both lists and repeats every other path in the prompt. A single shared helper (and cap) would remove the duplication and the double listing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair; we'll fold them into one renderer in a cleanup pass.

* be dropped, or null to keep it. Centralizing this here (vs prompt text) is
* what keeps the false-positive rate down deterministically.
*/
/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: the exclusionReason JSDoc now documents this constant

The /** Apply the rubric's exclusion predicates ... */ block above now attaches to PROJECT_BREAKING_RULES, so exclusionReason loses its doc and editors/TS show the wrong description for it. Move the constant above the JSDoc, or give the constant its own comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the doc placement; we'll move it in the next touch of rubric.ts. (kilo, misplaced exclusionReason JSDoc)

@kilo-code-bot

kilo-code-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/rubric.ts 74 PROJECT_BREAKING_RULES still omits dbt-project-rules:ref-target-missing, whose finding body says dbt fails to parse the project (project-rules.ts:874), so a typo'd ref() under dev/, sandbox/ or scratch/ is still dropped with skipNonProdModels on by default

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 142 The test-gap ## OTHER FILES THIS PR CHANGES list is appended to plan.messages at send time (ai-review.ts:952), after the "Return ONLY the JSON array per the output contract." line (ai-review.ts:855), so the contract is no longer final and plan.messages differs from what runAiReview sends
packages/opencode/src/altimate/review/run.ts 626 packageDocNames reads packages.yml from the commit tree but dbt_packages docs from the working tree; an uncommitted packages.yml yields an authoritative empty set and a false doc-block-missing (mirror projectResourceNames' working-tree merge at run.ts:661)
packages/opencode/test/altimate/review-project-rules.test.ts 806 The added single-quote escape case is excluded by the rule's `unless: /\b(?:log
Files Reviewed (10 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - clean
  • packages/opencode/src/altimate/review/project-rules.ts - clean
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - 1 issue

No review-file delta exists between the previously reviewed 7b5bb4b181 and current head a2d91cefe (git diff 7b5bb4b..HEAD is empty for every file this PR changes), so the four open findings were re-verified against current code and no new findings were raised. Previously verified in 7b5bb4b: packFocusedBatches enforces batchChars at packing, merge and balancing; credential paths are filtered from prFiles, the verifier allowlist, grep_project hits and the AI supporting-file list; the YAML anchor prefers added lines and no longer anchors to unrelated lines.

Previous Review Summaries (4 snapshots, latest commit 7b5bb4b)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 7b5bb4b)

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/rubric.ts 74 PROJECT_BREAKING_RULES still omits dbt-project-rules:ref-target-missing, whose finding body says dbt parse fails, so a typo'd ref() under dev/, sandbox/ or scratch/ is still dropped with skipNonProdModels on by default (unchanged by 7b5bb4b)

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 142 The test-gap ## OTHER FILES THIS PR CHANGES list is appended after the "Return ONLY the JSON array" contract, so the contract is no longer final and plan.messages differs from what runAiReview sends
packages/opencode/src/altimate/review/run.ts 626 packageDocNames reads packages.yml from the commit tree but dbt_packages docs from the working tree; an uncommitted packages.yml yields an authoritative empty set and a false doc-block-missing (mirror projectResourceNames' working-tree merge)
packages/opencode/test/altimate/review-project-rules.test.ts 806 The added single-quote escape case is excluded by the rule's log( unless clause, so it cannot fail and the new (?<!\\) guard is untested
Files Reviewed (10 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - clean
  • packages/opencode/src/altimate/review/project-rules.ts - clean
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - 1 issue

Verified in 7b5bb4b: packFocusedBatches now enforces batchChars at packing, merge and balancing; credential paths are filtered from prFiles, the verifier allowlist, grep_project hits and the AI supporting-file list; the YAML anchor prefers added lines and no longer anchors to unrelated lines. The remaining parse-anchor edge (a context-line anchor becomes preExisting) matches pre-existing behaviour and is not counted here.

Fix these issues in Kilo Cloud

Previous review (commit ddc4333)

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/rubric.ts 86 PROJECT_BREAKING_RULES omits dbt-project-rules:ref-target-missing, whose body says dbt fails to parse, so a typo'd ref() under dev/, sandbox/ or scratch/ is still dropped with skipNonProdModels on by default
packages/opencode/src/altimate/review/ai-review.ts 598 packFocusedBatches never consults budget.batchChars, so with focused + coverage flags one call can carry ~220k chars against the 120k per-call budget COVERAGE_BATCH_CHARS exists to enforce

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/run.ts 620 packageDocNames reads packages.yml from the commit tree but package docs from the working tree; an uncommitted packages.yml yields an authoritative empty set and a false doc-block-missing (mirror projectResourceNames' working-tree merge)
packages/opencode/src/altimate/review/ai-review.ts 911 The test-gap file list is appended after the "Return ONLY the JSON array" line, so the output contract is no longer final and plan.messages no longer matches what is sent
packages/opencode/test/altimate/review-project-rules.test.ts 787 The added single-quote escape case is excluded by the rule's log( unless clause, so it cannot fail and the new (?<!\\) guard is untested
Files Reviewed (17 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 2 issues
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/dbt-patterns.ts - clean
  • packages/opencode/src/altimate/review/diff-filter.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - clean
  • packages/opencode/src/altimate/review/project-rules.ts - clean
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/rule-catalog.ts - clean
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-coverage.txt - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-dbt-patterns.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - 1 issue
  • packages/opencode/test/altimate/review.test.ts - clean

Fix these issues in Kilo Cloud

Previous review (commit 31cb5c4)

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/rubric.ts 86 The new parse-breaking exemption set omits dbt-project-rules:ref-target-missing (whose body says dbt fails to parse), so a typo'd ref() under dev/, sandbox/ or scratch/ is still dropped with skipNonProdModels on by default
packages/opencode/src/altimate/review/ai-review.ts 583 packFocusedBatches never consults budget.batchChars, so with focused + coverage flags a single call can carry ~220k chars against the 120k per-call budget COVERAGE_BATCH_CHARS exists to enforce

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/run.ts 620 packageDocNames reads packages.yml from the commit tree but package docs from the working tree; an uncommitted packages.yml yields an authoritative empty set and a false doc-block-missing (mirror projectResourceNames' working-tree merge)
packages/opencode/src/altimate/review/ai-review.ts 894 The test-gap file list is appended after the "Return ONLY the JSON array" line, so the output contract is no longer final and plan.messages no longer matches what is sent
packages/opencode/test/altimate/review-project-rules.test.ts 736 The added single-quote escape case is excluded by the rule's log( unless clause, so it cannot fail and the new (?<!\\) guard is untested
Files Reviewed (17 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 2 issues
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/dbt-patterns.ts - clean
  • packages/opencode/src/altimate/review/diff-filter.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - clean
  • packages/opencode/src/altimate/review/project-rules.ts - clean
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/rule-catalog.ts - clean
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-coverage.txt - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-dbt-patterns.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - 1 issue
  • packages/opencode/test/altimate/review.test.ts - clean

Fix these issues in Kilo Cloud

Previous review (commit 1acd5d8)

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/project-rules.ts 506 New projectHasDocs() fallback treats any project docs block as proof the new doc() resolves, so a package-provided doc in a file whose only doc() is the new one is reported as a parse error (and the body text about "other doc() references" is false on this branch)

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2287 Generic docs fixed-string search can consume the 50-hit grepFixedAt cap on markdown prose before reaching the docs block, silently returning false
packages/opencode/src/altimate/review/ai-review.ts 113 Three copies of the "other PR paths" list renderer (test-gap, focused batches, verifier) with three separate 40 caps; both flags on duplicates every other path in a batch message
packages/opencode/src/altimate/review/rubric.ts 78 The exclusionReason JSDoc now attaches to the new PROJECT_BREAKING_RULES constant, leaving the function undocumented/misdocumented
Files Reviewed (16 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/dbt-patterns.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/project-rules.ts - 1 issue
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/rule-catalog.ts - clean
  • packages/opencode/src/altimate/review/run.ts - clean
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-coverage.txt - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-dbt-patterns.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - clean
  • packages/opencode/test/altimate/review.test.ts - clean

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 69.8K · Output: 8.7K · Cached: 566K

Review guidance: REVIEW.md from base branch feat/review-ai-input-coverage

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10 issues found across 16 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/review/run.ts">

<violation number="1" location="packages/opencode/src/altimate/review/run.ts:440">
P2: `prFiles` includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out `status === "deleted"` before mapping to paths.</violation>
</file>

<file name="packages/opencode/test/altimate/review-ai-test-gap.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review-ai-test-gap.test.ts:29">
P3: This test depends on the real native `altimate_core.review_ai_prompt` call, but it never stubs `Dispatcher.call`. If the native core is unavailable, `runAiReview` returns early and `LLM.stream` is never called, so the test fails on an undefined `off`. Stub it with `spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any)`, as the replay test does.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:117">
P2: For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-review.ts:579">
P2: With `ALTIMATE_REVIEW_AI_MAX_BATCHES` between 0 and 1 (e.g. 0.5), `Array.from({ length: maxBatches })` creates no bins and the `reduce` throws, and `runAiReview` rejects instead of returning a status. Clamp the bin count to a positive integer.</violation>
</file>

<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">

<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:933">
P2: A `join ... using (...)` has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a `USING (...)` list.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-verify.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-verify.ts:395">
P2: The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-verify.ts:783">
P2: The verifier's read allowlist now includes every PR file, including paths the user excluded with `exclude` or rubric `excludeGlobs` and out-of-project files. `prFiles` comes from the unfiltered `collected` list in run.ts, so those contents can reach the model provider. Build `prFiles` from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.</violation>
</file>

<file name="packages/opencode/src/altimate/review/orchestrate.ts">

<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:2287">
P2: The generic `docs` search can exhaust `grepFixedAt`’s 50-hit limit on Markdown prose before reaching a docs block, causing `projectHasDocs()` to suppress missing-`doc()` findings. Use a targeted, complete docs-block search instead.</violation>
</file>

<file name="packages/opencode/src/altimate/review/rule-catalog.ts">

<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:775">
P3: The lookbehind exempts any seam character preceded by a backslash, so a literal ending in `\\n` (escaped backslash plus `n`) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.</violation>
</file>

<file name="packages/opencode/src/altimate/review/rubric.ts">

<violation number="1" location="packages/opencode/src/altimate/review/rubric.ts:79">
P3: Move `PROJECT_BREAKING_RULES` above the `exclusionReason` JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))
return grepFixedAt(gitRoot, opts.head, pattern, specs.filter((d) => !d.startsWith("..")))
},
prFiles: collected.map((f) => f.path),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: prFiles includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out status === "deleted" before mapping to paths.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/run.ts, line 440:

<comment>`prFiles` includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out `status === "deleted"` before mapping to paths.</comment>

<file context>
@@ -437,6 +437,7 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise
       const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))
       return grepFixedAt(gitRoot, opts.head, pattern, specs.filter((d) => !d.startsWith("..")))
     },
+    prFiles: collected.map((f) => f.path),
     prTitle: opts.prTitle,
     prBody: opts.prBody,
</file context>
Suggested change
prFiles: collected.map((f) => f.path),
prFiles: collected.filter((f) => f.status !== "deleted").map((f) => f.path),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Filtering out deleted would also stop the verifier reading a deleted file's base side, so it needs a small design call first.

}
if (batches.length <= maxBatches) return batches
const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)
const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: With ALTIMATE_REVIEW_AI_MAX_BATCHES between 0 and 1 (e.g. 0.5), Array.from({ length: maxBatches }) creates no bins and the reduce throws, and runAiReview rejects instead of returning a status. Clamp the bin count to a positive integer.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 579:

<comment>With `ALTIMATE_REVIEW_AI_MAX_BATCHES` between 0 and 1 (e.g. 0.5), `Array.from({ length: maxBatches })` creates no bins and the `reduce` throws, and `runAiReview` rejects instead of returning a status. Clamp the bin count to a positive integer.</comment>

<file context>
@@ -431,6 +479,126 @@ export function batchAiFiles(files: AiReviewFile[], budget: AiInputBudget): AiRe
+  }
+  if (batches.length <= maxBatches) return batches
+  const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)
+  const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))
+  chunks
+    .map((c, i) => ({ i, chars: size(c) }))
</file context>
Suggested change
const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))
const bins = Array.from({ length: Math.max(1, Math.floor(maxBatches)) }, () => ({ chars: 0, chunks: [] as number[] }))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. The caller catches it and the lane reports error; Math.max(1, Math.floor(...)) fixes it.

if (!tail) {
joins.push({ preds: [], start: m.index!, end: i, inner })
// A join with no ON (`cross join params as p`): step over its alias so the chain continues.
const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A join ... using (...) has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a USING (...) list.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 933:

<comment>A `join ... using (...)` has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a `USING (...)` list.</comment>

<file context>
@@ -929,7 +929,9 @@ export function redundantJoinPredicates(sql: string): string[] {
     if (!tail) {
-      joins.push({ preds: [], start: m.index!, end: i, inner })
+      // A join with no ON (`cross join params as p`): step over its alias so the chain continues.
+      const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))
+      joins.push({ preds: [], start: m.index!, end: i + (bare?.[0].length ?? 0), inner })
       continue
</file context>
Suggested change
const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))
const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b|^\s+using\s*\([^)]*\)/i.exec(s.slice(i))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. The result is a missed redundant predicate, never a false one.

...ev.addedLines.keys(),
...graph.parents.map((p) => p.path),
...graph.downstream.map((d) => d.path),
...(ev.prFiles ?? []),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The verifier's read allowlist now includes every PR file, including paths the user excluded with exclude or rubric excludeGlobs and out-of-project files. prFiles comes from the unfiltered collected list in run.ts, so those contents can reach the model provider. Build prFiles from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 783:

<comment>The verifier's read allowlist now includes every PR file, including paths the user excluded with `exclude` or rubric `excludeGlobs` and out-of-project files. `prFiles` comes from the unfiltered `collected` list in run.ts, so those contents can reach the model provider. Build `prFiles` from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.</comment>

<file context>
@@ -763,6 +780,7 @@ export function verifierTools(
       ...ev.addedLines.keys(),
       ...graph.parents.map((p) => p.path),
       ...graph.downstream.map((d) => d.path),
+      ...(ev.prFiles ?? []),
     ]),
   ]
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 31cb5c4: prFiles (the verifier's read allowlist and the test-gap list) now drops paths that match the user's exclude globs, plus build output and vendored paths (isExcludedPath). Changed files outside the dbt project stay readable on purpose (a Python test elsewhere in the repo is evidence for a test-gap claim), as OrchestrateInput.prFiles documents.

const inBatch = new Set(batch.map((f) => f.path))
const others = prFiles.filter((p) => !inBatch.has(p))
if (!others.length) return message
const shown = others.slice(0, TEST_GAP_FILES_MAX)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 117:

<comment>For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.</comment>

<file context>
@@ -76,6 +96,34 @@ export function aiDryRunEnabled(env: Record<string, string | undefined> = proces
+  const inBatch = new Set(batch.map((f) => f.path))
+  const others = prFiles.filter((p) => !inBatch.has(p))
+  if (!others.length) return message
+  const shown = others.slice(0, TEST_GAP_FILES_MAX)
+  return [
+    message,
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll list test-like paths first.

const parts = [`PR: ${(prTitle ?? "(untitled)").slice(0, 200)}`]
// A test-coverage claim is checked against the PR's own test files, which may sit outside the dbt project.
if (prFiles.length && candidates.some((c) => c.finding.category === "test_coverage")) {
const shown = prFiles.slice(0, PR_FILES_MAX)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 395:

<comment>The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.</comment>

<file context>
@@ -378,12 +380,27 @@ export function hunkAround(diff: string | undefined, line?: number): string {
+  const parts = [`PR: ${(prTitle ?? "(untitled)").slice(0, 200)}`]
+  // A test-coverage claim is checked against the PR's own test files, which may sit outside the dbt project.
+  if (prFiles.length && candidates.some((c) => c.finding.category === "test_coverage")) {
+    const shown = prFiles.slice(0, PR_FILES_MAX)
+    parts.push(
+      "",
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as r4627044412; tracked.

const seen: any[] = []
spyOn(Provider, "defaultModel").mockResolvedValue({ providerID: "fake", modelID: "m" } as any)
spyOn(Provider, "getModel").mockResolvedValue(FAKE_MODEL as any)
spyOn(LLM, "stream").mockImplementation(async (input: any) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This test depends on the real native altimate_core.review_ai_prompt call, but it never stubs Dispatcher.call. If the native core is unavailable, runAiReview returns early and LLM.stream is never called, so the test fails on an undefined off. Stub it with spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any), as the replay test does.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-ai-test-gap.test.ts, line 29:

<comment>This test depends on the real native `altimate_core.review_ai_prompt` call, but it never stubs `Dispatcher.call`. If the native core is unavailable, `runAiReview` returns early and `LLM.stream` is never called, so the test fails on an undefined `off`. Stub it with `spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any)`, as the replay test does.</comment>

<file context>
@@ -0,0 +1,58 @@
+    const seen: any[] = []
+    spyOn(Provider, "defaultModel").mockResolvedValue({ providerID: "fake", modelID: "m" } as any)
+    spyOn(Provider, "getModel").mockResolvedValue(FAKE_MODEL as any)
+    spyOn(LLM, "stream").mockImplementation(async (input: any) => {
+      seen.push(input)
+      return {
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll stub it so the test doesn't depend on the native prompt.

// Single- or double-quoted.
added:
// An escape (`\n`) at the seam is not a word.
/'[^'\n]*\s[^'\n]*(?<!\\)[A-Za-z0-9]'\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*'[A-Za-z0-9][^'\n]*\s[^'\n]*'|"[^"\n]*\s[^"\n]*(?<!\\)[A-Za-z0-9]"\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*"[A-Za-z0-9][^"\n]*\s[^"\n]*"/,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The lookbehind exempts any seam character preceded by a backslash, so a literal ending in \\n (escaped backslash plus n) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rule-catalog.ts, line 775:

<comment>The lookbehind exempts any seam character preceded by a backslash, so a literal ending in `\\n` (escaped backslash plus `n`) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.</comment>

<file context>
@@ -769,7 +769,10 @@ const CORRECTNESS: Rule[] = [
+      // Single- or double-quoted.
+      added:
+        // An escape (`\n`) at the seam is not a word.
+        /'[^'\n]*\s[^'\n]*(?<!\\)[A-Za-z0-9]'\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*'[A-Za-z0-9][^'\n]*\s[^'\n]*'|"[^"\n]*\s[^"\n]*(?<!\\)[A-Za-z0-9]"\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*"[A-Za-z0-9][^"\n]*\s[^"\n]*"/,
       macros: true,
     },
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing now: an escaped quote inside a Jinja string literal next to ~ is very rare in dbt code, and the cost is a missed suggestion.

* what keeps the false-positive rate down deterministically.
*/
/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */
const PROJECT_BREAKING_RULES = new Set(["dbt-project-rules:yaml-parse-error", "dbt-project-rules:doc-block-missing"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Move PROJECT_BREAKING_RULES above the exclusionReason JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rubric.ts, line 79:

<comment>Move `PROJECT_BREAKING_RULES` above the `exclusionReason` JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.</comment>

<file context>
@@ -75,12 +75,15 @@ export function blockingCategories(rubric: Rubric): Set<string> {
  * what keeps the false-positive rate down deterministically.
  */
+/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */
+const PROJECT_BREAKING_RULES = new Set(["dbt-project-rules:yaml-parse-error", "dbt-project-rules:doc-block-missing"])
+
 export function exclusionReason(finding: Finding, rubric: Rubric): string | null {
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the doc placement; we'll move it in the next touch of rubric.ts. (cubic, same)

@github-actions

Copy link
Copy Markdown

Thanks for updating your PR! It now meets our contributing guidelines. 👍

const isDev = /(^|\/)(dev|sandbox|scratch)\//.test(finding.file)

if (ex.skipNonProdModels && isDev) return "non-prod model (dev/sandbox/scratch)"
if (ex.skipNonProdModels && isDev && !PROJECT_BREAKING_RULES.has(finding.rule ?? "")) return "non-prod model (dev/sandbox/scratch)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: ref-target-missing is still dropped in dev/sandbox/scratch folders even though it breaks dbt parse

This exemption only keeps yaml-parse-error and doc-block-missing, but missingRefFindings emits dbt-project-rules:ref-target-missing, whose own body says "so dbt fails to parse the project" (project-rules.ts:828). skipNonProdModels defaults to true, so a ref('typo') added under models/dev/, sandbox/ or scratch/ is still filtered out here — the exact case the new constant is meant to stop. Add dbt-project-rules:ref-target-missing to the set (and jinja-unbalanced, which breaks compilation of every dependent model, if that is in scope for this fix).

else batches.push([...c])
}
if (batches.length <= maxBatches) return batches
const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: focused batching ignores the per-call batchChars budget

packFocusedBatches only uses sizes to balance chunks; it never splits a batch when the sections exceed budget.batchChars. batchAiFiles enforces that cap at line 475, and its default under ALTIMATE_REVIEW_AI_COVERAGE=1 is COVERAGE_BATCH_CHARS (120k) precisely so one call does not overrun the context window. With both flags on, whole chunks are packed into bins, so a bin can hold two 5-file chunks (~220k chars at the default ~22k/file), and a smaller ALTIMATE_REVIEW_AI_MAX_BATCHES makes it worse. Split bin contents against budget.batchChars the way batchAiFiles does, or otherwise clamp each call's section size.

if (!files) return undefined
const prefix = projectRootRel ? projectRootRel + "/" : ""
const names = new Set<string>()
if (!files.includes(prefix + "packages.yml") && !files.includes(prefix + "dependencies.yml")) return names

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: a working-tree review treats an uncommitted packages.yml as "no packages"

listFilesAt(cwd, head) falls back to git ls-tree HEAD when head is undefined, so a newly added/uncommitted packages.yml is not in files and this returns an empty set. Callers treat an empty set as authoritative (projectHasDocs then assumes the packages' blocks are known), while dbt_packages below is read from the working tree, so a doc() supplied by an installed package is reported missing. Mirror projectResourceNames, which merges listWorkingTreeFiles when head === undefined (run.ts:655), or return undefined here instead of an empty set.

for (let i = 0; i < batches.length; i += BATCH_CONCURRENCY) {
const slice = batches.slice(i, i + BATCH_CONCURRENCY)
results.push(...(await Promise.all(slice.map((b, j) => reviewBatch(plan.messages[i + j], b, system, model)))))
results.push(...(await Promise.all(slice.map((b, j) => reviewBatch(testGap ? testGapFileList(plan.messages[i + j], b, input.prFiles) : plan.messages[i + j], b, system, model)))))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: appending the test-gap file list at send time breaks the planned message

buildBatchMessage ends with "Return ONLY the JSON array per the output contract.", so the appended ## OTHER FILES THIS PR CHANGES section becomes the last thing the model reads and the contract is no longer final. It also means plan.messages — documented as the messages runAiReview sends exactly (line 818) and what the dry run reports — differs from what is actually sent when ALTIMATE_REVIEW_AI_TEST_GAP=1. Build the list inside buildBatchMessage (before the contract line) so planning and sending stay identical.

expect(seam('{% set q = "select " ~ cols ~ "from " ~ rel %}')).toBe(true)
expect(seam('{% set q = "select " ~ cols ~ " from " ~ rel %}')).toBe(false)
expect(seam('{{ elementary.debug_log("column_snapshot_query - \\n" ~ column_snapshot_query) }}')).toBe(false)
expect(seam("{{ log('query: \\n' ~ q) }}")).toBe(false)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: this case cannot fail, so the new single-quote escape guard is untested

fires() applies the rule's unless clause (rule-catalog.ts:799), and log( matches it, so the assertion is false whether or not (?<!\\) exists in the single-quoted alternative. Use a non-log/print context (a {% set %} or the dbt.datediff(...) call used above) to exercise the new lookbehind; line 749 already covers the log() exclusion.

@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-input-coverage branch from bf6e861 to c99cc33 Compare November 20, 2026 08:22
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-focused-batches branch from 31cb5c4 to ddc4333 Compare November 20, 2026 08:22

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review panel: Claude, Codex and lead. Verdict: request changes for one MAJOR (security guard bypass in the verifier); the rest is in good shape. Defaults are unchanged and the new paths are opt-in. Targeted review tests (333+) and typecheck of the touched code pass.

Counts: CRITICAL 0, MAJOR 1, MINOR 2, NIT 4.

Positives: every new behaviour is behind a switch with a default-output regression snapshot; paired fire/no-fire tests for each rule fix; new rules lean toward silence when unsure.

Missing tests: verifier access to credential paths; focused batching combined with coverage/ALTIMATE_REVIEW_AI_BATCH_CHARS; deletion-only YAML break; a direct test of packageDocNames (only its injected result is exercised).

Scope: matches the description; "three recall rules" is vague, as the package-docs filesystem walk and YAML re-anchoring also change behaviour and could be named. Note: bunx tsc --noEmit showed errors in packages/tui/src/component/dialog-move-session.tsx in one reviewer's environment (outside this diff), so "clean" may not hold everywhere.

Nits

  • ai-review.ts:64-76 and :261-266: with focused and test-gap both on, each call lists other files twice ("OTHER FILES CHANGED IN THIS PR..." and "OTHER FILES THIS PR CHANGES..."). Fix: add only PR paths the focused list doesn't already cover.
  • ai-review.ts:93-98: num() accepts non-integers, e.g. ALTIMATE_REVIEW_AI_FOCUS_BATCH_FILES=2.5. Fix: require Number.isInteger for the three focus settings.
  • rubric.ts:78-80: const PROJECT_BREAKING_RULES = new Set([...]) sits between exclusionReason's doc comment and the function, so the comment now attaches to the constant. Fix: move the constant above that comment.
  • Naming: ALTIMATE_REVIEW_AI_MAX_BATCHES only applies in focused mode; ALTIMATE_REVIEW_AI_FOCUS_MAX_BATCHES would match its FOCUS_MIN_FILES sibling. Also the packFocusedBatches comment says "exactly maxBatches" but it can return fewer.

// Files the PR changes that the AI may list and the verifier may read; never the user's exclusions.
const prFiles = (input.prFiles ?? input.changedFiles.map((f) => f.path)).filter(
(p) => !isExcludedPath(p, input.rubric.exclusions.excludeGlobs),
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] Verifier allowlist bypasses the credential-file exclusion

const prFiles = (input.prFiles ?? input.changedFiles.map((f) => f.path)).filter(
  (p) => !isExcludedPath(p, input.rubric.exclusions.excludeGlobs),
)

The AI file list above (line ~2599) deliberately drops profiles.yml via AI_CREDENTIAL_FILE, but prFiles only drops generated paths and user globs. Because prFiles is added to the verifierTools allowlist, read_file can open it. Scenario: a monorepo PR changes dbt/models/m.sql and config/profiles.yml; with ALTIMATE_REVIEW_AI_VERIFY=1 the verifier calls read_file("config/profiles.yml") and the contents are sent to the model provider. Reproduced with the changed verifierTools in isolation (synthetic file): returns the content with prFiles, returns "outside review scope" without it. .env, *.tfvars and *.pem changed by the PR are reachable the same way.

Fix: also filter AI_CREDENTIAL_FILE (and ideally a small secret-like deny list) when building prFiles, and enforce it in resolve() in ai-verify.ts so every allowlist source is covered. Add a regression test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed in 7b5bb4b (the "verifier credential scope" part). One isCredentialPath in ai-verify.ts covers profiles.yml/.yaml, .env*, .netrc, .pgpass, SSH keys, *.pem/*.key/*.p12/*.pfx and *.tfvars. It is applied in three places:

  • when prFiles is built in orchestrate.ts, so the AI lane's file list and the test-gap list never name these files;
  • inside verifierTools, on the merged allowlist (files, added lines, graph parents/children and prFiles), so no allowlist source can expose them;
  • on grep_project hits.

The AI lane's supporting-file filter now uses the same predicate. Tests: read_file refuses each credential path, even when it comes in through prFiles, addedLines or a graph parent, and grep drops a profiles.yml hit. A runReview-level test checks that the evidence's prFiles excludes them.

if (cur && cur.length + c.length <= batchFiles) cur.push(...c)
else batches.push([...c])
}
if (batches.length <= maxBatches) return batches

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Focused packing ignores batchChars

if (batches.length <= maxBatches) return batches
...
const bin = bins.reduce((best, b) => (b.chars < best.chars ? b : best))

Focused mode replaces batchAiFiles, the only code enforcing budget.batchChars (120k chars in coverage mode, meant to keep one call inside common context windows). Neither the initial packing nor the balancing step into maxBatches bins checks it. Reproduced: 30 model files of ~16k chars with coverage and focused both on gives calls of [7,7,8,8] files at ~[113k,113k,129k,129k] chars; the same input without focused stays under 120k per call. Also, 8 files of 2,500 chars with batchChars: 3000 yields batches of 5 and 3 files.

Fix: honour batchChars when packing/balancing; when it conflicts with maxBatches, either drop the overflow files and record them as unsent, or document that maxBatches wins. Add a test combining focused and coverage mode.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7b5bb4b ("focused batch size"). packFocusedBatches now applies batchChars at every step:

  • a group chunk over the budget is split (as batchAiFiles would split it);
  • consecutive chunks are only merged while they fit;
  • a balanced call that still goes over the budget is split again.

When the two limits conflict, batchChars wins over maxBatches, because it is the context-window guard. The cap is then exceeded rather than files being dropped, and the doc comment says so. Default behaviour is unchanged: outside coverage mode batchChars is unbounded. Tests cover your coverage+focused case (30 large files with a 30k budget: every call fits, more than 4 calls) and the "8 × 2,500 chars with batchChars: 3000" case.

// Some errors carry no line, or one past the edit (an unterminated quote fails where the scan
// ends): anchor on the added line at or before it, else the first added line.
const added = [...walkDiffRows(file.diff)].filter((r) => r.kind === "+")
const at = (err.line ? added.find((r) => r.line === err.line) ?? [...added].reverse().find((r) => r.line < err.line!) : undefined) ?? added[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] YAML parse finding loses its line anchor on deletion-only edits

const at = (err.line ? added.find((r) => r.line === err.line) ?? [...added].reverse().find((r) => r.line < err.line!) : undefined) ?? added[0]

The anchor now only considers added lines. If the breaking edit adds no lines (e.g. the diff deletes only columns:), at is undefined and startLine becomes null, whereas before the finding sat on the parser's error line (often a context line in the hunk). Reproduced: base models:\n - name: a\n columns:\n - name: id, diff deletes the columns: line; parser reports line 3; before = startLine: 3, now = null. Separately, when the error precedes every added line, added[0] may be an unrelated line further down.

Fix: when no added line is at or before err.line, fall back to err.line if it lies inside a hunk; use added[0] only when the error has no line. Add a deletion-only test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7b5bb4b ("YAML anchor"), as you suggested. The anchor order is now:

  1. the added line at the error line;
  2. else the last added line before it;
  3. else the error line itself, when it is a context line of a hunk (the deletion-only case);
  4. else no line anchor (a file-level finding).

added[0] is used only when the parser gives no line. Tests: your deletion-only columns: example (the finding stays on the parser's line), and an error before every added line and outside the hunks (it stays file-level instead of landing on an unrelated added line).

@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-input-coverage branch from c99cc33 to 31b384c Compare November 20, 2026 11:37
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-focused-batches branch from ddc4333 to 7b5bb4b Compare November 20, 2026 11:37
@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

Thanks Harish, and thanks to the panel for the reproductions. All three findings are fixed in 7b5bb4b ("fix(review): address review feedback: verifier credential scope, focused batch size, YAML anchor"), with a regression test for each, plus the nits below. On the missing tests: added tests for verifier access to credential paths, focused batching with coverage and ALTIMATE_REVIEW_AI_BATCH_CHARS, and the deletion-only YAML break. A direct packageDocNames test is not in this commit; I'll add it as a follow-up. On scope: I'll update the description to name the package-docs filesystem walk and the YAML re-anchoring. The packages/tui tsc errors are outside this diff, and tsc --noEmit is clean for everything outside /tui/ here.

Nits, all in the same commit:

  • Duplicate file lists: the test-gap list now leaves out paths the focused "OTHER FILES CHANGED" list already names (listedFiles).
  • The three focus settings now require positive integers, so 2.5 and 0 fall back to the defaults.
  • PROJECT_BREAKING_RULES moved above the exclusionReason doc comment.
  • Renamed ALTIMATE_REVIEW_AI_MAX_BATCHES to ALTIMATE_REVIEW_AI_FOCUS_MAX_BATCHES. The packFocusedBatches comment now says "at most" maxBatches, and explains when batchChars needs more calls.

return [
message,
"",
"## OTHER FILES THIS PR CHANGES (paths only, not shown; use them to judge test coverage)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: the test-gap list lands after the output contract and outside plan.messages

buildBatchMessage already ends with "Return ONLY the JSON array per the output contract.", and this helper appends ## OTHER FILES THIS PR CHANGES after it, so the last thing the model reads is a file list rather than the contract. It also means the string sent at runAiReview differs from plan.messages, which planAiInput documents as the exact user messages and the dry run reports. Build this section inside buildBatchMessage (before the contract line) so planning and sending stay identical.

RaghwendraSingh and others added 8 commits November 21, 2026 13:14
`ALTIMATE_REVIEW_AI_FOCUSED=1` reviews PRs with at least 7 selected files
in several smaller calls of related files instead of one large call.

- `groupAiFiles`: connected components over macro dispatch families
  (`default__x`, `snowflake__x`, `adapter.dispatch('x'`), `ref()` edges and
  yml `- name:` declarations, with a same-directory fallback for files that
  have no edge; components and members keep selection order
- `packFocusedBatches`: chunks of at most 5 files, never split across calls;
  above 4 calls the chunks are balanced by prompt size into exactly 4
- each focused call carries only its own files' grounding and lists the other
  changed files it must not comment on; `ai.input[].batch` records the call
- findings from all calls are deduplicated by file, line and normalized title
  before the verifier
- knobs: `ALTIMATE_REVIEW_AI_FOCUS_MIN_FILES`, `ALTIMATE_REVIEW_AI_FOCUS_BATCH_FILES`,
  `ALTIMATE_REVIEW_AI_MAX_BATCHES`; with the switch unset batches and messages
  are unchanged (coverage-only messages pinned by a new snapshot)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ules

- `doc-block-missing`: a yml whose only `doc()` is the new one now fires
  when the project keeps docs blocks of its own (before, it needed another
  `doc()` in the same file to resolve)
- `yaml-parse-error`: anchor on the added line at or before the parser's
  error line, or the first added line, when the error line is missing or
  past the edit (an unterminated quote)
- `jinja-concat-missing-space`: match double-quoted strings too, and
  ignore an escape such as `\n` at the seam

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`redundant-join-predicate` reset the chain at `cross join params as p`
because the alias was left in the gap before the next join. Step over the
alias of a join that has no ON clause so predicates of earlier inner joins
are still tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… folders

`skipNonProdModels` dropped every finding under `dev/`, `sandbox/` or
`scratch/`. A yml that no longer parses or a `doc()` with no docs block
breaks `dbt parse` for the whole project wherever the file lives, so
`yaml-parse-error` and `doc-block-missing` are exempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test-coverage claim on a dbt file was discarded as unverifiable when the
PR's tests lived outside the dbt project: the verifier could only read the
reviewed dbt files and their graph neighbours.

- `AiVerifyEvidence.prFiles`: every changed path in the PR, including tests
  and CI outside the dbt project; `read_file` may open them
- the verifier message lists those files when a candidate is a
  `test_coverage` claim (other batches are unchanged)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ALTIMATE_REVIEW_AI_TEST_GAP=1` appends one instruction to the core
reviewer prompt: for a changed macro, materialization or generic test whose
runtime behaviour changes, say whether the PR adds a test that runs it, and
report one `test_coverage` comment when none does. Each batch message also
lists the PR's other changed paths (tests and CI outside the dbt project
included), so the model can tell whether a test exists.

Unset, the system prompt and messages are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cs blocks

- `prFiles` (the AI test-gap list and the verifier's read allowlist) now drops
  paths matching the user's exclude globs and build output / vendored paths
  (new `isExcludedPath` in `diff-filter.ts`); out-of-project PR files stay readable
  by design.
- `doc-block-missing`: `doc()` names defined in installed packages' `.md` files
  (`dbt_packages/`, new `packageDocNames` in `run.ts`) count as defined. The
  no-other-`doc()` fallback now also requires the packages' docs blocks to be
  known (no packages declared, or installed), so a package-provided `doc()` is not
  reported as a parse failure. The finding body no longer claims other `doc()`
  references resolve when that path did not check them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sed batch size, YAML anchor

- Verifier: credential files (`profiles.yml`, `.env*`, `*.pem`/`*.key`,
  `*.tfvars`, SSH keys) are dropped from `prFiles` and from the verifier's
  readable set in `verifierTools`, whatever source lists them, and from
  `grep_project` hits; one `isCredentialPath` now backs the AI lane's
  supporting-file filter too
- Focused batches honour `batchChars`: oversized chunks split, packing stops at
  the char budget, and a balanced call over it is split again (`batchChars`
  wins over the call cap)
- `yamlParseFindings`: a deletion-only break anchors on the parser's error line
  when a hunk shows it, and stays file-level instead of on an unrelated added line
- Nits: focus settings require positive integers; `ALTIMATE_REVIEW_AI_MAX_BATCHES`
  renamed `ALTIMATE_REVIEW_AI_FOCUS_MAX_BATCHES`; the test-gap list skips paths
  the focused list already names; `PROJECT_BREAKING_RULES` moved above the
  `exclusionReason` doc comment
- Tests for each

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-input-coverage branch from 31b384c to bda6ef0 Compare November 21, 2026 09:45
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-focused-batches branch from 7b5bb4b to a2d91ce Compare November 21, 2026 09:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants