Skip to content

feat(review): AI evidence line, suggestions and base-branch repo instructions (needs core 0.10) - #1774

Draft
RaghwendraSingh wants to merge 3 commits into
feat/review-ai-verifierfrom
feat/review-ai-remit
Draft

RaghwendraSingh wants to merge 3 commits into
feat/review-ai-verifierfrom
feat/review-ai-remit

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Draft, not for merge yet. Stacked on #1769. It needs the @altimateai/altimate-core release that parses the new AI output fields (0.10.0, PR in the core repo). Against core 0.9.0 the new fields default and behaviour is unchanged.

What changed

  • Reads the AI reviewer's evidence_line, pre_existing, suggestion and cites and maps them to the finding: quoted line, pre-existing tag, a GitHub suggestion block when valid, and (per <file>).
  • Repo review instructions (REVIEW.md, .altimate/REVIEW.md, AGENTS.md, CLAUDE.md at the root and above each changed path) are read from the base branch and given to the AI as conventions, so a PR can't change its own instructions. Capped at 4k chars per file and 12k in total.
  • AI findings that quote a line but give no line number are anchored to the diff line they quote.

Results so far

  • Widening the AI's remit to macro, test and yml bugs did not help, so that part was dropped:
    • Tuning repos: +2 catches (95% CI −3 to +7).
    • Held-out repos: −1 catch, with 8% wrong comments.
  • This PR keeps only the output fields, repo instructions and anchoring. Measuring those alone is still to do: one tuning-repo run before the core release.

How did you verify your code works?

  • bun test test/altimate/review: 678 pass.
  • bunx tsc --noEmit: clean.

Cost side effects

No new model calls, warehouse queries or cloud resources. When the AI lane is enabled, base-branch repo instructions add at most 12k characters (about 3k tokens) of input per AI call (4k per file), and the new output fields add a few output tokens per finding.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Maps the AI reviewer's newer output fields onto findings, anchors AI findings that quote a line without a line number, and supplies the reviewer with the repo's own review instructions read from the base branch. Requires altimate-core 0.10.0; against 0.9.0 the new fields default to absent and behavior is unchanged.

  • Reads evidence_line, pre_existing, suggestion and cites from the AI reviewer, mapping them to the finding's quoted line, pre-existing tag, GitHub suggestion block, and a (per <file>) body suffix; malformed suggestions are dropped, not fatal.
  • Reads REVIEW.md, .altimate/REVIEW.md, AGENTS.md and CLAUDE.md at the repo root and each ancestor directory of a changed path from the base commit, so a PR can't rewrite its own instructions; each file is capped at 4k chars, 12k total.
  • Anchors AI findings that quote a line without a line number to the matching diff line.

Written for commit 73c7c7e. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

Thanks for updating your PR! It now meets our contributing guidelines. 👍

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: no CRITICAL/MAJOR; six MINORs (details inline) plus nits. Panel: Claude, Codex and lead; every finding was premise-checked against the code. Draft PR, reviewed at 14b3d2c.

Contract check against altimate-core PR 786: matches. evidence_line, pre_existing (core already coerces "true"), suggestion {start_line,end_line,replacement} (integral, >= 1, start <= end, <= 2000 chars) and cites have the right names and types. Core returns a path already normalized onto valid_files, so the exact byFile.get lookup is correct, and line is a number or null, matching typeof item.line === "number". With 0.9.0 every field is absent and each reader type-checks, so parsing behaviour is unchanged (32/32 tests in review-ai-inputs.test.ts pass). Instruction loading reads the base commit, so a PR cannot rewrite its own rules.

Counts: CRITICAL 0, MAJOR 0, MINOR 6, NIT 6. APPROVE as a draft; MINORs are worth fixing before it leaves draft.

Nits

  • ai-review.ts:429,443 slice(0, 200): counts UTF-16 units, can split a surrogate pair; core counts code points. Same for replacement.length > 2_000 (line 417): 1,001 emoji pass core but are dropped here (suggestion only). Fix: count code points.
  • ai-review.ts:417 s.replacement.length > 2_000: no 200-line span cap mirroring core; validateSuggestions builds an array as long as the span if core ever relaxes it. Fix: end - start >= 200 returns undefined.
  • config.ts:302 opts.readBase(c.path): a symlinked CLAUDE.md -> AGENTS.md returns the link target text and a directory returns a tree listing. Fix: skip non-blobs.
  • config.ts:285 "Root first, then shallower directories": root is the shallowest; reword to "then deeper directories".
  • ai-replay.ts:149 (untouched, now stale) "re-derived on the next run": AI-set preExisting/evidenceLine are not re-derivable, so replay drops them. Fix: keep AI-origin values through unverify.
  • PR description says behaviour is unchanged on core 0.9.0: the instructions block and "set cites" hint are still sent to the model, so only parsing is unchanged. Also aiOptionalFields returns a required body (name suggests optional fields only).

Missing tests: an end-to-end reviewBatch test with a mocked Dispatcher returning core-0.10-shaped JSON (nulls included); budget priority between deep REVIEW.md and large root files; wrong numeric line with a correct quote; cites naming an unloaded file.

return {
body: String(item.body) + (cites ? ` (per \`${cites}\`)` : ""),
...(evidence ? { evidenceLine: evidence } : {}),
...(item.pre_existing === true ? { preExisting: true as const } : {}),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Model's pre_existing flag overrides the diff

...(item.pre_existing === true ? { preExisting: true as const } : {}),

anchorToDiff only ever sets preExisting; it never clears it on an added line, and the PR's own test pins that behaviour. A model that mislabels a line it did see added (or is steered by a SQL comment in the diff) moves a real inline comment into the collapsed pre-existing section (format.ts inlineEligible). With ALTIMATE_REVIEW_AI_VERIFY=1, ai-verify.ts:192 sets targetsUnchangedCode = true, so the finding is discarded unless the verifier confirms it. Reproduced by both external reviewers' probes. The diff is the authority on "a line the PR did not add".

Suggested fix: when the finding has a line on the diff, let the diff decide (clear the flag on added lines / new files); keep the model's flag only for line-less findings.

for (let i = 1; i <= parts.length; i++) scopes.add(parts.slice(0, i).join("/"))
}
const depth = (scope: string) => (scope ? scope.split("/").length : 0)
const ordered = [...scopes].sort((a, b) => depth(a) - depth(b) || a.localeCompare(b))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Instruction budget is spent breadth-first, so scoped files are starved

const ordered = [...scopes].sort((a, b) => depth(a) - depth(b) || a.localeCompare(b))

Shallowest scope first, and no priority of REVIEW.md over AGENTS.md/CLAUDE.md. Root AGENTS.md (6k) + CLAUDE.md (6k) + REVIEW.md consume the 12k, so analytics/dbt/REVIEW.md (the most relevant file) is never loaded, with no truncation marker. The budget is also global while the prompt filter is per batch, so a batch can receive neither its own scoped file nor the sibling text that used the budget.

Suggested fix: rank review-specific files above agent files and deeper scopes first, or apply the budget per batch inside buildUserMessage; add a (truncated) marker.

const candidates = ordered.flatMap((scope) =>
REPO_INSTRUCTION_FILES.map((name) => ({ scope, path: scope ? `${scope}/${name}` : name })),
)
const texts = await Promise.all(candidates.map((c) => opts.readBase(c.path).catch(() => undefined)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Unbounded concurrent git show calls; errors silently swallowed

const texts = await Promise.all(candidates.map((c) => opts.readBase(c.path).catch(() => undefined)))

4 x (distinct ancestor dirs) reads start at once: a PR touching 100 directories launches 408 git show processes even when root files already fill the 12k budget. run.ts also does this whenever AI is not disabled by flag/config, even if the risk tier excludes the ai_review lane. A failure (e.g. EMFILE) is turned into "file absent", so instructions vanish silently (note showAtRef deliberately surfaces non-absence errors).

Suggested fix: bound concurrency (or one git cat-file --batch), stop once the budget is full, load lazily when the lane will run, and only swallow the "not in commit" case.

/** Evidence for an AI finding. A quoted line with no line number lets `anchorByEvidence` place it on the diff. */
export function aiEvidenceResult(item: Record<string, unknown>): Record<string, unknown> {
const quoted = typeof item.evidence_line === "string" ? item.evidence_line.trim() : ""
return { confidence: item.confidence, ...(typeof item.line !== "number" && quoted ? { line: quoted.slice(0, 200) } : {}) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Evidence anchoring ignores a wrong line, and short quotes match the first occurrence

return { confidence: item.confidence, ...(typeof item.line !== "number" && quoted ? { line: quoted.slice(0, 200) } : {}) }

(a) Evidence anchoring runs only when line is not a number. If the model gives line: 40 (off the diff) with a correct evidence_line, anchorToDiff strips the line and the finding drops to the summary although the quote would have placed it. If the line is on the diff but its text differs from the quote, the comment shows file:N with a quote that is not on line N. (b) anchorByEvidence takes the first added line with equal trimmed text, so a quote like ) or end, anchors on the first such line, not the intended one (confirmed in a probe).

Suggested fix: fall back to evidence anchoring when the numeric line is off-diff or its text does not match the quote; require a minimum length or a unique match for AI quotes.

const cites = typeof item.cites === "string" ? item.cites.trim() : ""
const suggestion = aiSuggestion(item.suggestion)
return {
body: String(item.body) + (cites ? ` (per \`${cites}\`)` : ""),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] cites is not checked against the instruction files actually sent

body: String(item.body) + (cites ? ` (per \`${cites}\`)` : ""),

Any path the model invents is appended as (per X), giving false authority, including when no instructions were sent (e.g. none exist, or the older core ignores the hint). Core only bounds length/characters.

Suggested fix: pass the loaded instruction paths into aiOptionalFields and drop cites that is not in the set.

const s = raw as { start_line?: unknown; end_line?: unknown; replacement?: unknown } | null | undefined
const start = s?.start_line
const end = s?.end_line
if (!Number.isInteger(start) || !Number.isInteger(end) || typeof s?.replacement !== "string") return undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Suggestion range is not tied to the finding's anchor

if (!Number.isInteger(start) || !Number.isInteger(end) || typeof s?.replacement !== "string") return undefined

The inline comment is posted on s.startLine..s.endLine (format.ts inlineComments) while the evidence header uses f.startLine. Core's prompt asks the range to cover the quoted evidence but neither core nor this code enforces it: a suggestion for lines 10-11 on a finding anchored at line 3 posts a comment on line 11 quoting line 3. Also, a valid suggestion on a finding that ends up summary-only, demoted or over the inline budget is never rendered (format.ts summary bullets ignore suggestion; validateSuggestions only text-blocks invalid ones), so it is lost silently.

Suggested fix: in validateSuggestions require startLine <= f.startLine <= endLine when anchored, and render a Suggested fix: block for summary-only findings.

RaghwendraSingh and others added 3 commits November 21, 2026 13:14
…, suggestion and cites

Newer cores return `evidence_line`, `pre_existing`, `suggestion` and
`cites` on each AI finding. `aiOptionalFields` maps them onto the finding
(`evidenceLine`, `preExisting`, `suggestion`, and a `(per \`<file>\`)`
body suffix for `cites`). Each defaults to absent, so the lane works
unchanged against the pinned core 0.9.0. A malformed suggestion is
dropped rather than failing the finding's schema check.

The existing pipeline already handles the rest: `anchorToDiff` only ever
sets `preExisting`, so an AI-set flag survives anchoring, and AI
suggestions pass through `validateSuggestions` before posting. Tests cover
both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e base branch

- `loadRepoInstructions` reads `REVIEW.md`, `.altimate/REVIEW.md`,
  `AGENTS.md` and `CLAUDE.md` at the repo root and at each ancestor
  directory of a changed path, from the BASE commit via `showAtRef`, so a
  PR cannot rewrite the instructions it is reviewed under. Each file is cut
  to 4,000 chars and the set to 12,000; the root comes first.
- `run.ts` loads them only when the AI lane runs and a git base resolved;
  `OrchestrateInput.repoInstructions` carries them to `AiReviewInput`.
- `buildUserMessage` adds a "REPO REVIEW INSTRUCTIONS" section per call,
  listing only files whose directory contains a file in that call, and
  asks the model to set `cites` when a finding relies on one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umber

When the AI reviewer returns `evidence_line` without `line`, the quoted
text now goes into `evidence.result.line`, so `anchorByEvidence` places
the finding on the matching diff line, as it does for rule findings.
The logic lives in a small exported `aiEvidenceResult` helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants