Repository navigation
feat(review): AI evidence line, suggestions and base-branch repo instructions (needs core 0.10) - #1774
feat(review): AI evidence line, suggestions and base-branch repo instructions (needs core 0.10)#1774RaghwendraSingh wants to merge 3 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
8aba994 to
f3c5e86
Compare
be62a4d to
58b9300
Compare
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
f3c5e86 to
a74e400
Compare
58b9300 to
14b3d2c
Compare
hkaltai
left a comment
There was a problem hiding this comment.
Verdict: no CRITICAL/MAJOR; six MINORs (details inline) plus nits. Panel: Claude, Codex and lead; every finding was premise-checked against the code. Draft PR, reviewed at 14b3d2c.
Contract check against altimate-core PR 786: matches. evidence_line, pre_existing (core already coerces "true"), suggestion {start_line,end_line,replacement} (integral, >= 1, start <= end, <= 2000 chars) and cites have the right names and types. Core returns a path already normalized onto valid_files, so the exact byFile.get lookup is correct, and line is a number or null, matching typeof item.line === "number". With 0.9.0 every field is absent and each reader type-checks, so parsing behaviour is unchanged (32/32 tests in review-ai-inputs.test.ts pass). Instruction loading reads the base commit, so a PR cannot rewrite its own rules.
Counts: CRITICAL 0, MAJOR 0, MINOR 6, NIT 6. APPROVE as a draft; MINORs are worth fixing before it leaves draft.
Nits
ai-review.ts:429,443slice(0, 200): counts UTF-16 units, can split a surrogate pair; core counts code points. Same forreplacement.length > 2_000(line 417): 1,001 emoji pass core but are dropped here (suggestion only). Fix: count code points.ai-review.ts:417s.replacement.length > 2_000: no 200-line span cap mirroring core;validateSuggestionsbuilds an array as long as the span if core ever relaxes it. Fix:end - start >= 200returns undefined.config.ts:302opts.readBase(c.path): a symlinkedCLAUDE.md -> AGENTS.mdreturns the link target text and a directory returns a tree listing. Fix: skip non-blobs.config.ts:285"Root first, then shallower directories": root is the shallowest; reword to "then deeper directories".ai-replay.ts:149(untouched, now stale) "re-derived on the next run": AI-setpreExisting/evidenceLineare not re-derivable, so replay drops them. Fix: keep AI-origin values throughunverify.- PR description says behaviour is unchanged on core 0.9.0: the instructions block and "set cites" hint are still sent to the model, so only parsing is unchanged. Also
aiOptionalFieldsreturns a requiredbody(name suggests optional fields only).
Missing tests: an end-to-end reviewBatch test with a mocked Dispatcher returning core-0.10-shaped JSON (nulls included); budget priority between deep REVIEW.md and large root files; wrong numeric line with a correct quote; cites naming an unloaded file.
| return { | ||
| body: String(item.body) + (cites ? ` (per \`${cites}\`)` : ""), | ||
| ...(evidence ? { evidenceLine: evidence } : {}), | ||
| ...(item.pre_existing === true ? { preExisting: true as const } : {}), |
There was a problem hiding this comment.
[MINOR] Model's pre_existing flag overrides the diff
...(item.pre_existing === true ? { preExisting: true as const } : {}),
anchorToDiff only ever sets preExisting; it never clears it on an added line, and the PR's own test pins that behaviour. A model that mislabels a line it did see added (or is steered by a SQL comment in the diff) moves a real inline comment into the collapsed pre-existing section (format.ts inlineEligible). With ALTIMATE_REVIEW_AI_VERIFY=1, ai-verify.ts:192 sets targetsUnchangedCode = true, so the finding is discarded unless the verifier confirms it. Reproduced by both external reviewers' probes. The diff is the authority on "a line the PR did not add".
Suggested fix: when the finding has a line on the diff, let the diff decide (clear the flag on added lines / new files); keep the model's flag only for line-less findings.
| for (let i = 1; i <= parts.length; i++) scopes.add(parts.slice(0, i).join("/")) | ||
| } | ||
| const depth = (scope: string) => (scope ? scope.split("/").length : 0) | ||
| const ordered = [...scopes].sort((a, b) => depth(a) - depth(b) || a.localeCompare(b)) |
There was a problem hiding this comment.
[MINOR] Instruction budget is spent breadth-first, so scoped files are starved
const ordered = [...scopes].sort((a, b) => depth(a) - depth(b) || a.localeCompare(b))
Shallowest scope first, and no priority of REVIEW.md over AGENTS.md/CLAUDE.md. Root AGENTS.md (6k) + CLAUDE.md (6k) + REVIEW.md consume the 12k, so analytics/dbt/REVIEW.md (the most relevant file) is never loaded, with no truncation marker. The budget is also global while the prompt filter is per batch, so a batch can receive neither its own scoped file nor the sibling text that used the budget.
Suggested fix: rank review-specific files above agent files and deeper scopes first, or apply the budget per batch inside buildUserMessage; add a (truncated) marker.
| const candidates = ordered.flatMap((scope) => | ||
| REPO_INSTRUCTION_FILES.map((name) => ({ scope, path: scope ? `${scope}/${name}` : name })), | ||
| ) | ||
| const texts = await Promise.all(candidates.map((c) => opts.readBase(c.path).catch(() => undefined))) |
There was a problem hiding this comment.
[MINOR] Unbounded concurrent git show calls; errors silently swallowed
const texts = await Promise.all(candidates.map((c) => opts.readBase(c.path).catch(() => undefined)))
4 x (distinct ancestor dirs) reads start at once: a PR touching 100 directories launches 408 git show processes even when root files already fill the 12k budget. run.ts also does this whenever AI is not disabled by flag/config, even if the risk tier excludes the ai_review lane. A failure (e.g. EMFILE) is turned into "file absent", so instructions vanish silently (note showAtRef deliberately surfaces non-absence errors).
Suggested fix: bound concurrency (or one git cat-file --batch), stop once the budget is full, load lazily when the lane will run, and only swallow the "not in commit" case.
| /** Evidence for an AI finding. A quoted line with no line number lets `anchorByEvidence` place it on the diff. */ | ||
| export function aiEvidenceResult(item: Record<string, unknown>): Record<string, unknown> { | ||
| const quoted = typeof item.evidence_line === "string" ? item.evidence_line.trim() : "" | ||
| return { confidence: item.confidence, ...(typeof item.line !== "number" && quoted ? { line: quoted.slice(0, 200) } : {}) } |
There was a problem hiding this comment.
[MINOR] Evidence anchoring ignores a wrong line, and short quotes match the first occurrence
return { confidence: item.confidence, ...(typeof item.line !== "number" && quoted ? { line: quoted.slice(0, 200) } : {}) }
(a) Evidence anchoring runs only when line is not a number. If the model gives line: 40 (off the diff) with a correct evidence_line, anchorToDiff strips the line and the finding drops to the summary although the quote would have placed it. If the line is on the diff but its text differs from the quote, the comment shows file:N with a quote that is not on line N. (b) anchorByEvidence takes the first added line with equal trimmed text, so a quote like ) or end, anchors on the first such line, not the intended one (confirmed in a probe).
Suggested fix: fall back to evidence anchoring when the numeric line is off-diff or its text does not match the quote; require a minimum length or a unique match for AI quotes.
| const cites = typeof item.cites === "string" ? item.cites.trim() : "" | ||
| const suggestion = aiSuggestion(item.suggestion) | ||
| return { | ||
| body: String(item.body) + (cites ? ` (per \`${cites}\`)` : ""), |
There was a problem hiding this comment.
[MINOR] cites is not checked against the instruction files actually sent
body: String(item.body) + (cites ? ` (per \`${cites}\`)` : ""),
Any path the model invents is appended as (per X), giving false authority, including when no instructions were sent (e.g. none exist, or the older core ignores the hint). Core only bounds length/characters.
Suggested fix: pass the loaded instruction paths into aiOptionalFields and drop cites that is not in the set.
| const s = raw as { start_line?: unknown; end_line?: unknown; replacement?: unknown } | null | undefined | ||
| const start = s?.start_line | ||
| const end = s?.end_line | ||
| if (!Number.isInteger(start) || !Number.isInteger(end) || typeof s?.replacement !== "string") return undefined |
There was a problem hiding this comment.
[MINOR] Suggestion range is not tied to the finding's anchor
if (!Number.isInteger(start) || !Number.isInteger(end) || typeof s?.replacement !== "string") return undefined
The inline comment is posted on s.startLine..s.endLine (format.ts inlineComments) while the evidence header uses f.startLine. Core's prompt asks the range to cover the quoted evidence but neither core nor this code enforces it: a suggestion for lines 10-11 on a finding anchored at line 3 posts a comment on line 11 quoting line 3. Also, a valid suggestion on a finding that ends up summary-only, demoted or over the inline budget is never rendered (format.ts summary bullets ignore suggestion; validateSuggestions only text-blocks invalid ones), so it is lost silently.
Suggested fix: in validateSuggestions require startLine <= f.startLine <= endLine when anchored, and render a Suggested fix: block for summary-only findings.
a74e400 to
08af665
Compare
14b3d2c to
c7b0dc8
Compare
…, suggestion and cites Newer cores return `evidence_line`, `pre_existing`, `suggestion` and `cites` on each AI finding. `aiOptionalFields` maps them onto the finding (`evidenceLine`, `preExisting`, `suggestion`, and a `(per \`<file>\`)` body suffix for `cites`). Each defaults to absent, so the lane works unchanged against the pinned core 0.9.0. A malformed suggestion is dropped rather than failing the finding's schema check. The existing pipeline already handles the rest: `anchorToDiff` only ever sets `preExisting`, so an AI-set flag survives anchoring, and AI suggestions pass through `validateSuggestions` before posting. Tests cover both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e base branch - `loadRepoInstructions` reads `REVIEW.md`, `.altimate/REVIEW.md`, `AGENTS.md` and `CLAUDE.md` at the repo root and at each ancestor directory of a changed path, from the BASE commit via `showAtRef`, so a PR cannot rewrite the instructions it is reviewed under. Each file is cut to 4,000 chars and the set to 12,000; the root comes first. - `run.ts` loads them only when the AI lane runs and a git base resolved; `OrchestrateInput.repoInstructions` carries them to `AiReviewInput`. - `buildUserMessage` adds a "REPO REVIEW INSTRUCTIONS" section per call, listing only files whose directory contains a file in that call, and asks the model to set `cites` when a finding relies on one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umber When the AI reviewer returns `evidence_line` without `line`, the quoted text now goes into `evidence.result.line`, so `anchorByEvidence` places the finding on the matching diff line, as it does for rule findings. The logic lives in a small exported `aiEvidenceResult` helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
08af665 to
e3e6417
Compare
c7b0dc8 to
73c7c7e
Compare
Issue for this PR
Part of #1792
Type of change
What does this PR do?
Draft, not for merge yet. Stacked on #1769. It needs the
@altimateai/altimate-corerelease that parses the new AI output fields (0.10.0, PR in the core repo). Against core 0.9.0 the new fields default and behaviour is unchanged.What changed
evidence_line,pre_existing,suggestionandcitesand maps them to the finding: quoted line, pre-existing tag, a GitHub suggestion block when valid, and(per <file>).REVIEW.md,.altimate/REVIEW.md,AGENTS.md,CLAUDE.mdat the root and above each changed path) are read from the base branch and given to the AI as conventions, so a PR can't change its own instructions. Capped at 4k chars per file and 12k in total.Results so far
How did you verify your code works?
bun test test/altimate/review: 678 pass.bunx tsc --noEmit: clean.Cost side effects
No new model calls, warehouse queries or cloud resources. When the AI lane is enabled, base-branch repo instructions add at most 12k characters (about 3k tokens) of input per AI call (4k per file), and the new output fields add a few output tokens per finding.
Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code
Summary by cubic
Maps the AI reviewer's newer output fields onto findings, anchors AI findings that quote a line without a line number, and supplies the reviewer with the repo's own review instructions read from the base branch. Requires
altimate-core0.10.0; against 0.9.0 the new fields default to absent and behavior is unchanged.evidence_line,pre_existing,suggestionandcitesfrom the AI reviewer, mapping them to the finding's quoted line, pre-existing tag, GitHub suggestion block, and a(per <file>)body suffix; malformed suggestions are dropped, not fatal.REVIEW.md,.altimate/REVIEW.md,AGENTS.mdandCLAUDE.mdat the repo root and each ancestor directory of a changed path from the base commit, so a PR can't rewrite its own instructions; each file is capped at 4k chars, 12k total.Written for commit 73c7c7e. Summary will update on new commits.