Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions packages/opencode/src/altimate/llm/stream-text.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,42 @@ export async function readStreamText(stream: {
}
return text
}

/** A tool call the model made in one step, as the AI SDK reports it. */
export interface StepToolCall {
toolCallId: string
toolName: string
input: unknown
/** The SDK could not parse the call or the tool does not exist. */
invalid?: boolean
}

/**
* Read one AI SDK step without discarding its tool calls: the text, why the
* step ended, the tool calls it made, and the response messages to append to
* the conversation for the next step. Tools without an `execute` are not run
* by the SDK; the caller runs them and appends the results. Like
* `readStreamText`, a stream error is thrown rather than lost.
*/
export async function readStreamStep(stream: {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: readStreamStep adds a new error-propagation path and an invalid passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream error part (must throw) and for a tool call with invalid: true.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/llm/stream-text.ts, line 32:

<comment>`readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</comment>

<file context>
@@ -12,3 +12,42 @@ export async function readStreamText(stream: {
+ * by the SDK; the caller runs them and appends the results. Like
+ * `readStreamText`, a stream error is thrown rather than lost.
+ */
+export async function readStreamStep(stream: {
+  fullStream: AsyncIterable<{ type: string; error?: unknown }>
+  readonly text: PromiseLike<string>
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: nit. It is exercised through the verifier tests.

fullStream: AsyncIterable<{ type: string; error?: unknown }>
readonly text: PromiseLike<string>
readonly finishReason: PromiseLike<string>
readonly response: PromiseLike<{ messages: unknown[] }>
}): Promise<{ text: string; finishReason: string; toolCalls: StepToolCall[]; messages: unknown[] }> {
const toolCalls: StepToolCall[] = []
const text = await readStreamText({
text: stream.text,
fullStream: (async function* () {
for await (const part of stream.fullStream) {
if (part.type === "tool-call") {
const p = part as unknown as StepToolCall
toolCalls.push({ toolCallId: p.toolCallId, toolName: p.toolName, input: p.input, invalid: p.invalid })
}
yield part
}
})(),
})
const [finishReason, response] = await Promise.all([stream.finishReason, stream.response])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: finishReason/response are left unobserved when the stream errors

readStreamText throws on an error part (or a rejected .text), so line 51 never runs. .text is explicitly guarded against an unhandled rejection at line 9 for exactly this reason, but the two new SDK promises are not: if they reject when the stream failed, the process gets an unhandled rejection. Observe them before rethrowing (Promise.allSettled or void p.catch(() => {})), as readStreamText does for .text.

return { text, finishReason, toolCalls, messages: response?.messages ?? [] }
}
220 changes: 220 additions & 0 deletions packages/opencode/src/altimate/review/ai-replay.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
// altimate_change - new file: replay the AI lane's generator findings from an earlier envelope
//
// `ALTIMATE_REVIEW_AI_REPLAY=<path to a review --json envelope>` skips the
// generator model call and rebuilds its findings from that envelope: the AI
// findings it kept (with what the verifier added undone) plus the ones the
// verifier discarded (`ai.verifier.dropped`). The verifier then runs on them as
// usual, so verifier settings can be compared without paying for generation
// again. Ids are the stored ones; benchmark labels are keyed by them.
// Fidelity is approximate: discarded findings come back with bodies clamped to
// 600 characters, and kept ones after the orchestrator's anchoring/cleaning.
import { readFile } from "node:fs/promises"
import z from "zod"
import { Provider } from "@/provider/provider"
import { Log } from "@/altimate/util/log"
import { Finding, ReviewCategory, makeFinding, ruleId } from "./finding"
import type { AiLaneStatus, AiReviewInput, AiReviewResult } from "./ai-review"
import { aiVerifyEnabled, runAiVerify, type VerifierStatus } from "./ai-verify"
import { modelNameFromPath } from "./orchestrate"

const log = Log.create({ service: "ai-replay" })

/** The path of the envelope to replay, when replay is on. */
export function aiReplayPath(env: Record<string, string | undefined> = process.env): string | undefined {
return env.ALTIMATE_REVIEW_AI_REPLAY || undefined
}

const Tokens = z.object({
input: z.number(),
inputTotal: z.number(),
output: z.number(),
reasoning: z.number(),
cacheRead: z.number(),
})

/** The parts of an envelope replay reads; everything else is ignored. */
const ReplayEnvelope = z.object({
findings: z.array(z.unknown()),
ai: z
.object({
state: z.enum(["ok", "skipped", "timeout", "error"]),
reason: z.string().optional(),
model: z.string().optional(),
tokens: Tokens.optional(),
cost: z.number().optional(),
calls: z.number().optional(),
verifier: z
.object({
discarded: z.number().optional(),
dropped: z
.array(
z.object({
id: z.string(),
file: z.string(),
startLine: z.number().optional(),
title: z.string(),
body: z.string(),
category: z.string(),
severity: z.string(),
}),
)
.optional(),
decisions: z
.array(z.object({ id: z.string(), ts: z.object({ lowConfidence: z.boolean() }).partial().optional() }))
.optional(),
})
.optional(),
})
.optional(),
})

export interface ReplayedLane {
findings: Finding[]
/** The stored generator status: state, reason, model, tokens, cost, calls. */
status: Omit<AiLaneStatus, "durationMs" | "verifier">
/** Problems found while rebuilding (findings that could not be recovered). */
warnings: string[]
}

/**
* Rebuild the generator's findings from a parsed envelope (pure). Kept AI
* findings lose the verifier's record, its `Verified:` body suffix and the
* tiers set after generation (the generator sets none); discarded ones are
* rebuilt from `ai.verifier.dropped`. Order follows the verifier's
* `decisions` (the generator's order) when recorded.
*/
export function replayFindings(raw: unknown): ReplayedLane {
const env = ReplayEnvelope.parse(raw)
const ai = env.ai
if (!ai) throw new Error("the envelope has no AI lane status")
const status: ReplayedLane["status"] = {
state: ai.state,
...(ai.reason ? { reason: ai.reason } : {}),
...(ai.model ? { model: ai.model } : {}),
...(ai.tokens ? { tokens: ai.tokens } : {}),
...(ai.cost !== undefined ? { cost: ai.cost } : {}),
...(ai.calls !== undefined ? { calls: ai.calls } : {}),
replayed: true,
}
const warnings: string[] = []
const out: Finding[] = []
for (const item of env.findings) {
const parsed = Finding.safeParse(item)
if (!parsed.success || parsed.data.evidence?.tool !== "ai-review") continue
out.push(unverify(parsed.data))
}

const v = ai.verifier
const lowConfidence = new Map((v?.decisions ?? []).map((d) => [d.id, d.ts?.lowConfidence === true]))
for (const d of v?.dropped ?? []) {
const category = ReviewCategory.safeParse(d.category)
if (!category.success) {
warnings.push(`discarded finding ${d.id} has an unknown category "${d.category}"`)
continue
}
// The envelope does not keep a discarded finding's confidence; the verifier reads only whether it is low.
const confidence = lowConfidence.get(d.id) ? ("low" as const) : ("medium" as const)
out.push(
makeFinding({
id: d.id,
severity: d.severity === "warning" ? "warning" : "suggestion",
category: category.data,
title: d.title,
body: d.body,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Discarded findings are rebuilt from the envelope's body, which applyTriage clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any replayIncomplete flag. Note the truncation in the replay status, or stop clamping for replayed records.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-replay.ts, line 121:

<comment>Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</comment>

<file context>
@@ -0,0 +1,203 @@
+        severity: d.severity === "warning" ? "warning" : "suggestion",
+        category: category.data,
+        title: d.title,
+        body: d.body,
+        file: d.file,
+        model: modelNameFromPath(d.file),
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, low priority (ai-replay.ts:121, clamped bodies not flagged). Leaving this for a follow-up PR rather than widening this one.

file: d.file,
model: modelNameFromPath(d.file),
...(d.startLine ? { startLine: d.startLine, endLine: d.startLine } : {}),
confidence,
evidence: { tool: "ai-review", result: { confidence } },
rule: ruleId("ai-review", d.category),
}),
)
}

const listed = v?.dropped?.length ?? 0
if ((v?.discarded ?? 0) > listed)
warnings.push(`the verifier discarded ${v!.discarded} findings but the envelope lists ${listed}`)
const have = new Set(out.map((f) => f.id))
const missing = (v?.decisions ?? []).filter((d) => !have.has(d.id)).length

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A merged repeat is reported as a missing decision on every replay

decisions is recorded inside runAiVerify on the generator's ids, but env.findings is written after cleanFindings merges repeats (orchestrate.ts:2456, finding.ts:305-311) and the merged finding gets a synthetic f_<hash> id. have therefore never contains the merged group's original ids, so any envelope with a merged AI repeat reports "N findings the verifier decided are not in the envelope" and sets replayIncomplete even though every finding is represented. Treat a merged id as covering its group, or record decisions after merging.

if (missing) warnings.push(`${missing} findings the verifier decided are not in the envelope`)

if (v?.decisions?.length) {
const order = new Map(v.decisions.map((d, i) => [d.id, i]))
out.sort((a, b) => (order.get(a.id) ?? Infinity) - (order.get(b.id) ?? Infinity))
}
return { findings: out, status: warnings.length ? { ...status, replayIncomplete: true } : status, warnings }
}

/** A kept finding close to how the generator returned it: no verification record, body suffix,
* tier, origin, or the orchestrator's `preExisting` / `evidenceLine` (re-derived on the next run). */
function unverify(f: Finding): Finding {
const { tier: _tier, origin: _origin, preExisting: _pre, evidenceLine: _line, ...rest } = f
const prior = f.evidence?.result
if (!prior || typeof prior !== "object") return rest
const { verification, ...result } = prior as Record<string, unknown> & {
verification?: { rule?: string; evidence?: string }
}
let body = f.body
const suffix = verification?.rule === "verified" && verification.evidence ? `\n\nVerified: ${verification.evidence}` : ""
if (suffix && body.endsWith(suffix)) body = body.slice(0, -suffix.length)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: unverify only strips a Verified: suffix that is the tail of the body, which mergeRepeats breaks

applyTriage appends the suffix, but cleanFindings then runs mergeRepeats (finding.ts:291-295), which rewrites the first finding of a group to ${first.body}\n\nSame rule, also here:…. If a verified finding wins that tie-break, the stored body is …Verified: <evidence>\n\nSame rule, also here:…, so body.endsWith(suffix) is false and replay keeps a Verified: claim the new verifier never made (it is fed f.body by buildVerifyMessage/extractFeatures, and is posted if re-kept). The merged (+N more) title is likewise not undone.

Remove the last occurrence of the suffix instead of requiring a tail match, and strip the (+N more) marker when rebuilding the generator's finding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, low priority (ai-replay.ts:156, unverify suffix). Leaving this for a follow-up PR rather than widening this one.

return { ...rest, body, evidence: { tool: "ai-review", result } }
}

/**
* The AI lane in replay mode: no generator call. Fails loudly (stderr) and
* reports `state: "error"` when the envelope cannot be read; never falls back
* to calling the generator.
*/
export async function runAiReplay(input: AiReviewInput, path: string): Promise<AiReviewResult> {
const startedAt = Date.now()
const done = (findings: Finding[], status: Omit<AiLaneStatus, "durationMs">): AiReviewResult => ({
findings,
status: { ...status, durationMs: Date.now() - startedAt },
})
let lane: ReplayedLane
try {
lane = replayFindings(JSON.parse(await readFile(path, "utf8")))
} catch (err) {
const reason = `cannot replay ${path}: ${err instanceof Error ? err.message : String(err)}`.slice(0, 300)
process.stderr.write(`altimate review: ALTIMATE_REVIEW_AI_REPLAY: ${reason}\n`)
log.error("ai replay failed", { path, error: err })
return done([], { state: "error", reason, replayed: true })
}
for (const w of lane.warnings) {
process.stderr.write(`altimate review: ALTIMATE_REVIEW_AI_REPLAY: ${w}\n`)
log.warn("ai replay incomplete", { path, warning: w })
}
log.info("ai replay", { path, findings: lane.findings.length })
if (lane.status.state !== "ok") return done([], lane.status)

let findings = lane.findings
let verifier: VerifierStatus | undefined
if (input.evidence && aiVerifyEnabled()) {
let model: Awaited<ReturnType<typeof Provider.getModel>>
try {
const chosen = input.model ? Provider.parseModel(input.model) : await Provider.defaultModel()
model = await Provider.getModel(chosen.providerID, chosen.modelID)
} catch (err) {
// Fail open: the replayed generator findings stand, unverified.
const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200)
const n = findings.length
const skipped: VerifierStatus = {
state: "skipped",
reason,
durationMs: 0,
turns: 0,
toolCalls: 0,
reviewed: 0,
kept: n,
discarded: 0,
summary: 0,
unreviewed: n,
}
return done(findings, { ...lane.status, verifier: skipped })
}
const v = await runAiVerify({ findings, grounding: input.grounding, evidence: input.evidence, model })
findings = v.findings
verifier = v.status
}
return done(findings, { ...lane.status, ...(verifier ? { verifier } : {}) })
}
24 changes: 22 additions & 2 deletions packages/opencode/src/altimate/review/ai-review.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ import { redactSecrets } from "@/installation"
import { Dispatcher } from "../native"
import { type Finding, type ReviewCategory, type Severity, makeFinding, ruleId } from "./finding"
import { readStreamText } from "../llm/stream-text"
import { type AiVerifyEvidence, type VerifierStatus, aiVerifyEnabled, runAiVerify } from "./ai-verify"
import { aiReplayPath, runAiReplay } from "./ai-replay"

const log = Log.create({ service: "ai-review" })

Expand Down Expand Up @@ -138,6 +140,8 @@ export interface AiReviewInput {
prBody?: string
/** Explicit `provider/model`; omit to use the configured default model. */
model?: string
/** What the verifier may read; it runs only when this is set and `ALTIMATE_REVIEW_AI_VERIFY=1`. */
evidence?: AiVerifyEvidence
}

/** What happened in the advisory lane, reported on the envelope and in the comment. */
Expand All @@ -155,6 +159,13 @@ export interface AiLaneStatus {
cost?: number
/** Model calls made (one per batch). */
calls?: number
/** The dbt-grounded verifier's outcome, when it ran (opt-in). */
verifier?: VerifierStatus
/** The generator's findings came from an earlier envelope (`ALTIMATE_REVIEW_AI_REPLAY`);
* tokens, cost and calls are that run's, so only `verifier` was spent now. */
replayed?: boolean
/** The replayed envelope did not list every generator finding. */
replayIncomplete?: boolean
}
/** Token counts as `Session.getUsage` reports them: `input` excludes cached
* tokens, `inputTotal` includes them. */
Expand Down Expand Up @@ -253,6 +264,8 @@ export function buildUserMessage(input: AiReviewInput, maxFileChars = MAX_DIFF_C
* envelope and the PR comment.
*/
export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult> {
const replay = aiReplayPath()
if (replay) return runAiReplay(input, replay)
const startedAt = Date.now()
const done = (findings: Finding[], status: Omit<AiLaneStatus, "durationMs">): AiReviewResult => ({
findings,
Expand Down Expand Up @@ -291,15 +304,22 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
const slice = batches.slice(i, i + BATCH_CONCURRENCY)
results.push(...(await Promise.all(slice.map((b) => reviewBatch(input, b, budget, system, model)))))
}
const findings = results.flatMap((r) => r.findings)
let findings = results.flatMap((r) => r.findings)
const failed = results.filter((r) => r.state !== "ok")
log.info("ai review complete", { findings: findings.length, batches: batches.length, failed: failed.length })
// The lane is ok when any call answered; a partial failure is reported, not hidden.
const spent = laneUsage(results)
if (failed.length === results.length)
return done([], { state: failed[0].state, reason: failed[0].reason, model: modelLabel, ...spent })
const reason = failed.length ? `${failed.length} of ${results.length} calls failed: ${failed[0].reason}` : undefined
return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent })
// Opt-in: try to refute each finding with dbt evidence. Never throws.
let verifier: VerifierStatus | undefined
if (input.evidence && aiVerifyEnabled()) {
const v = await runAiVerify({ findings, grounding: input.grounding, evidence: input.evidence, model })
findings = v.findings
verifier = v.status

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A fail-closed verifier failure is invisible in the report

When runAiVerify hits an error/timeout with ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed, it returns no findings and state: "error"/"timeout", but this line only stores that status and done(...) still reports the lane as state: "ok". coverageNotes (format.ts:278-284) reads only env.ai.state/env.ai.reason, so the PR comment shows a clean AI lane while every AI finding disappeared. Fold the verifier's non-ok state/reason into the lane status (or render it in the comment) so the loss is stated.

}
return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: With the verifier on, lane-level tokens, cost and calls cover only the generator, so ai_cost/ai_tokens_total/ai_calls in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the tokens/calls JSDoc to match.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 312:

<comment>With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</comment>

<file context>
@@ -281,15 +294,22 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
+      findings = v.findings
+      verifier = v.status
+    }
+    return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })
   } catch (err) {
     log.error("ai review failed", { error: err })
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the verifier is reported separately as ai_verify_tokens_* so lane and verifier spend stay distinguishable.

} catch (err) {
log.error("ai review failed", { error: err })
return done([], { state: "error", reason: safeReason(err), model: modelLabel })
Expand Down
Loading
Loading