Skip to content

fix(review): failures are never clean: engine errors reported, per-file isolation, PII abstention, gate dismissal - #1723

Open
RaghwendraSingh wants to merge 5 commits into
fix/review-verdict-trustfrom
fix/review-failures-gate
Open

RaghwendraSingh wants to merge 5 commits into
fix/review-verdict-trustfrom
fix/review-failures-gate

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes since review: af1ebcc (review feedback): a PII abstention keeps the block, unverified kinds are typed, stale-review dismissal is safer, and --head runs deepen history when needed. 071d1e8 (re-review): a model with no compiled head SQL makes the run incomplete (no APPROVE, no dismissal), dismissal requires the run to be proven on the PR head (head or its refs/pull/N/merge commit), only bot-authored reviews are dismissed, and core parse/unsupported-dialect errors abstain instead of counting as engine errors.

Issue for this PR

Closes #1792
Stacked on #1722 (← #1721 ← #1720 ← #1719); merge in that order.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

A broken review could look like a clean one:

  • Silent engine failures: every ReviewRunner method caught its errors and returned empty. A failed native core read as "no findings" and could produce APPROVE.
  • One file aborts everything: the per-model precompute and the lane Promise.all had no error handling, so one file throwing rejected the whole review.
  • PII "no columns" ambiguity: an empty PII result looked the same whether the model was clean or core couldn't parse it.
  • Gate never clears: a blocking review stayed blocking after a clean rerun.

Fixes:

  • Engine errors: the dispatcher runner records each throwing call as method, first reason and count (13 call sites), and still returns its empty result. These are signed into the envelope as engineErrors, named in the comment, and counted in review_run.engine_errors. An empty review with engine errors is COMMENT, never APPROVE.
  • Per-file isolation: a failure preparing one file, or a lane throwing for one file, marks that file unverified with the reason; the rest is reviewed. The PR-level lanes (sibling consistency, topology equivalence) and the core-ran check are guarded too. A test caught the core-ran check aborting the review after a lane failure.
  • PII abstention: detectPii returns abstained when core couldn't analyse the SQL (check didn't run, or pii.parse_error). The model is listed as not fully verified.
  • Gate lifecycle: when a rerun's event is non-blocking, earlier altimate CHANGES_REQUESTED reviews are dismissed with a pointer to the new verdict. postGitHubReview takes an optional client, so dedup, dismissal and skip are tested without the network.

Not in this PR: wiring the break-glass override (applyOverride) behind maintainer authorization. That needs a decision on who may override and where the audit record lives.

Follow-up after an adversarial review (commit 2f865225ec):

  • A failed run never clears a block. A stale CHANGES_REQUESTED review is dismissed only when the new run completed (no engineErrors, no file that failed). Before, an engine crash on the next push made the findings disappear, and the block was dismissed.
  • Shallow CI checkouts work. In CI, a missing target branch is fetched from origin, and a shallow history is deepened (50, 200, 1000) until the merge-base exists. The default actions/checkout therefore works without fetch-depth: 0. It's still an error when that fails.
  • Upgrade note for gate users in the docs. Pattern-rule findings now comment instead of blocking, unless they're listed in rubric.blockingRules.
  • Tests: a crashed run and a failed file each leave the block in place; resolving the base from a real git clone --depth 1 with only GITHUB_BASE_REF set.

How did you verify your code works?

  • New test/altimate/review-failures-gate.test.ts (9 tests):
    • engine failure → COMMENT with a note; engineErrors signed and tamper-evident; an empty clean run still approves
    • unreadable content and a throwing lane each degrade only their file
    • PII abstention reported
    • with a fake GitHub client: a clean rerun dismisses only the bot's blocking review; a still-blocking rerun doesn't dismiss; already-posted findings aren't re-posted and no repeat review is created
  • 1,892 tests across the review files and release gates pass. bun typecheck is clean.

Cost side effects

None. At most one dismissReview call per stale blocking review on a non-blocking rerun.

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Fixes the review so engine failures, per-file crashes, and PII parse errors can no longer read as a clean pass, a stale blocking review is cleared only by a completed non-blocking rerun on the PR's current head, and a shallow CI checkout can fetch the base it needs.

Bug Fixes

  • Dispatcher runner records each throwing engine call (method, first reason, count) instead of silently returning empty; errors are signed into the envelope as engineErrors, shown in the comment, and counted in telemetry, and an empty review with engine errors is COMMENT, never APPROVE.
  • A failure preparing one file, or a lane throwing for one file, marks that file unverified; the rest of the PR is still reviewed, and PR-level lanes and the core-ran check are guarded the same way. An unanalysable base no longer reports every head PII column as newly exposed.
  • detectPii reports abstained when core couldn't analyse the SQL (check didn't run, or its PII parser errored), listing the model as not fully verified instead of PII-clean; parse/dialect failures abstain rather than counting as engine errors.
  • A blocking review is dismissed only by a completed rerun on the PR's current head, posted by this project's bot or the token's own user; the GitHub client is injectable so dedup, dismissal, and skip rules are tested without the network.
  • In CI, a missing target branch is fetched from origin and history deepened until the merge-base exists, so the default shallow actions/checkout works without fetch-depth: 0; if that fails, the review errors instead of diffing another branch.
  • Docs: pattern-rule findings no longer block gate setups unless listed in rubric.blockingRules.

Written for commit 20decc7. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5d07a19d-7354-4c76-a18b-385fb3744f6e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 10 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/review/format.ts">

<violation number="1" location="packages/opencode/src/altimate/review/format.ts:210">
P3: Grouping by the full reason text fragments per-file error messages into one note each, so a widespread failure produces many notes. Group by a normalized reason category, or cap the number of distinct notes, so the comment stays compact.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/format.ts:220">
P3: `e.reason` is not sanitized here, and PR-level lane errors can contain newlines. That breaks the `> ℹ️` blockquote in the posted comment. Collapse whitespace before interpolating the reason.</violation>
</file>

<file name="packages/opencode/src/altimate/review/post-github.ts">

<violation number="1" location="packages/opencode/src/altimate/review/post-github.ts:146">
P2: A failed dismissal must not fall through to the no-new-review skip path. Track failed IDs and only skip after every stale blocking review is dismissed; otherwise surface the incomplete cleanup and retry or leave an explicit failure signal instead of claiming no review was needed.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/docs/usage/dbt-pr-review.md Resolved
"Run `dbt compile` (and compile the base into `target-base/`) for equivalence and lineage checks.",
)
const byReason = new Map<string, string[]>()
for (const u of unverified) byReason.set(u.reason, [...(byReason.get(u.reason) ?? []), u.path])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Grouping by the full reason text fragments per-file error messages into one note each, so a widespread failure produces many notes. Group by a normalized reason category, or cap the number of distinct notes, so the comment stays compact.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 210:

<comment>Grouping by the full reason text fragments per-file error messages into one note each, so a widespread failure produces many notes. Group by a normalized reason category, or cap the number of distinct notes, so the comment stays compact.</comment>

<file context>
@@ -206,13 +206,19 @@ export function coverageNotes(env: VerdictEnvelope): string[] {
-        "Run `dbt compile` (and compile the base into `target-base/`) for equivalence and lineage checks.",
-    )
+  const byReason = new Map<string, string[]>()
+  for (const u of unverified) byReason.set(u.reason, [...(byReason.get(u.reason) ?? []), u.path])
+  for (const [reason, paths] of byReason) {
+    const head = paths.slice(0, 5).map((p) => `\`${p}\``).join(", ")
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We will group per-file failures by reason prefix.

notes.push(`${paths.length} model(s) not fully verified — ${reason}: ${head}${more}.${hint}`)
}
if (env.engineErrors?.length) {
const list = env.engineErrors.map((e) => `${e.method} ×${e.count} (${e.reason})`).join("; ")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: e.reason is not sanitized here, and PR-level lane errors can contain newlines. That breaks the > ℹ️ blockquote in the posted comment. Collapse whitespace before interpolating the reason.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 220:

<comment>`e.reason` is not sanitized here, and PR-level lane errors can contain newlines. That breaks the `> ℹ️` blockquote in the posted comment. Collapse whitespace before interpolating the reason.</comment>

<file context>
@@ -206,13 +206,19 @@ export function coverageNotes(env: VerdictEnvelope): string[] {
+    notes.push(`${paths.length} model(s) not fully verified — ${reason}: ${head}${more}.${hint}`)
+  }
+  if (env.engineErrors?.length) {
+    const list = env.engineErrors.map((e) => `${e.method} ×${e.count} (${e.reason})`).join("; ")
+    notes.push(`Some engine checks failed and did not run: ${list}. This verdict is not a clean approval.`)
   }
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. prLevel will collapse whitespace the way fail() does.

const piiRes = engineNewSql ? await input.runner.detectPii(engineNewSql, dialect) : { columns: [] }
if (piiRes.abstained) unverified.push({ path: file.path, reason: "the PII check could not analyse this model" })
const pii = piiRes.columns
const complex = engineNewSql && input.runner.isComplex ? await input.runner.isComplex(engineNewSql) : false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: The per-file try is too coarse — a failure in a supplementary call discards the whole file.

ctxByPath.set is the last statement of the block (line 1190), so a throw from input.runner.impact(...) (1184), detectPii(...) (1186) or isComplex(...) (1189) leaves the file out of ctxByPath entirely. isComplex in particular is only a tiering hint, yet its failure means the file gets no quality / semantic-change / dbt-patterns lanes, no lineage lane, and no blast-radius / PII / complexity signal for tiering (blastRadiusOf / touchesPiiOf / isComplexOf at 1227-1231) — zero review even though its SQL was read successfully.

Consider per-call fallbacks (impact → zeroed result, pii: [], complex: false) and always calling ctxByPath.set once the content has been fetched, so an enrichment failure degrades the file instead of skipping it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Giving impact, detectPii and isComplex separate fallbacks is reasonable. Today a throw there still lands the file in unverified, and with this commit that also blocks APPROVE.

const impact = await input.runner.impact(model)
if (impact.hasManifest) anyManifest = true
const piiRes = engineNewSql ? await input.runner.detectPii(engineNewSql, dialect) : { columns: [] }
if (piiRes.abstained) unverified.push({ path: file.path, reason: "the PII check could not analyse this model" })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: An abstained PII result is reported but then treated as "no PII" for tiering. pii is assigned piiRes.columns, which is empty, so touchesPiiOf (line 1230) returns false and the FULL-tier "PII column touched" hard floor cannot fire. On trivial/lite tiers pii_exposure is not in TIER_LANES, so piiLane never runs for this file either, leaving only the coarse regex twins. Abstention means unknown, not clean — consider promoting the file (or keeping the broad fallback) when abstained is true.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Treating a PII abstention as "touches PII" for tiering is a fair suggestion.

scope: ReviewScope.optional(),
/** Engine calls that failed during the review. Their checks did not run, so
* the verdict is never a clean APPROVE when this is present. */
engineErrors: z

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: An empty engineErrors array satisfies this schema, and buildEnvelope (line 256) tests input.engineErrors?.length, so engineErrors: [] still produces APPROVE — contradicting the field doc above ("the verdict is never a clean APPROVE when this is present"). runReview normalises empty to undefined, but buildEnvelope is exported and called directly (including from tests). Add .min(1) or normalise empty to undefined in buildEnvelope.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: buildEnvelope tests engineErrors?.length and runReview normalises an empty list to undefined, so an empty array never changes the verdict.

@kilo-code-bot

kilo-code-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 4
SUGGESTION 0

The merge-commit head detection, fail-closed SHA check, bot-identity/marker scoping, dismiss-after-post ordering and the shallow-checkout deepening now behave as documented. The remaining holes are all in the "failures are never clean" contract: one base-side exemption in reviewIncomplete, one over-broad parse pattern, one unrecorded parse abstention, and one unreadable-head case.

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/verdict.ts 288 A base-side not_compiled entry is exempt from reviewIncomplete, so runCompleted() is true and an earlier block is dismissed although the base comparison never ran (semanticChangeLane/structuralChangeLane return nothing, piiLane downgrades to a warning). Contradicts the dismissal wording added in this PR (dbt-pr-review.md:309-312).
packages/opencode/src/altimate/review/runner.ts 100 PARSE_FAILURE_RE also matches genuine engine faults: handlers JSON.parse core-returned strings (native/altimate-core.ts:361-417), so malformed output surfaces as SyntaxError: Unexpected token …, and expected … found/got matches argument/type errors. Those become CoreParseError and are dropped, so a real fault leaves engineErrors empty.
packages/opencode/src/altimate/review/runner.ts 126 A parse abstention is recorded nowhere. Only check reports ran:false (→ PII abstention → unverified); structuralDiff, columnLineage, grain, dataDiff and sourceFilters return empty results that their lanes read as "no findings". A base-side parse failure silently skips the comparison and the run can still APPROVE.
packages/opencode/src/altimate/review/orchestrate.ts 1218 An unreadable head file is never recorded as unverified: the newSql && guard at line 1206 skips the head entry, and this else if records the file's only entry as side: base, which reviewIncomplete exempts. With a compiled base nothing is recorded at all, so a model whose head SQL was never read can APPROVE and clear a block.
Files Reviewed (14 files)
  • docs/docs/usage/dbt-pr-review.md — no new issues (dismissal sentence contradicts the base-side exemption, see verdict.ts:288)
  • packages/opencode/src/altimate/review/format.ts — no issues (per-reason grouping and engine-error note verified)
  • packages/opencode/src/altimate/review/git.ts — no issues (commitParents reads the commit object; all failure paths fail closed; shallow deepening targets the configured head)
  • packages/opencode/src/altimate/review/orchestrate.ts — 1 issue
  • packages/opencode/src/altimate/review/post-github.ts — no issues (bot/self identity, project marker, PR-head proof, dismiss only after a successful post)
  • packages/opencode/src/altimate/review/run.ts — no issues
  • packages/opencode/src/altimate/review/runner.ts — 2 issues
  • packages/opencode/src/altimate/review/telemetry.ts — no issues
  • packages/opencode/src/altimate/review/verdict.ts — 1 issue
  • packages/opencode/src/altimate/telemetry/index.ts — no issues
  • packages/opencode/src/cli/cmd/review.ts — no issues
  • packages/opencode/test/altimate/review-failures-gate.test.ts — no new issues (dismissal, merge-commit, fail-closed and marker round-trip cases are meaningful)
  • packages/opencode/test/altimate/review-runner.test.ts — no new issues (the isParseFailure cases encode the over-broad match noted above)
  • packages/opencode/test/altimate/review-scope-diff.test.ts — no issues
Previous Review Summaries (3 snapshots, latest commit 071d1e8)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 071d1e8)

Status: 7 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 5
SUGGESTION 2

The merge-commit head detection, the fail-closed SHA check and bot-identity gating now work as documented (commitParents reads parents from the commit object, reviewedPrHead accepts refs/pull/N/merge and keeps the block when either SHA is unknown). The remaining problems are all in the new "parse failure = abstention" path and in the base-side not_compiled exemption.

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/runner.ts 100 New: PARSE_FAILURE_RE also matches non-parser faults — unexpected token / parse error from the core handlers' own JSON.parse failures (altimate-core.ts:90-92 returns String(error)), and expected … found validation wording. Those become CoreParseError and are silently dropped, so a genuinely broken engine call leaves engineErrors empty and the run can still APPROVE. The lookahead also misses failed to parse the project manifest and could not parse dbt_project.yml.
packages/opencode/src/altimate/review/runner.ts 126 New: a parse abstention is dropped here with no record. Only check/detectPii turn ran:false into scope.unverified; grade, grain, dbtConfigLint, structuralDiff, sourceFilters, isComplex, columnLineage and equivalence return empty results that read as "no findings". A base-side parse failure in check(head, dialect, oldSql) is uncovered, so the base comparison can silently never run and the verdict still APPROVE.
packages/opencode/src/altimate/review/verdict.ts 288 New: exempting base-side not_compiled also makes runCompleted() true, so a run that never compared against the base can dismiss an earlier block. With the base uncompiled, semanticChangeLane/structuralChangeLane return nothing and piiLane downgrades the new-PII finding to a warning (orchestrate.ts:408, 1373-1374, 1116-1135), so the blocking finding is not reproduced and the block is dismissed — contradicting the dismissal docs added in this PR. Use a stricter predicate for runCompleted than for the APPROVE → COMMENT downgrade.
packages/opencode/src/altimate/review/orchestrate.ts 1213 New: when the head content could not be read (newSql undefined — makeContentResolver returns undefined, not a throw, at git.ts:127-148) and the base is uncompiled, this else if records the file's only entry as side: "base", which the new predicate exempts. The never-read head can then APPROVE and clear a block with no failed/abstained entry.
packages/opencode/src/altimate/review/orchestrate.ts 1226 Still present: the per-file try is too coarse — a throw from impact/detectPii/isComplex (tiering hints) leaves the file out of ctxByPath, so it gets no lanes and no tiering signals. Move ctxByPath.set before the enrichment calls and give each a fallback.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 1223 Still present: abstained PII is recorded but stored as pii = [], so touchesPiiOf is false and the FULL-tier PII hard floor cannot promote the file; on lite/trivial the broad piiLane fallback never runs either. Unknown PII should promote the file, not read as clean.
packages/opencode/src/altimate/review/verdict.ts 178 Still present: engineErrors: [] passes the schema and .length is falsy in reviewIncomplete, so buildEnvelope({ engineErrors: [] }) still yields APPROVE. Add .min(1) or normalise empty to undefined.
Files Reviewed (11 files)
  • docs/docs/usage/dbt-pr-review.md — no new issues (dismissal wording now contradicts the base-side exemption, see verdict.ts:288)
  • packages/opencode/src/altimate/review/git.ts — commitParents correct (raw commit object, all failure paths → [] → fail closed); no new issues
  • packages/opencode/src/altimate/review/orchestrate.ts — 2 issues (1 new, 1 carried)
  • packages/opencode/src/altimate/review/post-github.ts — no new issues (fail-open SHA check and merge-commit detection fixed; isBotReview/marker scoping now gated on bot/self identity)
  • packages/opencode/src/altimate/review/run.ts — no new issues
  • packages/opencode/src/altimate/review/runner.ts — 2 new issues
  • packages/opencode/src/altimate/review/verdict.ts — 1 new issue + 1 carried
  • packages/opencode/test/altimate/adversarial.test.ts — no new issues (DuckDB mock removed; registry closed in afterAll)
  • packages/opencode/test/altimate/review-failures-gate.test.ts — no new issues (merge-commit, fail-closed, not-compiled and marker round-trip cases are meaningful)
  • packages/opencode/test/altimate/review-runner.test.ts — no new issues (the parse-abstention test is real, but the isParseFailure false cases do not cover the false positives above)
  • packages/opencode/test/altimate/review-scope-diff.test.ts — title-only change

Fix these issues in Kilo Cloud

Previous review (commit 27a9a99)

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 2
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/post-github.ts 176 New: scope.headSha is HEAD, which is GitHub's PR merge commit under the documented pull_request actions/checkout and never equals pulls.get().head.sha, so onCurrentHead is always false and the gate is never dismissed. !commitId also fails open when the PR fetch fails.
packages/opencode/src/altimate/review/orchestrate.ts 1220 Still present: the per-file try is too coarse — a throw from impact/detectPii/isComplex (tiering hints) leaves the file out of ctxByPath, so it gets no lanes and no tiering signals. Move ctxByPath.set before the enrichment calls and give each a fallback.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 1218 Still present: abstained PII is recorded with kind: "abstained" but stored as pii = [], so touchesPiiOf is false and the FULL-tier PII hard floor cannot promote the file; on lite/trivial the broad piiLane fallback never runs either. Unknown PII should promote the file, not read as clean.
packages/opencode/src/altimate/review/verdict.ts 167 Still present: engineErrors: [] passes the schema and .length is falsy in reviewIncomplete, so buildEnvelope({ engineErrors: [] }) still yields APPROVE, contradicting the field doc. Add .min(1) or normalise empty to undefined.
Files Reviewed (14 files)
  • docs/docs/usage/dbt-pr-review.md — no new issues
  • packages/opencode/src/altimate/review/format.ts — no new issues
  • packages/opencode/src/altimate/review/git.ts — no new issues (existing review comments cover the fetchTargetHistory double find() and raw-SHA deepen arg)
  • packages/opencode/src/altimate/review/orchestrate.ts — 2 issues
  • packages/opencode/src/altimate/review/post-github.ts — 1 new issue
  • packages/opencode/src/altimate/review/run.ts — no new issues
  • packages/opencode/src/altimate/review/runner.ts — no new issues
  • packages/opencode/src/altimate/review/telemetry.ts — no new issues
  • packages/opencode/src/altimate/review/verdict.ts — 1 issue
  • packages/opencode/src/altimate/telemetry/index.ts — no new issues
  • packages/opencode/src/cli/cmd/review.ts — no new issues
  • packages/opencode/test/altimate/review-failures-gate.test.ts — no new issues
  • packages/opencode/test/altimate/review-runner.test.ts — no new issues
  • packages/opencode/test/altimate/review-scope-diff.test.ts — no new issues

Fix these issues in Kilo Cloud

Previous review (commit 7e2a8c7)

Status: 3 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 2
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 1189 Per-file try is too coarse: a throw from impact/detectPii/isComplex (a tiering hint only) leaves the file out of ctxByPath, so it gets no lanes and no tiering signals despite its SQL being readable. Move ctxByPath.set before the enrichment calls and give each call a fallback.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 1187 abstained PII is recorded but stored as pii = [], so touchesPiiOf is false and the FULL-tier PII hard floor cannot fire; on trivial/lite tiers piiLane never runs either. Unknown PII should promote the file, not read as clean.
packages/opencode/src/altimate/review/verdict.ts 159 engineErrors: [] passes the schema and .length is falsy, so buildEnvelope({ engineErrors: [] }) still yields APPROVE, contradicting the field doc. Add .min(1) or normalise empty to undefined.
Files Reviewed (10 files)
  • docs/docs/usage/dbt-pr-review.md — no new issues
  • packages/opencode/src/altimate/review/format.ts — no new issues
  • packages/opencode/src/altimate/review/orchestrate.ts — 2 issues
  • packages/opencode/src/altimate/review/post-github.ts — no new issues (existing review comments cover the dismissal gating and bot-identification concerns)
  • packages/opencode/src/altimate/review/runner.ts — no new issues (existing review comments cover success: false handling and base-side PII abstention)
  • packages/opencode/src/altimate/review/telemetry.ts — no new issues
  • packages/opencode/src/altimate/review/verdict.ts — 1 issue
  • packages/opencode/src/altimate/telemetry/index.ts — no new issues
  • packages/opencode/src/cli/cmd/review.ts — no new issues
  • packages/opencode/test/altimate/review-failures-gate.test.ts — no new issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 151.5K · Output: 38.5K · Cached: 3.8M

Review guidance: REVIEW.md from base branch fix/review-verdict-trust

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/review/git.ts">

<violation number="1" location="packages/opencode/src/altimate/review/git.ts:212">
P2: The deepen fetch adds `headSha` as a raw SHA request, which requires the remote to allow reachable SHA wants and can make otherwise-valid CI remotes reject the fetch. Remove this extra argument; the target-branch refspec is sufficient for deepening the shallow checkout.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/git.ts:267">
P3: The result of `find()` inside `fetchTargetHistory` is thrown away and `find()` runs again here. Have `fetchTargetHistory` return the resolved base (or undefined) so the caller uses it directly, which also removes the boolean-plus-callback pattern.</violation>
</file>

Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

View guided diff | Turn on auto-fix | Re-trigger cubic

try {
if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return false
const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()
await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The deepen fetch adds headSha as a raw SHA request, which requires the remote to allow reachable SHA wants and can make otherwise-valid CI remotes reject the fetch. Remove this extra argument; the target-branch refspec is sufficient for deepening the shallow checkout.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 212:

<comment>The deepen fetch adds `headSha` as a raw SHA request, which requires the remote to allow reachable SHA wants and can make otherwise-valid CI remotes reject the fetch. Remove this extra argument; the target-branch refspec is sufficient for deepening the shallow checkout.</comment>

<file context>
@@ -191,6 +191,33 @@ async function mergeBase(a: string, b: string, cwd: string): Promise<string | un
+    try {
+      if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return false
+      const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()
+      await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)
+    } catch {
+      return false
</file context>
Suggested change
await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)
await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`], cwd)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. GitHub accepts the raw-SHA want, and on other remotes a failed fetch only means the base isn't found. A retry without the SHA is a cheap addition.

}
return undefined
}
const found = (await find()) ?? ((await fetchTargetHistory(target.branch, opts.cwd, find)) ? await find() : undefined)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The result of find() inside fetchTargetHistory is thrown away and find() runs again here. Have fetchTargetHistory return the resolved base (or undefined) so the caller uses it directly, which also removes the boolean-plus-callback pattern.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 267:

<comment>The result of `find()` inside `fetchTargetHistory` is thrown away and `find()` runs again here. Have `fetchTargetHistory` return the resolved base (or undefined) so the caller uses it directly, which also removes the boolean-plus-callback pattern.</comment>

<file context>
@@ -230,10 +257,15 @@ export async function resolveReviewBase(opts: ResolveBaseOptions): Promise<Resol
+      }
+      return undefined
     }
+    const found = (await find()) ?? ((await fetchTargetHistory(target.branch, opts.cwd, find)) ? await find() : undefined)
+    if (found) return found
     throw new Error(
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing for now: the cost is one extra git merge-base after a deepen fetch. Duplicate of the dev-punia D2/D3 comments.

@dev-punia-altimate dev-punia-altimate left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Code Review — OpenCodeReview (Gemini) — 6 finding(s)

  • 6 anchored to a line (posted inline when the comment stream is on)
  • 0 without a line anchor
All findings (full text)

1. packages/opencode/src/altimate/review/format.ts (L209-L210)

[🟠 MEDIUM] Using the array spread operator within a loop to accumulate items creates a new array on each iteration. This can lead to O(N²) time complexity and unnecessary memory allocations when there are many items. It is more efficient to mutate the existing array by pushing new elements.

Suggested change:

  const byReason = new Map<string, string[]>()
  for (const u of unverified) {
    const paths = byReason.get(u.reason)
    if (paths) paths.push(u.path)
    else byReason.set(u.reason, [u.path])
  }

2. packages/opencode/src/altimate/review/git.ts (L194-L219)

[🟠 MEDIUM] By making fetchTargetHistory return a boolean, the caller is forced to execute await find() an extra time when the fetch succeeds. Since find() invokes an external git merge-base process, this results in redundant and potentially expensive operations. It is recommended to use a generic type so the function can return the found result directly, avoiding double evaluation.

Suggested change:

/**
 * A default CI checkout is shallow (`actions/checkout` fetches one commit) and
 * lacks the target branch. Fetch the branch, then deepen the history until a
 * merge-base exists, so the review works without `fetch-depth: 0`. Returns
 * the resolved value once `found` succeeds; undefined when it never does (no remote, no access).
 */
async function fetchTargetHistory<T>(branch: string, cwd: string, found: () => Promise<T | undefined>): Promise<T | undefined> {
  if (branch.startsWith("-")) return undefined
  try {
    await git(["fetch", "--no-tags", "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`], cwd)
  } catch {
    return undefined
  }
  let result = await found()
  if (result) return result
  for (const depth of [50, 200, 1000]) {
    try {
      if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return undefined
      const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()
      await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)
    } catch {
      return undefined
    }
    result = await found()
    if (result) return result
  }
  return undefined
}

3. packages/opencode/src/altimate/review/git.ts (L267-L268)

[🔵 LOW] Following the suggested refactoring of fetchTargetHistory, you can significantly simplify this caller logic. This effectively removes the complex ternary expression and avoids the redundant invocation of find().

Suggested change:

    const found = (await find()) ?? (await fetchTargetHistory(target.branch, opts.cwd, find))
    if (found) return found

4. packages/opencode/src/altimate/review/runner.ts (L269)

[🔵 LOW] Using any bypasses TypeScript's type checking and goes against best practices. Since data is already typed as Record<string, any>, data.pii is implicitly any, so the explicit as any cast is redundant. However, it's better to provide a more specific type structure, such as !!(data.pii as { parse_error?: boolean })?.parse_error, or properly type the pii property in the data interface to ensure type safety.

Suggested change:

        piiAbstained: !!(data.pii as { parse_error?: boolean })?.parse_error,

5. packages/opencode/src/altimate/review/orchestrate.ts (L1433-L1436)

[🔴 HIGH] High Severity: Risk of synchronous exceptions bypassing .catch()

The fault-tolerance logic heavily relies on chaining .catch() directly to the promise returned by input.runner.check(...). However, check is an interface method. If the underlying implementation is not a strict async function and throws a synchronous exception (e.g., during argument validation before returning a Promise), the exception will bypass the .catch() and crash the Promise.all block.

To safely handle both synchronous exceptions and asynchronous rejections, it's recommended to wrap the await within a try/catch block.

Suggested change:

        let cr: CheckResult = { ran: false }
        try {
          cr = await input.runner.check(ctx.engineNewSql, dialect, ctx.engineOldSql)
        } catch {
          // ignore failures
        }
        if (cr.ran) coreRanFiles.add(ctx.file.path)

6. packages/opencode/src/altimate/review/orchestrate.ts (L1156-L1166)

[🔵 LOW] Low Severity: Error object stringification obscuring details & Duplication of unverified files

  1. Object Stringification: Using String(err) on plain object errors (e.g., { code: 500, message: "failed" }) will evaluate to "[object Object]", making debugging difficult. We should serialize plain objects using JSON.stringify().
  2. Duplication in unverified: A single file might be pushed to unverified multiple times (e.g., once for PII abstained, and again here if a lane crashes). The downstream consumer or UI may not expect duplicate paths. It's recommended to deduplicate or aggregate reasons for the same path.

You can create a helper to format errors and ensure duplicate entries for the same file path are merged or ignored.

Suggested change:

  const formatErr = (err: unknown) => err instanceof Error ? err.message : (typeof err === "object" && err !== null ? JSON.stringify(err) : String(err))
  const laneErrors: Array<{ method: string; reason: string; count: number }> = []
  
  // Consider taking a function `() => Promise<Finding[]>` and using try/catch if lanes can throw synchronously
  const prLevel = (method: string, lane: Promise<Finding[]>) =>
    lane.catch((err) => {
      laneErrors.push({ method, reason: formatErr(err).slice(0, 200), count: 1 })
      return [] as Finding[]
    })
  const failedFile = (file: { path: string }, err: unknown) => {
    const reason = `review failed for this file: ${formatErr(err).replace(/\s+/g, " ").slice(0, 160)}`
    const existing = unverified.find((u) => u.path === file.path)
    if (existing) {
      if (!existing.reason.includes(reason)) existing.reason += `; ${reason}`
    } else {
      unverified.push({ path: file.path, reason })
    }
  }

Comment on lines +209 to +210
const byReason = new Map<string, string[]>()
for (const u of unverified) byReason.set(u.reason, [...(byReason.get(u.reason) ?? []), u.path])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🟠 MEDIUM] Using the array spread operator within a loop to accumulate items creates a new array on each iteration. This can lead to O(N²) time complexity and unnecessary memory allocations when there are many items. It is more efficient to mutate the existing array by pushing new elements.

Suggested change:

Suggested change
const byReason = new Map<string, string[]>()
for (const u of unverified) byReason.set(u.reason, [...(byReason.get(u.reason) ?? []), u.path])
const byReason = new Map<string, string[]>()
for (const u of unverified) {
const paths = byReason.get(u.reason)
if (paths) paths.push(u.path)
else byReason.set(u.reason, [u.path])
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, taken in 6fb2411. Paths are now pushed into the existing array. The list is small in practice (a handful of unverified files), but the change costs nothing.

Comment on lines +194 to +219
/**
* A default CI checkout is shallow (`actions/checkout` fetches one commit) and
* lacks the target branch. Fetch the branch, then deepen the history until a
* merge-base exists, so the review works without `fetch-depth: 0`. Returns
* true once `found` resolves; false when it never does (no remote, no access).
*/
async function fetchTargetHistory(branch: string, cwd: string, found: () => Promise<unknown>): Promise<boolean> {
if (branch.startsWith("-")) return false
try {
await git(["fetch", "--no-tags", "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`], cwd)
} catch {
return false
}
if (await found()) return true
for (const depth of [50, 200, 1000]) {
try {
if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return false
const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()
await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)
} catch {
return false
}
if (await found()) return true
}
return false
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🟠 MEDIUM] By making fetchTargetHistory return a boolean, the caller is forced to execute await find() an extra time when the fetch succeeds. Since find() invokes an external git merge-base process, this results in redundant and potentially expensive operations. It is recommended to use a generic type so the function can return the found result directly, avoiding double evaluation.

Suggested change:

Suggested change
/**
* A default CI checkout is shallow (`actions/checkout` fetches one commit) and
* lacks the target branch. Fetch the branch, then deepen the history until a
* merge-base exists, so the review works without `fetch-depth: 0`. Returns
* true once `found` resolves; false when it never does (no remote, no access).
*/
async function fetchTargetHistory(branch: string, cwd: string, found: () => Promise<unknown>): Promise<boolean> {
if (branch.startsWith("-")) return false
try {
await git(["fetch", "--no-tags", "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`], cwd)
} catch {
return false
}
if (await found()) return true
for (const depth of [50, 200, 1000]) {
try {
if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return false
const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()
await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)
} catch {
return false
}
if (await found()) return true
}
return false
}
/**
* A default CI checkout is shallow (`actions/checkout` fetches one commit) and
* lacks the target branch. Fetch the branch, then deepen the history until a
* merge-base exists, so the review works without `fetch-depth: 0`. Returns
* the resolved value once `found` succeeds; undefined when it never does (no remote, no access).
*/
async function fetchTargetHistory<T>(branch: string, cwd: string, found: () => Promise<T | undefined>): Promise<T | undefined> {
if (branch.startsWith("-")) return undefined
try {
await git(["fetch", "--no-tags", "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`], cwd)
} catch {
return undefined
}
let result = await found()
if (result) return result
for (const depth of [50, 200, 1000]) {
try {
if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return undefined
const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()
await git(["fetch", "--no-tags", `--deepen=${depth}`, "origin", `+refs/heads/${branch}:refs/remotes/origin/${branch}`, headSha], cwd)
} catch {
return undefined
}
result = await found()
if (result) return result
}
return undefined
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This is correct, but the cost is one extra git merge-base, and only after a deepen fetch, so we're leaving it as is for this PR. It is noted as a cleanup for when fetchTargetHistory is next touched.

Comment on lines +267 to +268
const found = (await find()) ?? ((await fetchTargetHistory(target.branch, opts.cwd, find)) ? await find() : undefined)
if (found) return found

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🔵 LOW] Following the suggested refactoring of fetchTargetHistory, you can significantly simplify this caller logic. This effectively removes the complex ternary expression and avoids the redundant invocation of find().

Suggested change:

Suggested change
const found = (await find()) ?? ((await fetchTargetHistory(target.branch, opts.cwd, find)) ? await find() : undefined)
if (found) return found
const found = (await find()) ?? (await fetchTargetHistory(target.branch, opts.cwd, find))
if (found) return found

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Same as D2: a readability cleanup with no behaviour change, deferred together with it.

issues,
piiColumns: [...new Set(piiColumns)],
ran: true,
piiAbstained: !!(data.pii as any)?.parse_error,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🔵 LOW] Using any bypasses TypeScript's type checking and goes against best practices. Since data is already typed as Record<string, any>, data.pii is implicitly any, so the explicit as any cast is redundant. However, it's better to provide a more specific type structure, such as !!(data.pii as { parse_error?: boolean })?.parse_error, or properly type the pii property in the data interface to ensure type safety.

Suggested change:

Suggested change
piiAbstained: !!(data.pii as any)?.parse_error,
piiAbstained: !!(data.pii as { parse_error?: boolean })?.parse_error,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The any here reads loosely typed engine JSON whose shape changes across core versions; every field read is guarded (asArray, typeof checks). Typing the engine payloads properly is a broader change than this PR.

Comment on lines +1433 to 1436
const cr = await input.runner
.check(ctx.engineNewSql, dialect, ctx.engineOldSql)
.catch(() => ({ ran: false }) as CheckResult)
if (cr.ran) coreRanFiles.add(ctx.file.path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🔴 HIGH] High Severity: Risk of synchronous exceptions bypassing .catch()

The fault-tolerance logic heavily relies on chaining .catch() directly to the promise returned by input.runner.check(...). However, check is an interface method. If the underlying implementation is not a strict async function and throws a synchronous exception (e.g., during argument validation before returning a Promise), the exception will bypass the .catch() and crash the Promise.all block.

To safely handle both synchronous exceptions and asynchronous rejections, it's recommended to wrap the await within a try/catch block.

Suggested change:

Suggested change
const cr = await input.runner
.check(ctx.engineNewSql, dialect, ctx.engineOldSql)
.catch(() => ({ ran: false }) as CheckResult)
if (cr.ran) coreRanFiles.add(ctx.file.path)
let cr: CheckResult = { ran: false }
try {
cr = await input.runner.check(ctx.engineNewSql, dialect, ctx.engineOldSql)
} catch {
// ignore failures
}
if (cr.ran) coreRanFiles.add(ctx.file.path)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this. In this case it can't happen. ReviewRunner.check returns runCheck(...), which is an async function (runner.ts check(sql, dialect, baseSql) { return runCheck(sql, dialect, baseSql) }), so any throw inside it becomes a rejected promise and .catch() handles it. The test runners in the suite are async as well. Separately, 6fb2411 moves the remaining synchronous detectors into the per-file task list.

Comment on lines +1156 to +1166
const laneErrors: Array<{ method: string; reason: string; count: number }> = []
const prLevel = (method: string, lane: Promise<Finding[]>) =>
lane.catch((err) => {
laneErrors.push({ method, reason: (err instanceof Error ? err.message : String(err)).slice(0, 200), count: 1 })
return [] as Finding[]
})
const failedFile = (file: { path: string }, err: unknown) =>
unverified.push({
path: file.path,
reason: `review failed for this file: ${(err instanceof Error ? err.message : String(err)).replace(/\s+/g, " ").slice(0, 160)}`,
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🔵 LOW] Low Severity: Error object stringification obscuring details & Duplication of unverified files

  1. Object Stringification: Using String(err) on plain object errors (e.g., { code: 500, message: "failed" }) will evaluate to "[object Object]", making debugging difficult. We should serialize plain objects using JSON.stringify().
  2. Duplication in unverified: A single file might be pushed to unverified multiple times (e.g., once for PII abstained, and again here if a lane crashes). The downstream consumer or UI may not expect duplicate paths. It's recommended to deduplicate or aggregate reasons for the same path.

You can create a helper to format errors and ensure duplicate entries for the same file path are merged or ignored.

Suggested change:

Suggested change
const laneErrors: Array<{ method: string; reason: string; count: number }> = []
const prLevel = (method: string, lane: Promise<Finding[]>) =>
lane.catch((err) => {
laneErrors.push({ method, reason: (err instanceof Error ? err.message : String(err)).slice(0, 200), count: 1 })
return [] as Finding[]
})
const failedFile = (file: { path: string }, err: unknown) =>
unverified.push({
path: file.path,
reason: `review failed for this file: ${(err instanceof Error ? err.message : String(err)).replace(/\s+/g, " ").slice(0, 160)}`,
})
const formatErr = (err: unknown) => err instanceof Error ? err.message : (typeof err === "object" && err !== null ? JSON.stringify(err) : String(err))
const laneErrors: Array<{ method: string; reason: string; count: number }> = []
// Consider taking a function `() => Promise<Finding[]>` and using try/catch if lanes can throw synchronously
const prLevel = (method: string, lane: Promise<Finding[]>) =>
lane.catch((err) => {
laneErrors.push({ method, reason: formatErr(err).slice(0, 200), count: 1 })
return [] as Finding[]
})
const failedFile = (file: { path: string }, err: unknown) => {
const reason = `review failed for this file: ${formatErr(err).replace(/\s+/g, " ").slice(0, 160)}`
const existing = unverified.find((u) => u.path === file.path)
if (existing) {
if (!existing.reason.includes(reason)) existing.reason += `; ${reason}`
} else {
unverified.push({ path: file.path, reason })
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Errors on this path are Error instances in practice (we read .message), and String(err) is only a fallback. Duplicate paths are harmless because the notes group by reason. We'll keep it as is.

@kilo-code-bot

kilo-code-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

Kilo Code Review could not run — your account is out of credits.

Add credits or switch to a free model to enable reviews on this change.

@dev-punia-altimate

Copy link
Copy Markdown
Contributor

🤖 Code Review — OpenCodeReview (Gemini) — No Issues Found

No supported files changed.

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: changes requested. Panel: Claude, Codex and a lead review (independent). Scope: this PR's diff against its stacked base only.

The design is sound (per-method failure recording, signed engineErrors, per-file isolation, injectable client). But the title's invariant only holds for thrown engine errors: core's in-band success:false, per-file failures and PII abstentions can still produce APPROVE, and PII abstention can still clear an earlier block.

Counts: CRITICAL 0, MAJOR 3, MINOR 3, NIT 5.

Missing tests: native success:false responses and a real createDispatcherRunner failure path; verdict assertions for file failure and PII abstention; a block surviving PII abstention; human review quoting the headline; deepening beyond 50 commits.

Nits

  • post-github.ts:66 — orphaned doc comment /** Headline of a review this bot posted (see verdictHeadline). */ now sits above runCompleted; move it back above BOT_REVIEW_RE.
  • runner.ts:367 — fail("dbtConfigLint", err) in the dbt_config_diff branch names the wrong call; use "dbtConfigDiff".
  • runner.ts:360,368 — /* skip */ after the new fail(...) calls is now misleading; remove.
  • orchestrate.ts:1158 — laneErrors reasons ((err instanceof Error ? err.message : String(err)).slice(0, 200)) don't collapse whitespace like fail()/failedFile; a multi-line message breaks the markdown note. Use one shared helper.
  • Naming: runner failures() vs envelope engineErrors vs laneErrors; consider engineErrors(). runCompleted relies on matching reason text; a typed kind on unverified entries would fit the name.

piiColumns: [...new Set(piiColumns)],
ran: true,
piiAbstained: !!(data.pii as any)?.parse_error,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] Core's in-band success:false reads as a clean run

ran: true,
piiAbstained: !!(data.pii as any)?.parse_error,

(const data = (res.data ?? {}) at runner.ts:211 never checks res.success.)

Native handlers catch their own exceptions and return { success: false, data: {}, error } (native/altimate-core.ts, fail()); they do not reject. runCheck never looks at res.success, so a core failure yields ran: true, no issues, piiAbstained: false, and failures() stays empty. Reproduced by two reviewers with a fake altimate_core.check returning success:false: detectPii -> abstained:false, check().ran === true, no engine errors. A SQL the core cannot handle is therefore reported lint- and PII-clean and can still APPROVE and clear a block. The same pattern (res.data ?? {}) applies to grade/grain/source_filters/etc.

Fix: treat res.success === false like a throw: fail(method, res.error) and return the did-not-run result (ran: false), in every dispatcher-backed method.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reproducing it. Fixed in fix(review): address review comments: engine success:false, per-file isolation, failed-file verdict, base PII abstention, bot-review identity. Every altimate_core.* call goes through callCore, which turns success: false into a throw, so each method records fail(method, error) and returns its did-not-run result (check → ran: false → PII abstains). The new commit fix(review): address review feedback: PII abstention keeps the block, typed unverified kinds, safer dismissal, deepen --head`` also covers data.diff's in-band failure. Tests run a real `createDispatcherRunner` with a stubbed `success: false` Dispatcher.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — callCore (runner.ts:119) turns in-band success:false into a throw, so each method records fail(...) and check returns ran:false (PII then abstains). data.diff handled separately (runner.ts:476). review-runner.test.ts drives the real createDispatcherRunner with stubbed failures; removing the success check makes it fail.

// An engine failure means some checks did not run: "no findings" is then
// unknown, not clean, so the verdict cannot be APPROVE.
const computed = computeIdealVerdict(input.findings, rubric)
const ideal = computed === "APPROVE" && input.engineErrors?.length ? "COMMENT" : computed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] File failures and PII abstentions do not prevent APPROVE

const ideal = computed === "APPROVE" && input.engineErrors?.length ? "COMMENT" : computed

Only engineErrors downgrades APPROVE. A per-file failure (review failed for this file: ...) and a PII abstention are recorded only in scope.unverified, which the verdict ignores. Reproduced: a one-model PR whose getContent throws EACCES gives verdict=APPROVE, idealVerdict=APPROVE; a runner returning detectPii -> {columns:[], abstained:true} also APPROVEs. The comment says "Approved" while a note says a file was never reviewed, which contradicts the PR's "failures are never clean" claim.

Fix: pass a typed completeness flag (or failed-file/abstained count) into buildEnvelope and apply the same APPROVE -> COMMENT downgrade. Add a verdict assertion to the unreadable-content test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. File failures were fixed in the earlier commit. The new commit adds a typed kind to scope.unverified (failed/abstained/not_compiled), and reviewIncomplete downgrades APPROVE → COMMENT for engine errors, failed files and abstentions. The unreadable-content test and the PII-abstention test now assert idealVerdict: COMMENT.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — reviewIncomplete (verdict.ts:269) downgrades APPROVE→COMMENT for engine errors, failed and abstained entries; review-failures-gate.test.ts asserts idealVerdict: COMMENT for the unreadable-content and abstention cases (disabling the downgrade fails 7 tests). One gap of the same class remains for not_compiled entries (see the new review).

/** True when no engine call and no file failed, so missing findings mean "fixed", not "not checked". */
export function runCompleted(env: VerdictEnvelope): boolean {
if (env.engineErrors?.length) return false
return !(env.scope?.unverified ?? []).some((u) => u.reason.startsWith("review failed for this file"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] runCompleted treats PII abstention as a completed run, so an earlier block is dismissed

return !(env.scope?.unverified ?? []).some((u) => u.reason.startsWith("review failed for this file"))

Only the file-failure prefix counts as incomplete. Scenario: run 1 blocks on pii_exposure; the next push makes core's PII parser report parse_error, so piiAbstained is set and the finding disappears; runCompleted is true, the event is COMMENT, and dismissReview clears the block though nothing shows the PII was fixed (reproduced with a fake client: dismissReview was called).

Fix: treat abstentions as incomplete here too, preferably via a typed kind on unverified entries rather than matching reason text.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, agreed. Fixed in the new commit: runCompleted is !reviewIncomplete(env), which uses the typed kind rather than matching reason text, so an abstention never dismisses. There's a test of a block surviving an abstained rerun.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed for abstentions — runCompleted is now !reviewIncomplete(env) (post-github.ts:68) and a test keeps the block across an abstained rerun. Note kind: "not_compiled" is still treated as complete, so the same dismissal can happen for an uncompiled model (new finding in the review).

priorBotReview = reviews.some((r) => BOT_REVIEW_RE.test(r.body ?? ""))
const botReviews = reviews.filter((r) => BOT_REVIEW_RE.test(r.body ?? ""))
priorBotReview = botReviews.length > 0
staleBlocking = botReviews.filter((r) => r.state === "CHANGES_REQUESTED").map((r) => r.id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Dismissal is selected by body text only, and not tied to the run's commit

staleBlocking = botReviews.filter((r) => r.state === "CHANGES_REQUESTED").map((r) => r.id)

botReviews is matched by BOT_REVIEW_RE on the body alone. A human CHANGES_REQUESTED review that quotes the headline (e.g. "Bot said: Changes requested ... (full tier") is selected and dismissed if the token has permission. Also, a slow run on an older push that finishes non-blocking can dismiss the block posted by a newer push's run, and in a monorepo with several project_dir jobs one project's clean run dismisses another's block.

Fix: also require the bot identity (user.type === "Bot" or the token's login), dismiss only when this run's head SHA equals the PR's current head, and scope by a per-project marker.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The human-quote case was fixed in the earlier commit (hidden marker or Bot author). The new commit adds two checks. Dismissal needs this run's scope.headSha (new, signed) to equal the PR's current head. Reviews below the repo root carry <!-- altimate-code-review:verdict project=<dir> -->, and only the same project's reviews are dismissed or counted. Tests cover an older push and another project's block.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Partially fixed — project scoping and the head-SHA check were added, but (a) the hidden marker alone still identifies a review regardless of author (a human review copying the full body incl. marker is dismissed; the test at review-failures-gate.test.ts:262 expects that), (b) onCurrentHead fails open when either SHA is missing, and (c) under the documented pull_request checkout scope.headSha is the merge commit, so it never equals the PR head and nothing is ever dismissed. Details in the new review.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three remaining points are addressed in 071d1e8. (a) The marker alone no longer identifies the bot's review: a bot author or the token's own login is required, and the test at review-failures-gate.test.ts now expects a human's copied marker to be left alone. (b) onCurrentHead is now reviewedPrHead, which fails closed when either SHA is missing. (c) The refs/pull/N/merge checkout is recognised through the merge commit's parents. The misleading comment at post-github.ts:171-174 has been rewritten to match.

for (const depth of [50, 200, 1000]) {
try {
if ((await git(["rev-parse", "--is-shallow-repository"], cwd)).trim() !== "true") return false
const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Deepening fetches HEAD, not the configured head ref

const headSha = (await git(["rev-parse", "HEAD"], cwd)).trim()

find() merge-bases against opts.head, but deepening resolves HEAD. With --head <ref> different from the checkout, deepening never helps and the review errors after three wasted fetches. Related: the first fetch (without --depth) in a shallow repo pulls the target branch's whole history, defeating the shallow checkout.

Fix: pass the head ref into fetchTargetHistory, and start with --depth=50 when the repo is shallow.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in the new commit: fetchTargetHistory deepens the configured head ref, and a shallow checkout starts with --depth=50 instead of the target's whole history. There's a test with --head different from the checkout and more than 50 commits on each side.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — fetchTargetHistory resolves the configured head ref and starts with --depth=50 on shallow repos (git.ts:222-234). Caveat: reverting either change leaves review-scope-diff.test.ts:466 passing (--deepen moves the shallow boundary repo-wide), so the test does not pin the behavior.

// the new verdict. (Requires `pull-requests: write`; a failure is non-fatal.)
// Only a run that actually completed may clear a block: when an engine call
// or a file failed, the blocking finding may simply not have been produced.
if (event === "COMMENT" && staleBlocking.length && runCompleted(env)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Blocks are dismissed before the replacement review is posted

if (event === "COMMENT" && staleBlocking.length && runCompleted(env)) {

If the later createReview fails (5xx, rate limit), the block is already gone and only the summary comment remains.

Fix: dismiss after createReview succeeds, or after the skip decision.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. Fixed in the new commit: dismissal runs only after createReview succeeds (or on the skip path, where the summary is already updated). If both createReview attempts fail, the block stays (tested).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — dismissStale() now runs after createReview succeeds (post-github.ts:217,241) or on the skip path; the test with both createReview attempts failing asserts the block stays.

// not have been produced. A run on an older push (its head is no longer the
// PR's head) never clears the block a newer push's run may have posted.
const headSha = env.scope?.headSha
const onCurrentHead = !headSha || !commitId || headSha === commitId

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: This guard never matches under the documented CI setup, so the gate is never dismissed.

scope.headSha is filled by run.ts from git rev-parse HEAD. With the documented workflow (actions/checkout on a pull_request event and no --head), HEAD is GitHub's PR merge commit (refs/pull/N/merge), while commitId here comes from pulls.get().head.sha — the branch head. The SHAs differ on every run, onCurrentHead is always false, and dismissStale returns early, so the "gate clears itself" behaviour this PR adds never fires for the setup the docs describe.

Consider also accepting the merge ref (e.g. compare commitId against the merge commit's parents, or use the event payload's pull_request.head.sha), or have run.ts record the PR head SHA. Separately, !commitId currently reads as "on the current head": if pulls.get fails, an older push's run can still clear a newer block. That fallback should fail closed.

@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

@hkaltai, re your review:

Thanks for the careful review. Two commits on top address all three MAJORs and the MINORs:

  • In-band success:false → engine error (callCore, now also data.diff). File failures and PII abstentions block APPROVE and never clear a block, using a typed kind on unverified entries.
  • Dismissal only on the PR's current head and for the same project's marker, and only after the replacement review lands. Deepening follows --head, and shallow repos start at --depth=50.
  • Nits: doc comment back on BOT_REVIEW_RE; dbtConfigDiff named correctly; stale /* skip */ removed; one shared errorReason helper (one line, bounded) for runner failures, lane errors and failed files. I kept the failures() name because it's the runner interface across the stack.
  • Missing tests you listed: native success:false with the real runner, verdict assertions for file failure and abstention, a block surviving abstention, a human review quoting the headline, deepening past 50 commits. All added.

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: changes requested. Panel: Claude, Codex and a lead review (independent), re-review of b354b9f...27a9a99 (history was rewritten and the base moved since the previous review). Replies posted in each prior inline thread.

All three prior MAJORs are fixed for thrown engine errors, core success:false, failed files and typed abstained entries, with regression tests (review suite passes, typecheck clean). The claim "failures and PII abstentions can no longer pass as clean or dismiss a block" still has gaps: an uncompiled model can approve and dismiss a block, and the dismissal guard is both fail-open and never matches under the documented Action checkout.

Counts (new): CRITICAL 0, MAJOR 2, MINOR 2 inline, plus the body items below.

Prior findings

Prior Status
MAJOR core success:false reads clean Fixed (callCore, tested)
MAJOR file failure / PII abstention still APPROVE Fixed (reviewIncomplete, tested); not_compiled gap, see inline
MAJOR runCompleted treats abstention as complete Fixed for abstention; not_compiled gap, see inline
MINOR dismissal by body text / not tied to commit Partially fixed: project scoping and human-quote-without-marker fixed; copied marker, fail-open SHA and merge-commit mismatch remain (inline)
MINOR deepen fetches HEAD Fixed in code; test does not pin it (reverting leaves it green)
MINOR dismiss before createReview Fixed (tested)
Nit orphaned doc comment Fixed
Nit fail("dbtConfigLint") wrong name Fixed
Nit /* skip */ Fixed
Nit laneErrors whitespace Fixed (errorReason)
Nit naming failures() vs engineErrors Partially: typed kind added; failures() still named differently

Other findings (not anchorable to the diff)

  • MINOR cli/cmd/review.ts:211-213: in gate mode only REQUEST_CHANGES exits non-zero. An incomplete run (engine error, failed file, abstention) is downgraded to COMMENT, so the required CI check goes green on the very run that could not re-verify an earlier block. Consider a distinct non-zero exit for reviewIncomplete(env) in gate mode, with a test.
  • MINOR post-github.ts:76,91: reviews posted before the marker existed, in a project below the repo root, compare "" to projectRoot and are never matched (no auto-dismiss, wrong priorBotReview); project=(\S+) cannot match a directory containing spaces. Encode the root in the marker and treat unmarked Bot reviews as matching any project for dismissal.

Nits

  • verdict.ts:272 u.reason.startsWith(FAILED_FILE_REASON) is redundant now that failedFile sets kind: "failed"; drop it or comment it as legacy-envelope support.
  • review-scope-diff.test.ts:466 test name claims deepening follows --head but passes when HEAD is deepened; rename or add a case where only the --head history matters.
  • docs/docs/usage/dbt-pr-review.md:334 "at most COMMENT" is inaccurate: an incomplete run keeps REQUEST_CHANGES when a real blocking finding exists; say "an otherwise clean verdict becomes COMMENT". Also the "gate clears itself" bullet needs a caveat about the merge-commit checkout until the SHA check is fixed.
  • post-github.ts:171-174 comment says an older-push run never clears a block, but a missing SHA skips the check.
  • runner.ts:586 failures() vs engineErrors/laneErrors: consider engineErrors(); runCompleted/onCurrentHead names overstate behavior (true when unknown).

Missing tests

Merge-commit checkout vs PR head; gate exit code for an incomplete run; blocked rerun without compiled head must not dismiss; failed pulls.get / missing headSha must keep the block; human quoting the full body incl. marker; --head-only deepening and the initial --depth=50 fetch; synchronous-detector throw isolation.

export function reviewIncomplete(input: Pick<BuildEnvelopeInput, "engineErrors" | "scope">): boolean {
if (input.engineErrors?.length) return true
return (input.scope?.unverified ?? []).some(
(u) => u.kind === "failed" || u.kind === "abstained" || u.reason.startsWith(FAILED_FILE_REASON),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] not_compiled models count as a completed review: APPROVE and block dismissal with the PII check never run

(u) => u.kind === "failed" || u.kind === "abstained" || u.reason.startsWith(FAILED_FILE_REASON),

reviewIncomplete ignores kind: "not_compiled". When a Jinja model has no compiled head SQL, orchestrate.ts:1206-1218 records not_compiled and skips detectPii entirely (piiRes = { columns: [] }). Reproduced by two reviewers and the lead: a run with no findings and one head-side not_compiled entry gives idealVerdict=APPROVE, runCompleted=true.

Scenario: push 1 blocks on pii_exposure in models/m.sql. Push 2 has a missing/partial dbt compile (stale target/, compile in another job). No PII finding is produced, the run counts as complete, dismissStale removes the block. This is the same "nothing shows the PII was fixed" hole as the abstention MAJOR, through a different kind.

Fix: treat a head-side not_compiled as incomplete (at least for APPROVE and runCompleted); a base-side one can stay non-blocking. Add a test: blocked model, rerun without compiled head, block is not dismissed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. This is fixed in 071d1e8 ("fix(review): address re-review: not-compiled models, …"). Each not_compiled entry now records the side that had no compiled SQL. reviewIncomplete counts a head-side entry as incomplete, because that model's engine and PII checks never ran. So the run can't APPROVE, and runCompleted stays false, which keeps an earlier block. A base-side entry is still non-blocking as you suggested, and older envelopes without side are treated as head-side. As a result a parse-only run (no dbt compile) never approves a changed Jinja model. That is now stated in the docs. I added your scenario as a test ("a rerun without compiled head SQL for a blocked model does not clear the block"), plus verdict tests for each side and an orchestration test that checks side: "head".

// not have been produced. A run on an older push (its head is no longer the
// PR's head) never clears the block a newer push's run may have posted.
const headSha = env.scope?.headSha
const onCurrentHead = !headSha || !commitId || headSha === commitId

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] The new head check means auto-dismiss never fires with the documented GitHub Action setup (and fails open when a SHA is missing)

const onCurrentHead = !headSha || !commitId || headSha === commitId

scope.headSha is rev-parse HEAD (run.ts:241). On a pull_request event the documented workflow (actions/checkout@v4, docs dbt-pr-review.md:186) checks out refs/pull/N/merge, a synthetic merge commit; github/review/action.yml:191 passes --head only if the user sets the input. commitId is pulls.get().head.sha, the branch tip, so the two never match. Reproduced with a real merge-commit checkout and a fake client: dismissReview called 0 times. The gate therefore never clears itself in the default setup; the tests only use made-up SHAs ("old"/"abc").

Separately the guard fails open: if pulls.get throws (commitId undefined) or headSha is undefined (resolved falsy), a stale run on an older push can still dismiss a newer block (reproduced).

Fix: take the PR head from GITHUB_EVENT_PATH (pull_request.head.sha) or accept a merge-commit HEAD when one parent equals commitId; require both SHAs known and equal, otherwise keep the block. Add a merge-commit-checkout test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both counts, and thanks for reproducing it with a real merge checkout. Fixed in 071d1e8. run.ts now records scope.headParents when the reviewed commit is a merge commit. It reads them with git cat-file, so the depth-1 actions/checkout still sees them. The new reviewedPrHead treats the run as current when the reviewed commit equals pulls.get().head.sha or is a merge commit whose second parent is that SHA (the refs/pull/N/merge shape). It fails closed: if either SHA is unknown, including when pulls.get throws, the block stays. I went with the parent check rather than GITHUB_EVENT_PATH because it is tied to what was actually reviewed and also works with --head. Tests cover a merge checkout of the current head (dismisses), a merge built from an older push (keeps the block), a missing headSha and a failing pulls.get (both keep the block), and commitParents on a real depth-1 clone.

function isBotReview(r: { body?: string | null; user?: { type?: string } | null }, projectRoot: string | undefined): boolean {
const body = r.body ?? ""
if (!BOT_REVIEW_RE.test(body)) return false
const marker = BOT_REVIEW_MARKER_RE.exec(body)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] A copied marker still substitutes for author identity

if (!marker && r.user?.type !== "Bot") return false

Any author is accepted when the body contains the hidden marker, so a human CHANGES_REQUESTED review that quotes the full bot body (headline plus marker) is selected and dismissed if the token allows it. The new test at review-failures-gate.test.ts:262 asserts exactly that. Reproduced.

Fix: require the bot identity (user.type === "Bot" or the token's login) independently of the body; use the marker only for project/purpose.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 071d1e8. Author identity is now checked separately from the body: a review counts as the bot's only if user.type === "Bot" or the author is the token's own login. The login is looked up with users.getAuthenticated only when a marked review has a non-bot author, which covers personal tokens. The marker now only scopes the review to a project. The test that encoded the old behaviour now asserts that a human review copying the full body, marker included, is not dismissed. A second test covers the token-login case.

// catch records the failure; otherwise a failed call reads as a clean result.
const callCore: typeof Dispatcher.call = async (method, params, context) => {
const res: any = await Dispatcher.call(method, params, context)
if (res?.success === false) throw new Error(res.error || `${method} failed`)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Routine dialect parse failures now count as engine failures

if (res?.success === false) throw new Error(res.error || `${method} failed`)

Core handlers return success:false for any caught error, including "cannot parse this dialect's SQL" (the comment above altimate_core.equivalence describes that as an abstention). With callCore, equivalence, columnLineage, grain, sourceFilters on Snowflake/BigQuery-specific SQL now add engineErrors, which blocks APPROVE and prints "Some engine checks failed" on many ordinary PRs.

Fix: decide deliberately: document it, or distinguish parse-error abstentions from real engine faults in callCore.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that was a regression callCore introduced. Fixed in 071d1e8, and the behaviour is documented. callCore now classifies success:false errors. Parse and unsupported-dialect errors ("sql parser error", "ParseError", "cannot parse …", "unsupported dialect …", "expected … found …") become CoreParseError. These abstain: the method returns its "unknown" result, nothing is added to engineErrors, and check still reports ran:false, so a PII abstention still marks the model not fully verified. Any other error is still an engine failure. Schema, manifest and YAML parse errors are deliberately excluded. Tests: a review-runner case with a Snowflake QUALIFY parse error (no failures, equivalence undecided, PII abstained), plus a classifier table.

@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough re-review. Both MAJORs and both inline MINORs are fixed in 071d1e8, with a regression test for each (review-failures-gate.test.ts 29 tests, review-runner.test.ts 10). From the body:

  • The post-github.ts:76,91 MINOR is fixed in the same commit. A pre-marker bot review now matches any project, and the project in the marker is URI-encoded, so my project round-trips.
  • Nits fixed: the FAILED_FILE_REASON prefix check now applies only to legacy envelopes without kind. The review-scope-diff test is renamed to describe what it really checks. The docs now say "an otherwise clean verdict becomes COMMENT". The "gate clears itself" bullet now describes the merge-commit and bot-author rules.
  • Left open: failures() → engineErrors(). It is a ReviewRunner interface rename that every later branch in the stack implements, so I'd rather do it as a separate mechanical change.
  • Left open: the gate exit code for an incomplete run (cli/cmd/review.ts:211). That changes what a required check means for every gate user, so I've raised it for a decision rather than slipping it in here.
  • Not added yet: a test pinning --head-only deepening and the initial --depth=50 fetch. This is still on the missing-tests list.


/** Core error text for SQL it could not parse (unsupported dialect syntax). */
const PARSE_FAILURE_RE =
/\bparse ?error\b|\bparser error\b|(cannot|could not|failed to|unable to) parse (?!(the )?(schema|manifest|dbt|ya?ml|json|policy)\b)|\bsyntax error\b|unsupported (dialect|syntax|statement)|\bunexpected token\b|\bexpected\b[^\n]*\b(found|got)\b/i

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: PARSE_FAILURE_RE also matches non-parser faults, which are then dropped as abstentions instead of engine errors.

\bunexpected token\b and \bparse ?error\b match the core handlers' own JSON.parse failures (native/altimate-core.ts:90-92 returns String(error), so a malformed core payload becomes SyntaxError: Unexpected token … in JSON), and \bexpected\b[^\n]*\b(found|got)\b matches ordinary validation wording like expected 1 row, found 2. Those become CoreParseError in callCore and are silently ignored by fail(), so a genuinely broken engine call leaves engineErrors empty and the run can still APPROVE. The lookahead also does not exclude failed to parse the project manifest (project sits between the and manifest) or could not parse dbt_project.yml (_ is a word character after dbt).

Consider matching core's SQL-parser wording only (or keying off the call's SQL params), and keeping JSON/validation errors as faults. The false-positive cases are not covered by the new isParseFailure test.

// The engine could not parse this SQL (dialect syntax it does not support):
// the check abstains on that model, it did not fail. Each method still
// returns its "unknown" result (`check` reports ran:false, so PII abstains).
if (err instanceof CoreParseError) return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: A parse failure is dropped here without recording any abstention, so most lanes now read "not parsed" as "clean".

Only check/detectPii convert ran:false into scope.unverified (orchestrate.ts:1222-1224). For grade, grain, dbtConfigLint, structuralDiff, sourceFilters, isComplex, columnLineage and equivalence, fail() is a no-op and each returns its empty/false result, which the lanes treat as "no findings". Concretely, qualityLane calls check(head, dialect, oldSql) with the base SQL: a parse failure on that combined call returns {ran:false, issues:[]} with no unverified entry, so the base comparison silently never ran while the head PII check is clean and the verdict can still APPROVE. isComplex returning false also under-tiers a model with no record.

Consider recording a parse abstention per (method, file) in scope.unverified instead of returning silently.

(u) =>
u.kind === "failed" ||
u.kind === "abstained" ||
(u.kind === "not_compiled" && u.side !== "base") ||

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Exempting a base-side not_compiled here also makes runCompleted true, so a run that never compared against the base can dismiss an earlier block.

runCompleted() is !reviewIncomplete() (post-github.ts:69-71), and dismissStale requires only that. With the base uncompiled, engineOldSql is undefined: semanticChangeLane returns [] (orchestrate.ts:408), structuralChangeLane skips, and the newly-exposed-PII comparison is skipped with piiLane emitting a warning instead of a critical (orchestrate.ts:1373-1374, 1116-1135). A prior CHANGES_REQUESTED finding can therefore simply not be reproduced and the block is dismissed. The docs added in this PR ("a changed model had no compiled SQL, the earlier block stays") say the opposite.

Consider a stricter predicate for runCompleted (any not_compiled, including side: "base") while keeping the base exemption only for the APPROVE → COMMENT downgrade.

kind: "not_compiled",
side: "head",
})
else if (withOld && oldSql && !engineOldSql)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: When the head content could not be read, this else if records the file's only gap as base-side, which the new predicate exempts.

The head branch requires a truthy newSql. makeContentResolver returns undefined (not a throw) on a failed git show or read (git.ts:127-148), so for a modified/renamed Jinja model whose head file is unreadable the head branch is skipped; when the base is also uncompiled the else if records {kind:"not_compiled", side:"base"}. reviewIncomplete ignores base-side entries, so the run can APPROVE (and clear a block) on a file that was never read, with no failed/abstained entry either.

Consider recording a head-side entry when newSql is missing for a non-deleted model, before falling through to the base branch.

RaghwendraSingh and others added 5 commits November 21, 2026 13:14
…le isolation, PII abstention, gate dismissal

- The dispatcher runner records every engine call that throws (method,
  first reason, count) instead of silently returning empty. They are signed
  into the envelope (`engineErrors`), named in the comment, counted in
  `review_run.engine_errors`, and an empty review with engine errors is
  COMMENT, never APPROVE.
- A failure while preparing one file, or a lane throwing for one file, marks
  that file unverified with the reason; the rest of the PR is still reviewed.
  PR-level lanes and the core-ran check are guarded the same way.
- `detectPii` reports `abstained` when core could not analyse the SQL (check
  did not run, or its PII parser reported a parse error); the model is listed
  as not fully verified instead of passing as PII-clean.
- When a rerun no longer blocks, earlier blocking altimate reviews are
  dismissed so the gate does not stay closed. The GitHub client is injectable
  so dedup, dismissal and skip rules are tested without the network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fetch the base; upgrade note for gate users

- a stale `CHANGES_REQUESTED` review is dismissed only when the new run completed (no `engineErrors`, no file that failed); an engine crash no longer unblocks a PR by making its findings disappear
- in CI, a missing target branch is fetched from `origin` and a shallow history deepened (50/200/1000) until the merge-base exists, so the default `actions/checkout` works without `fetch-depth: 0`; still an error when that fails
- docs: what changes for existing `gate` setups (pattern rules no longer block unless listed in `rubric.blockingRules`) and the new checkout behavior

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e isolation, failed-file verdict, base PII abstention, bot-review identity

- `runner.ts`: `altimate_core.*` handlers return `{ success: false }` instead of
  throwing; a `callCore` wrapper rethrows so each method records the failure in
  `engineErrors` and a failed check no longer reads as clean and PII-free
- `orchestrate.ts`: the synchronous `lineageBreakageLane` / `detectModelPatterns`
  calls run as per-file tasks, and each schema.yml file is guarded, so one throw
  degrades only that file instead of aborting the review
- `orchestrate.ts`: an unanalysable base no longer reports every head PII column
  as newly exposed; the file is marked unverified instead
- `verdict.ts`: a file whose review failed downgrades an otherwise clean
  verdict to COMMENT (shared `FAILED_FILE_REASON`)
- `post-github.ts`: stale blocking reviews must carry a hidden marker or a bot
  author, so a human review quoting the headline is never dismissed
- `format.ts`: group unverified paths without re-spreading the array per item

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… typed unverified kinds, safer dismissal, deepen `--head`

- `scope.unverified` entries carry a typed `kind` (`failed`, `abstained`,
  `not_compiled`). `reviewIncomplete` (engine errors, failed files, abstained
  checks) drives both the APPROVE -> COMMENT downgrade and `runCompleted`, so a
  PII abstention neither approves nor dismisses an earlier block.
- Dismissal: only when this run's head (`scope.headSha`, new) is the PR's
  current head; only reviews carrying this project's marker
  (`<!-- altimate-code-review:verdict project=<dir> -->` below the repo root);
  and only after the replacement review was posted (or skipped). A failed
  `createReview` keeps the block.
- `fetchTargetHistory` deepens the configured head ref, not `HEAD`, and a
  shallow checkout starts with `--depth=50` instead of the target's whole
  history.
- Runner: `dataDiff` records an in-band `success: false`; the config-diff
  failure is named `dbtConfigDiff`; stale `/* skip */` comments removed.
- One `errorReason` helper (one line, bounded) for runner failures, lane
  errors and failed files, so a multi-line message cannot break the note.
- Docs: abstentions block APPROVE and keep the block; dismissal rules.
- Tests: verdict on unreadable content and PII abstention, a block surviving
  abstention, older-push and other-project runs not dismissing, no dismissal
  when the replacement review fails, deepening past 50 commits with `--head`,
  `dataDiff` in-band failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l check, bot-authored dismissals, dialect parse failures

- `reviewIncomplete` treats a model with no compiled head SQL (`kind: "not_compiled"`, new `side: "head"`) as incomplete: its engine and PII checks were skipped, so the run cannot APPROVE or dismiss a block; a base-side gap stays non-blocking; legacy entries without `side` count as head-side
- dismissal requires the run to be proven on the PR head: `reviewedPrHead` accepts the head itself or the `refs/pull/N/merge` commit built from it (`scope.headParents`, read with `git cat-file` so shallow checkouts work) and fails closed when either SHA is unknown
- `isBotReview` requires a bot author (or the token's own login via `users.getAuthenticated`); the marker only scopes the project. Unmarked bot reviews match any project; the project in the marker is URI-encoded so paths with spaces round-trip
- `callCore` maps core `success:false` parse/unsupported-dialect errors to `CoreParseError`, which abstains (unknown result, PII still abstains) instead of adding `engineErrors`
- the legacy `FAILED_FILE_REASON` prefix check applies only to envelopes without `kind`; misleading dismissal comment and test name fixed
- docs: an otherwise clean verdict becomes COMMENT; parse-only runs never APPROVE a changed Jinja model; the gate's head check and bot-author rule
- tests: not-compiled head/base verdicts and orchestration `side`, blocked rerun without compiled head, merge-commit checkout (fake client and a real depth-1 clone), unknown SHA / failed `pulls.get`, copied marker by a human, token-login reviews, legacy and spaced project markers, parse-failure abstention

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(u) =>
u.kind === "failed" ||
u.kind === "abstained" ||
(u.kind === "not_compiled" && u.side !== "base") ||

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: a base-side not_compiled entry leaves runCompleted() true, so an earlier block can be dismissed without the base comparison ever running.

runCompleted() is !reviewIncomplete(env) (post-github.ts:69), and dismissStale() clears every bot CHANGES_REQUESTED review when it is true. With no engineOldSql, semanticChangeLane returns [] (orchestrate.ts:408), structuralChangeLane returns [] (orchestrate.ts:936), and piiLane downgrades a new-PII critical to a warning (baseUnknown, orchestrate.ts:1373-1374, 1116-1135). The blocking finding therefore cannot be reproduced, yet the block is cleared — while the docs added in this PR say the earlier block stays when a changed model had no compiled SQL (dbt-pr-review.md:309-312). Use a stricter predicate for runCompleted than for the APPROVE → COMMENT downgrade (for example, treat any not_compiled entry as incomplete, or require no unverified entries at all).


/** Core error text for SQL it could not parse (unsupported dialect syntax). */
const PARSE_FAILURE_RE =
/\bparse ?error\b|\bparser error\b|(cannot|could not|failed to|unable to) parse (?!(the )?(schema|manifest|dbt|ya?ml|json|policy)\b)|\bsyntax error\b|unsupported (dialect|syntax|statement)|\bunexpected token\b|\bexpected\b[^\n]*\b(found|got)\b/i

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: this pattern also matches engine faults, which are then dropped as abstentions.

The handlers JSON.parse a string returned by the core (native/altimate-core.ts:361,371,383,392,401,408,417) and wrap a throw as { success: false, error: String(error) } (native/altimate-core.ts:90-92). Malformed core output surfaces as SyntaxError: Unexpected token …, matched by \bunexpected token\b; expected … (found|got) likewise matches argument/type errors. callCore turns that into CoreParseError, fail() returns early without an engineErrors entry, and the method returns its empty result — a genuine core fault then reads as no findings. Match SQL-parse wording specifically (require sql/query/dialect context) rather than generic JS error text.

// The engine could not parse this SQL (dialect syntax it does not support):
// the check abstains on that model, it did not fail. Each method still
// returns its "unknown" result (`check` reports ran:false, so PII abstains).
if (err instanceof CoreParseError) return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: a parse abstention is dropped with no record, and for every method except check the unknown result is indistinguishable from a clean one.

check reports ran: false, which makes detectPii abstain and adds a scope.unverified entry. But structuralDiff, columnLineage, grain, dataDiff and sourceFilters return []/{}/null here, and their lanes read that as no findings (for example structuralChangeLane, orchestrate.ts:939-962). A base-side parse failure — check(head, dialect, baseSql) can still succeed because its uses of baseSql are individually guarded, while structural_diff(base, head) fails — silently skips the base comparison, and with no engineErrors/unverified entry the verdict can still APPROVE. Record the abstention (or mark the file unverified) in the non-check methods too.

path: file.path,
reason: "no compiled SQL for the base (compile the base into target-base)",
kind: "not_compiled",
side: "base",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: an unreadable head file is never recorded as unverified, so this base-side entry can be the file's only record.

getContent returns undefined rather than throwing when git show <head>:<path> fails or the path escapes the root (git.ts:126-149). For a modified model with no head content, the newSql && guard at line 1206 skips the head entry, and this else if records the file's only entry as side: base, which reviewIncomplete exempts (verdict.ts:288). When the base is compiled, nothing is recorded at all. Either way the file gets no engine lanes (ctx.newSql is undefined) and no unverified marker, so the verdict can be APPROVE and a stale block cleared. Record a failed/abstained entry when the file is not deleted and newSql is undefined.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants