Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/docs/data-engineering/tools/dbt-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,12 +373,12 @@ WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})
Beyond the agent-facing tools above, altimate-code ships **harness-side
validators** that fire automatically after the agent declares done. They run
`altimate-dbt test` and `altimate-dbt schema-verify` against every model
modified during the session and block "done" if anything failed.
modified during the session.

This is **opt-in** today via either `ALTIMATE_VALIDATORS_ENABLED=1`
(enforcement mode — failing validators block "done" with synthetic
retries) or `ALTIMATE_VALIDATORS_SHADOW=1` (telemetry-only mode — runs
without blocking, useful for measuring "would have caught" rates). When
With `ALTIMATE_VALIDATORS_ENABLED=1`, confirmed findings block completion
with synthetic retries and inconclusive checks leave advisories. Set
`ALTIMATE_VALIDATORS_STRICT=1` to restore the previous retry policy.
`ALTIMATE_VALIDATORS_SHADOW=1` alone runs the checks for telemetry. When
neither flag is set the dispatch path is completely skipped and there
is zero overhead. See the [Validators page](../validators.md) for the
full reference, env var catalogue, performance characteristics, and
Expand Down
30 changes: 22 additions & 8 deletions docs/docs/data-engineering/validators.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@ description: "Harness-side validators run after the agent declares done, catchin

Validators are harness-side checks that run **after** the agent declares done.
Unlike tools, they are not visible to the agent — the framework dispatches
them automatically once the LLM emits `finishReason === "stop"`. If a
validator fails, the framework injects a synthetic user turn with the
failure body, giving the agent one more turn to fix the issue (bounded by
a per-session retry budget).
them automatically once the LLM emits `finishReason === "stop"`.
With `ALTIMATE_VALIDATORS_ENABLED=1`, a confirmed finding gives the agent
one more turn to fix the issue within the retry budget. An inconclusive
check leaves a note in the next model request's context. Set
`ALTIMATE_VALIDATORS_STRICT=1` to restore the previous retry policy.
Shadow-only mode records results in telemetry without injecting messages.

This catches a class of error that prompt engineering alone cannot: the
agent claiming "done" while the work is actually broken (failing dbt
Expand Down Expand Up @@ -53,7 +55,8 @@ deterministic-validators work and are currently soak-tested in shadow mode

After the agent declares done, runs `altimate-dbt test --model <name>`
against every dbt model the agent modified during this session. Refuses
to terminate if any model's tests fail or error.
to terminate if a model's test run reports failures. If the test tool cannot
return a usable result, the validator reports an advisory.

**Catches**: row-data correctness errors (`relationships`, `unique`,
`not_null`, `accepted_values`, `AUTO_*_equality` tests) — the kind of
Expand All @@ -66,7 +69,8 @@ counts.
After the agent declares done, runs `altimate-dbt schema-verify --model
<name>` on every modified model. Reports a mismatch if the produced
column shape diverges from the `schema.yml` spec (extra, missing,
reordered, or type-mismatched columns).
reordered, or type-mismatched columns). If the schema verification tool
cannot complete the check, the validator reports an advisory.

**Catches**: column-level drift that wouldn't be caught by `dbt build`
alone — equality tests against the spec would fail later but the
Expand Down Expand Up @@ -137,7 +141,7 @@ Two opt-in modes:

| Env var | Effect |
|---|---|
| `ALTIMATE_VALIDATORS_ENABLED=1` | Full enforcement. Failing validators inject a synthetic user turn for the agent to fix (bounded by retries). |
| `ALTIMATE_VALIDATORS_ENABLED=1` | Enforces findings within the retry budget and adds inconclusive checks as advisories. |
| `ALTIMATE_VALIDATORS_SHADOW=1` | Telemetry-only. Validators run and emit `validator_check` events (with `enforced: false`), but do **not** block the session. Use this to measure "would have caught a real bug" rates against production traffic. |

Set in your shell, your `~/.altimate-code/altimate-code.json`'s `env`
Expand All @@ -158,6 +162,7 @@ export ALTIMATE_VALIDATORS_SHADOW=1
|---|---|---|
| `ALTIMATE_VALIDATORS_ENABLED` | unset (off) | Master enforcement switch |
| `ALTIMATE_VALIDATORS_SHADOW` | unset (off) | Telemetry-only mode |
| `ALTIMATE_VALIDATORS_STRICT` | unset (off) | Set to `1` to restore the prior retry policy for inconclusive checks and suppress advisories. Requires enforcement mode. |
| `ALTIMATE_VALIDATORS_MAX_RETRIES` | `3` | How many synthetic-message retries per session before giving up |
| `ALTIMATE_VALIDATORS_TIMEOUT_MS` | `60000` | Per-subprocess kill timeout (NaN/0/negative falls back to default) |
| `ALTIMATE_VALIDATORS_CONCURRENCY` | `4` | Max concurrent `altimate-dbt` subprocesses (clamped to `items.length`) |
Expand Down Expand Up @@ -186,7 +191,11 @@ minute of wall time.
## Telemetry

When validators run (either mode), they emit one
`validator_check` event per applied validator:
`validator_check` event per applied validator. The optional `classification`
field is `finding` or `unverifiable`. The `ok` field retains its legacy
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
meaning. `enforced` is false for advisory-only results when
`ALTIMATE_VALIDATORS_ENABLED=1` is set without
`ALTIMATE_VALIDATORS_STRICT=1`.

```json
{
Expand Down Expand Up @@ -221,8 +230,13 @@ When a validator runs, it returns:
```ts
{
ok: boolean
classification?: "finding" | "unverifiable"
advisory?: string // what could not be checked and why
strictFailure?: true // preserves a legacy failure in strict mode
reason?: string // human-readable failure summary
fixHint?: string // the body injected into the synthetic user turn
strictReason?: string // legacy failure summary in strict mode
strictFixHint?: string // legacy retry guidance in strict mode
details: {
models_touched: number
dbt_root: string | null
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ We collect the following categories of events:
| `error_fingerprint` | Error pattern for grouping — SHA-256 hash of the masked error message, error class, tool name, whether recovery succeeded, plus `error_message`. `error_message` contains only words from a fixed list of common error terms taken from the first line of the error; every other word is replaced with `<x>`, so no user text, identifiers, paths, values or SQL can appear (max 200 characters). |
| `sql_fingerprint` | SQL structural shape via AST parsing — statement types, table count, function count, subquery/aggregation/window function presence, and AST node count. No table names, column names, or SQL content. |
| `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
| `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
| `validator_check` | A completion-gate validator ran after the model attempted to finish — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow-only mode and for `unverifiable` results when `ALTIMATE_VALIDATORS_STRICT=1` is unset), optional `classification` (`finding` or `unverifiable`), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
| `validator_retries_exhausted` | A session terminated with unresolved validator failures after exhausting the synthetic-retry budget — names of the failing validators (no failure body content). |
| `onboarding_started` | The first-run setup gate opened (fresh launch with no usable model). |
| `model_picker_shown` | The provider picker was displayed. `trigger` distinguishes the first run from `/connect` and from the prompt gate (`altimate_base_back` is defined in the schema but no longer emitted — it recorded backing out of the old consent dialog, which no longer exists). |
Expand Down
4 changes: 4 additions & 0 deletions packages/opencode/src/altimate/observability/tracing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ export const LLMRequestEvent = BusEvent.define(
variant: z.string().optional(),
startTime: z.number(),
system: z.string(),
uncachedTurnContext: z.string().optional(),
instructions: z.string().optional(),
tools: z.array(
z.object({
Expand Down Expand Up @@ -1363,6 +1364,9 @@ export class Trace {
},
input: {
system: snapshot("system", request.system),
...(request.uncachedTurnContext !== undefined && {
uncachedTurnContext: snapshot("uncachedTurnContext", request.uncachedTurnContext),
}),
tools: snapshot("tools", tools),
...(request.instructions !== undefined && { instructions: snapshot("instructions", request.instructions) }),
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ export * as SyncDecision from "./decide"

import type { LanguageModelV3Prompt } from "@ai-sdk/provider"
import { Fingerprint } from "./fingerprint"
import { TurnContext } from "@/session/turn-context"
import type { SessionRecord } from "./record"

export function decide(prompt: LanguageModelV3Prompt, record?: SessionRecord.Record) {
Expand All @@ -24,8 +25,10 @@ export function decide(prompt: LanguageModelV3Prompt, record?: SessionRecord.Rec
if (reply.role !== "assistant" || reply.content.some((part) => part.type === "tool-call")) break
matched++
}
const start = groups[matched]?.start ?? prompt.length
if (record?.interrupted && start === prompt.length) return { kind: "changed", start: 0 } as const
const next = groups[matched]
if (record?.interrupted && !next) return { kind: "changed", start: 0 } as const
const tail = prompt.at(-1)
const start = next?.start ?? (tail && TurnContext.isTransient(tail) ? prompt.length - 1 : prompt.length)
const missed = groups.slice(matched)
const kind = missed.some((group) => group.role !== "user") ? "catch-up" : "continue"
return { kind, start } as const
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export * as Fingerprint from "./fingerprint"

import { createHash } from "node:crypto"
import { TurnContext } from "@/session/turn-context"
import type { LanguageModelV3Message, LanguageModelV3Prompt } from "@ai-sdk/provider"

export interface Group {
Expand Down Expand Up @@ -92,7 +93,7 @@ export function groups(prompt: LanguageModelV3Prompt, prefix: readonly { readonl
const message = prompt[i]!
const start = i
i++
if (message.role === "system") continue
if (message.role === "system" || TurnContext.isTransient(message)) continue
const parts = visible(message)
if (parts.length === 0) continue
const members = [hash(parts)]
Expand All @@ -102,7 +103,7 @@ export function groups(prompt: LanguageModelV3Prompt, prefix: readonly { readonl
const checkpoint = prefix[groups.length]
if (checkpoint && hash({ role: message.role, members: members.toSorted() }) === checkpoint.hash) break
const next = prompt[i]!
if (next.role !== "user") break
if (next.role !== "user" || TurnContext.isTransient(next)) break
members.push(hash(visible(next)))
i++
}
Expand Down
2 changes: 2 additions & 0 deletions packages/opencode/src/altimate/validators/dbt-build-green.ts
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,7 @@ export const DbtBuildGreenValidator: Validator = {
: `You edited ${touchedPaths.length} model(s) but the only build artifact (${sanitizeForPrompt(artifact.path, 160)}) predates this session. Your edits have never been built.`
return {
ok: false,
classification: "finding",
reason,
fixHint: freshButNotABuild
? `That command only parses and renders SQL — it never runs it against the warehouse. Run \`dbt build\` for the models you changed and confirm it finishes without errors before declaring done.`
Expand Down Expand Up @@ -523,6 +524,7 @@ export const DbtBuildGreenValidator: Validator = {

return {
ok: false,
classification: "finding",
reason: `The build is not green: ${reasonParts.join("; ")}.`,
fixHint: hintLines.join("\n"),
details,
Expand Down
72 changes: 31 additions & 41 deletions packages/opencode/src/altimate/validators/dbt-deliverable-names.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ import {
findDbtProjectRoot,
findTaskInstructionFiles,
extractRequiredDeliverables,
collectProducedNodeNames,
collectProducedNodeInventory,
modelsModifiedSince,
modelNameFromPath,
resolveWithinRoot,
Expand Down Expand Up @@ -138,7 +138,7 @@ export const DbtDeliverableNamesValidator: Validator = {
return { ok: true, details: { skipped: "no literal contract", session_id: ctx.sessionID } }
}

const produced = await collectProducedNodeNames(dbtRoot)
const { names: produced, manifestUnreadable } = await collectProducedNodeInventory(dbtRoot)
const missingModels = contract.required.models.filter((name) => !produced.has(name))
const missingFiles: string[] = []
for (const relative of contract.required.files) {
Expand Down Expand Up @@ -176,52 +176,42 @@ export const DbtDeliverableNamesValidator: Validator = {
),
)

const reasonParts: string[] = []
if (missingModels.length > 0) {
reasonParts.push(
`the task names ${missingModels.length} deliverable(s) this project does not define: ${missingModels.join(", ")}`,
)
}
if (missingFiles.length > 0) {
reasonParts.push(`required file(s) missing: ${missingFiles.join(", ")}`)
}
// An unreadable manifest leaves alias claims unresolved. A literal file
// absence is independently verifiable and still has its own retry.
const advisory = manifestUnreadable && missingModels.length > 0
? "Could not verify the missing dbt deliverable names against aliases because manifest.json could not be read or parsed."
: undefined
const includeModelFindings = missingModels.length > 0 && (!advisory || missingFiles.length === 0)
const modelReason = missingModels.length > 0
? `the task names ${missingModels.length} deliverable(s) this project does not define: ${missingModels.join(", ")}`
: undefined
const fileReason = missingFiles.length > 0
? `required file(s) missing: ${missingFiles.join(", ")}`
: undefined
const reason = (includeModels: boolean) =>
`Deliverable-name mismatch: ${[includeModels && modelReason, fileReason].filter(Boolean).join("; ")}.`

// The workspace directory is repository/environment-controlled and can in
// principle carry adversarial text; `contract.taskFile` is built from it.
// `dbt-nothing-built` already routes its own task-file path through
// `sanitizeForPrompt` for this reason — this interpolation is the
// matching gap in this file.
// Task paths and model filenames come from the repository. Quote and
// bound them before including them in synthetic retry messages.
const safeTaskFile = sanitizeForPrompt(contract.taskFile, 160)
const hintLines: string[] = [
const safeUnrequested = unrequested.map((name) => sanitizeForPrompt(name, 80))
const fixHint = (includeModels: boolean) => [
`The task document (${safeTaskFile}) states these names literally. A model that does the right thing under a different name does not satisfy the task, and self-verification against the renamed output will not detect it.`,
]
if (missingModels.length > 0) {
hintLines.push(` • Create or rename to exactly: ${missingModels.join(", ")}`)
}
if (missingFiles.length > 0) {
hintLines.push(` • Create at exactly these paths: ${missingFiles.join(", ")}`)
}
if (unrequested.length > 0) {
// `unrequested` is derived from ACTUAL FILENAMES ON DISK, which — unlike
// `missingModels`/`missingFiles` (constrained to identifier/path-shaped
// code-span tokens by `extractRequiredDeliverables`) — can contain
// arbitrary bytes, including newlines, on a POSIX filesystem. Spliced
// verbatim into `fixHint`, a hostile or merely adversarial filename
// breaks out of the bullet it is quoted in and lands at instruction
// position in the synthetic retry turn `prompt.ts` builds from this.
const safeUnrequested = unrequested.map((n) => sanitizeForPrompt(n, 80))
hintLines.push(
includeModels && modelReason && ` • Create or rename to exactly: ${missingModels.join(", ")}`,
fileReason && ` • Create at exactly these paths: ${missingFiles.join(", ")}`,
includeModels && safeUnrequested.length > 0 &&
` • Models you created this session that the task did not name: ${safeUnrequested.join(", ")}. If one of them is a renamed version of a required deliverable, rename the file (and any \`ref()\` to it) back to the required name.`,
)
}
hintLines.push(
" • If a required deliverable is produced under an alias, set `alias` in its config so the required name is the relation name.",
)
includeModels &&
" • If a required deliverable is produced under an alias, set `alias` in its config so the required name is the relation name.",
].filter(Boolean).join("\n")

return {
ok: false,
reason: `Deliverable-name mismatch: ${reasonParts.join("; ")}.`,
fixHint: hintLines.join("\n"),
classification: advisory && missingFiles.length === 0 ? "unverifiable" : "finding",
advisory,
reason: reason(includeModelFindings),
fixHint: fixHint(includeModelFindings),
...(!includeModelFindings && { strictReason: reason(true), strictFixHint: fixHint(true) }),
details: { ...details, unrequested_models: unrequested },
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,7 @@ export const DbtDialectGuardValidator: Validator = {

return {
ok: false,
classification: "finding",
// Repository-derived model names reach a synthetic `role: "user"` turn
// through dispatch; sanitized as in `dbt-build-green`.
reason: `${findings.length} unguarded warehouse-specific construct(s) in ${byModel.size} model(s) you edited: ${Array.from(byModel.keys()).map((m) => sanitizeForPrompt(m, 80)).join(", ")}.`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -473,6 +473,7 @@ export const DbtIncrementalConfigValidator: Validator = {

return {
ok: false,
classification: "finding",
// Model names come from repository filenames and reach a synthetic
// `role: "user"` turn through dispatch, so they are sanitized exactly as
// `dbt-build-green` sanitizes node names and artifact paths.
Expand Down
Loading
Loading