Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d407af0
Enforce trace-only evidence for selected session sources
jazz127 Sep 27, 2026
117aeff
no-mistakes(review): Drop prompt-phrase assertion from trace-only ses…
jazz127 Sep 27, 2026
0fde7bf
no-mistakes(review): Exclude distill truncation annotations from trac…
jazz127 Sep 27, 2026
1263447
no-mistakes(document): Document trace-only quote anchoring rules in R…
jazz127 Sep 27, 2026
357f642
Isolate trace evidence and bind provenance across runs
jazz127 Sep 27, 2026
fc37e48
no-mistakes(review): Keep persisted gap ledger intact; fingerprint se…
jazz127 Sep 27, 2026
17968f5
no-mistakes(review): Fingerprint pi/opencode seats by selected provid…
jazz127 Sep 27, 2026
be1b2d6
no-mistakes(review): Block revoked trace priors; fingerprint probes b…
jazz127 Sep 27, 2026
e9f48fa
no-mistakes(document): Document session-source apply binding and rout…
jazz127 Sep 27, 2026
7fa2598
no-mistakes(lint): Fix typecheck error in agents test auth stub
jazz127 Sep 27, 2026
79fdf49
Add private run context and restricted child environment
jazz127 Sep 27, 2026
3c78869
no-mistakes(review): Trust root-owned system ancestor symlinks in pri…
jazz127 Sep 27, 2026
6ab64d3
no-mistakes(review): Scope probe cache and credential seats to child …
jazz127 Sep 27, 2026
bb8e146
no-mistakes(review): Canonicalize explicit state dir before containme…
jazz127 Sep 27, 2026
caf20d8
no-mistakes(document): Document system-link trust and child-env probe…
jazz127 Sep 27, 2026
6a10c6e
test: cover external session source end to end
jazz127 Sep 27, 2026
40ff4ec
no-mistakes(review): Collapse copied external sessions by content; ti…
jazz127 Sep 27, 2026
db033ee
no-mistakes(document): Document selected-source revision and copy ded…
jazz127 Sep 27, 2026
fc6e8fe
Merge pull request #8 from jazz127/fm/backpass-s12-e2e
jazz127 Sep 27, 2026
52cbfa0
Merge pull request #6 from jazz127/fm/backpass-s9-isolation
jazz127 Sep 27, 2026
9e25f18
Merge pull request #5 from jazz127/fm/backpass-s8-trace-only
jazz127 Sep 27, 2026
940f03c
Merge pull request #9 from jazz127/fm/backpass-s10-run-context
jazz127 Sep 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 48 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,26 @@ State lives in `$XDG_CONFIG_HOME/backpass/user/` (default
`.backpass/`. User-scope evidence, ledgers, proposals, and apply surfaces stay in
that one directory.

Use `--state-dir <dir>` with `scan`, `analyze`, `propose`, `status`, and `apply` to
keep a run in a dedicated private state directory. Run `scan` first, then pass the
same directory and scope to each later command. The scan pins the source snapshot,
selected corpus, and an inventory of the exact memory and skill file bytes and
resolved pointer targets. `analyze` and `propose` refuse changed inputs; `apply`
checks the saved inventory again. A new `scan` starts a new run in that directory.
The directory is owner-only (0700), and Backpass refuses links in the path (other than
root-owned system links above your own directories, such as macOS `/tmp`), permissive
permissions, or reuse for a different scope or repository. The default state paths
continue to work as before.

`--child-env restricted` starts probes and model children with a minimal environment
containing runtime basics such as `PATH`, `HOME`, and temporary-directory settings.
Inherited provider credentials, endpoints, MCP settings, and plugin roots are omitted;
an embedding runner can add explicitly approved variables through the child-environment
hook. This mode also rejects memory pointers to files outside the configured memory
surface. The default child environment remains the native inherited environment.
`--prompt-retries <n>` and `--timeout <seconds>` override their configured values for
one command only; neither flag changes a config file or harness default.

Harness load paths, verified for v1:

- **Claude Code** loads `CLAUDE.md` from `CLAUDE_CONFIG_DIR` (default `~/.claude`)
Expand Down Expand Up @@ -260,14 +280,28 @@ the same tiers, sample and cap - see [Your other machines](#your-other-machines)

To select a normalized `external-session-source/v1` snapshot instead, pass
`--session-source <snapshot-directory-or-manifest>` to `scan`, `analyze`, `propose`,
`status`, or the default run. Selection is exclusive: no local harness stores or SSH
`status`, `apply`, or the default run. Selection is exclusive: no local harness stores or SSH
hosts are scanned. `--session-source-mode exclusive` makes that choice explicit; other
modes are unsupported. A malformed or unsafe selected snapshot fails the run by name.
The selected path is resolved once; everything inside the snapshot must be owned by the
current user and private, with regular files without hard links and no symlinks. The source's recorded
cwd, git root, and remotes go through the same association tiers. Its origin harness
remains visible in the corpus mix, while its source identity stays stable when the
snapshot directory moves.
Only the latest revision of each session is collected, and sessions whose screened events
match (ignoring event ids and source references) are kept once, so a copy republished under
another session id counts as a single session toward evidence floors.
Analysis of selected-source sessions uses only the screened trace. Its prompt offers
abstention when retained events do not support a claim, and evidence quotes must match
one retained event; stored citations include the approved revision, event, span, and
opaque source references. Native transcript analysis retains its existing behavior.
A proposal built from a selected source records that source's snapshot digest, and
`apply` refuses it unless the same unchanged snapshot is passed again with
`--session-source`.
For quote matching, runs of whitespace fold to one space and leading/trailing
whitespace is ignored; spans still point into the rendered event field. A quote that
overlaps text the distiller added (truncation markers, output-size notes, redaction
placeholders) is rejected, and a reported raw-transcript read fails the analysis.

Collection is incremental. Codex alone can hold 10,000+ rollouts, so verdicts are cached in
`.backpass/scan-cache.json` by path, mtime and size - re-scans cost only the new files.
Expand Down Expand Up @@ -338,9 +372,10 @@ coining a paraphrase of it.
items are discarded - the single most important defence against a model confabulating
influence - and so are quotes that do not actually appear in the distilled trace they claim
to come from, compared whitespace-folded so the trace's line wrapping never rejects a real
quote. A paraphrase is a claim without evidence. The check is skipped only when the analysis
reports `usedRawTranscript`, because then the quote may legitimately come from text the
distiller truncated or elided. When a run discards quotes this way it says so on stderr:
quote. A paraphrase is a claim without evidence. For native transcripts, the check is
skipped only when the analysis reports `usedRawTranscript`, because then the quote may
legitimately come from text the distiller truncated or elided; selected-source sessions
never skip it (see [Collect samples](#1-collect-samples---which-sessions-belong-to-this-repo)). When a run discards quotes this way it says so on stderr:
a model that paraphrases instead of copying produces fewer findings, not cleaner ones.
Negative evidence is weighted highest, but its class determines what it supports:
non-compliance supports reinforcement, while only harm supports removal.
Expand All @@ -353,7 +388,9 @@ draw repeated non-compliance, synthesis is steered to restructure the paragraph
items instead of adding a cosmetic label.

Results are cached per transcript, keyed to the transcript's content, the effective
memory-surface hash, and the analysis-index version. The surface hash covers the memory-file
memory-surface hash, and the analysis-index version, plus the distiller version, a
selected source's snapshot, revision, and screening policy, and the analysis route (agent,
model, effort, and credential-seat fingerprint). The surface hash covers the memory-file
set plus every project skill's name and description. Edit a memory file or skill description
and the evidence correctly re-computes; edit only a skill body and the cache remains valid
because bodies are inspected only for failed-trigger confirmation. A repo without skills
Expand Down Expand Up @@ -394,7 +431,7 @@ majority-excluded cluster, including a pure-orchestration cluster, remains clear
as a report-only diagnostic rather than becoming an instruction in the project's memory
file; an uncorroborated pure-orchestration singleton stays hidden.

Only evidence with the current transcript, memory-surface, and analysis-index cache key,
Only evidence whose full cache key (above) is current,
stamped with one of the two interaction categories, and belonging to this run's selected
sample is folded into a proposal. A transcript that fell outside the time window or
`maxTranscripts` cap, disappeared, or still has legacy evidence can leave an evidence file
Expand Down Expand Up @@ -653,9 +690,11 @@ because its adapter accepts sessions while logged out). Session creation may wai
minutes for a cold-starting adapter; the remaining probe operations retain their shorter
10-20 second limits. A potentially transient busy-harness miss retries once and is not cached;
durable verdicts are cached in
`.backpass/agent-probe-cache.json` for 12h (30min for negatives). Pi and OpenCode entries
are re-probed when their credential or auth-file state changes; `--force` re-probes every
entry. The probe is a filter, not a promise: if the chosen harness answers `AUTH_REQUIRED`,
`.backpass/agent-probe-cache.json` for 12h (30min for negatives). An entry is re-probed
when the session source, memory surface, effort, or `--child-env` mode changes, or when its harness's
credential seat (credential home, account identity, or the selected provider's API key)
changes; `--force` re-probes every entry. The probe is a filter, not a promise: if the
chosen harness answers `AUTH_REQUIRED`,
rejects the model, or returns a clean exit with no output at all (a provider account out
of quota or credits, often swallowed before it reaches stderr) mid-run, backpass falls
through to the next candidate and says so. The one blank exit that never falls through is
Expand Down
49 changes: 37 additions & 12 deletions src/agents.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ import { UserError, color, info, warn } from "./logger.js";
import { DEFAULT_EFFORT, LEGACY_DEFAULT_AGENTS } from "./config.js";
import { AcpxError, SESSION_CREATE_TIMEOUT_MS, acpxVersion, classifyAcpxFailure, probeSession } from "./acpx.js";
import { ambiguousModelDetail, providerAuthState, rankCollidingIds, readProviderAuthTypes } from "./provider-auth.js";
import { runCapture } from "./subprocess.js";
import { currentChildEnvironmentMode, runCapture } from "./subprocess.js";
import { sha256 } from "./state.js";

/**
* Ordered agent selection (the ordered-defaults design, section 8).
Expand All @@ -28,7 +29,8 @@ import { runCapture } from "./subprocess.js";
*
* Verdicts are cached in `.backpass/agent-probe-cache.json` (12h for ok, 30min for
* negatives) and memoized for the run. An acpx version change invalidates every entry;
* Pi and OpenCode entries are also keyed to credential environment and auth-file state.
* Probe keys include the source and memory context, effort, and a credential-seat
* fingerprint (`providerAuthState`) of the model the candidate resolves to.
*
* A busy harness (another backpass run, a wedged ACP session) looks like a probe
* timeout, a bare `exit 1`, or an empty advertised-model list. Those retry once with
Expand All @@ -45,7 +47,6 @@ const PROBE_RETRY_BACKOFF_MS = 1_000;

/** Adapters whose model list is open-ended: any id is forwarded, none can be verified. */
const TRUSTING_MODEL_AGENTS = new Set(["claude"]);
const PROVIDER_AUTH_SENSITIVE_AGENTS = new Set(["pi", "opencode"]);

export const VERDICT_LABELS = {
ok: "ok",
Expand Down Expand Up @@ -340,8 +341,14 @@ export class AgentResolver {
}

async verdictFor(candidate) {
const key = candidateKey(candidate);
if (this.memo.has(key)) return this.memo.get(key);
const key = this.probeKey(candidate);
const memo = this.memo.get(key);
if (
memo &&
memo.authState === this.credentialFingerprint(candidate, memo.resolvedModel) &&
memo.scope === this.probeScope()
)
return memo;
if (this.inflight.has(key)) return this.inflight.get(key);
const pending = this.probeAndRecord(candidate, key).finally(() => this.inflight.delete(key));
this.inflight.set(key, pending);
Expand All @@ -351,11 +358,10 @@ export class AgentResolver {
async probeAndRecord(candidate, key) {
const cache = await this.loadCache();
const cached = cache.entries[key];
const authState = PROVIDER_AUTH_SENSITIVE_AGENTS.has(candidate.agent)
? this.providerAuthState(candidate.agent)
: null;
const authStateMatches = authState === null || cached?.authState === authState;
if (!this.bypassCache && authStateMatches && isProbeEntryFresh(cached, { now: this.now() })) {
const authStateMatches = cached?.authState === this.credentialFingerprint(candidate, cached?.resolvedModel);
const scopeMatches =
cached?.scope === undefined ? this.probeScope() === "native:::" : cached?.scope === this.probeScope();
if (!this.bypassCache && authStateMatches && scopeMatches && isProbeEntryFresh(cached, { now: this.now() })) {
this.memo.set(key, { ...cached, cached: true });
return this.memo.get(key);
}
Expand All @@ -373,12 +379,14 @@ export class AgentResolver {
sessionName: `backpass-probe-${process.pid}-${this.probeCount}`,
});
}
const authState = this.credentialFingerprint(candidate, result.resolvedModel);
const entry = {
verdict: result.verdict,
detail: result.detail || "",
resolvedModel: result.resolvedModel || null,
checkedAt: new Date(this.now()).toISOString(),
...(authState === null ? {} : { authState }),
scope: this.probeScope(),
...(result.tieBreak ? { tieBreak: result.tieBreak } : {}),
};
this.memo.set(key, entry);
Expand All @@ -391,6 +399,22 @@ export class AgentResolver {
return entry;
}

credentialFingerprint({ agent, model }, resolvedModel = null) {
return this.providerAuthState(agent, { model: resolvedModel || model });
}

probeScope() {
const scope = `${this.config.sourceFingerprint || "native"}:${this.config.memoryFingerprint || ""}:${this.config.analysis?.effort || ""}:${this.config.synthesis?.effort || ""}`;
const mode = currentChildEnvironmentMode();
return mode === "native" ? scope : `${scope}:${mode}`;
}

probeKey(candidate) {
const base = candidateKey(candidate);
if (!this.config.enforceEvidenceRoute) return base;
return `${base}:${sha256(`${this.probeScope()}:${this.credentialFingerprint(candidate)}`).slice(0, 24)}`;
}

/** The candidates for a role, in order, as `{ agent, model }`. */
ladder(role) {
return flattenLadder(this.config.ladders[role]);
Expand Down Expand Up @@ -485,16 +509,17 @@ export class AgentResolver {
*/
async demote(role, pick, verdict, detail = "", stderr = "") {
if (pick.pinned) return false;
const key = candidateKey({ agent: pick.agent, model: pick.ladderModel });
const key = this.probeKey({ agent: pick.agent, model: pick.ladderModel });
if (this.memo.get(key)?.verdict === "ok") {
// First worker to see the failure records it; the rest just re-resolve.
const authState = PROVIDER_AUTH_SENSITIVE_AGENTS.has(pick.agent) ? this.providerAuthState(pick.agent) : null;
const authState = this.credentialFingerprint({ agent: pick.agent, model: pick.ladderModel });
const entry = {
verdict,
detail,
resolvedModel: null,
checkedAt: new Date(this.now()).toISOString(),
...(authState === null ? {} : { authState }),
scope: this.probeScope(),
...(stderr ? { stderr } : {}),
};
this.memo.set(key, entry);
Expand Down
Loading
Loading