diff --git a/CHANGELOG.md b/CHANGELOG.md index 895f5e0..0024d15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,61 @@ All notable changes to the Toolpath workspace are documented here. +## Conversation items IR + compaction provenance — 2026-06-22 + +`ConversationView` now exposes a single ordered `items` stream (the new +`Item` enum) in place of the separate `turns`/`events` lists, giving every +provider one timeline to populate and every consumer one timeline to walk. +On top of that, compaction is now first-class provenance: when an agent +summarizes and drops earlier context, we record what was kept and why +instead of silently losing the boundary. + +- **`toolpath-convo`** (BREAKING): `ConversationView.turns`/`events` are + unified into `ConversationView.items: Vec`, with `turns()`, + `events()`, and `compactions()` accessors for walking the stream. New + public types `Compaction` and `CompactionTrigger` model a + context-compaction boundary — `kept` is the set of prior turn ids that + survived, and `trigger` records what caused it. `derive_path` resolves + step-id collisions as it emits steps — a byte-identical re-emission is + dropped, a same-id-but-different step is re-IDed to `#` — so it + is infallible (returns `Path`, not `Result`) and the result is always + collision-free. The per-provider `derive::derive_path` / `derive_project` + (and pi's `derive_graph`) wrappers are likewise infallible now; only the + disk-reading entry points (e.g. pi's `derive_project`) still return + `Result`. +- **New step type**: `conversation.compact` records a compaction event as + a Step in the derived `Path`. +- **Per-provider compaction population**: `toolpath-claude`, + `toolpath-codex`, `toolpath-opencode`, and `toolpath-pi` emit + `Compaction` items from their on-disk compaction signals. + `toolpath-gemini` and `toolpath-cursor` participate in the items IR but + persist no compaction: Gemini records none, and Cursor's `/summarize` + writes only a boundary marker (the summary and kept set live server-side + and are unrecoverable from local data), so the marker is skipped on read. +- **`agent-coding-session` kind → v1.2.0**: the kind constant is now + `https://toolpath.net/kinds/agent-coding-session/v1.2.0`, extending + v1.1.0 (token usage) with the `conversation.compact` step type. The + v1.2.0 schema is bundled in both `toolpath` and `path-cli` alongside the + retained v1.0.0/v1.1.0 schemas; producers (the shared + `toolpath_convo::derive_path` and every provider built on it) emit the + new URI. +- **Token-usage idempotency fixes** (surfaced by the cross-harness + round-trip matrix once compaction fixtures were added): message totals + are canonicalized per `group_id` across the whole turn sequence rather + than per consecutive run, so a group interrupted by an intervening turn + no longer double-counts on re-read (`toolpath-claude`); consecutive + same-id Gemini lines (one split message) now share a `group_id` so their + repeated `tokens` snapshot is counted once (`toolpath-gemini`); and + Codex's cumulative `token_count` advances by a group's total once, on the + group's last turn (`toolpath-codex`). + +Crates bumped: `toolpath` 0.8.0, `toolpath-convo` 0.12.0, +`toolpath-claude` 0.13.0, `toolpath-codex` 0.7.0, `toolpath-gemini` +0.7.0, `toolpath-opencode` 0.6.0, `toolpath-pi` 0.7.0, +`toolpath-cursor` 0.3.0, `path-cli` 0.15.0, `toolpath-cli` 0.15.0. +Unchanged: `toolpath-git`, `toolpath-github`, `toolpath-dot`, +`toolpath-md`, `pathbase-client`. + ## Token usage: once per message, with per-step attribution + kind v1.1.0 — 2026-06-17 Fixes token over-counting in derived documents (~3× output-token diff --git a/CLAUDE.md b/CLAUDE.md index 01c70e9..558ebae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -242,9 +242,10 @@ Build the site after changes: `cd site && pnpm run build` (should produce 11 pag - `toolpath-gemini` treats main file + sibling sub-agent UUID dir as one conversation. Sub-agent files are folded into `DelegatedWork` with populated `turns` (unlike `toolpath-claude`, whose sub-agent turns live in separate session files and stay empty). See `docs/agents/formats/gemini.md` for the full format reference. - Provider-specific extras convention: `Turn.extra` and `WatcherEvent::Progress.data` use provider-namespaced keys (e.g. `extra["claude"]`, `extra["gemini"]`). `toolpath-claude` populates `Turn.extra["claude"]` from `ConversationEntry.extra`; `toolpath-gemini` populates `Turn.extra["gemini"]` with the full `tokens` struct, per-thought metadata, and tool-call status. This lets trait-only consumers access provider metadata without importing provider types. - Shared derivation: `toolpath-convo` provides a provider-agnostic `ConversationView → Path` mapping via `toolpath_convo::derive_path`. New conversation providers should build on it rather than re-implementing the mapping. -- Path kinds: `toolpath::v1::PathMeta.kind` is an optional URI naming a hosted kind spec; URIs are immutable and semver-versioned. The only one defined so far is `https://toolpath.net/kinds/agent-coding-session/v1.1.0` (constant `toolpath::v1::PATH_KIND_AGENT_CODING_SESSION`; `…_V1_0_0` names the superseded URI); every conversation → `Path` derivation sets it via the shared `toolpath_convo::derive_path` or each provider crate's own. Carried through the JSONL form via `PathOpen.meta` and `PathMeta` patch lines. Spec sources live in `site/kinds///{index.md,schema.json}` (schema.json is a symlink into `crates/path-cli/kinds/`, which `path p validate` bundles — both versions) and publish under `https://toolpath.net/kinds/`; the registry index is `site/kinds/index.md`. RFC: "Document Kind". JSON Schema: `$defs/pathMeta`. +- Path kinds: `toolpath::v1::PathMeta.kind` is an optional URI naming a hosted kind spec; URIs are immutable and semver-versioned. The only kind defined so far is `agent-coding-session`, currently at `https://toolpath.net/kinds/agent-coding-session/v1.2.0` (constant `toolpath::v1::PATH_KIND_AGENT_CODING_SESSION`; `…_V1_0_0` names the superseded v1.0.0 URI); v1.2.0 adds the `conversation.compact` step type for context-compaction boundaries on top of v1.1.0's message-level token accounting, and the earlier v1.1.0 and v1.0.0 URIs stay registered (their schemas kept in `KIND_SCHEMAS`) and documented for backward compatibility. Every conversation → `Path` derivation sets it via the shared `toolpath_convo::derive_path` or each provider crate's own. Carried through the JSONL form via `PathOpen.meta` and `PathMeta` patch lines. Spec sources live in `site/kinds///{index.md,schema.json}` (schema.json is a symlink into `crates/path-cli/kinds/`, which `path p validate` bundles — all versions) and publish under `https://toolpath.net/kinds/`; the registry index is `site/kinds/index.md`. RFC: "Document Kind". JSON Schema: `$defs/pathMeta`. - Token accounting (kind v1.1.0): two keys on `conversation.append`/`Turn`, both optional. `token_usage` = "the total for a message" (on the group's final step; `Σ` over a path = session total). `attributed_token_usage` = "this step's own attributed spend", populated only where the source genuinely reports per-step spend (its own key, so the sum is unaffected; remainder = group total − Σ attributed, computed not stored). One provider message can span several steps (Claude writes one JSONL line per content block); `Turn.group_id` groups them. `toolpath-claude` fills `group_id` from `message.id` and takes the **field-wise-max** group total (line order not trusted). Claude's per-line `usage` is a cumulative *streaming snapshot* (Anthropic streaming API: `message_start` seeds output near 0, `message_delta` is cumulative), NOT a per-block cost — so Claude emits no `attributed_token_usage`; the projector re-expands the total onto every line. `toolpath-codex` differences the cumulative `total_token_usage` (dedup-safe: never sum `last_token_usage` — Codex re-emits it stale; openai/codex #14489), attributes each per-call delta to the step it follows, and derives the round total from those attributions. pi/opencode decode all-zero wire counters as `None`. Never stamp a cumulative counter, a repeated message total, or zero-filled placeholders onto a step; never derive attribution from Claude's streaming snapshots. - Token usage `breakdowns` (kind v1.1.0, additive): an optional third key on `TokenUsage` — a decomposition of a top-level class into named sub-classes, keyed by class (e.g. `"output"`), inner map sub-class → tokens (e.g. `breakdowns["output"]["reasoning"] = 243`). INFORMATIONAL ONLY: **never summed into any total** (the parent class already counts those tokens, so the session-total guarantee is untouched); invariant `Σ(inner) ≤ parent`; omitted when empty; rides both `token_usage` and `attributed_token_usage`. Per-provider reality: **Gemini** reports `thoughts` (reasoning) as an additive sibling that the derivation used to **drop** (under-counting output) — it's now folded into `output_tokens` *and* recorded as `breakdowns["output"]["reasoning"]`, with the projector un-folding it on the reverse path for a lossless round-trip (`Some(0)` preserved as a real Gemini-3 zero-reasoning signal). **OpenCode** folds `reasoning` into output and records the same breakdown. **Codex** differences `reasoning_output_tokens` (⊆ output, cumulative) into `breakdowns["output"]["reasoning"]` on both per-step `attributed_token_usage` and per-round `token_usage`. **Claude** records no breakdown (its JSONL `usage` doesn't itemize thinking tokens). +- Compaction boundary (kind v1.2.0): the `conversation.compact` step type — a context-compaction boundary recorded as its own step between the turns it separates (turns after the boundary parent on it, so the `head`-ancestry walk crosses it in order). `structural` fields are all optional but `type`: `trigger` (`auto`|`manual`), `summary`, `pre_tokens`, and `kept` (ids of prior turns surviving verbatim into the post-compaction window; non-contiguous allowed; empty = wholesale). It is not a turn (no `text`/`role`/`tool_uses`); `step.actor` is `tool:`. Populated by each provider's `Item::Compaction` derivation and projected back to harness markers. - Pi provider: `toolpath-pi` reads Pi session JSONL from `~/.pi/agent/sessions/`. Sessions use a tree (id/parentId) in a single file, and may link to a parent file via `parentSession` in the header. The tree is preserved as a DAG in the derived `Path`. - Codex provider: `toolpath-codex` reads Codex CLI rollout files from `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`. Sessions are date-bucketed (not project-keyed). File-change fidelity is excellent — Codex's `patch_apply_end` events carry either the unified diff (for updates) or the full file content (for adds), so the derived `Path` gets a real `raw` perspective on every file artifact. See `docs/agents/formats/codex.md` for the full format reference. - opencode provider: `toolpath-opencode` reads a SQLite database at `~/.local/share/opencode/opencode.db` (opened read-only). Each session's messages and 12 typed part variants (text, reasoning, tool, step-start/-finish, snapshot, patch, file, agent, subtask, retry, compaction) land as one step per message with tool invocations attached. File diffs come from a sibling bare git repo at `snapshot//[]/` via `git2` tree↔tree diffs — opencode respects the user's `.gitignore`, so changes under gitignored paths fall back to tool-input-derived structural changes with no `raw` perspective. Project id is the SHA of the repo's first root commit. See `docs/agents/formats/opencode.md` for the full format reference. diff --git a/Cargo.lock b/Cargo.lock index 77e8650..0e80f37 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2308,7 +2308,7 @@ dependencies = [ [[package]] name = "path-cli" -version = "0.14.0" +version = "0.15.0" dependencies = [ "anyhow", "assert_cmd", @@ -3885,7 +3885,7 @@ dependencies = [ [[package]] name = "toolpath" -version = "0.7.0" +version = "0.8.0" dependencies = [ "serde", "serde_json", @@ -3893,7 +3893,7 @@ dependencies = [ [[package]] name = "toolpath-claude" -version = "0.12.0" +version = "0.13.0" dependencies = [ "anyhow", "chrono", @@ -3910,7 +3910,7 @@ dependencies = [ [[package]] name = "toolpath-codex" -version = "0.6.0" +version = "0.7.0" dependencies = [ "anyhow", "chrono", @@ -3924,7 +3924,7 @@ dependencies = [ [[package]] name = "toolpath-convo" -version = "0.11.0" +version = "0.12.0" dependencies = [ "chrono", "jsonschema", @@ -3937,7 +3937,7 @@ dependencies = [ [[package]] name = "toolpath-cursor" -version = "0.2.0" +version = "0.3.0" dependencies = [ "anyhow", "chrono", @@ -3960,7 +3960,7 @@ dependencies = [ [[package]] name = "toolpath-gemini" -version = "0.6.0" +version = "0.7.0" dependencies = [ "anyhow", "chrono", @@ -4006,7 +4006,7 @@ dependencies = [ [[package]] name = "toolpath-opencode" -version = "0.5.0" +version = "0.6.0" dependencies = [ "anyhow", "chrono", @@ -4024,7 +4024,7 @@ dependencies = [ [[package]] name = "toolpath-pi" -version = "0.6.0" +version = "0.7.0" dependencies = [ "anyhow", "chrono", diff --git a/Cargo.toml b/Cargo.toml index b03c2d2..2ac32cf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -23,19 +23,19 @@ edition = "2024" license = "Apache-2.0" [workspace.dependencies] -toolpath = { version = "0.7.0", path = "crates/toolpath" } -toolpath-convo = { version = "0.11.0", path = "crates/toolpath-convo" } +toolpath = { version = "0.8.0", path = "crates/toolpath" } +toolpath-convo = { version = "0.12.0", path = "crates/toolpath-convo" } toolpath-git = { version = "0.6.0", path = "crates/toolpath-git" } -toolpath-claude = { version = "0.12.0", path = "crates/toolpath-claude", default-features = false } -toolpath-gemini = { version = "0.6.0", path = "crates/toolpath-gemini", default-features = false } -toolpath-codex = { version = "0.6.0", path = "crates/toolpath-codex" } -toolpath-opencode = { version = "0.5.0", path = "crates/toolpath-opencode" } -toolpath-cursor = { version = "0.2.0", path = "crates/toolpath-cursor" } +toolpath-claude = { version = "0.13.0", path = "crates/toolpath-claude", default-features = false } +toolpath-gemini = { version = "0.7.0", path = "crates/toolpath-gemini", default-features = false } +toolpath-codex = { version = "0.7.0", path = "crates/toolpath-codex" } +toolpath-opencode = { version = "0.6.0", path = "crates/toolpath-opencode" } +toolpath-cursor = { version = "0.3.0", path = "crates/toolpath-cursor" } toolpath-github = { version = "0.6.0", path = "crates/toolpath-github" } toolpath-dot = { version = "0.5.0", path = "crates/toolpath-dot" } toolpath-md = { version = "0.7.0", path = "crates/toolpath-md" } -toolpath-pi = { version = "0.6.0", path = "crates/toolpath-pi" } -path-cli = { version = "0.14.0", path = "crates/path-cli" } +toolpath-pi = { version = "0.7.0", path = "crates/toolpath-pi" } +path-cli = { version = "0.15.0", path = "crates/path-cli" } pathbase-client = { version = "0.2.0", path = "crates/pathbase-client" } reqwest = { version = "0.13", default-features = false, features = ["blocking", "json", "rustls"] } diff --git a/RFC.md b/RFC.md index fa81952..6886c18 100644 --- a/RFC.md +++ b/RFC.md @@ -296,10 +296,14 @@ unrecognized URIs should be treated as a generic path. Kind URIs are immutable, semver-versioned, and revisions ship at a new version URI. Defined kinds are listed at . The only one defined -so far is `https://toolpath.net/kinds/agent-coding-session/v1.0.0` — a path -recording an AI coding conversation, where each conversational-turn step -carries a `"conversation.append"` structural change with the turn's role, -text, and so on. See the linked spec for the full contract. +so far is `agent-coding-session`, currently at +`https://toolpath.net/kinds/agent-coding-session/v1.2.0` — a path recording an +AI coding conversation, where each conversational-turn step carries a +`"conversation.append"` structural change with the turn's role, text, and so +on, and context-compaction boundaries carry a `"conversation.compact"` step. +The earlier `v1.1.0` URI (message-level token accounting, without +`conversation.compact`) and `v1.0.0` URI both remain valid and documented. See +the linked spec for the full contract. #### Actor Definitions diff --git a/crates/path-cli/Cargo.toml b/crates/path-cli/Cargo.toml index 01eea17..7a6fa87 100644 --- a/crates/path-cli/Cargo.toml +++ b/crates/path-cli/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "path-cli" -version = "0.14.0" +version = "0.15.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/path-cli/kinds/agent-coding-session/v1.2.0/schema.json b/crates/path-cli/kinds/agent-coding-session/v1.2.0/schema.json new file mode 100644 index 0000000..99e91d8 --- /dev/null +++ b/crates/path-cli/kinds/agent-coding-session/v1.2.0/schema.json @@ -0,0 +1,281 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://toolpath.net/kinds/agent-coding-session/v1.2.0/schema.json", + "title": "Toolpath kind: agent-coding-session v1.2.0", + "description": "Additive constraints on a Toolpath `Path` whose `meta.kind` is the agent-coding-session v1.2.0 URI. Apply alongside the base Toolpath schema; the path is valid when both pass. These constraints are by structural `type`, not by artifact key — a `change` entry is checked only when its `structural.type` matches one this kind defines (`conversation.append`, `file.write`, `conversation.event`, `conversation.compact`). Everything is additive: unmentioned properties are allowed, so producer-specific extras never invalidate a path. New in v1.1.0: `group_id` on the turn payload, and the group-level accounting rule — within a `group_id` group, the last step (document order) carries the group's total `token_usage` verbatim from the source and the other steps carry none; values are per-group amounts (never cumulative session counters); summing `token_usage` over a path's steps therefore yields the session totals. The once-per-group rule is normative prose (JSON Schema cannot express it); producers must enforce it. New in v1.2.0: the `conversation.compact` step type, a context-compaction boundary recorded as its own step between the turns it separates. The human-readable contract is at https://toolpath.net/kinds/agent-coding-session/v1.2.0/.", + "type": "object", + + "$defs": { + "tokenUsage": { + "type": "object", + "description": "Token accounting for one source group (a message for Claude, a round for Codex). `token_usage` always means the total for a group, verbatim from the source — never a cumulative session counter, never a per-step share. Within a `group_id` group the total sits on the group's last step (document order) and the other steps carry none; a step without a `group_id` is a group of one, so its value is that group's total. Summing over a path's steps therefore yields the session totals. `input_tokens`/`output_tokens` are always emitted (possibly null); cache counters appear only when the source records them.", + "properties": { + "input_tokens": { "type": ["integer", "null"] }, + "output_tokens": { "type": ["integer", "null"] }, + "cache_read_tokens": { "type": "integer" }, + "cache_write_tokens": { "type": "integer" }, + "breakdowns": { + "type": "object", + "description": "Optional decomposition of a top-level class into named sub-classes, keyed by the class being broken down (e.g. \"output\"); each value maps sub-class → tokens. INFORMATIONAL: never summed into the total — the parent class already counts these. Invariant: Σ(inner) ≤ the parent class's value.", + "additionalProperties": { "type": "object", "additionalProperties": { "type": "integer" } } + } + }, + "required": ["input_tokens", "output_tokens"] + }, + + "attributedTokenUsage": { + "type": "object", + "description": "This step's own attributed spend, when the source provides step-aligned data — distinct from `token_usage` (the group total). Optional and orthogonal: it rides its own key so summing `token_usage` over steps is unaffected. Within a `group_id` group, the sum of `attributed_token_usage` over its steps is the group's attributed spend; the unattributed remainder (`group token_usage − Σ attributed`) is computed by consumers, never recorded. Same field shape as `tokenUsage`. A producer populates it only when the source genuinely reports per-step spend — among current producers, Codex does (each step is a separate API call with a reported per-call delta); Claude does not (its per-block usage is a cumulative streaming snapshot, not a per-block cost), so Claude-derived steps carry the group total only.", + "properties": { + "input_tokens": { "type": ["integer", "null"] }, + "output_tokens": { "type": ["integer", "null"] }, + "cache_read_tokens": { "type": "integer" }, + "cache_write_tokens": { "type": "integer" } + } + }, + + "toolResult": { + "type": "object", + "properties": { + "content": { "type": "string" }, + "is_error": { "type": "boolean" } + }, + "required": ["content", "is_error"] + }, + + "toolUse": { + "type": "object", + "description": "One tool invocation. `input` is producer-specific JSON (left unconstrained). `category` is Toolpath's classification, or null when the tool is unrecognized. `result` is present only when the result was available in the same turn.", + "properties": { + "id": { "type": "string" }, + "name": { "type": "string" }, + "input": true, + "category": { + "type": ["string", "null"], + "enum": [ + "file_read", + "file_write", + "file_search", + "shell", + "network", + "delegation", + null + ] + }, + "result": { "$ref": "#/$defs/toolResult" } + }, + "required": ["id", "name", "input", "category"] + }, + + "environment": { + "type": "object", + "description": "Working environment captured at the turn. All fields optional.", + "properties": { + "working_dir": { "type": "string" }, + "vcs_branch": { "type": "string" }, + "vcs_revision": { "type": "string" } + } + }, + + "delegation": { + "type": "object", + "description": "Sub-agent work spawned from a turn. `turns` carries the sub-agent's own turns when the producer inlines them.", + "properties": { + "agent_id": { "type": "string" }, + "prompt": { "type": "string" }, + "turns": { "type": "array" }, + "result": { "type": "string" } + }, + "required": ["agent_id", "prompt"] + }, + + "conversationAppend": { + "type": "object", + "description": "The turn payload: the `structural` object of the one `change` entry whose `type` is `conversation.append`. `role` and `text` are always present (text may be empty); everything else appears only when the turn carries it. `group_id` is the provider's identifier for the source accounting unit this turn was derived from — a message for Claude (`message.id`), a round for Codex (`turn_id`). A grouping key, not a step identifier: steps sharing a `group_id` came from one accounting unit (Claude Code writes one JSONL line per content block; a Codex round emits several turns).", + "properties": { + "type": { "const": "conversation.append" }, + "role": { "type": "string" }, + "text": { "type": "string" }, + "thinking": { "type": "string" }, + "group_id": { "type": "string" }, + "tool_uses": { + "type": "array", + "items": { "$ref": "#/$defs/toolUse" } + }, + "token_usage": { "$ref": "#/$defs/tokenUsage" }, + "attributed_token_usage": { "$ref": "#/$defs/attributedTokenUsage" }, + "stop_reason": { "type": "string" }, + "delegations": { + "type": "array", + "items": { "$ref": "#/$defs/delegation" } + }, + "environment": { "$ref": "#/$defs/environment" } + }, + "required": ["type", "role", "text"] + }, + + "fileWrite": { + "type": "object", + "description": "The `structural` object of a sibling `file.write` change keyed by file path. The unified diff (when present) lives on the artifact change's `raw`, not here. `tool_id`/`tool` link the mutation to the `ToolInvocation` that caused it when attributable.", + "properties": { + "type": { "const": "file.write" }, + "tool_id": { "type": "string" }, + "tool": { "type": "string" }, + "operation": { "type": "string" }, + "before": { "type": "string" }, + "after": { "type": "string" }, + "rename_to": { "type": "string" } + }, + "required": ["type"] + }, + + "conversationEvent": { + "type": "object", + "description": "The `structural` object of a `conversation.event` change — a non-turn entry (attachment, preamble line, snapshot, …) preserved for round-trip fidelity. `entry_type` names the source entry kind; the producer's flattened event data rides alongside.", + "properties": { + "type": { "const": "conversation.event" }, + "entry_type": { "type": "string" }, + "event_source_id": { "type": "string" } + }, + "required": ["type", "entry_type"] + }, + + "conversationCompact": { + "type": "object", + "description": "The `structural` object of a `conversation.compact` change — a context-compaction boundary. Sits as its own step between the turns it separates; only `type` is required and every other field appears only when the source records it.", + "properties": { + "type": { "const": "conversation.compact" }, + "trigger": { "type": "string", "enum": ["auto", "manual"] }, + "summary": { "type": "string" }, + "pre_tokens": { "type": "number" }, + "kept": { + "type": "array", + "description": "Ids of the prior turns that survive verbatim into the post-compaction context window. May be non-contiguous. Empty = wholesale (the summary replaced everything).", + "items": { "type": "string" } + } + }, + "required": ["type"] + }, + + "artifactChange": { + "type": "object", + "description": "An artifact change, constrained only when its `structural.type` is one this kind defines.", + "allOf": [ + { + "if": { + "type": "object", + "properties": { + "structural": { + "type": "object", + "properties": { "type": { "const": "conversation.append" } }, + "required": ["type"] + } + }, + "required": ["structural"] + }, + "then": { + "properties": { + "structural": { "$ref": "#/$defs/conversationAppend" } + } + } + }, + { + "if": { + "type": "object", + "properties": { + "structural": { + "type": "object", + "properties": { "type": { "const": "file.write" } }, + "required": ["type"] + } + }, + "required": ["structural"] + }, + "then": { + "properties": { + "structural": { "$ref": "#/$defs/fileWrite" } + } + } + }, + { + "if": { + "type": "object", + "properties": { + "structural": { + "type": "object", + "properties": { "type": { "const": "conversation.event" } }, + "required": ["type"] + } + }, + "required": ["structural"] + }, + "then": { + "properties": { + "structural": { "$ref": "#/$defs/conversationEvent" } + } + } + }, + { + "if": { + "type": "object", + "properties": { + "structural": { + "type": "object", + "properties": { "type": { "const": "conversation.compact" } }, + "required": ["type"] + } + }, + "required": ["structural"] + }, + "then": { + "properties": { + "structural": { "$ref": "#/$defs/conversationCompact" } + } + } + } + ] + } + }, + + "properties": { + "meta": { + "type": "object", + "description": "Path metadata. `kind` pins this spec; `source` names the producing harness; `producer`/`files_changed`/`vcs_remote` are flattened session-level fields (PathMeta carries `extra` via serde flatten, so they sit directly under `meta`, not under `meta.extra`).", + "properties": { + "kind": { + "const": "https://toolpath.net/kinds/agent-coding-session/v1.2.0" + }, + "source": { "type": "string" }, + "files_changed": { + "type": "array", + "items": { "type": "string" } + }, + "vcs_remote": { "type": "string" }, + "producer": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "version": { "type": "string" } + }, + "required": ["name"] + } + }, + "required": ["kind"] + }, + + "steps": { + "type": "array", + "items": { + "type": "object", + "properties": { + "change": { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/artifactChange" } + } + } + } + } + }, + + "required": ["meta"] +} diff --git a/crates/path-cli/src/cmd_export.rs b/crates/path-cli/src/cmd_export.rs index 9447f76..2b6fc22 100644 --- a/crates/path-cli/src/cmd_export.rs +++ b/crates/path-cli/src/cmd_export.rs @@ -1405,9 +1405,9 @@ fn build_cursor_session( // Reuse the existing id when present, otherwise pre-create a // workspaceStorage entry so Cursor adopts ours on next open. let resolver = PathResolver::new(); - if let Ok(ensured) = resolver.ensure_workspace_storage_entry(&canonical, |path| { - stable_workspace_id_for(path) - }) { + if let Ok(ensured) = + resolver.ensure_workspace_storage_entry(&canonical, stable_workspace_id_for) + { projector = projector.with_workspace_id(ensured.id); if ensured.created { eprintln!( @@ -1535,13 +1535,8 @@ fn cursor_open_hints(workspace: &std::path::Path) -> Vec { } } - #[cfg(not(target_os = "emscripten"))] -fn upsert_cursor_kv( - tx: &rusqlite::Transaction<'_>, - key: &str, - value: &str, -) -> Result<()> { +fn upsert_cursor_kv(tx: &rusqlite::Transaction<'_>, key: &str, value: &str) -> Result<()> { tx.execute( "INSERT OR REPLACE INTO cursorDiskKV (key, value) VALUES (?1, ?2)", rusqlite::params![key, value], diff --git a/crates/path-cli/src/cmd_import.rs b/crates/path-cli/src/cmd_import.rs index 7dac653..8083351 100644 --- a/crates/path-cli/src/cmd_import.rs +++ b/crates/path-cli/src/cmd_import.rs @@ -437,7 +437,9 @@ fn derive_claude_with_manager( .map_err(|e| anyhow::anyhow!("{}", e))? .ok_or_else(|| anyhow::anyhow!("No conversations found for project: {}", p))?; let cfg = make_config(&p); - return wrap_paths_claude(vec![toolpath_claude::derive::derive_path(&convo, &cfg)]); + return wrap_paths_claude(vec![toolpath_claude::derive::derive_path( + &convo, &cfg, + )]); } } (None, _, _) => { @@ -660,7 +662,9 @@ fn derive_gemini_with_manager( .map_err(|e| anyhow::anyhow!("{}", e))? .ok_or_else(|| anyhow::anyhow!("No conversations found for project: {}", p))?; let cfg = make_config(&p); - return wrap_paths_gemini(vec![toolpath_gemini::derive::derive_path(&convo, &cfg)]); + return wrap_paths_gemini(vec![toolpath_gemini::derive::derive_path( + &convo, &cfg, + )]); } } (None, _, _) => { @@ -1144,16 +1148,15 @@ fn derive_cursor( Ok(toolpath_cursor::derive_path(&s, &cfg)) }; - let workspace_filter = project.as_deref().map(|p| { - std::fs::canonicalize(p).unwrap_or_else(|_| PathBuf::from(p)) - }); + let workspace_filter = project + .as_deref() + .map(|p| std::fs::canonicalize(p).unwrap_or_else(|_| PathBuf::from(p))); let workspace_match = |m: &toolpath_cursor::CursorSessionMetadata| -> bool { match (&workspace_filter, &m.workspace_path) { (None, _) => true, (Some(_), None) => false, (Some(want), Some(have)) => { - let canonical = - std::fs::canonicalize(have).unwrap_or_else(|_| have.clone()); + let canonical = std::fs::canonicalize(have).unwrap_or_else(|_| have.clone()); &canonical == want } } @@ -1352,7 +1355,8 @@ fn derive_pi_with_manager( .ok_or_else(|| { anyhow::anyhow!("No Pi sessions found for project: {}", p) })?; - let doc = Graph::from_path(toolpath_pi::derive::derive_path(&session, &config)); + let doc = + Graph::from_path(toolpath_pi::derive::derive_path(&session, &config)); let cache_id = make_id("pi", &doc_inner_id(&doc)); return Ok(vec![DerivedDoc { cache_id, doc }]); } diff --git a/crates/path-cli/src/cmd_share.rs b/crates/path-cli/src/cmd_share.rs index fba314e..bb5ba95 100644 --- a/crates/path-cli/src/cmd_share.rs +++ b/crates/path-cli/src/cmd_share.rs @@ -836,10 +836,7 @@ fn harness_status_pi(bundle: &HarnessBundle, home: Option<&std::path::Path>) -> } } -fn harness_status_cursor( - bundle: &HarnessBundle, - home: Option<&std::path::Path>, -) -> HarnessStatus { +fn harness_status_cursor(bundle: &HarnessBundle, home: Option<&std::path::Path>) -> HarnessStatus { let Some(mgr) = &bundle.cursor else { return HarnessStatus::unresolved(); }; diff --git a/crates/path-cli/src/schema.rs b/crates/path-cli/src/schema.rs index 381ef7d..41b744f 100644 --- a/crates/path-cli/src/schema.rs +++ b/crates/path-cli/src/schema.rs @@ -33,6 +33,10 @@ const KIND_SCHEMAS: &[(&str, &str)] = &[ "https://toolpath.net/kinds/agent-coding-session/v1.1.0", include_str!("../kinds/agent-coding-session/v1.1.0/schema.json"), ), + ( + "https://toolpath.net/kinds/agent-coding-session/v1.2.0", + include_str!("../kinds/agent-coding-session/v1.2.0/schema.json"), + ), ]; fn validator() -> &'static Validator { @@ -229,7 +233,7 @@ mod tests { validate(&doc).expect("base is optional on path identity"); } - const ACS_KIND: &str = "https://toolpath.net/kinds/agent-coding-session/v1.1.0"; + const ACS_KIND: &str = "https://toolpath.net/kinds/agent-coding-session/v1.2.0"; fn acs_graph(append: serde_json::Value) -> serde_json::Value { json!({ @@ -301,7 +305,8 @@ mod tests { .join("../../test-fixtures/claude/convo.jsonl"); let convo = toolpath_claude::ConversationReader::read_conversation(&fixture) .expect("read claude fixture"); - let path = toolpath_claude::derive::derive_path(&convo, &Default::default()); + let path = + toolpath_claude::derive::derive_path(&convo, &Default::default()); assert_eq!( path.meta.as_ref().and_then(|m| m.kind.as_deref()), Some(ACS_KIND), diff --git a/crates/path-cli/tests/cross_harness_matrix.rs b/crates/path-cli/tests/cross_harness_matrix.rs index 34e55fe..0b1afd7 100644 --- a/crates/path-cli/tests/cross_harness_matrix.rs +++ b/crates/path-cli/tests/cross_harness_matrix.rs @@ -27,6 +27,12 @@ trait Harness { /// root) through the harness's native reader and forward it to IR. /// Returns `None` if the fixture isn't on disk. fn load_fixture(&self) -> Option; + /// The `convo-compacted.{jsonl,json}` fixture — a captured session that + /// contains a real context compaction. `None` when the harness has no + /// compaction concept or no such fixture on disk. + fn load_compacted_fixture(&self) -> Option { + None + } /// Project the IR view, serialize the native output to its on-disk /// wire format, and re-parse through the harness's own reader. /// Returns Err with a descriptive message if the wire round-trip @@ -34,6 +40,13 @@ trait Harness { /// Some harnesses don't have a JSON/JSONL wire (opencode is SQL); /// those can return Ok(()) with the rationale documented inline. fn schema_validates(&self, view: &ConversationView) -> Result<(), String>; + /// Whether our derive↔project pipeline round-trips a context compaction + /// for this harness. When false, `compaction_survives` skips it as a + /// translation target — either the on-disk format carries no boundary + /// (gemini) or the provider doesn't implement compaction yet (cursor). + fn roundtrips_compaction(&self) -> bool { + true + } } fn fixtures_dir() -> PathBuf { @@ -62,6 +75,15 @@ impl Harness for ClaudeHarness { .expect("claude fixture parse"); Some(toolpath_claude::provider::to_view(&convo)) } + fn load_compacted_fixture(&self) -> Option { + let path = fixtures_dir().join("claude/convo-compacted.jsonl"); + if !path.exists() { + return None; + } + let convo = toolpath_claude::ConversationReader::read_conversation(&path) + .expect("claude compacted fixture parse"); + Some(toolpath_claude::provider::to_view(&convo)) + } fn schema_validates(&self, view: &ConversationView) -> Result<(), String> { let projector = toolpath_claude::ClaudeProjector; let convo = projector @@ -104,6 +126,15 @@ impl Harness for CodexHarness { toolpath_codex::RolloutReader::read_session(&path).expect("codex fixture parse"); Some(toolpath_codex::to_view(&session)) } + fn load_compacted_fixture(&self) -> Option { + let path = fixtures_dir().join("codex/convo-compacted.jsonl"); + if !path.exists() { + return None; + } + let session = toolpath_codex::RolloutReader::read_session(&path) + .expect("codex compacted fixture parse"); + Some(toolpath_codex::to_view(&session)) + } fn schema_validates(&self, view: &ConversationView) -> Result<(), String> { let projector = toolpath_codex::project::CodexProjector::new(); let session = projector @@ -142,6 +173,15 @@ impl Harness for PiHarness { let session = toolpath_pi::reader::read_session_from_file(&path).expect("pi fixture parse"); Some(toolpath_pi::session_to_view(&session)) } + fn load_compacted_fixture(&self) -> Option { + let path = fixtures_dir().join("pi/convo-compacted.jsonl"); + if !path.exists() { + return None; + } + let session = + toolpath_pi::reader::read_session_from_file(&path).expect("pi compacted fixture parse"); + Some(toolpath_pi::session_to_view(&session)) + } fn schema_validates(&self, view: &ConversationView) -> Result<(), String> { let projector = toolpath_pi::project::PiProjector::new(); let session = projector @@ -167,6 +207,11 @@ impl Harness for GeminiHarness { fn name(&self) -> &'static str { "gemini" } + fn roundtrips_compaction(&self) -> bool { + // Gemini compresses context in memory but persists no boundary to the + // chat file, so there's nothing to round-trip. + false + } fn roundtrip(&self, view: &ConversationView) -> ConversationView { let projector = toolpath_gemini::project::GeminiProjector::default(); let convo = projector.project(view).expect("gemini project"); @@ -242,6 +287,15 @@ impl Harness for OpencodeHarness { let session = parse_opencode_export(&json); Some(toolpath_opencode::to_view(&session)) } + fn load_compacted_fixture(&self) -> Option { + let path = fixtures_dir().join("opencode/convo-compacted.json"); + if !path.exists() { + return None; + } + let json = std::fs::read_to_string(&path).expect("opencode compacted fixture read"); + let session = parse_opencode_export(&json); + Some(toolpath_opencode::to_view(&session)) + } fn schema_validates(&self, view: &ConversationView) -> Result<(), String> { // Opencode's wire format is SQLite + JSON-in-TEXT columns; // a full wire round-trip would need to spin up a temporary @@ -267,6 +321,14 @@ impl Harness for CursorHarness { fn name(&self) -> &'static str { "cursor" } + fn roundtrips_compaction(&self) -> bool { + // Cursor DOES compact (`/summarize`) and writes a `capabilityType: 22` + // boundary marker bubble, but the summary text and kept set live + // server-side — not in the local store — so there's nothing + // reconstructable to round-trip. Like gemini, we don't derive a + // compaction. See docs/agents/formats/cursor.md. + false + } fn roundtrip(&self, view: &ConversationView) -> ConversationView { let projector = toolpath_cursor::project::CursorProjector::new(); let session = projector.project(view).expect("cursor project"); @@ -466,10 +528,7 @@ mod invariants { } fn meaningful_turns(view: &ConversationView) -> Vec<&Turn> { - view.turns - .iter() - .filter(|t| !is_system_envelope(t)) - .collect() + view.turns().filter(|t| !is_system_envelope(t)).collect() } pub fn turn_count_and_role_sequence( @@ -498,6 +557,45 @@ mod invariants { } } + /// Every compaction in `original` must survive: the boundary count and + /// each one's summary *text* (whitespace-normalized) are preserved. + /// Comparing the text — not just its presence — catches a boundary that + /// keeps *a* summary but the wrong one (e.g. a multi-compaction session + /// collapsing every boundary onto the first summary). An empty summary + /// normalizes to `None`, so an absent and an empty summary compare equal + /// — Codex always carries a (possibly empty) message, and that empty + /// survives as empty. `kept` is still not compared — wholesale harnesses + /// (Codex) drop it by design, so it isn't a cross-harness invariant. + pub fn compaction_survives( + original: &ConversationView, + result: &ConversationView, + failures: &mut Vec, + ) { + let want = original.compactions().count(); + let got = result.compactions().count(); + if want != got { + failures.push(format!( + "compaction count diverged: first={want} second={got}" + )); + return; + } + let summary = |c: &toolpath_convo::Compaction| -> Option { + c.summary.as_deref().map(norm).filter(|s| !s.is_empty()) + }; + for (i, (a, b)) in original.compactions().zip(result.compactions()).enumerate() { + let (sa, sb) = (summary(a), summary(b)); + if sa != sb { + let clip = + |s: &Option| s.as_deref().map(|t| t.chars().take(80).collect::()); + failures.push(format!( + "compaction {i} summary diverged:\n first: {:?}\n second: {:?}", + clip(&sa), + clip(&sb) + )); + } + } + } + fn norm(s: &str) -> String { s.split_whitespace().collect::>().join(" ") } @@ -647,8 +745,7 @@ mod invariants { // on input/output — the fields every wire carries (codex has no // cache_write analog, cursor carries no cache counters at all). let usage_seq = |v: &ConversationView| -> Vec<(Option, Option)> { - v.turns - .iter() + v.turns() .filter(|t| matches!(t.role, Role::Assistant)) .filter_map(|t| t.token_usage.as_ref()) .map(|u| (u.input_tokens, u.output_tokens)) @@ -702,13 +799,11 @@ mod invariants { failures: &mut Vec, ) { let pre: Vec<&Turn> = before_target - .turns - .iter() + .turns() .filter(|t| matches!(t.role, Role::Assistant)) .collect(); let post: Vec<&Turn> = after_target - .turns - .iter() + .turns() .filter(|t| matches!(t.role, Role::Assistant)) .collect(); for (i, (a, b)) in pre.iter().zip(post.iter()).enumerate() { @@ -770,8 +865,7 @@ mod invariants { failures: &mut Vec, ) { let edges = |v: &ConversationView| -> BTreeSet<(String, Option)> { - v.turns - .iter() + v.turns() .map(|t| (t.id.clone(), t.parent_id.clone())) .collect() }; @@ -821,7 +915,7 @@ mod invariants { failures: &mut Vec, ) { let count = - |v: &ConversationView| -> usize { v.turns.iter().map(|t| t.delegations.len()).sum() }; + |v: &ConversationView| -> usize { v.turns().map(|t| t.delegations.len()).sum() }; let o = count(original); let f = count(final_); if o != f { @@ -832,7 +926,7 @@ mod invariants { return; } - for (i, (a, b)) in original.turns.iter().zip(final_.turns.iter()).enumerate() { + for (i, (a, b)) in original.turns().zip(final_.turns()).enumerate() { if a.delegations.len() != b.delegations.len() { failures.push(format!( "turn {} delegation count diverged: first={} second={}", @@ -895,14 +989,12 @@ mod invariants { failures: &mut Vec, ) { let agent_ids = |v: &ConversationView| -> BTreeSet { - v.turns - .iter() + v.turns() .flat_map(|t| t.delegations.iter().map(|d| d.agent_id.clone())) .collect() }; let tool_use_ids = |v: &ConversationView| -> BTreeSet { - v.turns - .iter() + v.turns() .flat_map(|t| t.tool_uses.iter().map(|tu| tu.id.clone())) .collect() }; @@ -975,6 +1067,9 @@ fn run_cell( invariants::delegations(&view_first, &view_second, &mut failures); invariants::delegations_survive(&view_after_source, &view_first, &mut failures); invariants::files_changed(&view_first, &view_second, &mut failures); + if target.roundtrips_compaction() { + invariants::compaction_survives(&view_after_source, &view_first, &mut failures); + } failures } @@ -1054,7 +1149,11 @@ fn matrix_translation() { h.name() ) }); - eprintln!("loaded {} fixture: {} turns", h.name(), view.turns.len()); + eprintln!( + "loaded {} fixture: {} turns", + h.name(), + view.turns().count() + ); sources.push((h.name().to_string(), view)); } run_matrix("matrix (real fixtures)", &sources); @@ -1096,3 +1195,31 @@ fn matrix_schema_validation() { ); } } + +#[test] +fn matrix_translation_compacted() { + // The same matrix, sourced from each harness's captured COMPACTED + // session, so `compaction_survives` in every cell exercises a real + // boundary alongside the usual translation invariants. Gemini has no + // compaction fixture and is skipped as a source. + let harnesses = all_harnesses(); + let mut sources: Vec<(String, ConversationView)> = Vec::new(); + for h in &harnesses { + if let Some(view) = h.load_compacted_fixture() { + assert!( + view.compactions().count() > 0, + "{} compacted fixture carries no compaction", + h.name() + ); + eprintln!( + "loaded {} compacted fixture: {} turns, {} compaction(s)", + h.name(), + view.turns().count(), + view.compactions().count() + ); + sources.push((h.name().to_string(), view)); + } + } + assert!(!sources.is_empty(), "no compacted fixtures on disk"); + run_matrix("matrix (compacted fixtures)", &sources); +} diff --git a/crates/path-cli/tests/roundtrip.rs b/crates/path-cli/tests/roundtrip.rs index e768ba2..693b473 100644 --- a/crates/path-cli/tests/roundtrip.rs +++ b/crates/path-cli/tests/roundtrip.rs @@ -65,19 +65,18 @@ fn roundtrip_claude_conversation() { // The extracted view comes from the toolpath Path which also emits // conversation.append steps only for real turns (not tool-result-only entries). assert_eq!( - extracted_view.turns.len(), - original_view.turns.len(), + extracted_view.turns().count(), + original_view.turns().count(), "turn count mismatch: extracted {} vs original {}", - extracted_view.turns.len(), - original_view.turns.len(), + extracted_view.turns().count(), + original_view.turns().count(), ); // ── Assertions: turn content ──────────────────────────────────── for (i, (ext, orig)) in extracted_view - .turns - .iter() - .zip(original_view.turns.iter()) + .turns() + .zip(original_view.turns()) .enumerate() { assert_eq!( diff --git a/crates/toolpath-claude/Cargo.toml b/crates/toolpath-claude/Cargo.toml index 5f46341..a0aaf7a 100644 --- a/crates/toolpath-claude/Cargo.toml +++ b/crates/toolpath-claude/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-claude" -version = "0.12.0" +version = "0.13.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-claude/src/derive.rs b/crates/toolpath-claude/src/derive.rs index 8d9cb61..1ef82dc 100644 --- a/crates/toolpath-claude/src/derive.rs +++ b/crates/toolpath-claude/src/derive.rs @@ -22,7 +22,10 @@ pub struct DeriveConfig { } /// Derive a Toolpath [`Path`] from a Claude [`Conversation`]. -pub fn derive_path(conversation: &Conversation, config: &DeriveConfig) -> Path { +pub fn derive_path( + conversation: &Conversation, + config: &DeriveConfig, +) -> Path { let view = to_view(conversation); let prefix: String = conversation.session_id.chars().take(8).collect(); let base_uri = config.project_path.as_ref().map(|p| { @@ -42,7 +45,10 @@ pub fn derive_path(conversation: &Conversation, config: &DeriveConfig) -> Path { } /// Derive Toolpath Paths from multiple conversations in a project. -pub fn derive_project(conversations: &[Conversation], config: &DeriveConfig) -> Vec { +pub fn derive_project( + conversations: &[Conversation], + config: &DeriveConfig, +) -> Vec { conversations .iter() .map(|c| derive_path(c, config)) diff --git a/crates/toolpath-claude/src/project.rs b/crates/toolpath-claude/src/project.rs index 85969d7..e0be98b 100644 --- a/crates/toolpath-claude/src/project.rs +++ b/crates/toolpath-claude/src/project.rs @@ -12,7 +12,8 @@ use crate::types::{ use serde_json::json; use std::collections::HashMap; use toolpath_convo::{ - ConversationProjector, ConversationView, ConvoError, Result, Role, ToolInvocation, Turn, + Compaction, CompactionTrigger, ConversationProjector, ConversationView, ConvoError, Result, + Role, ToolInvocation, Turn, }; // ── ClaudeProjector ─────────────────────────────────────────────────── @@ -65,7 +66,7 @@ fn project_view(view: &ConversationView) -> std::result::Result std::result::Result> = HashMap::new(); - for event in &view.events { + for event in view.events() { if event.event_type != TOOL_RESULT_USER_EVENT { continue; } @@ -114,7 +115,7 @@ fn project_view(view: &ConversationView) -> std::result::Result = HashMap::new(); - for turn in &view.turns { + for turn in view.turns() { if let (Some(mid), Some(usage)) = (turn.group_id.as_deref(), &turn.token_usage) { group_total .entry(mid) @@ -123,7 +124,31 @@ fn project_view(view: &ConversationView) -> std::result::Result t, + toolpath_convo::Item::Compaction(c) => { + // Emit the boundary (+ summary) that `to_view` re-folds into + // this `Item::Compaction`. `kept` rides in + // compactMetadata.preservedMessages, which is how re-read + // recovers it. + let effective_parent = c + .parent_id + .as_ref() + .and_then(|pid| parent_rewrites.get(pid).cloned()) + .or_else(|| c.parent_id.clone()); + for entry in compaction_entries(c, &view.id, effective_parent) { + convo.add_entry(entry); + } + continue; + } + toolpath_convo::Item::Event(_) => continue, + }; + // Pre-rewrite this turn's parent_id if a synthesized tool_result // was emitted between it and its IR-recorded parent. let effective_parent = turn @@ -202,7 +227,7 @@ fn project_view(view: &ConversationView) -> std::result::Result std::result::Result, +) -> Vec { + let mut entries: Vec = Vec::new(); + + let mut compact_metadata = serde_json::Map::new(); + if let Some(trigger) = c.trigger { + let s = match trigger { + CompactionTrigger::Auto => "auto", + CompactionTrigger::Manual => "manual", + }; + compact_metadata.insert("trigger".into(), json!(s)); + } + if let Some(pre_tokens) = c.pre_tokens { + compact_metadata.insert("preTokens".into(), json!(pre_tokens)); + } + if !c.kept.is_empty() { + compact_metadata.insert("preservedMessages".into(), json!({ "uuids": c.kept })); + } + + let mut boundary_extra: HashMap = HashMap::new(); + boundary_extra.insert("subtype".into(), json!("compact_boundary")); + if let Some(parent) = &effective_parent { + boundary_extra.insert("logicalParentUuid".into(), json!(parent)); + } + boundary_extra.insert( + "compactMetadata".into(), + serde_json::Value::Object(compact_metadata), + ); + + let boundary = ConversationEntry { + uuid: c.id.clone(), + // The boundary's own parentUuid is always null on the wire; the + // logical parent rides in compactMetadata's logicalParentUuid. + parent_uuid: None, + is_sidechain: false, + entry_type: "system".to_string(), + timestamp: c.timestamp.clone(), + session_id: Some(session_id.to_string()), + cwd: None, + git_branch: None, + message: None, + version: None, + user_type: None, + request_id: None, + tool_use_result: None, + snapshot: None, + message_id: None, + extra: boundary_extra, + }; + + entries.push(boundary); + + if let Some(summary) = &c.summary { + let mut summary_extra: HashMap = HashMap::new(); + summary_extra.insert("isCompactSummary".into(), json!(true)); + + entries.push(ConversationEntry { + uuid: format!("{}-summary", c.id), + parent_uuid: Some(c.id.clone()), + is_sidechain: false, + entry_type: "user".to_string(), + timestamp: c.timestamp.clone(), + session_id: Some(session_id.to_string()), + cwd: None, + git_branch: None, + message: Some(Message { + role: MessageRole::User, + content: Some(MessageContent::Text(summary.clone())), + model: None, + id: None, + message_type: None, + stop_reason: None, + stop_sequence: None, + usage: None, + }), + version: None, + user_type: None, + request_id: None, + tool_use_result: None, + snapshot: None, + message_id: None, + extra: summary_extra, + }); + } + + entries +} + /// Rebuild a Claude tool-result user entry verbatim from a preserved event. /// /// The event was emitted by [`crate::derive::derive_path`] when reading the @@ -324,11 +461,9 @@ fn apply_turn_metadata(entry: &mut ConversationEntry, turn: &Turn) { } } - // Source-format details (`version`, `user_type`, `request_id`, - // per-entry catch-all) used to ride through `Turn.extra["claude"]` for - // claude → IR → claude round-trip. The IR no longer carries - // provider-specific extras; the projected entry's fields stay `None` - // and the harness fills in defaults at write time. + // The IR carries no provider-specific extras, so source-format details + // (`version`, `user_type`, `request_id`, per-entry catch-all) stay `None` + // here and the harness fills in defaults at write time. } /// Build a `ConversationEntry` for a user turn. @@ -1021,12 +1156,11 @@ mod tests { id: id.to_string(), started_at: None, last_activity: None, - turns, + items: turns.into_iter().map(toolpath_convo::Item::Turn).collect(), total_usage: None, provider_id: None, files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } @@ -1137,7 +1271,10 @@ mod tests { // Wire: the total is stamped on every line of the split, each tagged // with the shared message.id. - for entry in content_entries(&convo).iter().filter(|e| e.entry_type == "assistant") { + for entry in content_entries(&convo) + .iter() + .filter(|e| e.entry_type == "assistant") + { let msg = entry.message.as_ref().unwrap(); assert_eq!(msg.id.as_deref(), Some("msg_A")); assert_eq!(msg.usage.as_ref().unwrap().output_tokens, Some(164)); @@ -1145,7 +1282,7 @@ mod tests { // Re-read: total back on the final turn only; no fabricated attribution. let back = crate::provider::to_view(&convo); - let a: Vec<&Turn> = back.turns.iter().filter(|t| t.role == Role::Assistant).collect(); + let a: Vec<&Turn> = back.turns().filter(|t| t.role == Role::Assistant).collect(); assert!(a[0].token_usage.is_none()); assert_eq!(a[1].token_usage.as_ref().unwrap().output_tokens, Some(164)); assert!(a.iter().all(|t| t.attributed_token_usage.is_none())); diff --git a/crates/toolpath-claude/src/provider.rs b/crates/toolpath-claude/src/provider.rs index 1896d1e..bbea844 100644 --- a/crates/toolpath-claude/src/provider.rs +++ b/crates/toolpath-claude/src/provider.rs @@ -12,8 +12,9 @@ use crate::types::{Conversation, ConversationEntry, Message, MessageContent, Mes #[cfg(any(feature = "watcher", test))] use toolpath_convo::WatcherEvent; use toolpath_convo::{ - ConversationMeta, ConversationProvider, ConversationView, ConvoError, DelegatedWork, - EnvironmentSnapshot, Role, TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, + Compaction, CompactionTrigger, ConversationMeta, ConversationProvider, ConversationView, + ConvoError, DelegatedWork, EnvironmentSnapshot, Item, Role, TokenUsage, ToolCategory, + ToolInvocation, ToolResult, Turn, }; // ── Conversion helpers ─────────────────────────────────────────────── @@ -309,6 +310,39 @@ fn merge_tool_results(turns: &mut [Turn], msg: &Message) -> bool { merged } +/// Mutable accessor for the turn inside an [`Item`], if it is one. +fn item_turn_mut(item: &mut Item) -> Option<&mut Turn> { + match item { + Item::Turn(t) => Some(t), + _ => None, + } +} + +/// Merge a tool-result-only message into the turns already pushed onto +/// `items`. Equivalent to [`merge_tool_results`] but operating on the +/// interleaved item stream — non-turn items (events, compaction) are skipped. +fn merge_tool_results_into_items(items: &mut [Item], msg: &Message) -> bool { + let mut turns: Vec<&mut Turn> = items.iter_mut().filter_map(item_turn_mut).collect(); + let mut merged = false; + for tr in msg.tool_results() { + for turn in turns.iter_mut().rev() { + if let Some(invocation) = turn + .tool_uses + .iter_mut() + .find(|tu| tu.id == tr.tool_use_id && tu.result.is_none()) + { + invocation.result = Some(ToolResult { + content: tr.content.text(), + is_error: tr.is_error, + }); + merged = true; + break; + } + } + } + merged +} + fn entry_to_turn(entry: &ConversationEntry) -> Option { entry .message @@ -316,39 +350,216 @@ fn entry_to_turn(entry: &ConversationEntry) -> Option { .map(|msg| message_to_turn(entry, msg)) } +/// Returns true if this entry is Claude's inline compaction boundary marker. +/// +/// Claude writes the boundary either as a top-level `type: "compact_boundary"` +/// entry or as `type: "system"` with `subtype: "compact_boundary"`. The +/// `subtype` field isn't in [`ConversationEntry`]'s typed fields, so it lands +/// in `extra`. +fn is_compact_boundary(entry: &ConversationEntry) -> bool { + entry.entry_type == "compact_boundary" + || entry + .extra + .get("subtype") + .and_then(|v| v.as_str()) + .map(|s| s == "compact_boundary") + .unwrap_or(false) +} + +/// Returns true if this entry is the synthetic compaction summary that Claude +/// writes immediately after a boundary (`isCompactSummary: true`). +fn is_compact_summary(entry: &ConversationEntry) -> bool { + entry + .extra + .get("isCompactSummary") + .and_then(|v| v.as_bool()) + .unwrap_or(false) +} + +/// Build a [`Compaction`] from Claude's boundary marker and (optionally) the +/// synthetic summary that follows it. +/// +/// All the boundary's compaction-specific data lives in `entry.extra` +/// (`logicalParentUuid`, `compactMetadata.{trigger,preTokens,preservedMessages}`). +/// `summary` comes from the following `isCompactSummary` entry's message text. +/// +/// `replayed` is the set of turn-ids that Claude re-emitted just before this +/// boundary (detected by duplicate-uuid stripping in [`conversation_to_view`]): +/// the tool_use/tool_result entries it "pins" into the post-compaction window. +/// `kept` becomes the de-duplicated union, in order, of (a) the explicit +/// preserved tail (`compactMetadata.preservedMessages.uuids`) and (b) that +/// replayed set — every turn-id that survives verbatim into the post-compaction +/// context. +fn compaction_from_boundary( + boundary: &ConversationEntry, + summary: Option, + replayed: &[String], +) -> Compaction { + let extra = &boundary.extra; + + // The pre-compaction message the boundary logically continues from. + // `parentUuid` is always null on the boundary, so use logicalParentUuid. + let parent_id = extra + .get("logicalParentUuid") + .and_then(|v| v.as_str()) + .map(|s| s.to_string()); + + let meta = extra.get("compactMetadata"); + + let trigger = meta + .and_then(|m| m.get("trigger")) + .and_then(|v| v.as_str()) + .and_then(|s| match s { + "auto" => Some(CompactionTrigger::Auto), + "manual" => Some(CompactionTrigger::Manual), + _ => None, + }); + + let pre_tokens = meta + .and_then(|m| m.get("preTokens")) + .and_then(|v| v.as_u64()); + + // The turn-ids that survive verbatim into the post-compaction window: the + // explicit preserved tail (preservedMessages.uuids) plus the re-emitted + // "pinned" set we detected via duplicate-uuid stripping. Union in order, + // de-duplicated — the two sets are disjoint in practice but we guard anyway. + let mut kept: Vec = Vec::new(); + let mut seen = std::collections::HashSet::new(); + let preserved_tail = meta + .and_then(|m| m.get("preservedMessages")) + .and_then(|p| p.get("uuids")) + .and_then(|v| v.as_array()); + if let Some(uuids) = preserved_tail { + for u in uuids.iter().filter_map(|v| v.as_str()) { + if seen.insert(u.to_string()) { + kept.push(u.to_string()); + } + } + } + for u in replayed { + if seen.insert(u.clone()) { + kept.push(u.clone()); + } + } + + Compaction { + id: boundary.uuid.clone(), + parent_id, + timestamp: boundary.timestamp.clone(), + trigger, + summary, + pre_tokens, + kept, + } +} + /// Convert a full conversation to a view with cross-entry tool result assembly. /// /// Tool-result-only user entries are absorbed into the preceding assistant /// turn's `ToolInvocation.result` fields rather than emitted as separate turns. +/// +/// Compaction boundaries are detected and emitted as [`Item::Compaction`] at +/// their position in the ordered item stream: the boundary's `compactMetadata` +/// becomes the `Compaction`, and the immediately-following synthetic summary +/// entry is folded into `Compaction.summary` rather than surfaced as a turn. fn conversation_to_view(convo: &Conversation) -> ConversationView { - let mut turns: Vec = Vec::new(); - let mut events: Vec = Vec::new(); + // Items are built in source order so a compaction boundary lands at its + // true position between the turns it separates. Preamble events come + // first — they precede all entries in the file. + let mut items: Vec = Vec::new(); // Headerless preamble lines (ai-title, last-prompt, queue-operation, // permission-mode, file-history-snapshot, etc.) become events so they // round-trip back to JSONL. for (idx, raw) in convo.preamble.iter().enumerate() { - events.push(preamble_to_event(idx, raw)); + items.push(Item::Event(preamble_to_event(idx, raw))); } // Map from "absorbed-or-skipped entry UUID" → "the previous - // turn-bearing entry's UUID". Used so that an assistant turn whose - // wire parentUuid points at a tool-result-only entry (or any other - // absorbed entry that didn't become a Turn) gets a Turn.parent_id - // that still maps onto a real Turn — keeping the IR's turn-to-turn - // chain intact for `derive_path`. The original UUID is preserved - // via the `tool_result_user` event. + // turn-or-compaction-bearing entry's UUID". Used so that a later turn + // whose wire parentUuid points at an absorbed entry (a tool-result-only + // entry, or the folded compaction summary) gets a `parent_id` that still + // maps onto a real Item — keeping the IR's chain intact for `derive_path`. let mut parent_rewrites: HashMap = HashMap::new(); - let mut last_turn_uuid: Option = None; + // The UUID of the last turn or compaction emitted into `items`, used to + // rewrite parents of subsequently absorbed entries. + let mut last_anchor_uuid: Option = None; + + // Duplicate-uuid stripping. When Claude compacts, it re-emits a block of + // earlier tool_use/tool_result entries — already-seen uuids — that it + // "pins" into the post-compaction context, immediately before the + // boundary. We keep only the FIRST occurrence of each uuid and capture the + // re-emitted ones as the boundary's `kept` set rather than surfacing them + // as turns. (`derive_path` would otherwise dedupe them itself — dropping a + // byte-identical re-emission, renaming a changed one — but doing it here is + // what lets the boundary record `kept`.) The stripped uuids since the last + // boundary are the "replayed" set, attached to the next boundary's + // `Compaction.kept`. + let mut seen_uuids: std::collections::HashSet = std::collections::HashSet::new(); + let mut replayed_since_boundary: Vec = Vec::new(); + + let entries = &convo.entries; + let mut i = 0; + while i < entries.len() { + let entry = &entries[i]; + + // Strip re-emitted entries: any non-boundary entry whose uuid already + // appeared earlier in this conversation. Boundary entries are exempt + // (their uuid is always unique). Record the uuid as replayed so the + // next boundary can list it in `kept`. + if !is_compact_boundary(entry) + && !entry.uuid.is_empty() + && !seen_uuids.insert(entry.uuid.clone()) + { + replayed_since_boundary.push(entry.uuid.clone()); + i += 1; + continue; + } + + // Compaction boundary: emit one Item::Compaction at this position, + // folding the immediately-following synthetic summary entry (if any) + // into Compaction.summary rather than surfacing it as a turn. + if is_compact_boundary(entry) { + let summary = entries.get(i + 1).filter(|next| is_compact_summary(next)); + let summary_text = summary.map(|s| s.text()); + let replayed = std::mem::take(&mut replayed_since_boundary); + let compaction = compaction_from_boundary(entry, summary_text, &replayed); + seen_uuids.insert(entry.uuid.clone()); + if let Some(s) = summary { + seen_uuids.insert(s.uuid.clone()); + } + // Rewire the compaction's logical parent through any prior + // absorption so it lands on a real Item in the derived DAG. + let mut compaction = compaction; + if let Some(pid) = compaction.parent_id.as_ref() + && let Some(real) = parent_rewrites.get(pid) + { + compaction.parent_id = Some(real.clone()); + } + let boundary_uuid = compaction.id.clone(); + items.push(Item::Compaction(compaction)); + // Later turns whose wire parentUuid points at the boundary (or at + // the folded summary) chain through the compaction. + if let Some(prev) = &last_anchor_uuid { + parent_rewrites.insert(entry.uuid.clone(), prev.clone()); + } + if let Some(s) = summary { + parent_rewrites.insert(s.uuid.clone(), boundary_uuid.clone()); + i += 1; // consume the folded summary entry + } + last_anchor_uuid = Some(boundary_uuid); + i += 1; + continue; + } - for entry in &convo.entries { let Some(msg) = &entry.message else { // Message-less entries (attachments, snapshots) survive as // events so the projector can re-emit them. - events.push(entry_to_event(entry)); - if let Some(prev) = &last_turn_uuid { + items.push(Item::Event(entry_to_event(entry))); + if let Some(prev) = &last_anchor_uuid { parent_rewrites.insert(entry.uuid.clone(), prev.clone()); } + i += 1; continue; }; @@ -362,10 +573,11 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { // but the Claude UI walks the chain by parentUuid, not by // specific UUIDs, so that's fine.) if is_tool_result_only(entry) { - merge_tool_results(&mut turns, msg); - if let Some(prev) = &last_turn_uuid { + merge_tool_results_into_items(&mut items, msg); + if let Some(prev) = &last_anchor_uuid { parent_rewrites.insert(entry.uuid.clone(), prev.clone()); } + i += 1; continue; } @@ -375,14 +587,17 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { { turn.parent_id = Some(real.clone()); } - last_turn_uuid = Some(turn.id.clone()); - turns.push(turn); + last_anchor_uuid = Some(turn.id.clone()); + items.push(Item::Turn(turn)); + i += 1; } - canonicalize_message_usage(&mut turns); + let mut turn_refs: Vec<&mut Turn> = items.iter_mut().filter_map(item_turn_mut).collect(); + canonicalize_message_usage(&mut turn_refs); + drop(turn_refs); // Re-derive delegation results now that tool results are merged - for turn in &mut turns { + for turn in items.iter_mut().filter_map(item_turn_mut) { for delegation in &mut turn.delegations { if delegation.result.is_none() && let Some(tu) = turn @@ -395,8 +610,8 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { } } - let total_usage = sum_usage(&turns); - let files_changed = extract_files_changed(&turns); + let total_usage = sum_usage(items.iter().filter_map(Item::as_turn)); + let files_changed = extract_files_changed(items.iter().filter_map(Item::as_turn)); // Pull path-level base/producer from the first entry that carries the // metadata (Claude records cwd / git_branch / version on every @@ -442,12 +657,11 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { id: convo.session_id.clone(), started_at: convo.started_at, last_activity: convo.last_activity, - turns, + items, total_usage, provider_id: Some("claude-code".into()), files_changed, session_ids: vec![], - events, base: view_base, producer, } @@ -561,55 +775,69 @@ pub(crate) fn max_usage(a: &TokenUsage, b: &TokenUsage) -> TokenUsage { /// per-step attribution from them, and — the format being undocumented — we /// do not trust line order. /// -/// For each consecutive `group_id` run this sets `token_usage` on the run's -/// **final** turn to the field-wise **maximum** across the run (the message -/// total — never under-counts whatever the stream order) and clears it from -/// the others, so summing `token_usage` over turns yields session totals. -fn canonicalize_message_usage(turns: &mut [Turn]) { - let mut i = 0; - while i < turns.len() { - let Some(mid) = turns[i].group_id.clone() else { - i += 1; +/// For each `group_id` this sets `token_usage` on the group's +/// **last-occurring** turn to the field-wise **maximum** across the group (the +/// message total — never under-counts whatever the stream order) and clears it +/// from the others, so summing `token_usage` over turns yields session totals. +/// +/// Grouping is by `group_id` across the whole sequence, not by consecutive run: +/// a single message's turns can be interrupted by an unrelated turn (e.g. a +/// `` user message lands between two assistant turns of +/// the same Codex round). Collapsing per run would leave the message total on +/// two turns — once per run — double-counting it. Keying on `group_id` lands it +/// exactly once. +fn canonicalize_message_usage(turns: &mut [&mut Turn]) { + // First pass: per group_id, the field-wise max usage and the index of the + // group's last-occurring turn. + let mut group_total: HashMap = HashMap::new(); + let mut group_last_idx: HashMap = HashMap::new(); + for (idx, t) in turns.iter().enumerate() { + let Some(mid) = t.group_id.clone() else { continue; }; - let mut j = i; - while j < turns.len() && turns[j].group_id.as_deref() == Some(mid.as_str()) { - j += 1; - } - - // Message total = field-wise max across the run (the final streaming - // snapshot, found without trusting line order). - let mut total: Option = None; - for t in &turns[i..j] { - if let Some(u) = &t.token_usage { - total = Some(match total { - Some(acc) => max_usage(&acc, u), - None => u.clone(), - }); - } + group_last_idx.insert(mid.clone(), idx); + if let Some(u) = &t.token_usage { + group_total + .entry(mid) + .and_modify(|acc| *acc = max_usage(acc, u)) + .or_insert_with(|| u.clone()); } + } - for t in &mut turns[i..j] { + // Second pass: clear usage off every grouped turn, then stamp each + // group's total back onto its last-occurring turn. + for t in turns.iter_mut() { + if t.group_id.is_some() { t.token_usage = None; } - if let Some(total) = total { - turns[j - 1].token_usage = Some(total); + } + for (mid, total) in group_total { + if let Some(&idx) = group_last_idx.get(&mid) { + turns[idx].token_usage = Some(total); } - i = j; } } /// Sum token usage across all turns. -fn sum_usage(turns: &[Turn]) -> Option { +fn sum_usage<'a>(turns: impl IntoIterator) -> Option { + let turns: Vec<&Turn> = turns.into_iter().collect(); + + // A message's usage repeats across every turn split from it; count it + // once, on the group's last-occurring turn. Key on `group_id` rather than + // adjacency so an interrupted group (a turn of another group landing in + // the middle) still counts once. + let mut group_last_idx: HashMap<&str, usize> = HashMap::new(); + for (idx, turn) in turns.iter().enumerate() { + if let Some(mid) = &turn.group_id { + group_last_idx.insert(mid.as_str(), idx); + } + } + let mut total = TokenUsage::default(); let mut any = false; for (idx, turn) in turns.iter().enumerate() { - // Turns split from one provider message all repeat that message's - // usage; count it once, on the run's last turn. if let Some(mid) = &turn.group_id - && turns - .get(idx + 1) - .is_some_and(|next| next.group_id.as_ref() == Some(mid)) + && group_last_idx.get(mid.as_str()) != Some(&idx) { continue; } @@ -637,7 +865,7 @@ fn sum_usage(turns: &[Turn]) -> Option { } /// Extract deduplicated file paths from file-write tool invocations. -fn extract_files_changed(turns: &[Turn]) -> Vec { +fn extract_files_changed<'a>(turns: impl IntoIterator) -> Vec { let mut seen = std::collections::HashSet::new(); let mut files = Vec::new(); for turn in turns { @@ -878,10 +1106,14 @@ mod tests { // (55) is NOT per-block attribution — it's where generation happened // to be when the line was flushed — so we never record it. let mut turns = vec![grp_turn("t1", "msg_A", 55), grp_turn("t2", "msg_A", 164)]; - canonicalize_message_usage(&mut turns); + let mut refs: Vec<&mut Turn> = turns.iter_mut().collect(); + canonicalize_message_usage(&mut refs); assert!(turns[0].token_usage.is_none(), "total only on final turn"); - assert_eq!(turns[1].token_usage.as_ref().unwrap().output_tokens, Some(164)); + assert_eq!( + turns[1].token_usage.as_ref().unwrap().output_tokens, + Some(164) + ); assert_eq!(turns[1].token_usage.as_ref().unwrap().input_tokens, Some(6)); for t in &turns { assert!( @@ -896,8 +1128,9 @@ mod tests { // Defensive: the complete total arrives FIRST (out of order). We // must still report 164 as the message total — the field-wise max, // not the last line's snapshot. - let mut turns = vec![grp_turn("t1", "msg_A", 164), grp_turn("t2", "msg_A", 55)]; - canonicalize_message_usage(&mut turns); + let mut turns = [grp_turn("t1", "msg_A", 164), grp_turn("t2", "msg_A", 55)]; + let mut refs: Vec<&mut Turn> = turns.iter_mut().collect(); + canonicalize_message_usage(&mut refs); assert_eq!( turns[1].token_usage.as_ref().unwrap().output_tokens, @@ -915,16 +1148,52 @@ mod tests { grp_turn("t2", "msg_A", 997), grp_turn("t3", "msg_A", 997), ]; - canonicalize_message_usage(&mut turns); + let mut refs: Vec<&mut Turn> = turns.iter_mut().collect(); + canonicalize_message_usage(&mut refs); assert!(turns[0].token_usage.is_none()); assert!(turns[1].token_usage.is_none()); - assert_eq!(turns[2].token_usage.as_ref().unwrap().output_tokens, Some(997)); + assert_eq!( + turns[2].token_usage.as_ref().unwrap().output_tokens, + Some(997) + ); for t in &turns { assert!(t.attributed_token_usage.is_none()); } } + #[test] + fn canonicalize_groups_across_an_interrupting_turn() { + // A message group can be interrupted by an unrelated turn (e.g. a + // `` user turn lands between two assistant + // turns of the same Codex round, both stamped with the group total). + // Grouping must key on `group_id`, not adjacency: the total lands on + // the group's LAST-occurring turn ONCE — collapsing per consecutive + // run would leave it on two turns, double-counting. + let mut t1 = grp_turn("t1", "msg_A", 997); + let mut interrupt = message_turn_stub("u1"); + interrupt.role = Role::User; + interrupt.group_id = None; + let mut t2 = grp_turn("t2", "msg_A", 997); + + { + let mut turns = [&mut t1, &mut interrupt, &mut t2]; + canonicalize_message_usage(&mut turns); + } + + assert!(t1.token_usage.is_none(), "earlier group turn cleared"); + assert!(interrupt.token_usage.is_none(), "ungrouped turn untouched"); + assert_eq!( + t2.token_usage.as_ref().unwrap().output_tokens, + Some(997), + "total lands once on the group's last-occurring turn" + ); + + // And the session sum counts the group exactly once. + let total = sum_usage([&t1, &interrupt, &t2]).expect("total"); + assert_eq!(total.output_tokens, Some(997)); + } + fn setup_provider() -> (TempDir, ClaudeConvo) { let temp = TempDir::new().unwrap(); let claude_dir = temp.path().join(".claude"); @@ -982,12 +1251,13 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "session-2") .unwrap(); - assert_eq!(view.turns.len(), 5); - assert!(view.turns[0].group_id.is_none(), "user lines carry no ID"); - for turn in &view.turns[1..=3] { + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 5); + assert!(turns[0].group_id.is_none(), "user lines carry no ID"); + for turn in &turns[1..=3] { assert_eq!(turn.group_id.as_deref(), Some("msg_A")); } - assert_eq!(view.turns[4].group_id.as_deref(), Some("msg_B")); + assert_eq!(turns[4].group_id.as_deref(), Some("msg_B")); } #[test] @@ -999,14 +1269,15 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "session-2") .unwrap(); - assert!(view.turns[1].token_usage.is_none()); - assert!(view.turns[2].token_usage.is_none()); + let turns: Vec<&Turn> = view.turns().collect(); + assert!(turns[1].token_usage.is_none()); + assert!(turns[2].token_usage.is_none()); assert_eq!( - view.turns[3].token_usage.as_ref().unwrap().output_tokens, + turns[3].token_usage.as_ref().unwrap().output_tokens, Some(997) ); assert_eq!( - view.turns[4].token_usage.as_ref().unwrap().output_tokens, + turns[4].token_usage.as_ref().unwrap().output_tokens, Some(11) ); } @@ -1033,52 +1304,50 @@ mod tests { .unwrap(); assert_eq!(view.id, "session-1"); + let turns: Vec<&Turn> = view.turns().collect(); // 7 entries collapse to 5 turns (2 tool-result-only entries absorbed) - assert_eq!(view.turns.len(), 5); + assert_eq!(turns.len(), 5); // Turn 0: user "Fix the bug" - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[0].text, "Fix the bug"); - assert!(view.turns[0].parent_id.is_none()); + assert_eq!(turns[0].role, Role::User); + assert_eq!(turns[0].text, "Fix the bug"); + assert!(turns[0].parent_id.is_none()); // Turn 1: assistant with tool use + assembled result - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[1].text, "I'll fix that."); - assert_eq!( - view.turns[1].thinking.as_deref(), - Some("The bug is in auth") - ); - assert_eq!(view.turns[1].tool_uses.len(), 1); - assert_eq!(view.turns[1].tool_uses[0].name, "Read"); - assert_eq!(view.turns[1].tool_uses[0].id, "t1"); + assert_eq!(turns[1].role, Role::Assistant); + assert_eq!(turns[1].text, "I'll fix that."); + assert_eq!(turns[1].thinking.as_deref(), Some("The bug is in auth")); + assert_eq!(turns[1].tool_uses.len(), 1); + assert_eq!(turns[1].tool_uses[0].name, "Read"); + assert_eq!(turns[1].tool_uses[0].id, "t1"); // Key assertion: result is populated from the next entry - let result = view.turns[1].tool_uses[0].result.as_ref().unwrap(); + let result = turns[1].tool_uses[0].result.as_ref().unwrap(); assert!(!result.is_error); assert!(result.content.contains("fn main()")); - assert_eq!(view.turns[1].model.as_deref(), Some("claude-opus-4-6")); - assert_eq!(view.turns[1].stop_reason.as_deref(), Some("tool_use")); - assert_eq!(view.turns[1].parent_id.as_deref(), Some("uuid-1")); + assert_eq!(turns[1].model.as_deref(), Some("claude-opus-4-6")); + assert_eq!(turns[1].stop_reason.as_deref(), Some("tool_use")); + assert_eq!(turns[1].parent_id.as_deref(), Some("uuid-1")); // Token usage - let usage = view.turns[1].token_usage.as_ref().unwrap(); + let usage = turns[1].token_usage.as_ref().unwrap(); assert_eq!(usage.input_tokens, Some(100)); assert_eq!(usage.output_tokens, Some(50)); // Turn 2: second assistant with tool use + assembled result - assert_eq!(view.turns[2].role, Role::Assistant); - assert_eq!(view.turns[2].text, "I see the issue. Let me fix it."); - assert_eq!(view.turns[2].tool_uses[0].name, "Edit"); - let result2 = view.turns[2].tool_uses[0].result.as_ref().unwrap(); + assert_eq!(turns[2].role, Role::Assistant); + assert_eq!(turns[2].text, "I see the issue. Let me fix it."); + assert_eq!(turns[2].tool_uses[0].name, "Edit"); + let result2 = turns[2].tool_uses[0].result.as_ref().unwrap(); assert_eq!(result2.content, "File written successfully"); // Turn 3: final assistant (no tools) - assert_eq!(view.turns[3].role, Role::Assistant); - assert_eq!(view.turns[3].text, "Done! The bug is fixed."); - assert!(view.turns[3].tool_uses.is_empty()); + assert_eq!(turns[3].role, Role::Assistant); + assert_eq!(turns[3].text, "Done! The bug is fixed."); + assert!(turns[3].tool_uses.is_empty()); // Turn 4: user "Thanks!" - assert_eq!(view.turns[4].role, Role::User); - assert_eq!(view.turns[4].text, "Thanks!"); + assert_eq!(turns[4].role, Role::User); + assert_eq!(turns[4].text, "Thanks!"); } #[test] @@ -1088,7 +1357,7 @@ mod tests { .unwrap(); // No turns should have empty text with User role (phantom turns) - for turn in &view.turns { + for turn in view.turns() { if turn.role == Role::User { assert!( !turn.text.is_empty(), @@ -1118,8 +1387,9 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); - assert_eq!(view.turns.len(), 2); // user + assistant (tool-result absorbed) - let result = view.turns[1].tool_uses[0].result.as_ref().unwrap(); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 2); // user + assistant (tool-result absorbed) + let result = turns[1].tool_uses[0].result.as_ref().unwrap(); assert!(result.is_error); assert_eq!(result.content, "File not found"); } @@ -1143,13 +1413,14 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); - assert_eq!(view.turns.len(), 2); - assert_eq!(view.turns[1].tool_uses.len(), 2); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 2); + assert_eq!(turns[1].tool_uses.len(), 2); - let r1 = view.turns[1].tool_uses[0].result.as_ref().unwrap(); + let r1 = turns[1].tool_uses[0].result.as_ref().unwrap(); assert_eq!(r1.content, "file a contents"); - let r2 = view.turns[1].tool_uses[1].result.as_ref().unwrap(); + let r2 = turns[1].tool_uses[1].result.as_ref().unwrap(); assert_eq!(r2.content, "file b contents"); } @@ -1171,9 +1442,10 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); - assert_eq!(view.turns.len(), 2); - assert_eq!(view.turns[0].text, "Hello"); - assert_eq!(view.turns[1].text, "Hi there!"); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 2); + assert_eq!(turns[0].text, "Hello"); + assert_eq!(turns[1].text, "Hi there!"); } #[test] @@ -1195,8 +1467,9 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); - assert_eq!(view.turns.len(), 2); - assert!(view.turns[1].tool_uses[0].result.is_none()); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 2); + assert!(turns[1].tool_uses[0].result.is_none()); } #[test] @@ -1231,7 +1504,7 @@ mod tests { .read_conversation("/test/project", "session-1") .unwrap(); let view = to_view(&convo); - assert_eq!(view.turns.len(), 5); + assert_eq!(view.turns().count(), 5); assert_eq!(view.title(20).unwrap(), "Fix the bug"); } @@ -1534,14 +1807,12 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "session-1") .unwrap(); + let turns: Vec<&Turn> = view.turns().collect(); // Turn 1 (assistant) has a Read tool - assert_eq!( - view.turns[1].tool_uses[0].category, - Some(ToolCategory::FileRead) - ); + assert_eq!(turns[1].tool_uses[0].category, Some(ToolCategory::FileRead)); // Turn 2 (assistant) has an Edit tool assert_eq!( - view.turns[2].tool_uses[0].category, + turns[2].tool_uses[0].category, Some(ToolCategory::FileWrite) ); } @@ -1564,14 +1835,15 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); + let turns: Vec<&Turn> = view.turns().collect(); // User turn has environment (entry has cwd and gitBranch) - let env = view.turns[0].environment.as_ref().unwrap(); + let env = turns[0].environment.as_ref().unwrap(); assert_eq!(env.working_dir.as_deref(), Some("/project/path")); assert_eq!(env.vcs_branch.as_deref(), Some("feat/auth")); assert!(env.vcs_revision.is_none()); // Assistant turn has no environment (entry has no cwd/gitBranch) - assert!(view.turns[1].environment.is_none()); + assert!(turns[1].environment.is_none()); } #[test] @@ -1592,7 +1864,7 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); - let usage = view.turns[1].token_usage.as_ref().unwrap(); + let usage = view.turns().nth(1).unwrap().token_usage.as_ref().unwrap(); assert_eq!(usage.cache_read_tokens, Some(500)); assert_eq!(usage.cache_write_tokens, Some(200)); } @@ -1662,8 +1934,9 @@ mod tests { ConversationProvider::load_conversation(&provider, "/test/project", "s1").unwrap(); // Assistant turn should have one delegation - assert_eq!(view.turns[1].delegations.len(), 1); - let d = &view.turns[1].delegations[0]; + let turn1 = view.turns().nth(1).unwrap(); + assert_eq!(turn1.delegations.len(), 1); + let d = &turn1.delegations[0]; assert_eq!(d.agent_id, "task-1"); assert_eq!(d.prompt, "Find the authentication bug"); assert!(d.turns.is_empty()); // Sub-agent turns are in separate files @@ -1722,7 +1995,7 @@ mod tests { .unwrap(); // No turns should have delegations (none use Task tool) - for turn in &view.turns { + for turn in view.turns() { assert!(turn.delegations.is_empty()); } } @@ -1767,11 +2040,12 @@ mod tests { // Should have turns from both segments (minus the bridge entry) // session-a: a1 (user), a2 (assistant) // session-b: b1 (user), b2 (assistant) — b0 is bridge, filtered - assert_eq!(view.turns.len(), 4); - assert_eq!(view.turns[0].text, "Fix the bug"); - assert_eq!(view.turns[1].text, "I'll fix that."); - assert_eq!(view.turns[2].text, "What about the tests?"); - assert_eq!(view.turns[3].text, "Tests pass now."); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 4); + assert_eq!(turns[0].text, "Fix the bug"); + assert_eq!(turns[1].text, "I'll fix that."); + assert_eq!(turns[2].text, "What about the tests?"); + assert_eq!(turns[3].text, "Tests pass now."); // Session IDs should be set assert_eq!(view.session_ids, vec!["session-a", "session-b"]); @@ -1785,7 +2059,7 @@ mod tests { .unwrap(); // Bridge entry text "Continue the fix" should NOT appear - for turn in &view.turns { + for turn in view.turns() { assert_ne!(turn.text, "Continue the fix"); } } @@ -1808,9 +2082,10 @@ mod tests { let view = ConversationProvider::load_conversation(&provider, "/test/project", "solo").unwrap(); - assert_eq!(view.turns.len(), 2); - assert_eq!(view.turns[0].text, "Hello"); - assert_eq!(view.turns[1].text, "Hi there!"); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 2); + assert_eq!(turns[0].text, "Hello"); + assert_eq!(turns[1].text, "Hi there!"); // Single segment — session_ids should be empty assert!(view.session_ids.is_empty()); } diff --git a/crates/toolpath-claude/src/reader.rs b/crates/toolpath-claude/src/reader.rs index 24aae75..0fd7591 100644 --- a/crates/toolpath-claude/src/reader.rs +++ b/crates/toolpath-claude/src/reader.rs @@ -116,8 +116,7 @@ impl ConversationReader { } if !entry.timestamp.is_empty() - && let Ok(timestamp) = - entry.timestamp.parse::>() + && let Ok(timestamp) = entry.timestamp.parse::>() { if started_at.is_none() || Some(timestamp) < started_at { started_at = Some(timestamp); diff --git a/crates/toolpath-claude/tests/compaction_roundtrip.rs b/crates/toolpath-claude/tests/compaction_roundtrip.rs index e728b46..dcf4dea 100644 --- a/crates/toolpath-claude/tests/compaction_roundtrip.rs +++ b/crates/toolpath-claude/tests/compaction_roundtrip.rs @@ -16,15 +16,13 @@ //! - The conversation can be re-projected to Claude JSONL and //! re-parsed by `ConversationReader` without error. //! -//! Known limitation (documented, not asserted): the `compact_boundary` -//! marker entry itself has no `message` field, so the current -//! provider drops it on the floor going Claude → IR. The synthetic -//! `isCompactSummary: true` summary entry is currently surfaced as a -//! plain `Role::User` turn — `toolpath-claude` does not yet recognize -//! the `isCompactSummary` flag. Both are acceptable losses for "good -//! UX" today (the compacted summary text still lands in the -//! transcript), but if/when we tighten this, this test gets -//! tightened with it. +//! Boundary handling — asserted in depth by `compaction_view.rs` — is now +//! first-class: the `compact_boundary` marker is read as an +//! `Item::Compaction`, and the synthetic `isCompactSummary: true` entry is +//! folded into that boundary's `summary` rather than surfaced as a +//! `Role::User` turn. This test covers the complement: the *surrounding* +//! content — the pre- and post-compact turns and their tool-call pairs — +//! surviving the derive → project → re-read round-trip intact. use std::path::{Path, PathBuf}; @@ -59,7 +57,7 @@ fn ir_roundtrip(view: &ConversationView) -> ConversationView { fn fixture_loads_without_panic() { let view = load_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "compaction fixture should produce turns" ); } @@ -73,29 +71,22 @@ fn pre_compact_content_survives_roundtrip() { let pre_assistant_text = "I'll start by reading the current auth code."; assert!( - original - .turns - .iter() - .any(|t| t.text.contains(pre_user_text)), + original.turns().any(|t| t.text.contains(pre_user_text)), "pre-compact user prompt missing from initial view" ); assert!( original - .turns - .iter() + .turns() .any(|t| t.text.contains(pre_assistant_text)), "pre-compact assistant response missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(pre_user_text)), + after.turns().any(|t| t.text.contains(pre_user_text)), "pre-compact user prompt dropped after roundtrip" ); assert!( - after - .turns - .iter() - .any(|t| t.text.contains(pre_assistant_text)), + after.turns().any(|t| t.text.contains(pre_assistant_text)), "pre-compact assistant response dropped after roundtrip" ); } @@ -111,11 +102,11 @@ fn post_compact_content_survives_roundtrip() { for needle in [post_user_text, post_assistant_text, post_summary_text] { assert!( - original.turns.iter().any(|t| t.text.contains(needle)), + original.turns().any(|t| t.text.contains(needle)), "post-compact text {needle:?} missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(needle)), + after.turns().any(|t| t.text.contains(needle)), "post-compact text {needle:?} dropped after roundtrip" ); } @@ -128,13 +119,11 @@ fn pre_compact_tool_call_pairs_survive_roundtrip() { let target_id = "t-pre-1"; let original_tool = original - .turns - .iter() + .turns() .find_map(|t| t.tool_uses.iter().find(|tu| tu.id == target_id)) .expect("pre-compact tool call missing from initial view"); let after_tool = after - .turns - .iter() + .turns() .find_map(|t| t.tool_uses.iter().find(|tu| tu.id == target_id)) .expect("pre-compact tool call dropped after roundtrip"); @@ -157,13 +146,11 @@ fn post_compact_tool_call_pairs_survive_roundtrip() { let target_id = "t-post-1"; let original_tool = original - .turns - .iter() + .turns() .find_map(|t| t.tool_uses.iter().find(|tu| tu.id == target_id)) .expect("post-compact tool call missing from initial view"); let after_tool = after - .turns - .iter() + .turns() .find_map(|t| t.tool_uses.iter().find(|tu| tu.id == target_id)) .expect("post-compact tool call dropped after roundtrip"); @@ -208,13 +195,11 @@ fn projector_output_is_re_parseable_by_reader() { fn role_distribution_is_sane() { let view = load_view(); let user_count = view - .turns - .iter() + .turns() .filter(|t| matches!(t.role, Role::User)) .count(); let assistant_count = view - .turns - .iter() + .turns() .filter(|t| matches!(t.role, Role::Assistant)) .count(); assert!( diff --git a/crates/toolpath-claude/tests/compaction_view.rs b/crates/toolpath-claude/tests/compaction_view.rs new file mode 100644 index 0000000..c25df2e --- /dev/null +++ b/crates/toolpath-claude/tests/compaction_view.rs @@ -0,0 +1,282 @@ +//! Compaction-boundary detection: loading the real captured Claude session +//! with an inline `compact_boundary` marker should surface exactly one +//! `Item::Compaction` at its true position in the ordered item stream — the +//! boundary's `compactMetadata` becomes the `Compaction`, the synthetic +//! `isCompactSummary` entry is folded into `Compaction.summary` (not surfaced +//! as a turn), and the surrounding turns are preserved. +//! +//! The fixture is `test-fixtures/claude/convo-compacted.jsonl` — a real Claude +//! Code 2.1.x session captured while running `/compact` (manual trigger). + +use std::path::{Path, PathBuf}; + +use toolpath::v1::Graph; +use toolpath_claude::{ClaudeProjector, ConversationReader}; +use toolpath_convo::{ + CompactionTrigger, ConversationProjector, ConversationView, DeriveConfig, Item, derive_path, + extract_conversation, +}; + +/// The real captured Claude session with one manual compaction boundary. +fn fixture_path() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("..") + .join("..") + .join("test-fixtures") + .join("claude") + .join("convo-compacted.jsonl") +} + +fn load_view() -> ConversationView { + let convo = ConversationReader::read_conversation(fixture_path()).expect("read fixture"); + toolpath_claude::provider::to_view(&convo) +} + +fn ir_roundtrip(view: &ConversationView) -> ConversationView { + let path = derive_path(view, &DeriveConfig::default()); + let graph = Graph::from_path(path); + let json = graph.to_json().expect("serialize Graph"); + let back = Graph::from_json(&json).expect("parse Graph"); + let path = back.into_single_path().expect("single path"); + extract_conversation(&path) +} + +/// Project the view back into a Claude `Conversation`, then re-read it with +/// the forward path. The compaction must survive: project re-emits the +/// boundary (+ summary) entries, and `to_view` re-folds them into one +/// `Item::Compaction`. +fn project_and_reread(view: &ConversationView) -> ConversationView { + let convo = ClaudeProjector.project(view).expect("project view"); + toolpath_claude::provider::to_view(&convo) +} + +fn only_compaction(view: &ConversationView) -> &toolpath_convo::Compaction { + let compactions: Vec<_> = view.items.iter().filter_map(Item::as_compaction).collect(); + assert_eq!( + compactions.len(), + 1, + "expected exactly one Item::Compaction, found {}", + compactions.len() + ); + compactions[0] +} + +#[test] +fn boundary_becomes_single_compaction_item_with_expected_fields() { + let view = load_view(); + let c = only_compaction(&view); + + assert_eq!( + c.trigger, + Some(CompactionTrigger::Manual), + "fixture ran /compact (manual trigger)" + ); + assert!( + c.summary.is_some(), + "summary folded from isCompactSummary entry" + ); + assert!( + c.pre_tokens.is_some(), + "preTokens carried from compactMetadata" + ); + assert_eq!( + c.kept, + vec![ + "8a1c3178-ba2b-43cc-a376-3ad159a03d25".to_string(), + "1b85db73-91ac-4095-a45e-6feb3e495282".to_string(), + ], + "kept = the de-duplicated union of preservedMessages.uuids and the \ + re-emitted (replayed) set; this fixture has no re-emission, so it's \ + exactly the two preserved-tail uuids" + ); + assert!( + c.parent_id.is_some(), + "logicalParentUuid maps to the compaction's parent" + ); +} + +#[test] +fn compaction_lands_between_surrounding_turns() { + let view = load_view(); + + let compaction_pos = view + .items + .iter() + .position(|i| matches!(i, Item::Compaction(_))) + .expect("compaction present"); + + // There must be at least one turn before and after the boundary. + let turns_before = view.items[..compaction_pos] + .iter() + .filter(|i| matches!(i, Item::Turn(_))) + .count(); + let turns_after = view.items[compaction_pos + 1..] + .iter() + .filter(|i| matches!(i, Item::Turn(_))) + .count(); + assert!(turns_before > 0, "pre-compaction turns missing"); + assert!(turns_after > 0, "post-compaction turns missing"); +} + +#[test] +fn summary_entry_is_not_surfaced_as_a_turn() { + let view = load_view(); + let c = only_compaction(&view); + let summary = c.summary.as_deref().expect("summary present"); + + // The synthetic summary's text must live on the Compaction, not on any + // turn (it was folded, not emitted). + let summary_head = &summary[..summary.len().min(60)]; + for turn in view.turns() { + assert!( + !turn.text.contains(summary_head), + "summary text leaked into a turn: {:?}", + turn.id + ); + } +} + +#[test] +fn compaction_roundtrips_through_derive_and_extract() { + let original = load_view(); + let orig_c = only_compaction(&original).clone(); + + let after = ir_roundtrip(&original); + let after_c = only_compaction(&after); + + assert_eq!(after_c.id, orig_c.id, "compaction id diverged"); + assert_eq!(after_c.trigger, orig_c.trigger, "trigger diverged"); + assert_eq!(after_c.summary, orig_c.summary, "summary diverged"); + assert_eq!(after_c.pre_tokens, orig_c.pre_tokens, "pre_tokens diverged"); + assert_eq!(after_c.kept, orig_c.kept, "kept ranges diverged"); + assert_eq!(after_c.parent_id, orig_c.parent_id, "parent_id diverged"); +} + +#[test] +fn surrounding_turns_survive_roundtrip() { + let original = load_view(); + let after = ir_roundtrip(&original); + + // A turn from before the boundary and one from after should both survive + // the derive→extract roundtrip. Use the first and last user turns as + // stable anchors keyed by id. + let orig_turn_ids: Vec = original.turns().map(|t| t.id.clone()).collect(); + assert!(orig_turn_ids.len() >= 2, "need at least two turns to test"); + + let after_turn_ids: std::collections::HashSet = + after.turns().map(|t| t.id.clone()).collect(); + + let first = &orig_turn_ids[0]; + let last = orig_turn_ids.last().unwrap(); + assert!( + after_turn_ids.contains(first), + "first turn {first} dropped after roundtrip" + ); + assert!( + after_turn_ids.contains(last), + "last turn {last} dropped after roundtrip" + ); +} + +#[test] +fn compaction_survives_projection_roundtrip() { + let original = load_view(); + let orig_c = only_compaction(&original).clone(); + + // view → project (emit boundary + summary entries) → to_view (re-fold). + let after = project_and_reread(&original); + let after_c = only_compaction(&after); + + assert_eq!(after_c.trigger, orig_c.trigger, "trigger diverged"); + assert_eq!( + after_c.summary.is_some(), + orig_c.summary.is_some(), + "summary presence diverged" + ); + assert_eq!(after_c.pre_tokens, orig_c.pre_tokens, "pre_tokens diverged"); + assert_eq!(after_c.kept, orig_c.kept, "kept ranges diverged"); + + // The re-folded compaction must sit between turns, not at an edge. + let pos = after + .items + .iter() + .position(|i| matches!(i, Item::Compaction(_))) + .expect("compaction present after projection roundtrip"); + let turns_before = after.items[..pos] + .iter() + .filter(|i| matches!(i, Item::Turn(_))) + .count(); + let turns_after = after.items[pos + 1..] + .iter() + .filter(|i| matches!(i, Item::Turn(_))) + .count(); + assert!(turns_before > 0, "no pre-compaction turn after projection"); + assert!(turns_after > 0, "no post-compaction turn after projection"); + + // The summary text must not have leaked into any turn — it stays folded + // on the Compaction. + let summary = after_c.summary.as_deref().expect("summary present"); + let summary_head = &summary[..summary.len().min(60)]; + for turn in after.turns() { + assert!( + !turn.text.contains(summary_head), + "summary text leaked into a turn after projection: {:?}", + turn.id + ); + } +} + +/// The re-emission strip keeps step ids unique so `derive_path` succeeds, the +/// `Compaction.kept` set is populated, every surviving turn appears exactly +/// once, and the compaction survives a project → re-read roundtrip with the +/// same `kept`. +#[test] +fn re_emission_is_stripped_and_kept_round_trips() { + use std::collections::HashSet; + + let view = load_view(); + + // Forward: derive_path must NOT error on duplicate step ids — the + // re-emitted (duplicate-uuid) entries were stripped during `to_view`. + let path = derive_path(&view, &DeriveConfig::default()); + let mut ids = HashSet::new(); + for step in &path.steps { + assert!( + ids.insert(step.step.id.clone()), + "duplicate step id leaked through: {}", + step.step.id + ); + } + + // Every turn in the view appears exactly once (re-emission stripped). + let mut turn_ids = HashSet::new(); + for turn in view.turns() { + assert!( + turn_ids.insert(turn.id.clone()), + "turn {} appears more than once — re-emission not stripped", + turn.id + ); + } + + // kept is populated. + let c = only_compaction(&view); + assert!(!c.kept.is_empty(), "Compaction.kept should be populated"); + + // Reverse: project (re-synthesizing the replay block) → re-read. The + // compaction survives with the same `kept`, and re-reading still produces + // unique step ids (the re-synthesized replay block is stripped again). + let after = project_and_reread(&view); + let after_c = only_compaction(&after); + assert_eq!(after_c.kept, c.kept, "kept diverged after projection"); + + let path2 = + derive_path(&after, &DeriveConfig::default()); + let mut ids2 = HashSet::new(); + for step in &path2.steps { + assert!( + ids2.insert(step.step.id.clone()), + "duplicate step id after projection roundtrip: {}", + step.step.id + ); + } +} diff --git a/crates/toolpath-claude/tests/real_fixture_roundtrip.rs b/crates/toolpath-claude/tests/real_fixture_roundtrip.rs index db9a55d..cd3d4b8 100644 --- a/crates/toolpath-claude/tests/real_fixture_roundtrip.rs +++ b/crates/toolpath-claude/tests/real_fixture_roundtrip.rs @@ -61,10 +61,7 @@ fn is_system_envelope(turn: &Turn) -> bool { } fn meaningful(view: &ConversationView) -> Vec<&Turn> { - view.turns - .iter() - .filter(|t| !is_system_envelope(t)) - .collect() + view.turns().filter(|t| !is_system_envelope(t)).collect() } fn norm(s: &str) -> String { @@ -75,7 +72,7 @@ fn norm(s: &str) -> String { fn fixture_loads() { let view = load_fixture_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "claude fixture should produce a non-empty view" ); let m = meaningful(&view); @@ -189,14 +186,14 @@ fn roundtrip_preserves_delegations() { let original = load_fixture_view(); let after = ir_roundtrip(&original); - let total_before: usize = original.turns.iter().map(|t| t.delegations.len()).sum(); - let total_after: usize = after.turns.iter().map(|t| t.delegations.len()).sum(); + let total_before: usize = original.turns().map(|t| t.delegations.len()).sum(); + let total_after: usize = after.turns().map(|t| t.delegations.len()).sum(); assert_eq!( total_before, total_after, "total delegation count diverged: {total_before} → {total_after}" ); - for (i, (a, b)) in original.turns.iter().zip(after.turns.iter()).enumerate() { + for (i, (a, b)) in original.turns().zip(after.turns()).enumerate() { assert_eq!( a.delegations.len(), b.delegations.len(), diff --git a/crates/toolpath-cli/Cargo.toml b/crates/toolpath-cli/Cargo.toml index da3b709..dd3ddc7 100644 --- a/crates/toolpath-cli/Cargo.toml +++ b/crates/toolpath-cli/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-cli" -version = "0.14.0" +version = "0.15.0" edition = "2024" license = "Apache-2.0" repository = "https://github.com/empathic/toolpath" @@ -14,7 +14,7 @@ name = "path" path = "src/main.rs" [dependencies] -path-cli = { path = "../path-cli", version = "0.14.0" } +path-cli = { path = "../path-cli", version = "0.15.0" } anyhow = "1.0" [workspace] diff --git a/crates/toolpath-codex/Cargo.toml b/crates/toolpath-codex/Cargo.toml index edf70bb..1df0dc1 100644 --- a/crates/toolpath-codex/Cargo.toml +++ b/crates/toolpath-codex/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-codex" -version = "0.6.0" +version = "0.7.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-codex/src/derive.rs b/crates/toolpath-codex/src/derive.rs index e48cbb1..6b30617 100644 --- a/crates/toolpath-codex/src/derive.rs +++ b/crates/toolpath-codex/src/derive.rs @@ -46,7 +46,10 @@ pub fn derive_path(session: &Session, config: &DeriveConfig) -> Path { } /// Derive a [`Path`] from multiple sessions. Used for bulk exports. -pub fn derive_project(sessions: &[Session], config: &DeriveConfig) -> Vec { +pub fn derive_project( + sessions: &[Session], + config: &DeriveConfig, +) -> Vec { sessions.iter().map(|s| derive_path(s, config)).collect() } @@ -173,7 +176,8 @@ mod tests { fn derive_project_per_session() { let (_t, mgr, id) = fixture_session(&minimal_body()); let s1 = mgr.read_session(&id).unwrap(); - let paths = derive_project(std::slice::from_ref(&s1), &DeriveConfig::default()); + let paths = + derive_project(std::slice::from_ref(&s1), &DeriveConfig::default()); assert_eq!(paths.len(), 1); } } diff --git a/crates/toolpath-codex/src/lib.rs b/crates/toolpath-codex/src/lib.rs index e15cf78..cc07a37 100644 --- a/crates/toolpath-codex/src/lib.rs +++ b/crates/toolpath-codex/src/lib.rs @@ -12,10 +12,10 @@ pub use io::ConvoIO; pub use paths::PathResolver; pub use reader::RolloutReader; pub use types::{ - BaseInstructions, ContentPart, CustomToolCall, CustomToolCallOutput, EventMsg, ExecCommandEnd, - FunctionCall, FunctionCallOutput, GitInfo, Message, PatchApplyEnd, PatchChange, Reasoning, - ResponseItem, RolloutItem, RolloutLine, SandboxPolicy, Session, SessionMeta, SessionMetadata, - TokenCountEvent, TokenCountInfo, TokenUsage, TurnContext, + BaseInstructions, CompactedItem, ContentPart, CustomToolCall, CustomToolCallOutput, EventMsg, + ExecCommandEnd, FunctionCall, FunctionCallOutput, GitInfo, Message, PatchApplyEnd, PatchChange, + Reasoning, ResponseItem, RolloutItem, RolloutLine, SandboxPolicy, Session, SessionMeta, + SessionMetadata, TokenCountEvent, TokenCountInfo, TokenUsage, TurnContext, }; pub mod provider; diff --git a/crates/toolpath-codex/src/project.rs b/crates/toolpath-codex/src/project.rs index 2ec3dd5..291d26b 100644 --- a/crates/toolpath-codex/src/project.rs +++ b/crates/toolpath-codex/src/project.rs @@ -30,12 +30,13 @@ use std::path::PathBuf; use serde_json::{Map, Value, json}; use toolpath_convo::{ - ConversationProjector, ConversationView, ConvoError, Result, Role, ToolInvocation, Turn, + Compaction, ConversationProjector, ConversationView, ConvoError, Item, Result, Role, + ToolInvocation, Turn, }; use crate::types::{ - ContentPart, CustomToolCall, CustomToolCallOutput, FunctionCall, FunctionCallOutput, Message, - Reasoning, RolloutLine, SessionMeta, TurnContext, + CompactedItem, ContentPart, CustomToolCall, CustomToolCallOutput, FunctionCall, + FunctionCallOutput, Message, Reasoning, RolloutLine, SessionMeta, TurnContext, }; // ── CodexProjector ─────────────────────────────────────────────────── @@ -116,8 +117,7 @@ fn project_view( .cwd .clone() .or_else(|| { - view.turns - .iter() + view.turns() .find_map(|t| t.environment.as_ref()?.working_dir.clone()) }) .unwrap_or_else(|| "/".to_string()); @@ -125,13 +125,13 @@ fn project_view( let model = cfg .model .clone() - .or_else(|| view.turns.iter().find_map(|t| t.model.clone())) + .or_else(|| view.turns().find_map(|t| t.model.clone())) .unwrap_or_else(|| "unknown".to_string()); let session_timestamp = view .started_at .map(|t| t.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)) - .or_else(|| view.turns.first().map(|t| t.timestamp.clone())) + .or_else(|| view.turns().next().map(|t| t.timestamp.clone())) .unwrap_or_else(|| "1970-01-01T00:00:00.000Z".to_string()); let mut lines: Vec = Vec::new(); @@ -141,9 +141,11 @@ fn project_view( // Find the last assistant turn so we can mark it `phase: "final"`. // Codex annotates every other assistant turn with `phase: "commentary"`, - // matching what real rollouts look like. + // matching what real rollouts look like. Indexed over the turn stream + // (events/compactions don't count), matching `turn_idx` below. let last_assistant_idx = view - .turns + .turns() + .collect::>() .iter() .rposition(|t| matches!(t.role, Role::Assistant)); @@ -162,13 +164,17 @@ fn project_view( // grouping survive the round-trip — the reader keys `Turn.group_id` // off the turn_context `turn_id`. let first_group = view - .turns - .iter() + .turns() .enumerate() .find(|(_, t)| matches!(t.role, Role::Assistant)) .map(|(i, t)| group_of(i, t)) .unwrap_or_else(|| view.id.clone()); - lines.push(make_turn_context_line(&first_group, &session_timestamp, &cwd, &model)); + lines.push(make_turn_context_line( + &first_group, + &session_timestamp, + &cwd, + &model, + )); let mut current_group = Some(first_group); // Running session-cumulative usage. Codex's `total_token_usage` is @@ -176,17 +182,50 @@ fn project_view( // emit it after the turn, so a re-read differences it back to the same // per-step spend. let mut running = toolpath_convo::TokenUsage::default(); - for (idx, turn) in view.turns.iter().enumerate() { - if matches!(turn.role, Role::Assistant) { - let group = group_of(idx, turn); - if current_group.as_deref() != Some(&group) { - lines.push(make_turn_context_line(&group, &turn.timestamp, &cwd, &model)); - current_group = Some(group); + + // Last turn-index of each message group, so a group whose total is + // repeated on every member turn (e.g. a Gemini split message) advances + // `running` exactly once — on the group's final turn — rather than once + // per member, which would double-count. + let group_last_idx = group_last_indices(view); + + // Walk the full ordered item stream so compaction boundaries land at + // their true position between the surrounding turns. Events have no + // Codex analog on the return path and are dropped; turns and compactions + // both project to rollout lines. + let mut turn_idx = 0usize; + for item in &view.items { + match item { + Item::Turn(turn) => { + if matches!(turn.role, Role::Assistant) { + let group = group_of(turn_idx, turn); + if current_group.as_deref() != Some(&group) { + lines.push(make_turn_context_line( + &group, + &turn.timestamp, + &cwd, + &model, + )); + current_group = Some(group); + } + } + let codex = codex_extras(turn).cloned().unwrap_or_default(); + let is_final_assistant = Some(turn_idx) == last_assistant_idx; + let contribution = running_contribution(turn, turn_idx, &group_last_idx); + emit_turn_lines( + turn, + &codex, + is_final_assistant, + contribution, + &cwd, + &mut lines, + &mut running, + ); + turn_idx += 1; } + Item::Compaction(c) => emit_compaction(c, &mut lines), + Item::Event(_) => {} } - let codex = codex_extras(turn).cloned().unwrap_or_default(); - let is_final_assistant = Some(idx) == last_assistant_idx; - emit_turn_lines(turn, &codex, is_final_assistant, &cwd, &mut lines, &mut running); } Ok(crate::types::Session { @@ -196,6 +235,43 @@ fn project_view( }) } +/// Last turn-index of each message group, indexed over the turn stream +/// (matching `turn_idx` in `project_view`). A turn with no `group_id` is its +/// own group and is omitted. +fn group_last_indices(view: &ConversationView) -> HashMap { + let mut last: HashMap = HashMap::new(); + for (idx, turn) in view.turns().enumerate() { + if let Some(mid) = &turn.group_id { + last.insert(mid.clone(), idx); + } + } + last +} + +/// The per-step usage this turn contributes to the session-cumulative +/// `running` counter (and thus the `token_count` line emitted after it): +/// +/// - `attributed_token_usage` when the source reports a per-step split +/// (codex-native): each member of a group carries its own slice, summing +/// to the group total. +/// - otherwise the turn's `token_usage` — but for a grouped turn whose total +/// is repeated on every member (no per-step split), only on the group's +/// last turn, so the message total is counted once, not once per member. +fn running_contribution<'a>( + turn: &'a Turn, + turn_idx: usize, + group_last_idx: &HashMap, +) -> Option<&'a toolpath_convo::TokenUsage> { + if let Some(attr) = turn.attributed_token_usage.as_ref() { + return Some(attr); + } + let usage = turn.token_usage.as_ref()?; + match &turn.group_id { + Some(mid) if group_last_idx.get(mid) != Some(&turn_idx) => None, + _ => Some(usage), + } +} + fn make_session_meta_line( cfg: &CodexProjector, view: &ConversationView, @@ -234,12 +310,7 @@ fn make_session_meta_line( } } -fn make_turn_context_line( - turn_id: &str, - timestamp: &str, - cwd: &str, - model: &str, -) -> RolloutLine { +fn make_turn_context_line(turn_id: &str, timestamp: &str, cwd: &str, model: &str) -> RolloutLine { let tc = TurnContext { turn_id: turn_id.to_string(), cwd: PathBuf::from(cwd), @@ -268,19 +339,46 @@ fn codex_extras(_turn: &Turn) -> Option<&'static Map> { None } +/// Emit a `compacted` rollout line for a [`Compaction`] boundary — the +/// inverse of `Builder::handle_compacted`. Codex's payload is +/// `{message, replacement_history?, window_id?}`; only `summary` survives +/// the forward path, so we round-trip it as `message` (defaulting to the +/// empty string) and leave the other fields absent. +fn emit_compaction(c: &Compaction, lines: &mut Vec) { + let payload = CompactedItem { + message: c.summary.clone().unwrap_or_default(), + replacement_history: None, + window_id: None, + extra: HashMap::new(), + }; + lines.push(RolloutLine { + timestamp: c.timestamp.clone(), + kind: "compacted".to_string(), + payload: serde_json::to_value(&payload).unwrap_or(Value::Null), + extra: HashMap::new(), + }); +} + fn emit_turn_lines( turn: &Turn, codex: &Map, is_final_assistant: bool, + contribution: Option<&toolpath_convo::TokenUsage>, session_cwd: &str, lines: &mut Vec, running: &mut toolpath_convo::TokenUsage, ) { match &turn.role { Role::User => emit_user_message(turn, lines), - Role::Assistant => { - emit_assistant(turn, codex, is_final_assistant, session_cwd, lines, running) - } + Role::Assistant => emit_assistant( + turn, + codex, + is_final_assistant, + contribution, + session_cwd, + lines, + running, + ), Role::System => emit_developer_message(turn, lines), Role::Other(_) => { // Unknown roles don't have a clean Codex analog; emit them @@ -350,6 +448,7 @@ fn emit_assistant( turn: &Turn, codex: &Map, is_final_assistant: bool, + contribution: Option<&toolpath_convo::TokenUsage>, session_cwd: &str, lines: &mut Vec, running: &mut toolpath_convo::TokenUsage, @@ -459,15 +558,12 @@ fn emit_assistant( } // Advance the session-cumulative counter by this step's contribution - // (its attributed per-step spend, or its group total when no per-step - // split exists), then emit `token_count` AFTER the turn — the reader - // differences the cumulative and attributes the delta to the step it - // follows. Mirrors how real Codex streams cumulative counts per step. - if let Some(contribution) = turn - .attributed_token_usage - .as_ref() - .or(turn.token_usage.as_ref()) - { + // (its attributed per-step spend, or its group total once when no + // per-step split exists — see `running_contribution`), then emit + // `token_count` AFTER the turn — the reader differences the cumulative + // and attributes the delta to the step it follows. Mirrors how real + // Codex streams cumulative counts per step. + if let Some(contribution) = contribution { add_codex_usage(running, contribution); lines.push(event_msg_line( &turn.timestamp, @@ -493,6 +589,15 @@ fn add_codex_usage(acc: &mut toolpath_convo::TokenUsage, delta: &toolpath_convo: add(&mut acc.output_tokens, delta.output_tokens); add(&mut acc.cache_read_tokens, delta.cache_read_tokens); add(&mut acc.cache_write_tokens, delta.cache_write_tokens); + // Breakdowns (e.g. output→reasoning) are cumulative subsets of their + // parent class; accumulate them the same way so the running cumulative + // carries reasoning and `convo_usage_to_codex_json` can re-emit it. + for (class, inner) in &delta.breakdowns { + let acc_inner = acc.breakdowns.entry(class.clone()).or_default(); + for (sub, v) in inner { + *acc_inner.entry(sub.clone()).or_insert(0) += v; + } + } } fn emit_tool_call( @@ -687,6 +792,12 @@ fn convo_usage_to_codex_json(u: &toolpath_convo::TokenUsage) -> Value { if let Some(v) = u.output_tokens { m.insert("output_tokens".to_string(), Value::from(v)); } + // `reasoning_output_tokens` ⊆ `output_tokens`; the reader differences it + // into `breakdowns["output"]["reasoning"]`, so emit it back from there to + // round-trip reasoning provenance (kind v1.2.0). + if let Some(r) = u.breakdowns.get("output").and_then(|m| m.get("reasoning")) { + m.insert("reasoning_output_tokens".to_string(), Value::from(*r)); + } Value::Object(m) } @@ -748,12 +859,11 @@ mod tests { id: "session-uuid".into(), started_at: None, last_activity: None, - turns, + items: turns.into_iter().map(toolpath_convo::Item::Turn).collect(), total_usage: None, provider_id: Some("codex".into()), files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } @@ -1000,6 +1110,95 @@ mod tests { } } + #[test] + fn grouped_turns_repeating_one_total_advance_running_once() { + // A message split into two turns that each repeat the SAME total + // (no per-step attribution — e.g. a Gemini split message) must + // advance the cumulative `token_count` ONCE (the group's total), + // not once per member. Otherwise a re-read differences a doubled + // cumulative and over-attributes. + let mut a1 = assistant_turn("a1", ""); + a1.group_id = Some("g".into()); + a1.token_usage = Some(TokenUsage { + input_tokens: Some(100), + output_tokens: Some(20), + ..Default::default() + }); + let mut a2 = assistant_turn("a2", "answer"); + a2.group_id = Some("g".into()); + a2.token_usage = Some(TokenUsage { + input_tokens: Some(100), + output_tokens: Some(20), + ..Default::default() + }); + + let s = CodexProjector::default() + .project(&view_with(vec![a1, a2])) + .unwrap(); + + // Every token_count is cumulative; the last one is the session total. + let totals: Vec<&Value> = s + .lines + .iter() + .filter(|l| l.payload.get("type").and_then(Value::as_str) == Some("token_count")) + .map(|l| &l.payload["info"]["total_token_usage"]) + .collect(); + assert!(!totals.is_empty(), "expected at least one token_count"); + let last = totals.last().unwrap(); + // Counted once: 20, not 40. + assert_eq!(last["output_tokens"], 20); + assert_eq!(last["input_tokens"], 100); + } + + #[test] + fn reasoning_breakdown_round_trips_through_projection() { + use std::collections::BTreeMap; + // A turn whose token_usage records a reasoning breakdown must project + // to a `token_count` carrying `reasoning_output_tokens`, and re-read + // back into the IR breakdown — otherwise reasoning provenance is lost + // on an IR→Codex→IR round-trip. + let mut breakdowns = BTreeMap::new(); + breakdowns.insert( + "output".to_string(), + BTreeMap::from([("reasoning".to_string(), 30u32)]), + ); + let mut a = assistant_turn("a1", "answer"); + a.token_usage = Some(TokenUsage { + input_tokens: Some(100), + output_tokens: Some(80), + breakdowns, + ..Default::default() + }); + + let s = CodexProjector::default() + .project(&view_with(vec![a])) + .unwrap(); + + // Projection: the emitted cumulative token_count carries reasoning. + let tc = s + .lines + .iter() + .rfind(|l| l.payload.get("type").and_then(Value::as_str) == Some("token_count")) + .expect("a token_count line"); + assert_eq!( + tc.payload["info"]["total_token_usage"]["reasoning_output_tokens"], + 30 + ); + + // Re-read: the breakdown survives back into the IR. + let view = crate::to_view(&s); + let turn = view + .turns() + .find(|t| t.role == Role::Assistant) + .expect("assistant turn"); + let usage = turn + .token_usage + .as_ref() + .or(turn.attributed_token_usage.as_ref()) + .expect("usage survives"); + assert_eq!(usage.breakdowns["output"]["reasoning"], 30); + } + #[test] fn jsonl_serializes_one_line_per_entry() { let s = CodexProjector::default() diff --git a/crates/toolpath-codex/src/provider.rs b/crates/toolpath-codex/src/provider.rs index 00bcdf4..aafe1af 100644 --- a/crates/toolpath-codex/src/provider.rs +++ b/crates/toolpath-codex/src/provider.rs @@ -41,9 +41,9 @@ use crate::types::{ }; use serde_json::Value; use toolpath_convo::{ - ConversationEvent, ConversationMeta, ConversationProvider, ConversationView, ConvoError, - EnvironmentSnapshot, FileMutation, ProducerInfo, Role, SessionBase, TokenUsage, ToolCategory, - ToolInvocation, ToolResult, Turn, + Compaction, ConversationEvent, ConversationMeta, ConversationProvider, ConversationView, + ConvoError, EnvironmentSnapshot, FileMutation, Item, ProducerInfo, Role, SessionBase, + TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, }; /// Provider for Codex sessions. @@ -182,10 +182,26 @@ pub fn to_turn(line_payload: &ResponseItem) -> Option { } } +/// A `compacted` marker captured during the walk, to be slotted into the +/// turn stream at assembly time. `prev_turn` is the buffer index of the +/// turn that immediately preceded the marker (`None` if it came before any +/// turn); the compaction is emitted right after that turn so it lands at +/// its true position between the turns it separates. +struct PendingCompaction { + compaction: Compaction, + prev_turn: Option, +} + struct Builder<'a> { session: &'a Session, turns: Vec, events: Vec, + /// `compacted` markers captured in source order, slotted into the turn + /// stream during assembly. Events stay grouped after the turns (as + /// before); only the compaction boundary moves into the turn sequence. + pending_compactions: Vec, + /// Running count of `compacted` markers, used to synthesize stable ids. + compact_count: usize, /// Plaintext reasoning summaries (rare — only in configurations where /// OpenAI exposes public reasoning). These land on `Turn.thinking`. pending_reasoning_plaintext: Vec, @@ -211,6 +227,8 @@ impl<'a> Builder<'a> { session, turns: Vec::new(), events: Vec::new(), + pending_compactions: Vec::new(), + compact_count: 0, pending_reasoning_plaintext: Vec::new(), current_round_id: None, pending_attributed: None, @@ -259,8 +277,7 @@ impl<'a> Builder<'a> { .push(event_from_raw(&line.timestamp, "session_state", &payload)); } RolloutItem::Compacted(payload) => { - self.events - .push(event_from_raw(&line.timestamp, "compacted", &payload)); + self.handle_compacted(&line.timestamp, &payload); } RolloutItem::Unknown { kind, payload } => { self.events @@ -309,36 +326,94 @@ impl<'a> Builder<'a> { // Filter empty carrier turns (no text, no thinking, no tool calls). // Previously done inside `derive_path_from_view`; moved here so the - // canonical `derive_path` sees only meaningful turns. - self.turns - .retain(|t| !(t.text.is_empty() && t.thinking.is_none() && t.tool_uses.is_empty())); + // canonical `derive_path` sees only meaningful turns. We compute a + // keep-mask instead of `retain`-ing in place so the buffer indices + // recorded for pending compactions stay valid. A turn that carries + // token accounting is NOT empty: `finalize_usage` (above) may have + // stamped a group's total `token_usage` onto an otherwise-bare + // group-final turn, and dropping it would make Σ token_usage < the + // session total. + let keep: Vec = self + .turns + .iter() + .map(|t| { + !(t.text.is_empty() + && t.thinking.is_none() + && t.tool_uses.is_empty() + && t.token_usage.is_none() + && t.attributed_token_usage.is_none()) + }) + .collect(); - // Assign synthetic ids to turns whose source message didn't carry - // one, then link sequentially via `parent_id` so the shared + // Assign synthetic ids to surviving turns whose source message didn't + // carry one, then link them sequentially via `parent_id` so the shared // `derive_path` can walk a connected DAG. Codex turns don't carry - // explicit parent ids on the wire; this preserves the linear - // ordering the old `derive_path_from_view` produced. + // explicit parent ids on the wire; this preserves the linear ordering + // the old `derive_path_from_view` produced. Numbering follows the + // post-filter position to match the prior `retain`-then-enumerate id. + let mut surviving = 0usize; + let mut prev: Option = None; + // Final id of each surviving turn, indexed by its position in + // `self.turns`; `None` for dropped turns. Used to resolve a + // compaction's `parent_id` back to a real turn step. + let mut turn_final_id: Vec> = vec![None; self.turns.len()]; for (idx, t) in self.turns.iter_mut().enumerate() { + if !keep[idx] { + continue; + } + surviving += 1; if t.id.is_empty() { - t.id = format!("codex-turn-{:04}", idx + 1); + t.id = format!("codex-turn-{:04}", surviving); } - } - let mut prev: Option = None; - for t in self.turns.iter_mut() { if t.parent_id.is_none() { t.parent_id = prev.clone(); } prev = Some(t.id.clone()); + turn_final_id[idx] = Some(t.id.clone()); + } + + // Resolve each pending compaction's `parent_id` to the nearest + // surviving turn at or before its marker, and bucket it by that + // turn's buffer index so it can be emitted right after that turn. + // Markers before any surviving turn go into `compactions_first`. + let mut compactions_after: HashMap> = HashMap::new(); + let mut compactions_first: Vec = Vec::new(); + for pending in self.pending_compactions { + let PendingCompaction { + mut compaction, + prev_turn, + } = pending; + match resolve_surviving_turn(prev_turn, &keep) { + Some(survivor_idx) => { + compaction.parent_id = turn_final_id[survivor_idx].clone(); + compactions_after + .entry(survivor_idx) + .or_default() + .push(compaction); + } + None => { + compaction.parent_id = None; + compactions_first.push(compaction); + } + } } // Disambiguate event ids. `event_from_raw` synthesizes // `-`, which collides when codex emits // multiple events of the same type at the same timestamp (rare // but real). Suffix duplicates with their position so each step - // gets a unique ID. + // gets a unique id. Compaction ids (`compact-`) are unique by + // construction and are reserved here so events can't collide with + // them either. let mut seen: std::collections::HashSet = std::collections::HashSet::new(); - for t in &self.turns { - seen.insert(t.id.clone()); + for id in turn_final_id.iter().flatten() { + seen.insert(id.clone()); + } + for c in compactions_first + .iter() + .chain(compactions_after.values().flatten()) + { + seen.insert(c.id.clone()); } for (i, e) in self.events.iter_mut().enumerate() { if !seen.insert(e.id.clone()) { @@ -347,11 +422,29 @@ impl<'a> Builder<'a> { } } + // Assemble the ordered stream: surviving turns first (with each + // compaction slotted in right after the turn it follows), then all + // events. Keeping events grouped after the turns reproduces the + // former layout, so the derived DAG stays a single connected + // ancestry; only the compaction boundary moves into the turn stream. + let mut items: Vec = Vec::with_capacity(self.turns.len() + self.events.len() + 1); + items.extend(compactions_first.into_iter().map(Item::Compaction)); + for (idx, turn) in self.turns.into_iter().enumerate() { + if !keep[idx] { + continue; + } + items.push(Item::Turn(turn)); + if let Some(cs) = compactions_after.remove(&idx) { + items.extend(cs.into_iter().map(Item::Compaction)); + } + } + items.extend(self.events.into_iter().map(Item::Event)); + ConversationView { id: self.session.id.clone(), started_at: self.session.started_at(), last_activity: self.session.last_activity(), - turns: self.turns, + items, total_usage: if self.total_usage_set { Some(self.total_usage) } else { @@ -360,7 +453,6 @@ impl<'a> Builder<'a> { provider_id: Some("codex".into()), files_changed: self.files_changed_order, session_ids: vec![], - events: self.events, base, producer, } @@ -610,6 +702,40 @@ impl<'a> Builder<'a> { self.turns.push(turn); } + /// Map a Codex `compacted` marker to a [`Compaction`], recorded for + /// slotting into the turn stream at its source position. Codex's payload + /// is `{message, replacement_history?, window_id?}` — only `message` is + /// consumed (as `summary`). The trigger (manual vs. auto) and + /// pre-compaction token count are never persisted to the rollout, and + /// `replacement_history` is a wholesale replacement we don't fold in, so + /// `trigger`/`pre_tokens` are `None` and `kept` is empty. The marker + /// carries no id of its own, so we synthesize a stable `compact-`. + /// See `docs/agents/formats/codex.md`. + fn handle_compacted(&mut self, timestamp: &str, payload: &Value) { + self.compact_count += 1; + let summary = payload + .get("message") + .and_then(Value::as_str) + .map(str::to_string); + let compaction = Compaction { + id: format!("compact-{}", self.compact_count), + parent_id: None, + timestamp: timestamp.to_string(), + trigger: None, + summary, + pre_tokens: None, + kept: Vec::new(), + }; + // Buffer index of the last turn pushed before this marker; the + // compaction is slotted right after it during assembly, and its + // final id becomes the compaction's `parent_id`. + let prev_turn = self.turns.len().checked_sub(1); + self.pending_compactions.push(PendingCompaction { + compaction, + prev_turn, + }); + } + fn drain_pending_onto(&mut self, turn: &mut Turn) { if turn.role != Role::Assistant { return; @@ -622,7 +748,11 @@ impl<'a> Builder<'a> { // A step's spend that arrived before any assistant turn existed // attaches to this, the first one. if let Some(pending) = self.pending_attributed.take() { - add_usage(turn.attributed_token_usage.get_or_insert_with(TokenUsage::default), &pending); + add_usage( + turn.attributed_token_usage + .get_or_insert_with(TokenUsage::default), + &pending, + ); } } @@ -690,9 +820,7 @@ impl<'a> Builder<'a> { let start = k; let mid = self.turns[assistants[k]].group_id.clone(); if mid.is_some() { - while k + 1 < assistants.len() - && self.turns[assistants[k + 1]].group_id == mid - { + while k + 1 < assistants.len() && self.turns[assistants[k + 1]].group_id == mid { k += 1; } } @@ -717,6 +845,21 @@ impl<'a> Builder<'a> { } } +/// Walk backward from a compaction's recorded predecessor-turn buffer index +/// to the nearest surviving turn at or before it. Empty carrier turns are +/// filtered out before assembly, so a dropped predecessor falls through to +/// the real prior turn the boundary hangs off of. `None` means the marker +/// preceded every surviving turn. +fn resolve_surviving_turn(prev_turn: Option, keep: &[bool]) -> Option { + let mut idx = prev_turn?; + loop { + if keep.get(idx).copied().unwrap_or(false) { + return Some(idx); + } + idx = idx.checked_sub(1)?; + } +} + // ── Patch → FileMutation conversion ───────────────────────────────── fn patch_change_to_file_mutation(path: &str, change: &PatchChange) -> FileMutation { @@ -1069,6 +1212,43 @@ mod tests { .join("\n") } + #[test] + fn empty_group_final_assistant_turn_keeps_its_usage() { + // A round whose only assistant message is empty (no text, no tools) + // but which still incurs token spend. `finalize_usage` stamps the + // round total onto that turn; the empty-carrier keep-mask must NOT + // drop it, or the spend disappears from per-step accounting while the + // session total still counts it (Σ token_usage < session total). + let body = [ + r#"{"timestamp":"2026-04-20T16:44:37.772Z","type":"session_meta","payload":{"id":"019dabc6-8fef-7681-a054-b5bb75fcb97d","timestamp":"2026-04-20T16:43:30.171Z","cwd":"/tmp/proj","originator":"codex-tui","cli_version":"0.118.0","source":"cli"}}"#, + r#"{"timestamp":"2026-04-20T16:44:37.773Z","type":"turn_context","payload":{"turn_id":"t1","cwd":"/tmp/proj","model":"gpt-5.4"}}"#, + r#"{"timestamp":"2026-04-20T16:44:37.800Z","type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"go"}]}}"#, + r#"{"timestamp":"2026-04-20T16:44:38.800Z","type":"event_msg","payload":{"type":"token_count","info":{"total_token_usage":{"input_tokens":100,"output_tokens":20,"cached_input_tokens":0,"total_tokens":120}}}}"#, + r#"{"timestamp":"2026-04-20T16:44:38.900Z","type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":""}],"phase":"final","end_turn":true}}"#, + r#"{"timestamp":"2026-04-20T16:44:39.000Z","type":"event_msg","payload":{"type":"task_complete","turn_id":"t1","last_agent_message":""}}"#, + ] + .join("\n"); + + let (_t, mgr, id) = setup_session_fixture(&body); + let session = mgr.read_session(&id).unwrap(); + let view = to_view(&session); + + let assistant = view + .turns() + .find(|t| t.role == Role::Assistant) + .expect("empty assistant turn carrying the round's usage must survive"); + let usage = assistant + .token_usage + .as_ref() + .or(assistant.attributed_token_usage.as_ref()) + .expect("surviving turn must carry the round's usage"); + assert_eq!( + usage.output_tokens, + Some(20), + "the round's output spend must land on the surviving turn" + ); + } + #[test] fn build_view_basic() { let (_t, mgr, id) = setup_session_fixture(&minimal_session()); @@ -1077,12 +1257,15 @@ mod tests { assert_eq!(view.id, "019dabc6-8fef-7681-a054-b5bb75fcb97d"); assert_eq!(view.provider_id.as_deref(), Some("codex")); - assert_eq!(view.turns.len(), 3); - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[0].text, "please do a thing"); - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[1].text, "working on it"); - assert_eq!(view.turns[1].model.as_deref(), Some("gpt-5.4")); + assert_eq!(view.turns().count(), 3); + assert_eq!(view.turns().next().unwrap().role, Role::User); + assert_eq!(view.turns().next().unwrap().text, "please do a thing"); + assert_eq!(view.turns().nth(1).unwrap().role, Role::Assistant); + assert_eq!(view.turns().nth(1).unwrap().text, "working on it"); + assert_eq!( + view.turns().nth(1).unwrap().model.as_deref(), + Some("gpt-5.4") + ); } /// Two API rounds. Codex's `token_count` events carry cumulative @@ -1116,12 +1299,12 @@ mod tests { let (_t, mgr, id) = setup_session_fixture(&two_round_session(true)); let view = to_view(&mgr.read_session(&id).unwrap()); - let first = view.turns[1].token_usage.as_ref().unwrap(); + let first = view.turns().nth(1).unwrap().token_usage.as_ref().unwrap(); assert_eq!(first.input_tokens, Some(100)); assert_eq!(first.output_tokens, Some(20)); assert_eq!(first.cache_read_tokens, Some(10)); - let second = view.turns[3].token_usage.as_ref().unwrap(); + let second = view.turns().nth(3).unwrap().token_usage.as_ref().unwrap(); assert_eq!(second.input_tokens, Some(200)); assert_eq!(second.output_tokens, Some(30)); assert_eq!(second.cache_read_tokens, Some(30)); @@ -1158,14 +1341,34 @@ mod tests { let (_t, mgr, id) = setup_session_fixture(&body); let view = to_view(&mgr.read_session(&id).unwrap()); - let assistants: Vec<&Turn> = view.turns.iter().filter(|t| t.role == Role::Assistant).collect(); + let assistants: Vec<&Turn> = view.turns().filter(|t| t.role == Role::Assistant).collect(); assert_eq!(assistants.len(), 2); // Per-step attribution: 40 then 60 — NOT 80/120 (which doubling gives). - assert_eq!(assistants[0].attributed_token_usage.as_ref().unwrap().output_tokens, Some(40)); - assert_eq!(assistants[1].attributed_token_usage.as_ref().unwrap().output_tokens, Some(60)); + assert_eq!( + assistants[0] + .attributed_token_usage + .as_ref() + .unwrap() + .output_tokens, + Some(40) + ); + assert_eq!( + assistants[1] + .attributed_token_usage + .as_ref() + .unwrap() + .output_tokens, + Some(60) + ); // Σ attributed == round total on the final turn. - assert_eq!(assistants[1].token_usage.as_ref().unwrap().output_tokens, Some(100)); - let sum: u32 = assistants.iter().filter_map(|t| t.attributed_token_usage.as_ref()?.output_tokens).sum(); + assert_eq!( + assistants[1].token_usage.as_ref().unwrap().output_tokens, + Some(100) + ); + let sum: u32 = assistants + .iter() + .filter_map(|t| t.attributed_token_usage.as_ref()?.output_tokens) + .sum(); assert_eq!(sum, 100); } @@ -1201,11 +1404,17 @@ mod tests { let (_t, mgr, id) = setup_session_fixture(&body); let view = to_view(&mgr.read_session(&id).unwrap()); - let assistants: Vec<&Turn> = view.turns.iter().filter(|t| t.role == Role::Assistant).collect(); + let assistants: Vec<&Turn> = view.turns().filter(|t| t.role == Role::Assistant).collect(); assert_eq!(assistants.len(), 2); // Per-step reasoning deltas, NOT cumulative (100/260) and NOT doubled. - assert_eq!(reasoning_of(assistants[0].attributed_token_usage.as_ref()), Some(100)); - assert_eq!(reasoning_of(assistants[1].attributed_token_usage.as_ref()), Some(160)); + assert_eq!( + reasoning_of(assistants[0].attributed_token_usage.as_ref()), + Some(100) + ); + assert_eq!( + reasoning_of(assistants[1].attributed_token_usage.as_ref()), + Some(160) + ); // Round total breakdown is the sum of attributions. let round = assistants[1].token_usage.as_ref().unwrap(); assert_eq!(reasoning_of(Some(round)), Some(260)); @@ -1233,8 +1442,14 @@ mod tests { ].join("\n"); let (_t, mgr, id) = setup_session_fixture(&body); let view = to_view(&mgr.read_session(&id).unwrap()); - let a = view.turns.iter().find(|t| t.role == Role::Assistant).unwrap(); - assert!(a.attributed_token_usage.as_ref().unwrap().breakdowns.is_empty()); + let a = view.turns().find(|t| t.role == Role::Assistant).unwrap(); + assert!( + a.attributed_token_usage + .as_ref() + .unwrap() + .breakdowns + .is_empty() + ); assert!(a.token_usage.as_ref().unwrap().breakdowns.is_empty()); } @@ -1259,15 +1474,16 @@ mod tests { let (_t, mgr, id) = setup_session_fixture(&body); let view = to_view(&mgr.read_session(&id).unwrap()); - assert_eq!(view.turns.len(), 3); - assert!(view.turns[0].group_id.is_none(), "user turn ungrouped"); - assert_eq!(view.turns[1].group_id.as_deref(), Some("round-1")); - assert_eq!(view.turns[2].group_id.as_deref(), Some("round-1")); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 3); + assert!(turns[0].group_id.is_none(), "user turn ungrouped"); + assert_eq!(turns[1].group_id.as_deref(), Some("round-1")); + assert_eq!(turns[2].group_id.as_deref(), Some("round-1")); assert!( - view.turns[1].token_usage.is_none(), + turns[1].token_usage.is_none(), "interior turn of the round must not carry usage" ); - let total = view.turns[2].token_usage.as_ref().unwrap(); + let total = turns[2].token_usage.as_ref().unwrap(); assert_eq!(total.output_tokens, Some(20)); assert_eq!(total.input_tokens, Some(100)); } @@ -1279,7 +1495,7 @@ mod tests { let (_t, mgr, id) = setup_session_fixture(&two_round_session(false)); let view = to_view(&mgr.read_session(&id).unwrap()); - let second = view.turns[3].token_usage.as_ref().unwrap(); + let second = view.turns().nth(3).unwrap().token_usage.as_ref().unwrap(); assert_eq!(second.input_tokens, Some(200)); assert_eq!(second.output_tokens, Some(30)); assert_eq!(second.cache_read_tokens, Some(30)); @@ -1292,7 +1508,7 @@ mod tests { // Turn.extra was removed, encrypted ciphertext is simply dropped. let (_t, mgr, id) = setup_session_fixture(&minimal_session()); let view = to_view(&mgr.read_session(&id).unwrap()); - let assistant = &view.turns[1]; + let assistant = view.turns().nth(1).unwrap(); assert!( assistant.thinking.is_none(), "encrypted ciphertext must not appear as thinking" @@ -1312,7 +1528,7 @@ mod tests { let (_t, mgr, id) = setup_session_fixture(&body); let view = to_view(&mgr.read_session(&id).unwrap()); assert_eq!( - view.turns[0].thinking.as_deref(), + view.turns().next().unwrap().thinking.as_deref(), Some("I should check the file") ); } @@ -1321,7 +1537,7 @@ mod tests { fn function_call_pairs_with_output() { let (_t, mgr, id) = setup_session_fixture(&minimal_session()); let view = to_view(&mgr.read_session(&id).unwrap()); - let assistant = &view.turns[1]; + let assistant = view.turns().nth(1).unwrap(); assert_eq!(assistant.tool_uses.len(), 2); let exec = &assistant.tool_uses[0]; assert_eq!(exec.name, "exec_command"); @@ -1334,7 +1550,7 @@ mod tests { fn custom_tool_call_preserves_raw_input() { let (_t, mgr, id) = setup_session_fixture(&minimal_session()); let view = to_view(&mgr.read_session(&id).unwrap()); - let assistant = &view.turns[1]; + let assistant = view.turns().nth(1).unwrap(); let apply = &assistant.tool_uses[1]; assert_eq!(apply.name, "apply_patch"); assert_eq!(apply.category, Some(ToolCategory::FileWrite)); @@ -1379,15 +1595,13 @@ mod tests { // Find the turn that hosts the `apply_patch` file mutation. The // mutation's `tool_id` should link back to the apply_patch tool. let apply_patch_id = view - .turns - .iter() + .turns() .flat_map(|t| t.tool_uses.iter()) .find(|tu| tu.name == "apply_patch") .map(|tu| tu.id.clone()) .expect("apply_patch tool invocation present"); let fm = view - .turns - .iter() + .turns() .flat_map(|t| t.file_mutations.iter()) .find(|fm| fm.path == "/tmp/proj/a.rs") .expect("file mutation present"); @@ -1410,7 +1624,7 @@ mod tests { fn events_preserve_non_turn_content() { let (_t, mgr, id) = setup_session_fixture(&minimal_session()); let view = to_view(&mgr.read_session(&id).unwrap()); - let kinds: Vec<&str> = view.events.iter().map(|e| e.event_type.as_str()).collect(); + let kinds: Vec<&str> = view.events().map(|e| e.event_type.as_str()).collect(); assert!(kinds.contains(&"session_meta")); assert!(kinds.contains(&"turn_context")); assert!(kinds.contains(&"task_started")); @@ -1446,7 +1660,7 @@ mod tests { "019dabc6-8fef-7681-a054-b5bb75fcb97d", ) .unwrap(); - assert_eq!(view.turns.len(), 3); + assert_eq!(view.turns().count(), 3); } #[test] @@ -1458,6 +1672,6 @@ mod tests { .join("\n"); let (_t, mgr, id) = setup_session_fixture(&body); let view = to_view(&mgr.read_session(&id).unwrap()); - assert_eq!(view.turns[0].role, Role::System); + assert_eq!(view.turns().next().unwrap().role, Role::System); } } diff --git a/crates/toolpath-codex/src/types.rs b/crates/toolpath-codex/src/types.rs index fea3c88..7bf11da 100644 --- a/crates/toolpath-codex/src/types.rs +++ b/crates/toolpath-codex/src/types.rs @@ -630,6 +630,32 @@ pub enum PatchChange { Unknown, } +// ── Compaction marker ─────────────────────────────────────────────── + +/// Payload of a `compacted` rollout line — Codex's context-compaction +/// boundary marker. The reader treats this payload as a raw `Value` and +/// consumes only `message` (as the summary); this struct exists so the +/// projector can emit a well-shaped `compacted` line that re-parses +/// cleanly. `replacement_history` is the wholesale-replaced prefix Codex +/// records but we don't reconstruct, and `window_id` (when present) +/// identifies the context window. Both are emitted as `null`/absent. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct CompactedItem { + /// The summary text that replaced the condensed prefix. Often empty + /// in real captures. + #[serde(default)] + pub message: String, + + #[serde(default, skip_serializing_if = "Option::is_none")] + pub replacement_history: Option, + + #[serde(default, skip_serializing_if = "Option::is_none")] + pub window_id: Option, + + #[serde(flatten, skip_serializing_if = "HashMap::is_empty", default)] + pub extra: HashMap, +} + // ── Logical session wrapping ──────────────────────────────────────── /// A parsed session: the sequence of lines plus derived first-line diff --git a/crates/toolpath-codex/tests/compaction_roundtrip.rs b/crates/toolpath-codex/tests/compaction_roundtrip.rs index 6ea4c17..2abaa7b 100644 --- a/crates/toolpath-codex/tests/compaction_roundtrip.rs +++ b/crates/toolpath-codex/tests/compaction_roundtrip.rs @@ -1,26 +1,36 @@ -//! Compaction-event roundtrip: a Codex rollout that includes a -//! `compacted` line in the middle should still preserve the -//! pre-compact and post-compact conversation content through the -//! projection round-trip. +//! Compaction handling for Codex rollouts. //! -//! Synthetic fixture is justified per project policy: real compaction -//! fires when the model context window fills mid-session and can't -//! reliably be triggered by a 5-minute capture prompt. +//! Codex appends a single `compacted` line when it condenses history +//! mid-session (same file, same session id). `toolpath-codex` now maps +//! that marker to an `Item::Compaction` positioned between the turns it +//! separates, rather than dropping it or surfacing it as a generic event. //! -//! What this test asserts (and why): +//! The marker payload is `{message, replacement_history?, window_id?}` +//! (see `docs/agents/formats/codex.md`). Only `message` is consumed, as +//! `Compaction.summary`. Codex never persists the manual-vs-auto trigger +//! or the pre-compaction token count, and we don't fold in +//! `replacement_history`, so `trigger`/`pre_tokens` are `None` and `kept` +//! is empty. //! -//! - The fixture loads via `RolloutReader::read_session` without +//! Two fixtures: +//! - synthetic `tests/fixtures/compacted_session.jsonl` — small, +//! deterministic pre/post turns around one compaction. (Justified per +//! project policy: real compaction fires only when the context window +//! fills mid-session, which a short capture prompt can't reliably +//! trigger.) Its `compacted` line uses the real +//! `{message, replacement_history}` shape. +//! - real `test-fixtures/codex/convo-compacted.jsonl` — a captured +//! production rollout that actually compacted (empty `message`). +//! +//! What these tests assert: +//! - The fixtures load via `RolloutReader::read_session` without //! crashing on the `compacted` line. -//! - Pre-compact user/assistant content survives the round-trip. -//! - Post-compact user/assistant content survives the round-trip. +//! - Exactly one `Item::Compaction` is emitted, with the field shape +//! the Codex payload supports. +//! - The compaction and its surrounding turns survive the +//! `derive_path` → `extract_conversation` round-trip. //! - The conversation projects back to JSONL that re-parses through //! `RolloutReader`. -//! -//! Known limitation (documented, not asserted): the `compacted` -//! rollout line itself carries an opaque payload (Codex doesn't model -//! its inner shape — `Compacted(Value)`). Today the IR drops it on the -//! floor. Acceptable loss for "good UX" — the surrounding messages -//! are what users actually read. use std::path::{Path, PathBuf}; @@ -28,18 +38,27 @@ use toolpath::v1::Graph; use toolpath_codex::project::CodexProjector; use toolpath_codex::{RolloutReader, to_view}; use toolpath_convo::{ - ConversationProjector, ConversationView, DeriveConfig, derive_path, extract_conversation, + ConversationProjector, ConversationView, DeriveConfig, Item, derive_path, extract_conversation, }; -fn fixture_path() -> PathBuf { +fn synthetic_fixture_path() -> PathBuf { Path::new(env!("CARGO_MANIFEST_DIR")) .join("tests") .join("fixtures") .join("compacted_session.jsonl") } -fn load_view() -> ConversationView { - let session = RolloutReader::read_session(fixture_path()).expect("read fixture"); +fn real_fixture_path() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("..") + .join("..") + .join("test-fixtures") + .join("codex") + .join("convo-compacted.jsonl") +} + +fn load_view(path: PathBuf) -> ConversationView { + let session = RolloutReader::read_session(path).expect("read fixture"); to_view(&session) } @@ -52,57 +71,191 @@ fn ir_roundtrip(view: &ConversationView) -> ConversationView { extract_conversation(&path) } +/// Index of the single compaction in the item stream, asserting there is +/// exactly one. +fn sole_compaction_index(view: &ConversationView) -> usize { + let indices: Vec = view + .items + .iter() + .enumerate() + .filter(|(_, it)| matches!(it, Item::Compaction(_))) + .map(|(i, _)| i) + .collect(); + assert_eq!( + indices.len(), + 1, + "expected exactly one Item::Compaction, got {}", + indices.len() + ); + indices[0] +} + +// ── Synthetic fixture ─────────────────────────────────────────────── + #[test] -fn fixture_loads_without_panic() { - let view = load_view(); +fn synthetic_fixture_loads_without_panic() { + let view = load_view(synthetic_fixture_path()); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "compaction fixture should produce turns" ); } #[test] -fn pre_compact_content_survives_roundtrip() { - let original = load_view(); - let after = ir_roundtrip(&original); +fn synthetic_emits_one_compaction_with_codex_field_shape() { + let view = load_view(synthetic_fixture_path()); + let idx = sole_compaction_index(&view); + let Item::Compaction(c) = &view.items[idx] else { + unreachable!() + }; - let needles = ["refactor the auth module", "reading the current auth code"]; - for n in needles { - assert!( - original.turns.iter().any(|t| t.text.contains(n)), - "pre-compact text {n:?} missing from initial view" - ); - assert!( - after.turns.iter().any(|t| t.text.contains(n)), - "pre-compact text {n:?} dropped after roundtrip" - ); - } + // `message` becomes the summary. + assert_eq!( + c.summary.as_deref(), + Some( + "Earlier in this session: read src/auth.rs, identified that login() lacks session-token validation." + ) + ); + // Codex never persists trigger or pre-token count; we don't consume + // replacement_history, so no kept ranges. + assert_eq!(c.trigger, None); + assert_eq!(c.pre_tokens, None); + assert!(c.kept.is_empty()); + // Synthesized stable id, and a parent that links to the prior turn. + assert_eq!(c.id, "compact-1"); + assert!( + c.parent_id.is_some(), + "compaction should parent on the prior turn" + ); + + // The compaction sits between the pre-compact and post-compact turns. + let turn_idx_before = view.items[..idx] + .iter() + .rposition(|it| matches!(it, Item::Turn(_))); + let turn_idx_after = view.items[idx + 1..] + .iter() + .position(|it| matches!(it, Item::Turn(_))); + assert!( + turn_idx_before.is_some(), + "a turn should precede the compaction" + ); + assert!( + turn_idx_after.is_some(), + "a turn should follow the compaction" + ); } #[test] -fn post_compact_content_survives_roundtrip() { - let original = load_view(); +fn synthetic_compaction_and_turns_survive_roundtrip() { + let original = load_view(synthetic_fixture_path()); let after = ir_roundtrip(&original); + // The compaction itself survives, with its summary intact. + let idx = sole_compaction_index(&after); + let Item::Compaction(c) = &after.items[idx] else { + unreachable!() + }; + assert!( + c.summary + .as_deref() + .unwrap() + .contains("session-token validation") + ); + assert!(c.parent_id.is_some()); + + // Surrounding pre/post turn content survives. let needles = [ + "refactor the auth module", + "reading the current auth code", "now add session validation", "added session validation to login()", ]; for n in needles { assert!( - original.turns.iter().any(|t| t.text.contains(n)), - "post-compact text {n:?} missing from initial view" + original.turns().any(|t| t.text.contains(n)), + "text {n:?} missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(n)), - "post-compact text {n:?} dropped after roundtrip" + after.turns().any(|t| t.text.contains(n)), + "text {n:?} dropped after roundtrip" ); } } #[test] -fn projector_output_is_re_parseable_by_reader() { - let view = load_view(); +fn synthetic_projector_output_is_re_parseable_by_reader() { + let view = load_view(synthetic_fixture_path()); + let after = ir_roundtrip(&view); + let projector = CodexProjector::new(); + let session = projector.project(&after).expect("project"); + + let mut lines: Vec = Vec::new(); + for line in &session.lines { + lines.push(serde_json::to_string(line).expect("serialize rollout line")); + } + + let tmp = tempfile::Builder::new() + .suffix(".jsonl") + .tempfile() + .expect("tempfile"); + std::fs::write(tmp.path(), lines.join("\n")).expect("write tempfile"); + RolloutReader::read_session(tmp.path()).expect("re-read projected JSONL"); +} + +// ── Real captured fixture ─────────────────────────────────────────── + +#[test] +fn real_fixture_emits_one_compaction() { + let view = load_view(real_fixture_path()); + let idx = sole_compaction_index(&view); + let Item::Compaction(c) = &view.items[idx] else { + unreachable!() + }; + + // The real capture's `message` is the empty string, so summary is + // `Some("")` — present but empty. The remaining fields follow the + // Codex payload shape: no trigger, no pre-token count, no kept ranges. + assert!( + c.summary.is_some(), + "summary should be Some (message field present, even if empty)" + ); + assert_eq!(c.trigger, None); + assert_eq!(c.pre_tokens, None); + assert!(c.kept.is_empty()); + assert!( + c.parent_id.is_some(), + "compaction should parent on the prior turn" + ); + assert_eq!(c.id, "compact-1"); +} + +#[test] +fn real_fixture_compaction_survives_roundtrip() { + let original = load_view(real_fixture_path()); + let pre_turns = original.turns().count(); + assert!(pre_turns > 0, "real fixture should have turns"); + + let after = ir_roundtrip(&original); + + // Exactly one compaction survives the round-trip. + let idx = sole_compaction_index(&after); + let Item::Compaction(c) = &after.items[idx] else { + unreachable!() + }; + assert!(c.summary.is_some()); + assert!(c.parent_id.is_some()); + + // Surrounding turns survive (count preserved through derive ↔ extract). + assert_eq!( + after.turns().count(), + pre_turns, + "turn count should survive the round-trip" + ); +} + +#[test] +fn real_fixture_projector_output_is_re_parseable_by_reader() { + let view = load_view(real_fixture_path()); let after = ir_roundtrip(&view); let projector = CodexProjector::new(); let session = projector.project(&after).expect("project"); @@ -119,3 +272,85 @@ fn projector_output_is_re_parseable_by_reader() { std::fs::write(tmp.path(), lines.join("\n")).expect("write tempfile"); RolloutReader::read_session(tmp.path()).expect("re-read projected JSONL"); } + +/// View → Codex `Session` → JSONL → `Session` → view: the compaction +/// boundary the projector now emits as a `compacted` line must reappear +/// as exactly one `Item::Compaction` when the projected session is read +/// back, preserving the summary and its position between turns. +/// +/// The real capture's `message` is empty, so the round-tripped summary is +/// `Some("")` rather than `None` — present but empty. +#[test] +fn real_fixture_projection_round_trips_compaction() { + let original = load_view(real_fixture_path()); + let orig_idx = sole_compaction_index(&original); + let Item::Compaction(orig) = &original.items[orig_idx] else { + unreachable!() + }; + let orig_summary = orig.summary.clone(); + + // Project directly (no IR detour) so we exercise the projector's + // `Item::Compaction` → `compacted` line path on its own. + let session = CodexProjector::new().project(&original).expect("project"); + + // Exactly one `compacted` line, carrying the summary as `message`. + let compacted: Vec<&toolpath_codex::RolloutLine> = session + .lines + .iter() + .filter(|l| l.kind == "compacted") + .collect(); + assert_eq!( + compacted.len(), + 1, + "projector should emit exactly one compacted line" + ); + assert_eq!( + compacted[0].payload.get("message").and_then(|m| m.as_str()), + orig_summary.as_deref(), + "compacted line `message` should carry the compaction summary" + ); + + // Serialize one JSON line per rollout entry and read it back through + // the crate's reader, then run the forward `to_view`. + let body = session + .lines + .iter() + .map(|l| serde_json::to_string(l).expect("serialize rollout line")) + .collect::>() + .join("\n"); + let tmp = tempfile::Builder::new() + .suffix(".jsonl") + .tempfile() + .expect("tempfile"); + std::fs::write(tmp.path(), body).expect("write tempfile"); + let reread = RolloutReader::read_session(tmp.path()).expect("re-read projected JSONL"); + let after = to_view(&reread); + + // Exactly one compaction survives, with the original summary intact + // and no trigger (Codex never persists it). + let idx = sole_compaction_index(&after); + let Item::Compaction(c) = &after.items[idx] else { + unreachable!() + }; + assert_eq!(c.summary, orig_summary, "summary should round-trip"); + assert_eq!(c.trigger, None, "Codex never persists the trigger"); + assert!(c.pre_tokens.is_none()); + assert!(c.kept.is_empty()); + + // The compaction sits between turns: a turn precedes it and a turn + // follows it in the re-read item stream. + let turn_before = after.items[..idx] + .iter() + .rposition(|it| matches!(it, Item::Turn(_))); + let turn_after = after.items[idx + 1..] + .iter() + .position(|it| matches!(it, Item::Turn(_))); + assert!( + turn_before.is_some(), + "a turn should precede the round-tripped compaction" + ); + assert!( + turn_after.is_some(), + "a turn should follow the round-tripped compaction" + ); +} diff --git a/crates/toolpath-codex/tests/fidelity.rs b/crates/toolpath-codex/tests/fidelity.rs index 0441c73..8c8a16c 100644 --- a/crates/toolpath-codex/tests/fidelity.rs +++ b/crates/toolpath-codex/tests/fidelity.rs @@ -88,7 +88,7 @@ fn turn_timestamps_match_source_message_lines() { let s = session(); let view = to_view(&s); - let turn_timestamps: HashSet<&str> = view.turns.iter().map(|t| t.timestamp.as_str()).collect(); + let turn_timestamps: HashSet<&str> = view.turns().map(|t| t.timestamp.as_str()).collect(); for line in &s.lines { if let RolloutItem::ResponseItem(ResponseItem::Message(_)) = line.item() { @@ -161,9 +161,9 @@ fn actor_scheme_matches_source_role() { let view = to_view(&s); let path = derive::derive_path(&s, &derive::DeriveConfig::default()); - let user_seen = view.turns.iter().any(|t| t.role == Role::User); - let assistant_seen = view.turns.iter().any(|t| t.role == Role::Assistant); - let system_seen = view.turns.iter().any(|t| t.role == Role::System); + let user_seen = view.turns().any(|t| t.role == Role::User); + let assistant_seen = view.turns().any(|t| t.role == Role::Assistant); + let system_seen = view.turns().any(|t| t.role == Role::System); let prefixes: HashSet<&str> = path .steps @@ -247,7 +247,7 @@ fn function_call_arguments_preserved_in_view() { let mut tool_by_id: std::collections::HashMap<&str, &toolpath_convo::ToolInvocation> = std::collections::HashMap::new(); - for t in &view.turns { + for t in view.turns() { for tu in &t.tool_uses { tool_by_id.insert(tu.id.as_str(), tu); } diff --git a/crates/toolpath-codex/tests/fixture_roundtrip.rs b/crates/toolpath-codex/tests/fixture_roundtrip.rs index d1bc593..6e82da0 100644 --- a/crates/toolpath-codex/tests/fixture_roundtrip.rs +++ b/crates/toolpath-codex/tests/fixture_roundtrip.rs @@ -30,17 +30,13 @@ fn view_has_expected_turn_count() { let view = to_view(&s); // From prior inspection: 1 user + 1 developer + 10 assistant messages. assert!( - view.turns.len() >= 10 && view.turns.len() <= 14, + view.turns().count() >= 10 && view.turns().count() <= 14, "expected 10-14 turns, got {}", - view.turns.len() + view.turns().count() ); - let users = view.turns.iter().filter(|t| t.role == Role::User).count(); - let assistants = view - .turns - .iter() - .filter(|t| t.role == Role::Assistant) - .count(); - let system = view.turns.iter().filter(|t| t.role == Role::System).count(); + let users = view.turns().filter(|t| t.role == Role::User).count(); + let assistants = view.turns().filter(|t| t.role == Role::Assistant).count(); + let system = view.turns().filter(|t| t.role == Role::System).count(); // The fixture has two user messages: the actual prompt plus a // `function_call_output`-style carrier that encodes tool output. // Accept either 1 or 2 so the test stays robust across wire variants. @@ -61,11 +57,10 @@ fn view_has_expected_turn_count() { fn tool_calls_pair_correctly() { let s = session(); let view = to_view(&s); - let total_tools: usize = view.turns.iter().map(|t| t.tool_uses.len()).sum(); + let total_tools: usize = view.turns().map(|t| t.tool_uses.len()).sum(); assert!(total_tools > 0); let with_result: usize = view - .turns - .iter() + .turns() .flat_map(|t| &t.tool_uses) .filter(|tu| tu.result.is_some()) .count(); @@ -83,8 +78,7 @@ fn exec_commands_surface_as_shell_category() { let s = session(); let view = to_view(&s); let shell_calls: Vec<&toolpath_convo::ToolInvocation> = view - .turns - .iter() + .turns() .flat_map(|t| &t.tool_uses) .filter(|tu| tu.category == Some(ToolCategory::Shell)) .collect(); @@ -99,8 +93,7 @@ fn apply_patch_preserved() { let s = session(); let view = to_view(&s); let apply_patches: Vec<&toolpath_convo::ToolInvocation> = view - .turns - .iter() + .turns() .flat_map(|t| &t.tool_uses) .filter(|tu| tu.name == "apply_patch") .collect(); @@ -169,14 +162,16 @@ fn reasoning_breakdown_differenced_dedup_safe_against_real_fixture() { // summing the twice-emitted counts, or stamping the cumulative — this would // overshoot. This is the dedup-safe / no-double-count proof on real data. let attributed_reasoning: u32 = view - .turns - .iter() + .turns() .map(|t| reasoning_of(t.attributed_token_usage.as_ref())) .sum(); - assert_eq!(attributed_reasoning, 979, "Σ attributed reasoning != cumulative"); + assert_eq!( + attributed_reasoning, 979, + "Σ attributed reasoning != cumulative" + ); // Per step, reasoning ⊆ output. - for t in &view.turns { + for t in view.turns() { if let Some(a) = t.attributed_token_usage.as_ref() { let r = reasoning_of(Some(a)); assert!( @@ -191,8 +186,7 @@ fn reasoning_breakdown_differenced_dedup_safe_against_real_fixture() { // Round (group) totals: Σ over group token_usage reasoning == 979 too, and // each round's reasoning ⊆ its output. let round_reasoning: u32 = view - .turns - .iter() + .turns() .filter(|t| t.token_usage.is_some()) .map(|t| { let u = t.token_usage.as_ref().unwrap(); @@ -201,7 +195,10 @@ fn reasoning_breakdown_differenced_dedup_safe_against_real_fixture() { r }) .sum(); - assert_eq!(round_reasoning, 979, "Σ round-total reasoning != cumulative"); + assert_eq!( + round_reasoning, 979, + "Σ round-total reasoning != cumulative" + ); } #[test] @@ -213,7 +210,7 @@ fn encrypted_reasoning_does_not_land_on_thinking() { // garbage). let s = session(); let view = to_view(&s); - let with_thinking = view.turns.iter().filter(|t| t.thinking.is_some()).count(); + let with_thinking = view.turns().filter(|t| t.thinking.is_some()).count(); assert_eq!( with_thinking, 0, "encrypted reasoning must not land on turn.thinking" @@ -224,13 +221,10 @@ fn encrypted_reasoning_does_not_land_on_thinking() { fn events_preserve_non_turn_content() { let s = session(); let view = to_view(&s); - let has_turn_context = view.events.iter().any(|e| e.event_type == "turn_context"); - let has_task_started = view.events.iter().any(|e| e.event_type == "task_started"); - let has_task_complete = view.events.iter().any(|e| e.event_type == "task_complete"); - let has_patch_apply = view - .events - .iter() - .any(|e| e.event_type == "patch_apply_end"); + let has_turn_context = view.events().any(|e| e.event_type == "turn_context"); + let has_task_started = view.events().any(|e| e.event_type == "task_started"); + let has_task_complete = view.events().any(|e| e.event_type == "task_complete"); + let has_patch_apply = view.events().any(|e| e.event_type == "patch_apply_end"); assert!(has_turn_context); assert!(has_task_started); assert!(has_task_complete); diff --git a/crates/toolpath-codex/tests/fixtures/compacted_session.jsonl b/crates/toolpath-codex/tests/fixtures/compacted_session.jsonl index 0478447..3ad5ba2 100644 --- a/crates/toolpath-codex/tests/fixtures/compacted_session.jsonl +++ b/crates/toolpath-codex/tests/fixtures/compacted_session.jsonl @@ -2,7 +2,7 @@ {"timestamp":"2026-06-01T10:00:01.000Z","type":"event_msg","payload":{"type":"task_started","turn_id":"turn-1","started_at":1722499201}} {"timestamp":"2026-06-01T10:00:02.000Z","type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"refactor the auth module"}]}} {"timestamp":"2026-06-01T10:00:03.000Z","type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"reading the current auth code"}],"phase":"final"}} -{"timestamp":"2026-06-01T10:30:00.000Z","type":"compacted","payload":{"trigger":"auto","preTokens":180000,"summary":"Earlier in this session: read src/auth.rs, identified that login() lacks session-token validation."}} +{"timestamp":"2026-06-01T10:30:00.000Z","type":"compacted","payload":{"message":"Earlier in this session: read src/auth.rs, identified that login() lacks session-token validation.","replacement_history":[{"type":"message","role":"user","content":[{"type":"input_text","text":"refactor the auth module"}]}]}} {"timestamp":"2026-06-01T10:31:00.000Z","type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"now add session validation"}]}} {"timestamp":"2026-06-01T10:31:01.000Z","type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"added session validation to login()"}],"phase":"final"}} {"timestamp":"2026-06-01T10:31:02.000Z","type":"event_msg","payload":{"type":"task_complete","turn_id":"turn-2","completed_at":1722500862}} diff --git a/crates/toolpath-codex/tests/real_fixture_roundtrip.rs b/crates/toolpath-codex/tests/real_fixture_roundtrip.rs index a0884e0..5beaec0 100644 --- a/crates/toolpath-codex/tests/real_fixture_roundtrip.rs +++ b/crates/toolpath-codex/tests/real_fixture_roundtrip.rs @@ -55,10 +55,7 @@ fn is_system_envelope(turn: &Turn) -> bool { } fn meaningful(view: &ConversationView) -> Vec<&Turn> { - view.turns - .iter() - .filter(|t| !is_system_envelope(t)) - .collect() + view.turns().filter(|t| !is_system_envelope(t)).collect() } fn norm(s: &str) -> String { @@ -69,7 +66,7 @@ fn norm(s: &str) -> String { fn fixture_loads() { let view = load_fixture_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "codex fixture should produce a non-empty view" ); let m = meaningful(&view); @@ -182,14 +179,14 @@ fn roundtrip_preserves_delegations() { let original = load_fixture_view(); let after = ir_roundtrip(&original); - let total_before: usize = original.turns.iter().map(|t| t.delegations.len()).sum(); - let total_after: usize = after.turns.iter().map(|t| t.delegations.len()).sum(); + let total_before: usize = original.turns().map(|t| t.delegations.len()).sum(); + let total_after: usize = after.turns().map(|t| t.delegations.len()).sum(); assert_eq!( total_before, total_after, "total delegation count diverged: {total_before} → {total_after}" ); - for (i, (a, b)) in original.turns.iter().zip(after.turns.iter()).enumerate() { + for (i, (a, b)) in original.turns().zip(after.turns()).enumerate() { assert_eq!( a.delegations.len(), b.delegations.len(), diff --git a/crates/toolpath-convo/Cargo.toml b/crates/toolpath-convo/Cargo.toml index e1009ed..6ec4f1d 100644 --- a/crates/toolpath-convo/Cargo.toml +++ b/crates/toolpath-convo/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-convo" -version = "0.11.0" +version = "0.12.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-convo/README.md b/crates/toolpath-convo/README.md index 90de3d5..d48eaad 100644 --- a/crates/toolpath-convo/README.md +++ b/crates/toolpath-convo/README.md @@ -54,7 +54,7 @@ fn show_conversation(provider: &dyn ConversationProvider) { } println!("Files changed: {:?}", view.files_changed); - for turn in &view.turns { + for turn in view.turns() { println!("[{}] {}", turn.role, turn.text); // Environment context diff --git a/crates/toolpath-convo/src/derive.rs b/crates/toolpath-convo/src/derive.rs index ad66857..23b23a7 100644 --- a/crates/toolpath-convo/src/derive.rs +++ b/crates/toolpath-convo/src/derive.rs @@ -13,7 +13,7 @@ use toolpath::v1::{ PathMeta, Step, StepIdentity, StructuralChange, }; -use crate::{ConversationView, Role, ToolCategory, ToolInvocation, Turn}; +use crate::{ConversationView, Item, Role, ToolCategory, ToolInvocation, Turn}; /// Configuration for [`derive_path`]. #[derive(Debug, Clone)] @@ -44,6 +44,13 @@ impl Default for DeriveConfig { } /// Derive a [`Path`] from a [`ConversationView`]. +/// +/// Step ids must be unique within a path (a toolpath invariant), so a +/// collision is resolved as the steps are emitted rather than surfaced as an +/// error: a byte-identical re-emission (e.g. a Claude compaction replay of an +/// unchanged message) is dropped, and a same-id-but-different step is renamed +/// to a fresh id. Either way the derivation always succeeds and the result is +/// collision-free. pub fn derive_path(view: &ConversationView, config: &DeriveConfig) -> Path { let provider = view.provider_id.as_deref().unwrap_or("unknown"); let id_prefix: String = view.id.chars().take(8).collect(); @@ -82,8 +89,7 @@ pub fn derive_path(view: &ConversationView, config: &DeriveConfig) -> Path { }) }) .or_else(|| { - view.turns - .iter() + view.turns() .find_map(|t| t.environment.as_ref()?.working_dir.clone()) .map(|wd| { let uri = if wd.starts_with('/') { @@ -101,324 +107,452 @@ pub fn derive_path(view: &ConversationView, config: &DeriveConfig) -> Path { let conv_artifact_key = format!("{}://{}", provider, view.id); - let mut steps: Vec = Vec::with_capacity(view.turns.len()); + let mut steps: Vec = Vec::with_capacity(view.items.len()); + // Final step id → index in `steps`, for resolving id collisions as steps + // are emitted (see `push_step`). + let mut by_id: HashMap = HashMap::new(); let mut turn_to_step: HashMap = HashMap::new(); let mut actors: HashMap = HashMap::new(); - for (idx, turn) in view.turns.iter().enumerate() { - // Step id: use the turn's native id when set so it round-trips - // through `extract_conversation`; otherwise synthesize sequentially. - let step_id = if turn.id.is_empty() { - format!("step-{:04}", idx + 1) - } else { - turn.id.clone() - }; - turn_to_step.insert(turn.id.clone(), step_id.clone()); - - let actor = actor_for_turn(turn, provider); - record_actor(&mut actors, &actor, turn, provider, view); - - let mut step = Step { - step: StepIdentity { - id: step_id, - parents: Vec::new(), - actor, - timestamp: turn.timestamp.clone(), - }, - change: HashMap::new(), - meta: None, - }; + // Single ordered pass over `view.items`, dispatching per variant so each + // emitted step lands at its true position relative to its neighbors — + // crucial for compaction boundaries, which must sit between the turns + // they separate. + // + // Per-variant counters drive the synthetic step ids: turns synthesize + // `step-{:04}` indexed by turn count, events `event-{:04}` by event count. + // + // `last_step_id` tracks the previously emitted step so that events (and + // compactions) without an explicit parent chain off whatever came before. + let mut turn_idx = 0usize; + let mut event_idx = 0usize; + let mut compact_idx = 0usize; + let mut last_step_id: Option = None; + + // Group ids in turn order, so a turn can tell whether it's the last of its + // message group (message-level token accounting, below). + let turn_groups: Vec> = view.turns().map(|t| t.group_id.clone()).collect(); + + for item in &view.items { + match item { + Item::Turn(turn) => { + let idx = turn_idx; + turn_idx += 1; + + // Step id: use the turn's native id when set so it round-trips + // through `extract_conversation`; otherwise synthesize sequentially. + let step_id = if turn.id.is_empty() { + format!("step-{:04}", idx + 1) + } else { + turn.id.clone() + }; - // Parent mapping - if let Some(parent_id) = &turn.parent_id - && let Some(parent_step_id) = turn_to_step.get(parent_id) - { - step.step.parents.push(parent_step_id.clone()); - } + let actor = actor_for_turn(turn, provider); + record_actor(&mut actors, &actor, turn, provider, view); + + let mut step = Step { + step: StepIdentity { + id: step_id.clone(), + parents: Vec::new(), + actor, + timestamp: turn.timestamp.clone(), + }, + change: HashMap::new(), + meta: None, + }; - // Build conversation.append structural change extras - let mut extra: HashMap = HashMap::new(); - extra.insert( - "role".to_string(), - serde_json::Value::String(turn.role.to_string()), - ); - extra.insert( - "text".to_string(), - serde_json::Value::String(turn.text.clone()), - ); + // Parent mapping + if let Some(parent_id) = &turn.parent_id + && let Some(parent_step_id) = turn_to_step.get(parent_id) + { + step.step.parents.push(parent_step_id.clone()); + } - if config.include_thinking - && let Some(thinking) = &turn.thinking - { - extra.insert( - "thinking".to_string(), - serde_json::Value::String(thinking.clone()), - ); - } + // Build conversation.append structural change extras + let mut extra: HashMap = HashMap::new(); + extra.insert( + "role".to_string(), + serde_json::Value::String(turn.role.to_string()), + ); + extra.insert( + "text".to_string(), + serde_json::Value::String(turn.text.clone()), + ); - if config.include_tool_uses && !turn.tool_uses.is_empty() { - let arr: Vec = turn - .tool_uses - .iter() - .map(|t| { - let mut obj = serde_json::json!({ - "id": t.id, - "name": t.name, - "input": t.input, - "category": t.category, - }); - if let Some(result) = &t.result - && let Ok(v) = serde_json::to_value(result) - { - obj.as_object_mut().unwrap().insert("result".to_string(), v); - } - obj - }) - .collect(); - extra.insert("tool_uses".to_string(), serde_json::Value::Array(arr)); - } + if config.include_thinking + && let Some(thinking) = &turn.thinking + { + extra.insert( + "thinking".to_string(), + serde_json::Value::String(thinking.clone()), + ); + } - // Message-level accounting lands exactly once per message: when a - // provider splits one message across several turns (group_id - // set on each), only the run's last turn carries token_usage, so - // summing over steps yields session totals. A turn without a - // group_id is its own accounting unit. - let last_of_message = match &turn.group_id { - None => true, - Some(mid) => view - .turns - .get(idx + 1) - .is_none_or(|next| next.group_id.as_ref() != Some(mid)), - }; - if last_of_message - && let Some(usage) = &turn.token_usage - && let Ok(v) = serde_json::to_value(usage) - { - extra.insert("token_usage".to_string(), v); - } + if config.include_tool_uses && !turn.tool_uses.is_empty() { + let arr: Vec = turn + .tool_uses + .iter() + .map(|t| { + let mut obj = serde_json::json!({ + "id": t.id, + "name": t.name, + "input": t.input, + "category": t.category, + }); + if let Some(result) = &t.result + && let Ok(v) = serde_json::to_value(result) + { + obj.as_object_mut().unwrap().insert("result".to_string(), v); + } + obj + }) + .collect(); + extra.insert("tool_uses".to_string(), serde_json::Value::Array(arr)); + } - // Per-step attributed spend rides its own key on every step that - // has it (independent of the once-per-message `token_usage`), so - // summing `token_usage` is unaffected while per-step cost stays - // readable structurally. - if let Some(attr) = &turn.attributed_token_usage - && let Ok(v) = serde_json::to_value(attr) - { - extra.insert("attributed_token_usage".to_string(), v); - } + // Message-group accounting: a message's total `token_usage` + // lands once, on the group's last turn, so summing over a + // path's steps yields session totals. A turn with no `group_id` + // is its own group. + let last_of_group = match &turn.group_id { + None => true, + Some(mid) => match turn_groups.get(idx + 1) { + Some(Some(next)) => next != mid, + _ => true, + }, + }; + if last_of_group + && let Some(usage) = &turn.token_usage + && let Ok(v) = serde_json::to_value(usage) + { + extra.insert("token_usage".to_string(), v); + } - if let Some(mid) = &turn.group_id { - extra.insert( - "group_id".to_string(), - serde_json::Value::String(mid.clone()), - ); - } + // Per-step attributed spend (when the source reports it) rides + // its own key, independent of the once-per-group `token_usage`. + if let Some(attr) = &turn.attributed_token_usage + && let Ok(v) = serde_json::to_value(attr) + { + extra.insert("attributed_token_usage".to_string(), v); + } - if !turn.delegations.is_empty() - && let Ok(v) = serde_json::to_value(&turn.delegations) - { - extra.insert("delegations".to_string(), v); - } + if let Some(mid) = &turn.group_id { + extra.insert( + "group_id".to_string(), + serde_json::Value::String(mid.clone()), + ); + } - if let Some(stop_reason) = &turn.stop_reason { - extra.insert( - "stop_reason".to_string(), - serde_json::Value::String(stop_reason.clone()), - ); - } + if !turn.delegations.is_empty() + && let Ok(v) = serde_json::to_value(&turn.delegations) + { + extra.insert("delegations".to_string(), v); + } - if let Some(env) = &turn.environment - && let Ok(v) = serde_json::to_value(env) - { - extra.insert("environment".to_string(), v); - } + if let Some(stop_reason) = &turn.stop_reason { + extra.insert( + "stop_reason".to_string(), + serde_json::Value::String(stop_reason.clone()), + ); + } - step.change.insert( - conv_artifact_key.clone(), - ArtifactChange { - raw: None, - structural: Some(StructuralChange { - change_type: "conversation.append".to_string(), - extra, - }), - }, - ); + if let Some(env) = &turn.environment + && let Ok(v) = serde_json::to_value(env) + { + extra.insert("environment".to_string(), v); + } - // File mutations → sibling `file.write` change entries. - // - // Preferred: each `Turn::file_mutations` entry comes from the - // provider's `to_view` with the resolved diff already in - // `raw_diff` (claude's git-HEAD lookup, codex's `apply_patch_end` - // parse, opencode's git2 tree↔tree, etc.). `tool_id` links back - // to a specific `ToolInvocation` when the provider can attribute. - // - // Fallback (un-migrated providers): for any `FileWrite`-category - // tool with no matching mutation, synthesize from `tool.input` - // via `file_write_change`. - let attributed: std::collections::HashSet = turn - .file_mutations - .iter() - .filter_map(|fm| fm.tool_id.clone()) - .collect(); - for fm in &turn.file_mutations { - let mut t_extra: HashMap = HashMap::new(); - if let Some(tid) = &fm.tool_id { - t_extra.insert( - "tool_id".to_string(), - serde_json::Value::String(tid.clone()), + step.change.insert( + conv_artifact_key.clone(), + ArtifactChange { + raw: None, + structural: Some(StructuralChange { + change_type: "conversation.append".to_string(), + extra, + }), + }, ); - if let Some(tool) = turn.tool_uses.iter().find(|t| &t.id == tid) { + + // File mutations → sibling `file.write` change entries. + // + // Preferred: each `Turn::file_mutations` entry comes from the + // provider's `to_view` with the resolved diff already in + // `raw_diff` (claude's git-HEAD lookup, codex's `apply_patch_end` + // parse, opencode's git2 tree↔tree, etc.). `tool_id` links back + // to a specific `ToolInvocation` when the provider can attribute. + // + // Fallback (un-migrated providers): for any `FileWrite`-category + // tool with no matching mutation, synthesize from `tool.input` + // via `file_write_change`. + let attributed: std::collections::HashSet = turn + .file_mutations + .iter() + .filter_map(|fm| fm.tool_id.clone()) + .collect(); + for fm in &turn.file_mutations { + let mut t_extra: HashMap = HashMap::new(); + if let Some(tid) = &fm.tool_id { + t_extra.insert( + "tool_id".to_string(), + serde_json::Value::String(tid.clone()), + ); + if let Some(tool) = turn.tool_uses.iter().find(|t| &t.id == tid) { + t_extra.insert( + "tool".to_string(), + serde_json::Value::String(tool.name.clone()), + ); + } + } + if let Some(op) = &fm.operation { + t_extra.insert( + "operation".to_string(), + serde_json::Value::String(op.clone()), + ); + } + if let Some(b) = &fm.before { + t_extra.insert("before".to_string(), serde_json::Value::String(b.clone())); + } + if let Some(a) = &fm.after { + t_extra.insert("after".to_string(), serde_json::Value::String(a.clone())); + } + if let Some(rt) = &fm.rename_to { + t_extra.insert( + "rename_to".to_string(), + serde_json::Value::String(rt.clone()), + ); + } + step.change.insert( + fm.path.clone(), + ArtifactChange { + raw: fm.raw_diff.clone(), + structural: Some(StructuralChange { + change_type: "file.write".to_string(), + extra: t_extra, + }), + }, + ); + } + for tool in &turn.tool_uses { + if tool.category != Some(ToolCategory::FileWrite) + || attributed.contains(&tool.id) + { + continue; + } + let Some(path) = extract_file_path(tool) else { + continue; + }; + let (raw, mut t_extra) = file_write_change(tool, &path, None); t_extra.insert( "tool".to_string(), serde_json::Value::String(tool.name.clone()), ); + t_extra.insert( + "tool_id".to_string(), + serde_json::Value::String(tool.id.clone()), + ); + step.change.insert( + path, + ArtifactChange { + raw, + structural: Some(StructuralChange { + change_type: "file.write".to_string(), + extra: t_extra, + }), + }, + ); } + + let final_id = push_step(&mut steps, &mut by_id, step); + // Map the turn's native id to whatever id its step ended up + // with, so later turns chaining off it resolve correctly even + // when this one was renamed or dropped as a duplicate. + turn_to_step.insert(turn.id.clone(), final_id.clone()); + last_step_id = Some(final_id); } - if let Some(op) = &fm.operation { - t_extra.insert( - "operation".to_string(), - serde_json::Value::String(op.clone()), + + // Emit `view.events` as `conversation.event` steps so that + // attachments, preamble lines (ai-title, last-prompt, + // queue-operation, permission-mode), and other non-turn entries + // survive the IR-to-Path-to-IR roundtrip. Without this, + // derive_path drops everything outside `turns`, so a Claude + // session loses ~10–25% of its lines on import/export. An event + // without an explicit `parent_id` chains off whatever step came + // before it. + Item::Event(event) => { + let idx = event_idx; + event_idx += 1; + + // Event step id: prefer the event's native id so it round-trips. + let step_id = if event.id.is_empty() { + format!("event-{:04}", idx + 1) + } else { + event.id.clone() + }; + let actor = format!("tool:{}", provider); + actors + .entry(actor.clone()) + .or_insert_with(|| ActorDefinition { + name: Some(provider.to_string()), + provider: Some(provider.to_string()), + ..Default::default() + }); + + // event.data is flattened into StructuralChange.extra. Strip keys + // that collide with the typed fields on StructuralChange itself — + // most importantly `type`, which serde renames `change_type` to. + // A Codex `user_message` event carries `data["type"] = "user_message"`, + // which would otherwise overwrite our `change_type = "conversation.event"` + // and break PathOrRef untagged-enum disambiguation on parse. + let mut extra: HashMap = event + .data + .iter() + .filter(|(k, _)| k.as_str() != "type") + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + // Stash the original `type` value under a non-colliding key so + // round-trip can recover it for providers that need it. + if let Some(t) = event.data.get("type") { + extra.insert("event_data_type".to_string(), t.clone()); + } + extra.insert( + "entry_type".to_string(), + serde_json::Value::String(event.event_type.clone()), ); - } - if let Some(b) = &fm.before { - t_extra.insert("before".to_string(), serde_json::Value::String(b.clone())); - } - if let Some(a) = &fm.after { - t_extra.insert("after".to_string(), serde_json::Value::String(a.clone())); - } - if let Some(rt) = &fm.rename_to { - t_extra.insert( - "rename_to".to_string(), - serde_json::Value::String(rt.clone()), + if !event.id.is_empty() { + extra.insert( + "event_source_id".to_string(), + serde_json::Value::String(event.id.clone()), + ); + } + + let parents: Vec = event + .parent_id + .as_ref() + .and_then(|pid| turn_to_step.get(pid).cloned()) + .or_else(|| last_step_id.clone()) + .into_iter() + .collect(); + + let mut step = Step { + step: StepIdentity { + id: step_id.clone(), + parents, + actor, + timestamp: event.timestamp.clone(), + }, + change: HashMap::new(), + meta: None, + }; + + step.change.insert( + conv_artifact_key.clone(), + ArtifactChange { + raw: None, + structural: Some(StructuralChange { + change_type: "conversation.event".to_string(), + extra, + }), + }, ); + last_step_id = Some(push_step(&mut steps, &mut by_id, step)); } - step.change.insert( - fm.path.clone(), - ArtifactChange { - raw: fm.raw_diff.clone(), - structural: Some(StructuralChange { - change_type: "file.write".to_string(), - extra: t_extra, - }), - }, - ); - } - for tool in &turn.tool_uses { - if tool.category != Some(ToolCategory::FileWrite) || attributed.contains(&tool.id) { - continue; - } - let Some(path) = extract_file_path(tool) else { - continue; - }; - let (raw, mut t_extra) = file_write_change(tool, &path, None); - t_extra.insert( - "tool".to_string(), - serde_json::Value::String(tool.name.clone()), - ); - t_extra.insert( - "tool_id".to_string(), - serde_json::Value::String(tool.id.clone()), - ); - step.change.insert( - path, - ArtifactChange { - raw, - structural: Some(StructuralChange { - change_type: "file.write".to_string(), - extra: t_extra, - }), - }, - ); - } - steps.push(step); - } - - // Emit `view.events` as `conversation.event` steps so that attachments, - // preamble lines (ai-title, last-prompt, queue-operation, permission-mode), - // and other non-turn entries survive the IR-to-Path-to-IR roundtrip. - // Without this, derive_path drops everything outside `turns`, so a - // Claude session loses ~10–25% of its lines on import/export. - // Track the last emitted step id so events without an explicit - // `parent_id` can chain off whatever step came before them. - let mut last_step_id: Option = steps.last().map(|s| s.step.id.clone()); - for (idx, event) in view.events.iter().enumerate() { - // Event step id: prefer the event's native id so it round-trips. - let step_id = if event.id.is_empty() { - format!("event-{:04}", idx + 1) - } else { - event.id.clone() - }; - let actor = format!("tool:{}", provider); - actors - .entry(actor.clone()) - .or_insert_with(|| ActorDefinition { - name: Some(provider.to_string()), - provider: Some(provider.to_string()), - ..Default::default() - }); - - // event.data is flattened into StructuralChange.extra. Strip keys - // that collide with the typed fields on StructuralChange itself — - // most importantly `type`, which serde renames `change_type` to. - // A Codex `user_message` event carries `data["type"] = "user_message"`, - // which would otherwise overwrite our `change_type = "conversation.event"` - // and break PathOrRef untagged-enum disambiguation on parse. - let mut extra: HashMap = event - .data - .iter() - .filter(|(k, _)| k.as_str() != "type") - .map(|(k, v)| (k.clone(), v.clone())) - .collect(); - // Stash the original `type` value under a non-colliding key so - // round-trip can recover it for providers that need it. - if let Some(t) = event.data.get("type") { - extra.insert("event_data_type".to_string(), t.clone()); - } - extra.insert( - "entry_type".to_string(), - serde_json::Value::String(event.event_type.clone()), - ); - if !event.id.is_empty() { - extra.insert( - "event_source_id".to_string(), - serde_json::Value::String(event.id.clone()), - ); - } + // A context-compaction boundary projects to one + // `conversation.compact` step, positioned between the turns it + // separates. Later turns whose `parent_id` references this + // compaction resolve to its step via `turn_to_step`, so the DAG + // is rewired through the boundary. + Item::Compaction(c) => { + let step_id = if c.id.is_empty() { + compact_idx += 1; + format!("compact-{:04}", compact_idx) + } else { + c.id.clone() + }; - let parents: Vec = event - .parent_id - .as_ref() - .and_then(|pid| turn_to_step.get(pid).cloned()) - .or_else(|| last_step_id.clone()) - .into_iter() - .collect(); + let actor = format!("tool:{}", provider); + actors + .entry(actor.clone()) + .or_insert_with(|| ActorDefinition { + name: Some(provider.to_string()), + provider: Some(provider.to_string()), + ..Default::default() + }); - let mut step = Step { - step: StepIdentity { - id: step_id.clone(), - parents, - actor, - timestamp: event.timestamp.clone(), - }, - change: HashMap::new(), - meta: None, - }; + let parents: Vec = c + .parent_id + .as_ref() + .and_then(|pid| turn_to_step.get(pid).cloned()) + // Fall back to whatever step came before (as events do) + // so a compaction with a missing/unresolvable parent_id + // still chains onto the DAG instead of becoming a + // disconnected root that orphans the pre-compaction turns. + .or_else(|| last_step_id.clone()) + .into_iter() + .collect(); + + let mut extra: HashMap = HashMap::new(); + if let Some(trigger) = &c.trigger { + let s = match trigger { + crate::CompactionTrigger::Auto => "auto", + crate::CompactionTrigger::Manual => "manual", + }; + extra.insert("trigger".to_string(), serde_json::Value::String(s.into())); + } + if let Some(summary) = &c.summary { + extra.insert( + "summary".to_string(), + serde_json::Value::String(summary.clone()), + ); + } + if let Some(pre_tokens) = c.pre_tokens { + extra.insert( + "pre_tokens".to_string(), + serde_json::Value::Number(pre_tokens.into()), + ); + } + if !c.kept.is_empty() + && let Ok(v) = serde_json::to_value(&c.kept) + { + extra.insert("kept".to_string(), v); + } - step.change.insert( - conv_artifact_key.clone(), - ArtifactChange { - raw: None, - structural: Some(StructuralChange { - change_type: "conversation.event".to_string(), - extra, - }), - }, - ); - steps.push(step); - last_step_id = Some(step_id); + let mut step = Step { + step: StepIdentity { + id: step_id.clone(), + parents, + actor, + timestamp: c.timestamp.clone(), + }, + change: HashMap::new(), + meta: None, + }; + step.change.insert( + conv_artifact_key.clone(), + ArtifactChange { + raw: None, + structural: Some(StructuralChange { + change_type: "conversation.compact".to_string(), + extra, + }), + }, + ); + let final_id = push_step(&mut steps, &mut by_id, step); + // Later turns whose `parent_id` references the boundary resolve + // through `turn_to_step`; map to the final (possibly renamed) id. + turn_to_step.insert(c.id.clone(), final_id.clone()); + last_step_id = Some(final_id); + } + } } + // The head is the last emitted step. Use `steps.last()` rather than + // `last_step_id`: when the final item is a byte-identical duplicate that + // `push_step` drops, `last_step_id` regresses to the earlier step it + // collapsed into, which would orphan any real step emitted after that + // earlier step (e.g. a `conversation.compact` between a turn and its + // replay) as a spurious dead end. The last surviving step keeps the whole + // chain on the head's ancestry. let head = steps.last().map(|s| s.step.id.clone()).unwrap_or_default(); // Meta @@ -473,9 +607,49 @@ pub fn derive_path(view: &ConversationView, config: &DeriveConfig) -> Path { } } +/// Push `step` into `steps`, resolving an id collision with an +/// already-emitted step. A byte-identical re-emission (same id, parents, +/// actor, timestamp, change) is dropped — keeping it would only duplicate a +/// step that already exists — and a same-id-but-different step is re-IDed to a +/// fresh `#` so the original id stays recoverable and no data is lost. +/// Returns the id the step ended up under (the surviving id when dropped, the +/// new id when re-IDed), which the caller records in `turn_to_step` / +/// `last_step_id` so the DAG keeps pointing at a real step. +fn push_step(steps: &mut Vec, by_id: &mut HashMap, mut step: Step) -> String { + let id = step.step.id.clone(); + let Some(&existing) = by_id.get(&id) else { + by_id.insert(id.clone(), steps.len()); + steps.push(step); + return id; + }; + if steps_content_eq(&steps[existing], &step) { + return id; + } + let mut n = 2u32; + let mut renamed = format!("{id}#{n}"); + while by_id.contains_key(&renamed) { + n += 1; + renamed = format!("{id}#{n}"); + } + step.step.id = renamed.clone(); + by_id.insert(renamed.clone(), steps.len()); + steps.push(step); + renamed +} + +/// Whether two steps are the same entry — equal once serialized, so dropping +/// one is lossless. Step doesn't implement `PartialEq`, and serializing only +/// happens on an actual id collision (rare), so the cost is negligible. +fn steps_content_eq(a: &Step, b: &Step) -> bool { + serde_json::to_value(a).ok() == serde_json::to_value(b).ok() +} + fn actor_for_turn(turn: &Turn, provider: &str) -> String { match &turn.role { Role::User => "human:user".to_string(), + Role::Assistant if turn.model.as_deref() == Some("") => { + format!("tool:{}", provider) + } Role::Assistant => { let model = turn.model.as_deref().unwrap_or("unknown"); format!("agent:{}", model) @@ -672,7 +846,9 @@ pub fn unified_diff(path: &str, before: &str, after: &str) -> String { #[cfg(test)] mod tests { use super::*; - use crate::{DelegatedWork, EnvironmentSnapshot, TokenUsage, ToolInvocation, ToolResult}; + use crate::{ + Compaction, CompactionTrigger, DelegatedWork, EnvironmentSnapshot, TokenUsage, ToolResult, + }; fn base_turn(id: &str, role: Role) -> Turn { Turn { @@ -697,7 +873,7 @@ mod tests { fn view_with(turns: Vec) -> ConversationView { ConversationView { id: "abcdef012345".to_string(), - turns, + items: turns.into_iter().map(Item::Turn).collect(), provider_id: Some("pi".to_string()), ..Default::default() } @@ -731,7 +907,7 @@ mod tests { // ...and survives a JSON round-trip. let json = serde_json::to_string(&path).unwrap(); assert!( - json.contains(r#""kind":"https://toolpath.net/kinds/agent-coding-session/v1.1.0""#) + json.contains(r#""kind":"https://toolpath.net/kinds/agent-coding-session/v1.2.0""#) ); } @@ -758,7 +934,10 @@ mod tests { let path = derive_path(&view, &DeriveConfig::default()); let extracted = crate::extract::extract_conversation(&path); - let usage = extracted.turns[0] + let usage = extracted + .turns() + .next() + .unwrap() .token_usage .as_ref() .expect("token_usage survives round-trip"); @@ -821,6 +1000,87 @@ mod tests { assert_eq!(path.steps[0].step.actor, "agent:unknown"); } + #[test] + fn test_duplicate_id_identical_content_is_dropped() { + // The same turn id can appear twice with byte-identical content (a + // Claude compaction replay re-emitting an unchanged message). The + // re-emission is dropped: one step survives, and the conversation + // head resolves to it. + let mut first = base_turn("dup", Role::User); + first.text = "same".into(); + let mid = base_turn("mid", Role::Assistant); + let mut second = base_turn("dup", Role::User); + second.text = "same".into(); + let view = view_with(vec![first, mid, second]); + + let path = derive_path(&view, &DeriveConfig::default()); + let ids: Vec<&str> = path.steps.iter().map(|s| s.step.id.as_str()).collect(); + assert_eq!(ids, vec!["dup", "mid"], "identical re-emission dropped"); + // Head is the last surviving step in document order, not the earlier + // step the dropped duplicate collapsed into — so any real step after + // that earlier one (here `mid`) stays on the head's ancestry. + assert_eq!(path.path.head, "mid", "head is the last surviving step"); + } + + #[test] + fn test_dropped_final_duplicate_keeps_compaction_on_head_ancestry() { + // Regression for the head-regression bug: a compaction sits between a + // turn and that turn's byte-identical replay at the end of the stream. + // The replay is dropped, but `head` must remain the compaction (the + // last surviving step) so the boundary is on the head's ancestry + // rather than orphaned as a dead end. + let mut a = base_turn("a", Role::User); + a.text = "same".into(); + let c = Compaction { + id: "c".into(), + parent_id: Some("a".into()), + timestamp: "2026-01-01T00:00:00Z".into(), + ..Default::default() + }; + let mut replay = base_turn("a", Role::User); + replay.text = "same".into(); + + let mut view = view_with(vec![a]); + view.items.push(Item::Compaction(c)); + view.items.push(Item::Turn(replay)); + + let path = derive_path(&view, &DeriveConfig::default()); + let ids: Vec<&str> = path.steps.iter().map(|s| s.step.id.as_str()).collect(); + assert_eq!(ids, vec!["a", "c"], "byte-identical replay dropped"); + assert_eq!(path.path.head, "c", "head is the compaction, not turn a"); + assert_eq!(path.steps[1].step.parents, vec!["a".to_string()]); + } + + #[test] + fn test_duplicate_id_different_content_is_renamed() { + // The same turn id with DIFFERENT content keeps both steps: the + // collision is resolved by renaming the later one to a fresh id so the + // path stays unique — never dropping data, never erroring. + let mut first = base_turn("dup", Role::User); + first.text = "original".into(); + let mid = base_turn("mid", Role::Assistant); + let mut second = base_turn("dup", Role::User); + second.text = "replayed".into(); + let view = view_with(vec![first, mid, second]); + + let path = derive_path(&view, &DeriveConfig::default()); + let ids: Vec<&str> = path.steps.iter().map(|s| s.step.id.as_str()).collect(); + assert_eq!( + ids, + vec!["dup", "mid", "dup#2"], + "differing duplicate re-IDed to `#`, not dropped" + ); + assert_eq!(path.path.head, "dup#2", "head is the re-IDed final step"); + assert_eq!( + conv_change(&path.steps[0]).extra["text"], + serde_json::json!("original") + ); + assert_eq!( + conv_change(&path.steps[2]).extra["text"], + serde_json::json!("replayed") + ); + } + #[test] fn test_system_role() { let turn = base_turn("t1", Role::System); @@ -858,13 +1118,13 @@ mod tests { let other = base_turn("t4", Role::Other("bash".into())); let mut view = view_with(vec![user, assistant, system, other]); - view.events.push(crate::ConversationEvent { + view.items.push(Item::Event(crate::ConversationEvent { id: "e1".into(), timestamp: "2026-01-01T00:00:00Z".into(), parent_id: None, event_type: "attachment".into(), data: HashMap::new(), - }); + })); let path = derive_path(&view, &DeriveConfig::default()); let graph = serde_json::json!({ @@ -952,13 +1212,13 @@ mod tests { other.text = "tool output".into(); let mut view = view_with(vec![user, assistant, system, other]); - view.events.push(crate::ConversationEvent { + view.items.push(Item::Event(crate::ConversationEvent { id: "e1".into(), timestamp: "2026-01-01T00:00:00Z".into(), parent_id: None, event_type: "attachment".into(), data: HashMap::new(), - }); + })); let path = derive_path(&view, &DeriveConfig::default()); assert_eq!( @@ -969,7 +1229,7 @@ mod tests { let schema_src = std::fs::read_to_string(concat!( env!("CARGO_MANIFEST_DIR"), - "/../path-cli/kinds/agent-coding-session/v1.1.0/schema.json" + "/../path-cli/kinds/agent-coding-session/v1.2.0/schema.json" )) .expect("read kind schema"); let schema: serde_json::Value = serde_json::from_str(&schema_src).unwrap(); @@ -1545,6 +1805,110 @@ mod tests { ); } + #[test] + fn test_compaction_emits_compact_step_and_rewires_dag() { + // [Turn(a), Compaction(c, parent=a), Turn(b, parent=c)] must derive + // to three steps in order, with the middle step carrying a + // `conversation.compact` change whose extras survive, parented on a, + // and turn b rewired through the boundary onto c. + let a = base_turn("a", Role::User); + let mut b = base_turn("b", Role::Assistant); + b.parent_id = Some("c".into()); + b.model = Some("m".into()); + + let c = Compaction { + id: "c".into(), + parent_id: Some("a".into()), + timestamp: "2026-01-01T00:00:00Z".into(), + trigger: Some(CompactionTrigger::Manual), + summary: Some("s".into()), + pre_tokens: Some(100), + kept: vec!["a".into(), "a".into()], + }; + + let mut view = view_with(vec![a]); + view.items.push(Item::Compaction(c)); + view.items.push(Item::Turn(b)); + + let path = derive_path(&view, &DeriveConfig::default()); + + let ids: Vec<&str> = path.steps.iter().map(|s| s.step.id.as_str()).collect(); + assert_eq!(ids, vec!["a", "c", "b"], "items emitted in order"); + + let compact = &path.steps[1]; + assert_eq!(compact.step.parents, vec!["a".to_string()]); + let sc = conv_change(compact); + assert_eq!(sc.change_type, "conversation.compact"); + assert_eq!(sc.extra["trigger"], serde_json::json!("manual")); + assert_eq!(sc.extra["summary"], serde_json::json!("s")); + assert_eq!(sc.extra["pre_tokens"], serde_json::json!(100)); + assert_eq!(sc.extra["kept"], serde_json::json!(["a", "a"])); + + // Turn b rewired through the boundary: its parent is the compact step. + assert_eq!(path.steps[2].step.parents, vec!["c".to_string()]); + } + + #[test] + fn test_compaction_omits_optional_extras_when_absent() { + let a = base_turn("a", Role::User); + let c = Compaction { + id: "c".into(), + parent_id: Some("a".into()), + timestamp: "2026-01-01T00:00:00Z".into(), + ..Default::default() + }; + let mut view = view_with(vec![a]); + view.items.push(Item::Compaction(c)); + + let path = derive_path(&view, &DeriveConfig::default()); + let sc = conv_change(&path.steps[1]); + assert_eq!(sc.change_type, "conversation.compact"); + assert!(!sc.extra.contains_key("trigger")); + assert!(!sc.extra.contains_key("summary")); + assert!(!sc.extra.contains_key("pre_tokens")); + assert!(!sc.extra.contains_key("kept")); + } + + #[test] + fn test_compaction_synthesizes_id_when_empty() { + let a = base_turn("a", Role::User); + let c = Compaction { + id: String::new(), + parent_id: Some("a".into()), + timestamp: "2026-01-01T00:00:00Z".into(), + ..Default::default() + }; + let mut view = view_with(vec![a]); + view.items.push(Item::Compaction(c)); + + let path = derive_path(&view, &DeriveConfig::default()); + assert_eq!(path.steps[1].step.id, "compact-0001"); + } + + #[test] + fn test_compaction_with_unresolvable_parent_chains_onto_previous_step() { + // A compaction whose parent_id resolves to no emitted step must chain + // onto whatever step came before it (the same fallback events use), + // not become a disconnected root that orphans the pre-compaction turns. + let a = base_turn("a", Role::User); + let c = Compaction { + id: "c".into(), + parent_id: Some("ghost".into()), // resolves to nothing + timestamp: "2026-01-01T00:00:00Z".into(), + ..Default::default() + }; + let mut view = view_with(vec![a]); + view.items.push(Item::Compaction(c)); + + let path = derive_path(&view, &DeriveConfig::default()); + assert_eq!(path.steps[1].step.id, "c"); + assert_eq!( + path.steps[1].step.parents, + vec!["a".to_string()], + "unresolvable parent falls back to the previous step, not an empty (root) parent set" + ); + } + #[test] fn test_serde_roundtrip() { let mut t1 = base_turn("t1", Role::User); diff --git a/crates/toolpath-convo/src/extract.rs b/crates/toolpath-convo/src/extract.rs index c1d8380..efb048f 100644 --- a/crates/toolpath-convo/src/extract.rs +++ b/crates/toolpath-convo/src/extract.rs @@ -13,8 +13,9 @@ use chrono::DateTime; use toolpath::v1::{Path, Step}; use crate::{ - ConversationEvent, ConversationView, DelegatedWork, EnvironmentSnapshot, FileMutation, - ProducerInfo, Role, SessionBase, TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, + Compaction, CompactionTrigger, ConversationEvent, ConversationView, DelegatedWork, + EnvironmentSnapshot, FileMutation, Item, ProducerInfo, Role, SessionBase, TokenUsage, + ToolCategory, ToolInvocation, ToolResult, Turn, }; /// Extract a [`ConversationView`] from a toolpath [`Path`] document. @@ -71,7 +72,8 @@ pub fn extract_conversation(path: &Path) -> ConversationView { view.producer = Some(p); } - // Map from step ID → index into view.turns, for parent lookups. + // Map from step ID → index into view.items (of a turn item), for + // parent lookups when attaching tool invocations. let mut step_to_turn: HashMap<&str, usize> = HashMap::new(); // Track files_changed for dedup in insertion order. let mut files_seen: HashSet = HashSet::new(); @@ -152,9 +154,9 @@ pub fn extract_conversation(path: &Path) -> ConversationView { if !step_mutations.is_empty() { turn.file_mutations = std::mem::take(&mut step_mutations); } - let idx = view.turns.len(); + let idx = view.items.len(); step_to_turn.insert(&step.step.id, idx); - view.turns.push(turn); + view.items.push(Item::Turn(turn)); } "conversation.event" => { let event_type = structural @@ -189,7 +191,39 @@ pub fn extract_conversation(path: &Path) -> ConversationView { event_type, data, }; - view.events.push(event); + view.items.push(Item::Event(event)); + } + "conversation.compact" => { + let trigger = structural + .extra + .get("trigger") + .and_then(|v| v.as_str()) + .and_then(|s| match s { + "auto" => Some(CompactionTrigger::Auto), + "manual" => Some(CompactionTrigger::Manual), + _ => None, + }); + let summary = structural + .extra + .get("summary") + .and_then(|v| v.as_str()) + .map(|s| s.to_string()); + let pre_tokens = structural.extra.get("pre_tokens").and_then(|v| v.as_u64()); + let kept = structural + .extra + .get("kept") + .and_then(|v| serde_json::from_value::>(v.clone()).ok()) + .unwrap_or_default(); + let compaction = Compaction { + id: step.step.id.clone(), + parent_id: step.step.parents.first().cloned(), + timestamp: step.step.timestamp.clone(), + trigger, + summary, + pre_tokens, + kept, + }; + view.items.push(Item::Compaction(compaction)); } "tool.invoke" => { let invocation = build_tool_invocation(&structural.extra); @@ -206,8 +240,9 @@ pub fn extract_conversation(path: &Path) -> ConversationView { // Attach to parent turn. if let Some(parent_id) = step.step.parents.first() && let Some(&turn_idx) = step_to_turn.get(parent_id.as_str()) + && let Some(Item::Turn(t)) = view.items.get_mut(turn_idx) { - view.turns[turn_idx].tool_uses.push(invocation); + t.tool_uses.push(invocation); } } _ => { @@ -220,7 +255,7 @@ pub fn extract_conversation(path: &Path) -> ConversationView { // Compute total_usage by summing across turns. let mut has_any_usage = false; let mut total = TokenUsage::default(); - for turn in &view.turns { + for turn in view.turns() { if let Some(usage) = &turn.token_usage { has_any_usage = true; total.input_tokens = add_opt(total.input_tokens, usage.input_tokens); @@ -233,14 +268,17 @@ pub fn extract_conversation(path: &Path) -> ConversationView { view.total_usage = Some(total); } - // Parse timestamps from first/last turns. - if let Some(first) = view.turns.first() { - view.started_at = DateTime::parse_from_rfc3339(&first.timestamp) + // Parse timestamps from first/last turns. Clone the strings out first + // so the `turns()` borrow ends before we assign back into `view`. + let first_ts = view.turns().next().map(|t| t.timestamp.clone()); + let last_ts = view.turns().last().map(|t| t.timestamp.clone()); + if let Some(ts) = first_ts { + view.started_at = DateTime::parse_from_rfc3339(&ts) .ok() .map(|dt| dt.with_timezone(&chrono::Utc)); } - if let Some(last) = view.turns.last() { - view.last_activity = DateTime::parse_from_rfc3339(&last.timestamp) + if let Some(ts) = last_ts { + view.last_activity = DateTime::parse_from_rfc3339(&ts) .ok() .map(|dt| dt.with_timezone(&chrono::Utc)); } @@ -629,7 +667,7 @@ mod tests { let path = make_path(vec![]); let view = extract_conversation(&path); assert!(view.id.is_empty()); - assert!(view.turns.is_empty()); + assert!(view.turns().next().is_none()); assert!(view.total_usage.is_none()); assert!(view.started_at.is_none()); assert!(view.last_activity.is_none()); @@ -700,13 +738,14 @@ mod tests { ]); let view = extract_conversation(&path); - assert_eq!(view.turns.len(), 2); - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[0].text, "Fix the bug"); - assert_eq!(view.turns[0].id, "step-002"); - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[1].text, "I'll fix that."); - assert_eq!(view.turns[1].model.as_deref(), Some("claude-opus-4-6")); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 2); + assert_eq!(turns[0].role, Role::User); + assert_eq!(turns[0].text, "Fix the bug"); + assert_eq!(turns[0].id, "step-002"); + assert_eq!(turns[1].role, Role::Assistant); + assert_eq!(turns[1].text, "I'll fix that."); + assert_eq!(turns[1].model.as_deref(), Some("claude-opus-4-6")); } #[test] @@ -728,7 +767,10 @@ mod tests { )]); let view = extract_conversation(&path); - assert_eq!(view.turns[0].group_id.as_deref(), Some("msg_01abc")); + assert_eq!( + view.turns().next().unwrap().group_id.as_deref(), + Some("msg_01abc") + ); } #[test] @@ -769,16 +811,14 @@ mod tests { ]); let view = extract_conversation(&path); - assert_eq!(view.turns.len(), 1); - assert_eq!(view.turns[0].tool_uses.len(), 1); - assert_eq!(view.turns[0].tool_uses[0].id, "tu-001"); - assert_eq!(view.turns[0].tool_uses[0].name, "Read"); - assert_eq!( - view.turns[0].tool_uses[0].category, - Some(ToolCategory::FileRead) - ); - assert!(view.turns[0].tool_uses[0].result.is_some()); - assert!(!view.turns[0].tool_uses[0].result.as_ref().unwrap().is_error); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 1); + assert_eq!(turns[0].tool_uses.len(), 1); + assert_eq!(turns[0].tool_uses[0].id, "tu-001"); + assert_eq!(turns[0].tool_uses[0].name, "Read"); + assert_eq!(turns[0].tool_uses[0].category, Some(ToolCategory::FileRead)); + assert!(turns[0].tool_uses[0].result.is_some()); + assert!(!turns[0].tool_uses[0].result.as_ref().unwrap().is_error); } #[test] @@ -863,9 +903,10 @@ mod tests { )]); let view = extract_conversation(&path); - assert_eq!(view.turns.len(), 1); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 1); assert_eq!( - view.turns[0].thinking.as_deref(), + turns[0].thinking.as_deref(), Some("Let me think about this carefully...") ); } @@ -904,8 +945,9 @@ mod tests { ]); let view = extract_conversation(&path); - assert!(view.turns[0].parent_id.is_none()); - assert_eq!(view.turns[1].parent_id.as_deref(), Some("step-001")); + let turns: Vec<&Turn> = view.turns().collect(); + assert!(turns[0].parent_id.is_none()); + assert_eq!(turns[1].parent_id.as_deref(), Some("step-001")); } #[test] @@ -940,8 +982,9 @@ mod tests { let view = extract_conversation(&path); // Only the conversation.append step becomes a turn. - assert_eq!(view.turns.len(), 1); - assert_eq!(view.turns[0].text, "hello"); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 1); + assert_eq!(turns[0].text, "hello"); } #[test] @@ -984,9 +1027,10 @@ mod tests { ]); let view = extract_conversation(&path); - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[2].role, Role::System); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns[0].role, Role::User); + assert_eq!(turns[1].role, Role::Assistant); + assert_eq!(turns[2].role, Role::System); } #[test] @@ -1043,10 +1087,11 @@ mod tests { ]); let view = extract_conversation(&path); - assert_eq!(view.turns.len(), 1); - assert_eq!(view.turns[0].tool_uses.len(), 2); - assert_eq!(view.turns[0].tool_uses[0].id, "tu-001"); - assert_eq!(view.turns[0].tool_uses[1].id, "tu-002"); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 1); + assert_eq!(turns[0].tool_uses.len(), 2); + assert_eq!(turns[0].tool_uses[0].id, "tu-001"); + assert_eq!(turns[0].tool_uses[1].id, "tu-002"); } #[test] @@ -1155,7 +1200,7 @@ mod tests { )]); let view = extract_conversation(&path); - assert!(view.turns.is_empty()); + assert!(view.turns().next().is_none()); } #[test] @@ -1178,7 +1223,7 @@ mod tests { )]); let view = extract_conversation(&path); - let env = view.turns[0].environment.as_ref().unwrap(); + let env = view.turns().next().unwrap().environment.as_ref().unwrap(); assert_eq!(env.working_dir.as_deref(), Some("/home/alex/project")); assert_eq!(env.vcs_branch.as_deref(), Some("feature/cool")); assert!(env.vcs_revision.is_none()); @@ -1202,7 +1247,7 @@ mod tests { )]); let view = extract_conversation(&path); - assert!(view.turns[0].environment.is_none()); + assert!(view.turns().next().unwrap().environment.is_none()); } #[test] @@ -1284,22 +1329,23 @@ mod tests { ]); let view = extract_conversation(&path); - assert!(view.turns.is_empty()); - assert_eq!(view.events.len(), 2); + let events: Vec<&ConversationEvent> = view.events().collect(); + assert!(view.turns().next().is_none()); + assert_eq!(events.len(), 2); - assert_eq!(view.events[0].id, "step-001"); - assert_eq!(view.events[0].event_type, "attachment"); + assert_eq!(events[0].id, "step-001"); + assert_eq!(events[0].event_type, "attachment"); assert_eq!( - view.events[0].data["cwd"], + events[0].data["cwd"], serde_json::json!("/home/alex/project") ); - assert_eq!(view.events[0].data["version"], serde_json::json!("1.0.30")); - assert!(view.events[0].parent_id.is_none()); + assert_eq!(events[0].data["version"], serde_json::json!("1.0.30")); + assert!(events[0].parent_id.is_none()); - assert_eq!(view.events[1].id, "step-002"); - assert_eq!(view.events[1].event_type, "file-history-snapshot"); - assert_eq!(view.events[1].parent_id.as_deref(), Some("step-001")); - assert!(view.events[1].data.contains_key("snapshot")); + assert_eq!(events[1].id, "step-002"); + assert_eq!(events[1].event_type, "file-history-snapshot"); + assert_eq!(events[1].parent_id.as_deref(), Some("step-001")); + assert!(events[1].data.contains_key("snapshot")); } #[test] @@ -1317,8 +1363,87 @@ mod tests { )]); let view = extract_conversation(&path); - assert_eq!(view.events.len(), 1); - assert_eq!(view.events[0].event_type, "unknown"); + let events: Vec<&ConversationEvent> = view.events().collect(); + assert_eq!(events.len(), 1); + assert_eq!(events[0].event_type, "unknown"); + } + + #[test] + fn test_compaction_round_trips_through_derive_and_extract() { + use crate::DeriveConfig; + + let a = Turn { + id: "a".into(), + parent_id: None, + role: Role::User, + timestamp: "2026-01-01T00:00:00Z".into(), + text: "first".into(), + thinking: None, + tool_uses: vec![], + model: None, + stop_reason: None, + token_usage: None, + environment: None, + delegations: vec![], + file_mutations: vec![], + group_id: None, + attributed_token_usage: None, + }; + let b = Turn { + id: "b".into(), + parent_id: Some("c".into()), + role: Role::Assistant, + timestamp: "2026-01-01T00:00:02Z".into(), + text: "second".into(), + thinking: None, + tool_uses: vec![], + model: Some("m".into()), + stop_reason: None, + token_usage: None, + environment: None, + delegations: vec![], + file_mutations: vec![], + group_id: None, + attributed_token_usage: None, + }; + let c = Compaction { + id: "c".into(), + parent_id: Some("a".into()), + timestamp: "2026-01-01T00:00:01Z".into(), + trigger: Some(CompactionTrigger::Manual), + summary: Some("condensed".into()), + pre_tokens: Some(4096), + kept: vec!["a".into(), "a".into()], + }; + + let source = ConversationView { + id: "sess-1".into(), + items: vec![Item::Turn(a), Item::Compaction(c), Item::Turn(b)], + provider_id: Some("claude-code".into()), + ..Default::default() + }; + + let path = crate::derive::derive_path(&source, &DeriveConfig::default()); + let view = extract_conversation(&path); + + // Item order [Turn, Compaction, Turn] is preserved. + assert_eq!(view.items.len(), 3); + assert!(matches!(view.items[0], Item::Turn(_))); + assert!(matches!(view.items[2], Item::Turn(_))); + + let Item::Compaction(rc) = &view.items[1] else { + panic!( + "middle item should be a compaction, got {:?}", + view.items[1] + ); + }; + assert_eq!(rc.id, "c"); + assert_eq!(rc.parent_id.as_deref(), Some("a")); + assert_eq!(rc.timestamp, "2026-01-01T00:00:01Z"); + assert_eq!(rc.trigger, Some(CompactionTrigger::Manual)); + assert_eq!(rc.summary.as_deref(), Some("condensed")); + assert_eq!(rc.pre_tokens, Some(4096)); + assert_eq!(rc.kept, vec!["a".to_string(), "a".to_string()]); } #[test] @@ -1352,9 +1477,11 @@ mod tests { ]); let view = extract_conversation(&path); - assert_eq!(view.turns.len(), 1); - assert_eq!(view.events.len(), 1); - assert_eq!(view.turns[0].text, "hello"); - assert_eq!(view.events[0].event_type, "system"); + let turns: Vec<&Turn> = view.turns().collect(); + let events: Vec<&ConversationEvent> = view.events().collect(); + assert_eq!(turns.len(), 1); + assert_eq!(events.len(), 1); + assert_eq!(turns[0].text, "hello"); + assert_eq!(events[0].event_type, "system"); } } diff --git a/crates/toolpath-convo/src/lib.rs b/crates/toolpath-convo/src/lib.rs index dcf3c3e..eeecda2 100644 --- a/crates/toolpath-convo/src/lib.rs +++ b/crates/toolpath-convo/src/lib.rs @@ -195,6 +195,102 @@ pub struct ConversationEvent { pub data: HashMap, } +/// How a context compaction was triggered. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CompactionTrigger { + /// The harness compacted automatically on context overflow. + Auto, + /// The user ran a compact command (e.g. `/compact`). + Manual, +} + +/// A context-compaction boundary: the agent summarized older turns and +/// continued. Every harness records this as an inline marker within one +/// session (never a new session), so it's modeled as one item in the +/// conversation's ordered stream and projects to a `conversation.compact` +/// step. All fields beyond identity are optional — harnesses vary in what +/// they persist (see `docs/agents/formats/`). +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct Compaction { + /// Unique identifier (the harness boundary id, or synthesized). + pub id: String, + + /// Logical parent — the last turn before compaction. Projects to the + /// step's `parents`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parent_id: Option, + + /// When the compaction occurred (ISO 8601). + pub timestamp: String, + + /// Auto (overflow) vs. manual (`/compact`). `None` when the harness + /// doesn't persist which (Codex, Pi). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub trigger: Option, + + /// The summary text that replaced the condensed prefix. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub summary: Option, + + /// Context token count just before compaction. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pre_tokens: Option, + + /// Ids of the prior turns that survive verbatim into the + /// post-compaction context window — the harness-agnostic "what's kept". + /// May be non-contiguous (Claude keeps a recent tail PLUS a scattered + /// set of pinned tool results). Empty = wholesale (the summary replaced + /// everything). Each harness's projector renders this set in its own + /// form: Claude re-emits these turns on-chain before the boundary; + /// opencode/Pi anchor a kept tail at the earliest id; Codex keeps none. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub kept: Vec, +} + +/// One element of a conversation's ordered stream — a turn, a +/// non-conversational event, or a compaction boundary. Keeping all three +/// in a single ordered `Vec` preserves their exact interleaving, so +/// `derive_path` ↔ `extract_conversation` round-trips losslessly and a +/// compaction boundary lands at its true position. +// `Turn` is much larger than the other variants but is also the dominant +// item in any conversation, so we keep it inline rather than box the hot +// path — boxing would add an allocation per turn to save space only on +// the rarer Event/Compaction items. +#[allow(clippy::large_enum_variant)] +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum Item { + Turn(Turn), + Event(ConversationEvent), + Compaction(Compaction), +} + +impl Item { + /// The turn, if this item is one. + pub fn as_turn(&self) -> Option<&Turn> { + match self { + Item::Turn(t) => Some(t), + _ => None, + } + } + + /// The event, if this item is one. + pub fn as_event(&self) -> Option<&ConversationEvent> { + match self { + Item::Event(e) => Some(e), + _ => None, + } + } + + /// The compaction, if this item is one. + pub fn as_compaction(&self) -> Option<&Compaction> { + match self { + Item::Compaction(c) => Some(c), + _ => None, + } + } +} + /// Toolpath's classification of what a tool invocation does. /// /// This is toolpath's ontology, not a provider-specific label. Provider @@ -333,8 +429,11 @@ pub struct ConversationView { /// When the conversation was last active. pub last_activity: Option>, - /// Ordered turns. - pub turns: Vec, + /// The conversation's ordered stream: turns, non-conversational + /// events, and compaction boundaries, interleaved in real order so the + /// interleaving survives derive ↔ extract round-trips. Use `turns()` / + /// `events()` / `compactions()` to read one kind. + pub items: Vec, /// Aggregate token usage across all turns. #[serde(default, skip_serializing_if = "Option::is_none")] @@ -355,12 +454,6 @@ pub struct ConversationView { #[serde(default, skip_serializing_if = "Vec::is_empty")] pub session_ids: Vec, - /// Non-conversational events (hooks, snapshots, metadata) in order. - /// These are provider-specific entries that aren't turns but need to - /// be preserved for round-trip fidelity. - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub events: Vec, - /// Path-level base: where this session was rooted (`cwd`, git /// commit/branch/remote). Projects directly to `Path.base`. #[serde(default, skip_serializing_if = "Option::is_none")] @@ -373,11 +466,25 @@ pub struct ConversationView { } impl ConversationView { + /// All turns, in order, skipping events and compaction boundaries. + pub fn turns(&self) -> impl Iterator { + self.items.iter().filter_map(Item::as_turn) + } + + /// All non-conversational events, in order. + pub fn events(&self) -> impl Iterator { + self.items.iter().filter_map(Item::as_event) + } + + /// All compaction boundaries, in order. + pub fn compactions(&self) -> impl Iterator { + self.items.iter().filter_map(Item::as_compaction) + } + /// Title derived from the first user turn, truncated to `max_len` characters. pub fn title(&self, max_len: usize) -> Option { let text = self - .turns - .iter() + .turns() .find(|t| t.role == Role::User && !t.text.is_empty()) .map(|t| &t.text)?; @@ -391,18 +498,18 @@ impl ConversationView { /// All turns with the given role. pub fn turns_by_role(&self, role: &Role) -> Vec<&Turn> { - self.turns.iter().filter(|t| &t.role == role).collect() + self.turns().filter(|t| &t.role == role).collect() } /// Turns added after the turn with the given ID. /// /// If the ID is not found, returns all turns. If the ID is the last - /// turn, returns an empty slice. - pub fn turns_since(&self, turn_id: &str) -> &[Turn] { - match self.turns.iter().position(|t| t.id == turn_id) { - Some(idx) if idx + 1 < self.turns.len() => &self.turns[idx + 1..], - Some(_) => &[], - None => &self.turns, + /// turn, returns an empty vec. + pub fn turns_since(&self, turn_id: &str) -> Vec<&Turn> { + let turns: Vec<&Turn> = self.turns().collect(); + match turns.iter().position(|t| t.id == turn_id) { + Some(idx) => turns[idx + 1..].to_vec(), + None => turns, } } } @@ -573,8 +680,8 @@ mod tests { id: "sess-1".into(), started_at: None, last_activity: None, - turns: vec![ - Turn { + items: vec![ + Item::Turn(Turn { id: "t1".into(), parent_id: None, group_id: None, @@ -590,8 +697,8 @@ mod tests { environment: None, delegations: vec![], file_mutations: Vec::new(), - }, - Turn { + }), + Item::Turn(Turn { id: "t2".into(), parent_id: Some("t1".into()), group_id: None, @@ -622,8 +729,8 @@ mod tests { environment: None, delegations: vec![], file_mutations: Vec::new(), - }, - Turn { + }), + Item::Turn(Turn { id: "t3".into(), parent_id: Some("t2".into()), group_id: None, @@ -639,13 +746,12 @@ mod tests { environment: None, delegations: vec![], file_mutations: Vec::new(), - }, + }), ], total_usage: None, provider_id: None, files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } @@ -670,12 +776,11 @@ mod tests { id: "empty".into(), started_at: None, last_activity: None, - turns: vec![], + items: vec![], total_usage: None, provider_id: None, files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() }; assert!(view.title(50).is_none()); @@ -730,7 +835,8 @@ mod tests { #[test] fn test_turn_serde_roundtrip() { - let turn = &sample_view().turns[1]; + let view = sample_view(); + let turn = view.turns().nth(1).unwrap(); let json = serde_json::to_string(turn).unwrap(); let back: Turn = serde_json::from_str(&json).unwrap(); assert_eq!(back.id, "t2"); @@ -746,15 +852,17 @@ mod tests { let json = serde_json::to_string(&view).unwrap(); let back: ConversationView = serde_json::from_str(&json).unwrap(); assert_eq!(back.id, "sess-1"); - assert_eq!(back.turns.len(), 3); + assert_eq!(back.turns().count(), 3); } #[test] fn test_watcher_event_variants() { - let turn_event = WatcherEvent::Turn(Box::new(sample_view().turns[0].clone())); + let view = sample_view(); + let turn_event = WatcherEvent::Turn(Box::new(view.turns().next().unwrap().clone())); assert!(matches!(turn_event, WatcherEvent::Turn(_))); - let updated_event = WatcherEvent::TurnUpdated(Box::new(sample_view().turns[1].clone())); + let updated_event = + WatcherEvent::TurnUpdated(Box::new(view.turns().nth(1).unwrap().clone())); assert!(matches!(updated_event, WatcherEvent::TurnUpdated(_))); let progress_event = WatcherEvent::Progress { @@ -766,7 +874,7 @@ mod tests { #[test] fn test_watcher_event_as_turn() { - let turn = sample_view().turns[0].clone(); + let turn = sample_view().turns().next().unwrap().clone(); let event = WatcherEvent::Turn(Box::new(turn.clone())); assert_eq!(event.as_turn().unwrap().id, "t1"); @@ -790,16 +898,17 @@ mod tests { assert_eq!(kind, "hook_progress"); assert_eq!(data["hookName"], "pre-commit"); - let turn = WatcherEvent::Turn(Box::new(sample_view().turns[0].clone())); + let turn = WatcherEvent::Turn(Box::new(sample_view().turns().next().unwrap().clone())); assert!(turn.as_progress().is_none()); } #[test] fn test_watcher_event_is_update() { - let turn = WatcherEvent::Turn(Box::new(sample_view().turns[0].clone())); + let turn = WatcherEvent::Turn(Box::new(sample_view().turns().next().unwrap().clone())); assert!(!turn.is_update()); - let updated = WatcherEvent::TurnUpdated(Box::new(sample_view().turns[0].clone())); + let updated = + WatcherEvent::TurnUpdated(Box::new(sample_view().turns().next().unwrap().clone())); assert!(updated.is_update()); let progress = WatcherEvent::Progress { @@ -811,10 +920,11 @@ mod tests { #[test] fn test_watcher_event_turn_id() { - let turn = WatcherEvent::Turn(Box::new(sample_view().turns[1].clone())); + let turn = WatcherEvent::Turn(Box::new(sample_view().turns().nth(1).unwrap().clone())); assert_eq!(turn.turn_id(), Some("t2")); - let updated = WatcherEvent::TurnUpdated(Box::new(sample_view().turns[0].clone())); + let updated = + WatcherEvent::TurnUpdated(Box::new(sample_view().turns().next().unwrap().clone())); assert_eq!(updated.turn_id(), Some("t1")); let progress = WatcherEvent::Progress { @@ -1014,7 +1124,7 @@ mod tests { id: "s1".into(), started_at: None, last_activity: None, - turns: vec![], + items: vec![], total_usage: Some(TokenUsage { input_tokens: Some(1000), output_tokens: Some(500), @@ -1025,7 +1135,6 @@ mod tests { provider_id: Some("claude-code".into()), files_changed: vec!["src/main.rs".into(), "src/lib.rs".into()], session_ids: vec![], - events: vec![], ..Default::default() }; let json = serde_json::to_string(&view).unwrap(); @@ -1041,8 +1150,8 @@ mod tests { #[test] fn test_conversation_view_old_format_deserializes() { - // Old-format JSON without total_usage/provider_id/files_changed - let json = r#"{"id":"s1","started_at":null,"last_activity":null,"turns":[]}"#; + // Minimal JSON without total_usage/provider_id/files_changed + let json = r#"{"id":"s1","started_at":null,"last_activity":null,"items":[]}"#; let view: ConversationView = serde_json::from_str(json).unwrap(); assert!(view.total_usage.is_none()); assert!(view.provider_id.is_none()); @@ -1107,25 +1216,24 @@ mod tests { id: "s1".into(), started_at: None, last_activity: None, - turns: vec![], - total_usage: None, - provider_id: None, - files_changed: vec![], - session_ids: vec![], - events: vec![ConversationEvent { + items: vec![Item::Event(ConversationEvent { id: "evt-1".into(), timestamp: "2026-01-01T00:00:00Z".into(), parent_id: None, event_type: "attachment".into(), data: HashMap::new(), - }], + })], + total_usage: None, + provider_id: None, + files_changed: vec![], + session_ids: vec![], ..Default::default() }; let json = serde_json::to_string(&view).unwrap(); - assert!(json.contains("events")); + assert!(json.contains("Event")); let back: ConversationView = serde_json::from_str(&json).unwrap(); - assert_eq!(back.events.len(), 1); - assert_eq!(back.events[0].event_type, "attachment"); + assert_eq!(back.events().count(), 1); + assert_eq!(back.events().next().unwrap().event_type, "attachment"); } #[test] @@ -1134,12 +1242,11 @@ mod tests { id: "s1".into(), started_at: None, last_activity: None, - turns: vec![], + items: vec![], total_usage: None, provider_id: None, files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() }; let json = serde_json::to_string(&view).unwrap(); @@ -1148,9 +1255,9 @@ mod tests { #[test] fn test_conversation_view_old_format_no_events() { - // Old-format JSON without events field should deserialize with empty vec - let json = r#"{"id":"s1","started_at":null,"last_activity":null,"turns":[]}"#; + // JSON without any event items should deserialize with no events + let json = r#"{"id":"s1","started_at":null,"last_activity":null,"items":[]}"#; let view: ConversationView = serde_json::from_str(json).unwrap(); - assert!(view.events.is_empty()); + assert!(view.events().next().is_none()); } } diff --git a/crates/toolpath-convo/src/project.rs b/crates/toolpath-convo/src/project.rs index 3a0a051..d6f3ab3 100644 --- a/crates/toolpath-convo/src/project.rs +++ b/crates/toolpath-convo/src/project.rs @@ -26,7 +26,7 @@ use std::any::Any; /// type Output = usize; /// /// fn project(&self, view: &ConversationView) -> Result { -/// Ok(view.turns.len()) +/// Ok(view.turns().count()) /// } /// } /// ``` @@ -77,7 +77,7 @@ where /// impl ConversationProjector for TurnCounter { /// type Output = usize; /// fn project(&self, view: &ConversationView) -> Result { -/// Ok(view.turns.len()) +/// Ok(view.turns().count()) /// } /// } /// @@ -134,7 +134,7 @@ impl AnyProjector { #[cfg(test)] mod tests { use super::*; - use crate::{Role, TokenUsage, ToolInvocation, ToolResult, Turn}; + use crate::{Item, Role, TokenUsage, ToolInvocation, ToolResult, Turn}; // ── helpers ────────────────────────────────────────────────────── @@ -143,12 +143,11 @@ mod tests { id: "sess-1".into(), started_at: None, last_activity: None, - turns: vec![], + items: vec![], total_usage: None, provider_id: None, files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } @@ -178,16 +177,15 @@ mod tests { id: "sess-2".into(), started_at: None, last_activity: None, - turns: vec![ - make_turn("t1", Role::User, "hello"), - make_turn("t2", Role::Assistant, "world"), - make_turn("t3", Role::User, "done"), + items: vec![ + Item::Turn(make_turn("t1", Role::User, "hello")), + Item::Turn(make_turn("t2", Role::Assistant, "world")), + Item::Turn(make_turn("t3", Role::User, "done")), ], total_usage: None, provider_id: Some("test-provider".into()), files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } @@ -198,7 +196,7 @@ mod tests { impl ConversationProjector for TurnCounter { type Output = usize; fn project(&self, view: &ConversationView) -> Result { - Ok(view.turns.len()) + Ok(view.turns().count()) } } @@ -306,7 +304,7 @@ mod tests { impl ConversationProjector for TextCollector { type Output = Vec; fn project(&self, view: &ConversationView) -> Result> { - Ok(view.turns.iter().map(|t| t.text.clone()).collect()) + Ok(view.turns().map(|t| t.text.clone()).collect()) } } @@ -315,8 +313,7 @@ mod tests { type Output = Vec; fn project(&self, view: &ConversationView) -> Result> { Ok(view - .turns - .iter() + .turns() .flat_map(|t| t.tool_uses.iter().map(|u| u.name.clone())) .collect()) } @@ -335,8 +332,7 @@ mod tests { id: "s3".into(), started_at: None, last_activity: None, - events: vec![], - turns: vec![Turn { + items: vec![Item::Turn(Turn { id: "t1".into(), parent_id: None, group_id: None, @@ -370,7 +366,7 @@ mod tests { environment: None, delegations: vec![], file_mutations: Vec::new(), - }], + })], total_usage: None, provider_id: None, files_changed: vec![], @@ -398,8 +394,7 @@ mod tests { type Output = u32; fn project(&self, view: &ConversationView) -> Result { Ok(view - .turns - .iter() + .turns() .filter_map(|t| t.token_usage.as_ref()) .filter_map(|u| u.input_tokens) .sum()) @@ -410,9 +405,8 @@ mod tests { id: "s4".into(), started_at: None, last_activity: None, - events: vec![], - turns: vec![ - Turn { + items: vec![ + Item::Turn(Turn { id: "t1".into(), parent_id: None, group_id: None, @@ -434,8 +428,8 @@ mod tests { environment: None, delegations: vec![], file_mutations: Vec::new(), - }, - Turn { + }), + Item::Turn(Turn { id: "t2".into(), parent_id: Some("t1".into()), group_id: None, @@ -457,7 +451,7 @@ mod tests { environment: None, delegations: vec![], file_mutations: Vec::new(), - }, + }), ], total_usage: None, provider_id: None, diff --git a/crates/toolpath-cursor/Cargo.toml b/crates/toolpath-cursor/Cargo.toml index ca09aa9..b02cf9c 100644 --- a/crates/toolpath-cursor/Cargo.toml +++ b/crates/toolpath-cursor/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-cursor" -version = "0.2.0" +version = "0.3.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-cursor/examples/dump_fixture.rs b/crates/toolpath-cursor/examples/dump_fixture.rs index e6fba08..77be98b 100644 --- a/crates/toolpath-cursor/examples/dump_fixture.rs +++ b/crates/toolpath-cursor/examples/dump_fixture.rs @@ -26,8 +26,8 @@ use std::path::PathBuf; use serde_json::Value; use toolpath_convo::{ - ConversationProjector, ConversationView, EnvironmentSnapshot, ProducerInfo, Role, SessionBase, - ToolInvocation, Turn, + ConversationProjector, ConversationView, EnvironmentSnapshot, Item, ProducerInfo, Role, + SessionBase, ToolInvocation, Turn, }; use toolpath_cursor::project::CursorProjector; use toolpath_cursor::provider::tool_category; @@ -81,7 +81,9 @@ fn capture_from_db( let chosen_id = composer_override.unwrap_or_else(|| { let mut chosen: Option<(String, usize)> = None; for id in &ids { - let Ok(s) = mgr.read_session(id) else { continue }; + let Ok(s) = mgr.read_session(id) else { + continue; + }; let n = s.bubbles.len(); eprintln!(" {} → {} bubbles", id, n); if chosen.as_ref().is_none_or(|(_, prev)| n > *prev) { @@ -126,8 +128,12 @@ fn capture_from_db( fn referenced_blob_hashes(session: &CursorSession) -> std::collections::HashSet { let mut needed = std::collections::HashSet::new(); for b in &session.bubbles { - let Some(tf) = &b.tool_former_data else { continue }; - let Ok(Some(result)) = tf.parse_result() else { continue }; + let Some(tf) = &b.tool_former_data else { + continue; + }; + let Ok(Some(result)) = tf.parse_result() else { + continue; + }; for field in ["beforeContentId", "afterContentId"] { if let Some(raw) = result.get(field).and_then(|v| v.as_str()) && let Some(hash) = raw.strip_prefix(CONTENT_PREFIX) @@ -142,15 +148,14 @@ fn referenced_blob_hashes(session: &CursorSession) -> std::collections::HashSet< // ── Mode 2: from a cursor-agent CLI JSONL transcript ────────────────── fn capture_from_jsonl(path: &str) -> CursorSession { - let content = fs::read_to_string(path) - .unwrap_or_else(|e| panic!("read {path}: {e}")); + let content = fs::read_to_string(path).unwrap_or_else(|e| panic!("read {path}: {e}")); let composer_id = composer_id_from_jsonl_path(path); let workspace = workspace_from_jsonl_path(path); let view = view_from_jsonl(&content, &composer_id, &workspace); eprintln!( "parsed JSONL: {} turns ({} tool uses)", - view.turns.len(), - view.turns.iter().map(|t| t.tool_uses.len()).sum::(), + view.turns().count(), + view.turns().map(|t| t.tool_uses.len()).sum::(), ); CursorProjector::new() .with_composer_id(composer_id) @@ -271,11 +276,7 @@ fn view_from_jsonl( role, // Synthesize plausible monotonic timestamps; the // transcript carries no real ones. - timestamp: format!( - "2026-06-01T{:02}:{:02}:00Z", - line_no / 60, - line_no % 60 - ), + timestamp: format!("2026-06-01T{:02}:{:02}:00Z", line_no / 60, line_no % 60), text, thinking: None, tool_uses, @@ -298,14 +299,12 @@ fn view_from_jsonl( version: Some("cursor-agent".into()), }), base: Some(SessionBase { - working_dir: workspace - .as_ref() - .map(|w| w.to_string_lossy().into_owned()), + working_dir: workspace.as_ref().map(|w| w.to_string_lossy().into_owned()), vcs_branch: None, vcs_revision: None, vcs_remote: None, }), - turns, + items: turns.into_iter().map(Item::Turn).collect(), ..Default::default() } } diff --git a/crates/toolpath-cursor/src/derive.rs b/crates/toolpath-cursor/src/derive.rs index 8642f0c..9549175 100644 --- a/crates/toolpath-cursor/src/derive.rs +++ b/crates/toolpath-cursor/src/derive.rs @@ -45,7 +45,10 @@ pub fn derive_path(session: &CursorSession, config: &DeriveConfig) -> Path { } /// Derive a `Path` from each of several Cursor sessions. -pub fn derive_project(sessions: &[CursorSession], config: &DeriveConfig) -> Vec { +pub fn derive_project( + sessions: &[CursorSession], + config: &DeriveConfig, +) -> Vec { sessions.iter().map(|s| derive_path(s, config)).collect() } diff --git a/crates/toolpath-cursor/src/io.rs b/crates/toolpath-cursor/src/io.rs index 8a40261..9ec690d 100644 --- a/crates/toolpath-cursor/src/io.rs +++ b/crates/toolpath-cursor/src/io.rs @@ -88,10 +88,7 @@ impl CursorIO { match db.load_session(&head.composer_id) { Ok(s) => out.push(to_metadata(&s)), Err(e) => { - eprintln!( - "Warning: skipping composer {}: {}", - head.composer_id, e - ); + eprintln!("Warning: skipping composer {}: {}", head.composer_id, e); } } } @@ -138,10 +135,7 @@ impl CursorIO { return None; } let slug = paths::slug_from_abs_path(&abs); - let p = self - .resolver - .transcript_path(&slug, session.id()) - .ok()?; + let p = self.resolver.transcript_path(&slug, session.id()).ok()?; p.exists().then_some(p) } } @@ -212,7 +206,10 @@ mod tests { assert_eq!(metas[0].id, "c1"); assert_eq!(metas[0].message_count, 3); assert_eq!(metas[0].first_user_message.as_deref(), Some("hello")); - assert_eq!(metas[0].workspace_path.as_deref(), Some(std::path::Path::new("/p"))); + assert_eq!( + metas[0].workspace_path.as_deref(), + Some(std::path::Path::new("/p")) + ); } #[test] diff --git a/crates/toolpath-cursor/src/paths.rs b/crates/toolpath-cursor/src/paths.rs index 91400f3..47d63e3 100644 --- a/crates/toolpath-cursor/src/paths.rs +++ b/crates/toolpath-cursor/src/paths.rs @@ -108,7 +108,10 @@ impl PathResolver { /// Path to the agent-transcripts folder for a project slug. pub fn project_transcripts_dir(&self, slug: &str) -> Result { - Ok(self.projects_dir()?.join(slug).join(AGENT_TRANSCRIPTS_SUBDIR)) + Ok(self + .projects_dir()? + .join(slug) + .join(AGENT_TRANSCRIPTS_SUBDIR)) } /// Path to the JSONL transcript file for a composer in a project. @@ -189,13 +192,10 @@ impl PathResolver { let Some(path_part) = folder_uri.strip_prefix("file://") else { continue; }; - let recorded = std::fs::canonicalize(path_part) - .unwrap_or_else(|_| PathBuf::from(path_part)); + let recorded = + std::fs::canonicalize(path_part).unwrap_or_else(|_| PathBuf::from(path_part)); if recorded == target { - let id = entry - .file_name() - .to_string_lossy() - .into_owned(); + let id = entry.file_name().to_string_lossy().into_owned(); return Ok(Some(id)); } } @@ -220,10 +220,7 @@ impl PathResolver { synthesize_id: impl FnOnce(&Path) -> String, ) -> Result { if let Some(id) = self.find_workspace_id(folder)? { - return Ok(EnsuredWorkspaceId { - id, - created: false, - }); + return Ok(EnsuredWorkspaceId { id, created: false }); } let id = synthesize_id(folder); let dir = self.workspace_storage_dir()?.join(&id); diff --git a/crates/toolpath-cursor/src/project.rs b/crates/toolpath-cursor/src/project.rs index 34bd0c7..7c96b5a 100644 --- a/crates/toolpath-cursor/src/project.rs +++ b/crates/toolpath-cursor/src/project.rs @@ -100,8 +100,7 @@ fn project_view( .workspace_path .clone() .or_else(|| { - view.turns - .iter() + view.turns() .find_map(|t| t.environment.as_ref()?.working_dir.clone()) .map(PathBuf::from) }) @@ -120,10 +119,7 @@ fn project_view( .filter(|s| !s.is_empty()) .unwrap_or_else(|| workspace_hash(&workspace_path_str)); - let title = cfg - .title - .clone() - .or_else(|| view.title(80)); + let title = cfg.title.clone().or_else(|| view.title(80)); let agent_backend = cfg .agent_backend @@ -134,33 +130,30 @@ fn project_view( .clone() .unwrap_or_else(|| DEFAULT_MODEL_NAME.to_string()); - let created_at = view - .started_at - .map(|t| t.timestamp_millis()) - .or_else(|| { - view.turns - .first() - .and_then(|t| parse_timestamp_ms(&t.timestamp)) - }); + let created_at = view.started_at.map(|t| t.timestamp_millis()).or_else(|| { + view.turns() + .next() + .and_then(|t| parse_timestamp_ms(&t.timestamp)) + }); let last_updated_at = view .last_activity .map(|t| t.timestamp_millis()) .or_else(|| { - view.turns + view.turns() .last() .and_then(|t| parse_timestamp_ms(&t.timestamp)) }) .or(created_at); let mut content_blobs: HashMap = HashMap::new(); - let mut bubbles: Vec = Vec::with_capacity(view.turns.len()); - let mut headers: Vec = Vec::with_capacity(view.turns.len()); + let mut bubbles: Vec = Vec::with_capacity(view.turns().count()); + let mut headers: Vec = Vec::with_capacity(view.turns().count()); let mut total_input: u64 = 0; let mut total_output: u64 = 0; let mut total_files: std::collections::HashSet = std::collections::HashSet::new(); - for turn in &view.turns { + for turn in view.turns() { if !is_projectable(turn) { continue; } @@ -295,10 +288,13 @@ fn build_bubble(turn: &Turn, content_blobs: &mut HashMap) -> Bub let model_info = if is_tool_bubble { None } else { - turn.model.as_deref().filter(|m| !m.is_empty()).map(|m| ModelInfo { - model_name: Some(m.to_string()), - extra: HashMap::new(), - }) + turn.model + .as_deref() + .filter(|m| !m.is_empty()) + .map(|m| ModelInfo { + model_name: Some(m.to_string()), + extra: HashMap::new(), + }) }; // Empty-text bubbles must omit richText — emitting an empty @@ -415,9 +411,7 @@ fn normalize_input_for_cursor(tool_id: u32, input: &Value) -> Value { }; let renames: &[(&str, &str)] = match tool_id { - crate::types::TOOL_RUN_TERMINAL_COMMAND_V2 => &[ - ("description", "commandDescription"), - ], + crate::types::TOOL_RUN_TERMINAL_COMMAND_V2 => &[("description", "commandDescription")], crate::types::TOOL_EDIT_FILE_V2 => &[ ("file_path", "relativeWorkspacePath"), ("filePath", "relativeWorkspacePath"), @@ -440,9 +434,7 @@ fn normalize_input_for_cursor(tool_id: u32, input: &Value) -> Value { ("target_directory", "path"), ("case_insensitive", "caseInsensitive"), ], - crate::types::TOOL_TASK_V2 => &[ - ("subagent_type", "subagentType"), - ], + crate::types::TOOL_TASK_V2 => &[("subagent_type", "subagentType")], _ => &[], }; @@ -450,7 +442,8 @@ fn normalize_input_for_cursor(tool_id: u32, input: &Value) -> Value { let mut renamed: std::collections::HashSet<&str> = std::collections::HashSet::new(); for (foreign, cursor) in renames { if let Some(v) = obj.get(*foreign) { - out.entry((*cursor).to_string()).or_insert_with(|| v.clone()); + out.entry((*cursor).to_string()) + .or_insert_with(|| v.clone()); renamed.insert(*foreign); } } @@ -463,11 +456,9 @@ fn normalize_input_for_cursor(tool_id: u32, input: &Value) -> Value { "parsingResult", "requestedSandboxPolicy", ]), - crate::types::TOOL_EDIT_FILE_V2 => Some(&[ - "relativeWorkspacePath", - "noCodeblock", - "cloudAgentEdit", - ]), + crate::types::TOOL_EDIT_FILE_V2 => { + Some(&["relativeWorkspacePath", "noCodeblock", "cloudAgentEdit"]) + } crate::types::TOOL_READ_FILE_V2 => Some(&[ "targetFile", "effectiveUri", @@ -476,24 +467,13 @@ fn normalize_input_for_cursor(tool_id: u32, input: &Value) -> Value { "startLineOneIndexed", "endLineOneIndexedInclusive", ]), - crate::types::TOOL_GLOB_FILE_SEARCH => Some(&[ - "globPattern", - "targetDirectory", - ]), - crate::types::TOOL_RIPGREP_RAW_SEARCH => Some(&[ - "pattern", - "path", - "caseInsensitive", - "includes", - "excludes", - ]), - crate::types::TOOL_TASK_V2 => Some(&[ - "description", - "prompt", - "subagentType", - "model", - "name", - ]), + crate::types::TOOL_GLOB_FILE_SEARCH => Some(&["globPattern", "targetDirectory"]), + crate::types::TOOL_RIPGREP_RAW_SEARCH => { + Some(&["pattern", "path", "caseInsensitive", "includes", "excludes"]) + } + crate::types::TOOL_TASK_V2 => { + Some(&["description", "prompt", "subagentType", "model", "name"]) + } _ => None, }; for (k, v) in obj { @@ -509,8 +489,10 @@ fn normalize_input_for_cursor(tool_id: u32, input: &Value) -> Value { } } if tool_id == crate::types::TOOL_EDIT_FILE_V2 { - out.entry("noCodeblock".to_string()).or_insert(Value::Bool(true)); - out.entry("cloudAgentEdit".to_string()).or_insert(Value::Bool(false)); + out.entry("noCodeblock".to_string()) + .or_insert(Value::Bool(true)); + out.entry("cloudAgentEdit".to_string()) + .or_insert(Value::Bool(false)); } Value::Object(out) } @@ -528,10 +510,7 @@ fn tool_id_and_name(tu: &ToolInvocation) -> (u32, String) { Some(ToolCategory::FileWrite) => (TOOL_EDIT_FILE_V2, "edit_file_v2".into()), Some(ToolCategory::FileRead) => (TOOL_READ_FILE_V2, "read_file_v2".into()), Some(ToolCategory::FileSearch) => (TOOL_GLOB_FILE_SEARCH, "glob_file_search".into()), - Some(ToolCategory::Network) => ( - crate::types::TOOL_WEB_SEARCH, - "web_search".into(), - ), + Some(ToolCategory::Network) => (crate::types::TOOL_WEB_SEARCH, "web_search".into()), Some(ToolCategory::Delegation) => (TOOL_TASK_V2, "task_v2".into()), None => (crate::types::TOOL_UNSPECIFIED, tu.name.clone()), } @@ -661,7 +640,9 @@ fn reconstruct_hunks_from_diff(diff: &str) -> (String, String) { fn register_blob(blobs: &mut HashMap, body: Option<&str>) -> Option { let body = body?; let hash = sha256_hex(body.as_bytes()); - blobs.entry(hash.clone()).or_insert_with(|| body.to_string()); + blobs + .entry(hash.clone()) + .or_insert_with(|| body.to_string()); Some(hash) } @@ -669,10 +650,12 @@ fn sha256_hex(bytes: &[u8]) -> String { use std::fmt::Write; let mut h = Sha256::new(); h.update(bytes); - h.finalize().iter().fold(String::with_capacity(64), |mut s, b| { - let _ = write!(s, "{b:02x}"); - s - }) + h.finalize() + .iter() + .fold(String::with_capacity(64), |mut s, b| { + let _ = write!(s, "{b:02x}"); + s + }) } fn workspace_hash(path: &str) -> String { @@ -684,7 +667,11 @@ fn parse_timestamp_ms(ts: &str) -> Option { DateTime::parse_from_rfc3339(ts) .ok() .map(|dt| dt.timestamp_millis()) - .or_else(|| ts.parse::>().ok().map(|dt| dt.timestamp_millis())) + .or_else(|| { + ts.parse::>() + .ok() + .map(|dt| dt.timestamp_millis()) + }) .or_else(|| ts.parse::().ok()) } @@ -975,16 +962,16 @@ mod tests { } fn view_with(turns: Vec) -> ConversationView { + use toolpath_convo::Item; ConversationView { id: "comp-1".into(), started_at: None, last_activity: None, - turns, + items: turns.into_iter().map(Item::Turn).collect(), total_usage: None, provider_id: Some("cursor".into()), files_changed: vec![], session_ids: vec![], - events: vec![], base: Some(SessionBase { working_dir: Some("/proj".into()), vcs_revision: None, @@ -1099,9 +1086,10 @@ mod tests { }]; let s = CursorProjector::new().project(&view_with(vec![t])).unwrap(); - assert!(s.content_blobs.contains_key( - "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" - )); + assert!( + s.content_blobs + .contains_key("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855") + ); let after_hash = sha256_hex("fn main() {}".as_bytes()); assert_eq!( s.content_blobs.get(&after_hash).map(String::as_str), @@ -1297,7 +1285,10 @@ mod tests { #[test] fn parse_hunk_header_basic() { assert_eq!(super::parse_hunk_header("@@ -1,3 +1,4 @@"), Some((1, 1))); - assert_eq!(super::parse_hunk_header("@@ -10,5 +20,7 @@ fn foo()"), Some((10, 20))); + assert_eq!( + super::parse_hunk_header("@@ -10,5 +20,7 @@ fn foo()"), + Some((10, 20)) + ); assert_eq!(super::parse_hunk_header("@@ -1 +1 @@"), Some((1, 1))); assert_eq!(super::parse_hunk_header("@@ -0,0 +1,3 @@"), Some((0, 1))); assert_eq!(super::parse_hunk_header("not a hunk"), None); @@ -1344,9 +1335,7 @@ mod tests { path: "/proj/x.rs".into(), tool_id: Some("tc1".into()), operation: Some("edit".into()), - raw_diff: Some( - "@@ -1,1 +1,1 @@\n-old\n+new\n".into(), - ), + raw_diff: Some("@@ -1,1 +1,1 @@\n-old\n+new\n".into()), before: None, after: None, rename_to: None, @@ -1361,8 +1350,14 @@ mod tests { assert_ne!(before_id, after_id); let before_hash = before_id.trim_start_matches("composer.content."); let after_hash = after_id.trim_start_matches("composer.content."); - assert_eq!(s.content_blobs.get(before_hash).map(String::as_str), Some("old\n")); - assert_eq!(s.content_blobs.get(after_hash).map(String::as_str), Some("new\n")); + assert_eq!( + s.content_blobs.get(before_hash).map(String::as_str), + Some("old\n") + ); + assert_eq!( + s.content_blobs.get(after_hash).map(String::as_str), + Some("new\n") + ); } #[test] @@ -1390,12 +1385,21 @@ mod tests { let s = CursorProjector::new().project(&view_with(vec![t])).unwrap(); let tf = s.bubbles[0].tool_former_data.as_ref().unwrap(); let result = tf.parse_result().unwrap().unwrap(); - let before_hash = result["beforeContentId"].as_str().unwrap() + let before_hash = result["beforeContentId"] + .as_str() + .unwrap() .trim_start_matches("composer.content."); - let after_hash = result["afterContentId"].as_str().unwrap() + let after_hash = result["afterContentId"] + .as_str() + .unwrap() .trim_start_matches("composer.content."); - assert_eq!(s.content_blobs.get(before_hash).map(String::as_str), Some("real before")); - assert_eq!(s.content_blobs.get(after_hash).map(String::as_str), Some("real after")); + assert_eq!( + s.content_blobs.get(before_hash).map(String::as_str), + Some("real before") + ); + assert_eq!( + s.content_blobs.get(after_hash).map(String::as_str), + Some("real after") + ); } - } diff --git a/crates/toolpath-cursor/src/provider.rs b/crates/toolpath-cursor/src/provider.rs index 02746ab..3b8f4cc 100644 --- a/crates/toolpath-cursor/src/provider.rs +++ b/crates/toolpath-cursor/src/provider.rs @@ -35,13 +35,13 @@ use crate::io::CursorIO; use crate::paths::PathResolver; use crate::reader::CONTENT_PREFIX; use crate::types::{ - Bubble, CursorSession, CursorSessionMetadata, ToolFormerData, BUBBLE_TYPE_ASSISTANT, - BUBBLE_TYPE_USER, TOOL_EDIT_FILE_V2, TOOL_RUN_TERMINAL_COMMAND_V2, tool_name_for_id, + BUBBLE_TYPE_ASSISTANT, BUBBLE_TYPE_USER, Bubble, CursorSession, CursorSessionMetadata, + TOOL_EDIT_FILE_V2, TOOL_RUN_TERMINAL_COMMAND_V2, ToolFormerData, tool_name_for_id, }; use toolpath_convo::{ ConversationMeta, ConversationProvider, ConversationView, ConvoError as ConvoTraitError, - EnvironmentSnapshot, FileMutation, ProducerInfo, Role, SessionBase, TokenUsage, ToolCategory, - ToolInvocation, ToolResult, Turn, unified_diff, + EnvironmentSnapshot, FileMutation, Item, ProducerInfo, Role, SessionBase, TokenUsage, + ToolCategory, ToolInvocation, ToolResult, Turn, unified_diff, }; /// The dispatch family used in `path.meta.source` and @@ -56,7 +56,9 @@ pub struct CursorConvo { impl CursorConvo { pub fn new() -> Self { - Self { io: CursorIO::new() } + Self { + io: CursorIO::new(), + } } pub fn with_resolver(resolver: PathResolver) -> Self { @@ -87,9 +89,10 @@ impl CursorConvo { // maintains roughly newest-first. Fall back to last_activity // when present so this stays right even if the header order // ever drifts. - let pick = metas - .iter() - .max_by_key(|m| m.last_activity.unwrap_or_else(chrono::DateTime::::default)); + let pick = metas.iter().max_by_key(|m| { + m.last_activity + .unwrap_or_else(chrono::DateTime::::default) + }); match pick { Some(m) => Ok(Some(self.read_session(&m.id)?)), None => Ok(None), @@ -178,24 +181,9 @@ fn category_by_name(name: &str) -> Option { // (`create_rm_files`, `save_file`, `undo_edit`, `apply_agent_diff`, // `reapply`), and the agent-side friendly aliases // `Write`/`StrReplace`/`edit`/`delete`/`Edit`. - "edit_file_v2" - | "edit_file" - | "edit" - | "Edit" - | "Write" - | "StrReplace" - | "delete_file" - | "delete" - | "new_edit" - | "new_file" - | "save_file" - | "reapply" - | "undo_edit" - | "apply_agent_diff" - | "create_rm_files" - | "add_test" - | "delete_test" - | "fix_lints" + "edit_file_v2" | "edit_file" | "edit" | "Edit" | "Write" | "StrReplace" | "delete_file" + | "delete" | "new_edit" | "new_file" | "save_file" | "reapply" | "undo_edit" + | "apply_agent_diff" | "create_rm_files" | "add_test" | "delete_test" | "fix_lints" | "fix_lints_subagent" => Some(ToolCategory::FileWrite), // ── FileRead ───────────────────────────────────────────── @@ -252,11 +240,9 @@ fn category_by_name(name: &str) -> Option { // outside the local fs / shell: web fetch + search, GitHub // PR retrieval, and MCP tool dispatch (`call_mcp_tool` // proxies a model-driven call to a remote MCP server). - "web_search" - | "web_fetch" - | "fetch_pull_request" - | "fetch" - | "call_mcp_tool" => Some(ToolCategory::Network), + "web_search" | "web_fetch" | "fetch_pull_request" | "fetch" | "call_mcp_tool" => { + Some(ToolCategory::Network) + } // ── Delegation ─────────────────────────────────────────── // `Task`/`task_v2` is the dispatch primitive. `TaskSubagent`, @@ -323,6 +309,14 @@ impl<'a> Builder<'a> { fn build(mut self) -> ConversationView { let mut prev_turn_id: Option = None; for bubble in &self.session.bubbles { + // Cursor's `/summarize` boundary marker (capabilityType 22) carries + // no recoverable summary or kept set — those live server-side, not + // in the local store — so there's nothing to derive a compaction + // from. Skip it rather than surface it as an empty turn. See + // docs/agents/formats/cursor.md. + if bubble.is_summarization() { + continue; + } let turn = match bubble.kind { BUBBLE_TYPE_USER => self.user_turn(bubble, prev_turn_id.as_deref()), BUBBLE_TYPE_ASSISTANT => self.assistant_turn(bubble, prev_turn_id.as_deref()), @@ -335,30 +329,22 @@ impl<'a> Builder<'a> { self.turns.push(turn); } - let started_at = self - .session - .started_at() - .or_else(|| { - self.session - .bubbles - .first() - .and_then(|b| b.created_at_utc()) - }); + let started_at = self.session.started_at().or_else(|| { + self.session + .bubbles + .first() + .and_then(|b| b.created_at_utc()) + }); let last_activity = self .session .last_activity() - .or_else(|| { - self.session - .bubbles - .last() - .and_then(|b| b.created_at_utc()) - }); + .or_else(|| self.session.bubbles.last().and_then(|b| b.created_at_utc())); ConversationView { id: self.session.id().to_string(), started_at, last_activity, - turns: self.turns, + items: self.turns.into_iter().map(Item::Turn).collect(), total_usage: if self.total_usage_set { Some(self.total_usage) } else { @@ -367,7 +353,6 @@ impl<'a> Builder<'a> { provider_id: Some(PROVIDER_ID.to_string()), files_changed: self.files_changed_order, session_ids: vec![self.session.id().to_string()], - events: Vec::new(), base: Some(SessionBase { working_dir: self .session @@ -552,11 +537,7 @@ impl<'a> Builder<'a> { } } - fn file_mutation_for_edit( - &self, - tf: &ToolFormerData, - tool_id: &str, - ) -> Option { + fn file_mutation_for_edit(&self, tf: &ToolFormerData, tool_id: &str) -> Option { let params = tf.parse_params().ok()?; let path = params .get("relativeWorkspacePath") @@ -633,14 +614,11 @@ fn result_to_text(tf: &ToolFormerData, v: &Value) -> String { // ids. We surface a deterministic summary so consumers // (audit, diff inspection) get something readable while the // structured file mutation carries the real payload. - let path = tf - .parse_params() - .ok() - .and_then(|p| { - p.get("relativeWorkspacePath") - .and_then(|v| v.as_str()) - .map(str::to_string) - }); + let path = tf.parse_params().ok().and_then(|p| { + p.get("relativeWorkspacePath") + .and_then(|v| v.as_str()) + .map(str::to_string) + }); match path { Some(p) => format!("edited {p}"), None => "edited file".into(), @@ -777,21 +755,25 @@ mod tests { assert_eq!(view.id, "c1"); assert_eq!(view.provider_id.as_deref(), Some("cursor")); - assert_eq!(view.turns.len(), 3); + let turns: Vec<&Turn> = view.turns().collect(); + assert_eq!(turns.len(), 3); - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[0].text, "hello"); + assert_eq!(turns[0].role, Role::User); + assert_eq!(turns[0].text, "hello"); - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[1].text, "hi back"); - assert_eq!(view.turns[1].model.as_deref(), Some("claude-opus-4-7")); - assert_eq!(view.turns[1].token_usage.as_ref().unwrap().input_tokens, Some(10)); + assert_eq!(turns[1].role, Role::Assistant); + assert_eq!(turns[1].text, "hi back"); + assert_eq!(turns[1].model.as_deref(), Some("claude-opus-4-7")); + assert_eq!( + turns[1].token_usage.as_ref().unwrap().input_tokens, + Some(10) + ); - assert_eq!(view.turns[2].role, Role::Assistant); - assert_eq!(view.turns[2].tool_uses.len(), 1); - assert_eq!(view.turns[2].tool_uses[0].name, "edit_file_v2"); + assert_eq!(turns[2].role, Role::Assistant); + assert_eq!(turns[2].tool_uses.len(), 1); + assert_eq!(turns[2].tool_uses[0].name, "edit_file_v2"); assert_eq!( - view.turns[2].tool_uses[0].category, + turns[2].tool_uses[0].category, Some(ToolCategory::FileWrite) ); } @@ -811,7 +793,7 @@ mod tests { fn file_mutation_populated_with_diff() { let (_t, mgr) = setup(); let view = session_to_view(&mgr.read_session("c1").unwrap()); - let edit_turn = &view.turns[2]; + let edit_turn = view.turns().nth(2).unwrap(); assert_eq!(edit_turn.file_mutations.len(), 1); let fm = &edit_turn.file_mutations[0]; assert_eq!(fm.path, "/p/x.rs"); @@ -833,9 +815,10 @@ mod tests { fn parent_id_chains_turns() { let (_t, mgr) = setup(); let view = session_to_view(&mgr.read_session("c1").unwrap()); - assert!(view.turns[0].parent_id.is_none()); - assert_eq!(view.turns[1].parent_id.as_deref(), Some("u1")); - assert_eq!(view.turns[2].parent_id.as_deref(), Some("a1")); + let turns: Vec<&Turn> = view.turns().collect(); + assert!(turns[0].parent_id.is_none()); + assert_eq!(turns[1].parent_id.as_deref(), Some("u1")); + assert_eq!(turns[2].parent_id.as_deref(), Some("a1")); } #[test] @@ -850,7 +833,7 @@ mod tests { let r = crate::reader::DbReader::open(f.path()).unwrap(); let s = r.load_session("cs").unwrap(); let view = session_to_view(&s); - let tool = &view.turns[0].tool_uses[0]; + let tool = &view.turns().next().unwrap().tool_uses[0]; assert_eq!(tool.category, Some(ToolCategory::Shell)); let result = tool.result.as_ref().unwrap(); assert!(!result.is_error); @@ -870,7 +853,7 @@ mod tests { let r = crate::reader::DbReader::open(f.path()).unwrap(); let s = r.load_session("ce").unwrap(); let view = session_to_view(&s); - let tool = &view.turns[0].tool_uses[0]; + let tool = &view.turns().next().unwrap().tool_uses[0]; assert!(tool.result.as_ref().unwrap().is_error); } @@ -886,7 +869,7 @@ mod tests { let r = crate::reader::DbReader::open(f.path()).unwrap(); let s = r.load_session("cu").unwrap(); let view = session_to_view(&s); - let tool = &view.turns[0].tool_uses[0]; + let tool = &view.turns().next().unwrap().tool_uses[0]; assert_eq!(tool.name, "future_thing_v9"); assert_eq!(tool.category, None); assert_eq!(tool.input["x"], 1); @@ -908,7 +891,7 @@ mod tests { let ids = ConversationProvider::list_conversations(&mgr, "").unwrap(); assert_eq!(ids, vec!["c1".to_string()]); let v = ConversationProvider::load_conversation(&mgr, "", "c1").unwrap(); - assert_eq!(v.turns.len(), 3); + assert_eq!(v.turns().count(), 3); let m = ConversationProvider::load_metadata(&mgr, "", "c1").unwrap(); assert_eq!(m.message_count, 3); } @@ -920,10 +903,22 @@ mod tests { tool_category(15, "run_terminal_command_v2"), Some(ToolCategory::Shell) ); - assert_eq!(tool_category(38, "edit_file_v2"), Some(ToolCategory::FileWrite)); - assert_eq!(tool_category(40, "read_file_v2"), Some(ToolCategory::FileRead)); - assert_eq!(tool_category(41, "ripgrep_raw_search"), Some(ToolCategory::FileSearch)); - assert_eq!(tool_category(42, "glob_file_search"), Some(ToolCategory::FileSearch)); + assert_eq!( + tool_category(38, "edit_file_v2"), + Some(ToolCategory::FileWrite) + ); + assert_eq!( + tool_category(40, "read_file_v2"), + Some(ToolCategory::FileRead) + ); + assert_eq!( + tool_category(41, "ripgrep_raw_search"), + Some(ToolCategory::FileSearch) + ); + assert_eq!( + tool_category(42, "glob_file_search"), + Some(ToolCategory::FileSearch) + ); assert_eq!(tool_category(48, "task_v2"), Some(ToolCategory::Delegation)); } @@ -932,7 +927,10 @@ mod tests { // Agent-side names from the JSONL transcript layer. assert_eq!(tool_category(9999, "Shell"), Some(ToolCategory::Shell)); assert_eq!(tool_category(9999, "Write"), Some(ToolCategory::FileWrite)); - assert_eq!(tool_category(9999, "StrReplace"), Some(ToolCategory::FileWrite)); + assert_eq!( + tool_category(9999, "StrReplace"), + Some(ToolCategory::FileWrite) + ); assert_eq!(tool_category(9999, "Read"), Some(ToolCategory::FileRead)); assert_eq!(tool_category(9999, "Glob"), Some(ToolCategory::FileSearch)); assert_eq!(tool_category(9999, "Grep"), Some(ToolCategory::FileSearch)); @@ -1093,7 +1091,10 @@ mod tests { #[test] fn tool_category_unknown_id_falls_through_to_name() { // Future numeric ids we haven't seen still classify via name. - assert_eq!(tool_category(7777, "edit_file_v2"), Some(ToolCategory::FileWrite)); + assert_eq!( + tool_category(7777, "edit_file_v2"), + Some(ToolCategory::FileWrite) + ); assert_eq!(tool_category(7777, "future_tool"), None); } } diff --git a/crates/toolpath-cursor/src/reader.rs b/crates/toolpath-cursor/src/reader.rs index 608a2aa..8dc4c93 100644 --- a/crates/toolpath-cursor/src/reader.rs +++ b/crates/toolpath-cursor/src/reader.rs @@ -110,12 +110,14 @@ impl DbReader { } })?; match raw { - Some(s) => serde_json::from_str(&s) - .map(Some) - .map_err(|e| CursorError::MalformedPayload { - what: format!("composerData:{composer_id}"), - detail: e.to_string(), - }), + Some(s) => { + serde_json::from_str(&s) + .map(Some) + .map_err(|e| CursorError::MalformedPayload { + what: format!("composerData:{composer_id}"), + detail: e.to_string(), + }) + } None => Ok(None), } } @@ -143,9 +145,7 @@ impl DbReader { Some(s) => match serde_json::from_str::(&s) { Ok(b) => Ok(Some(b)), Err(e) => { - eprintln!( - "Warning: bubble {composer_id}:{bubble_id} malformed: {e}; skipping" - ); + eprintln!("Warning: bubble {composer_id}:{bubble_id} malformed: {e}; skipping"); Ok(None) } }, @@ -260,7 +260,9 @@ impl DbReader { // Load content blobs referenced by tool results. let mut content_blobs = std::collections::HashMap::new(); for b in &bubbles { - let Some(tf) = &b.tool_former_data else { continue }; + let Some(tf) = &b.tool_former_data else { + continue; + }; let Ok(Some(result)) = tf.parse_result() else { continue; }; @@ -409,7 +411,8 @@ pub(crate) mod tests { #[test] fn malformed_composer_data_surfaces_error() { - let setup = r#"INSERT INTO cursorDiskKV (key, value) VALUES ('composerData:bad', '{not json}');"#; + let setup = + r#"INSERT INTO cursorDiskKV (key, value) VALUES ('composerData:bad', '{not json}');"#; let f = fixture_db(setup); let r = DbReader::open(f.path()).unwrap(); let err = r.read_composer_data("bad").unwrap_err(); diff --git a/crates/toolpath-cursor/src/types.rs b/crates/toolpath-cursor/src/types.rs index 53d29ec..0e15335 100644 --- a/crates/toolpath-cursor/src/types.rs +++ b/crates/toolpath-cursor/src/types.rs @@ -27,7 +27,11 @@ pub struct ComposerHead { pub subtitle: Option, #[serde(rename = "createdAt", default, skip_serializing_if = "Option::is_none")] pub created_at: Option, - #[serde(rename = "lastUpdatedAt", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "lastUpdatedAt", + default, + skip_serializing_if = "Option::is_none" + )] pub last_updated_at: Option, #[serde( rename = "conversationCheckpointLastUpdatedAt", @@ -35,7 +39,11 @@ pub struct ComposerHead { skip_serializing_if = "Option::is_none" )] pub conversation_checkpoint_last_updated_at: Option, - #[serde(rename = "unifiedMode", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "unifiedMode", + default, + skip_serializing_if = "Option::is_none" + )] pub unified_mode: Option, #[serde(rename = "forceMode", default, skip_serializing_if = "Option::is_none")] pub force_mode: Option, @@ -45,17 +53,37 @@ pub struct ComposerHead { pub is_draft: bool, #[serde(rename = "hasUnreadMessages", default)] pub has_unread_messages: bool, - #[serde(rename = "totalLinesAdded", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "totalLinesAdded", + default, + skip_serializing_if = "Option::is_none" + )] pub total_lines_added: Option, - #[serde(rename = "totalLinesRemoved", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "totalLinesRemoved", + default, + skip_serializing_if = "Option::is_none" + )] pub total_lines_removed: Option, - #[serde(rename = "filesChangedCount", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "filesChangedCount", + default, + skip_serializing_if = "Option::is_none" + )] pub files_changed_count: Option, - #[serde(rename = "contextUsagePercent", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "contextUsagePercent", + default, + skip_serializing_if = "Option::is_none" + )] pub context_usage_percent: Option, #[serde(rename = "numSubComposers", default)] pub num_sub_composers: u32, - #[serde(rename = "workspaceIdentifier", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "workspaceIdentifier", + default, + skip_serializing_if = "Option::is_none" + )] pub workspace_identifier: Option, #[serde(flatten, default, skip_serializing_if = "HashMap::is_empty")] pub extra: HashMap, @@ -118,29 +146,57 @@ pub struct ComposerData { pub subtitle: Option, #[serde(rename = "createdAt", default, skip_serializing_if = "Option::is_none")] pub created_at: Option, - #[serde(rename = "lastUpdatedAt", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "lastUpdatedAt", + default, + skip_serializing_if = "Option::is_none" + )] pub last_updated_at: Option, #[serde(rename = "isAgentic", default)] pub is_agentic: bool, #[serde(default, skip_serializing_if = "Option::is_none")] pub status: Option, - #[serde(rename = "unifiedMode", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "unifiedMode", + default, + skip_serializing_if = "Option::is_none" + )] pub unified_mode: Option, #[serde(rename = "forceMode", default, skip_serializing_if = "Option::is_none")] pub force_mode: Option, - #[serde(rename = "agentBackend", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "agentBackend", + default, + skip_serializing_if = "Option::is_none" + )] pub agent_backend: Option, - #[serde(rename = "modelConfig", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "modelConfig", + default, + skip_serializing_if = "Option::is_none" + )] pub model_config: Option, /// May contain more entries than there are `bubbleId:` rows on disk; /// don't use for iteration order. #[serde(rename = "fullConversationHeadersOnly", default)] pub full_conversation_headers_only: Vec, - #[serde(rename = "subComposerIds", default, skip_serializing_if = "Vec::is_empty")] + #[serde( + rename = "subComposerIds", + default, + skip_serializing_if = "Vec::is_empty" + )] pub sub_composer_ids: Vec, - #[serde(rename = "subagentComposerIds", default, skip_serializing_if = "Vec::is_empty")] + #[serde( + rename = "subagentComposerIds", + default, + skip_serializing_if = "Vec::is_empty" + )] pub subagent_composer_ids: Vec, - #[serde(rename = "latestChatGenerationUUID", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "latestChatGenerationUUID", + default, + skip_serializing_if = "Option::is_none" + )] pub latest_chat_generation_uuid: Option, #[serde(flatten, default, skip_serializing_if = "HashMap::is_empty")] pub extra: HashMap, @@ -171,7 +227,11 @@ pub struct ModelConfig { pub model_name: Option, #[serde(rename = "maxMode", default)] pub max_mode: bool, - #[serde(rename = "selectedModels", default, skip_serializing_if = "Vec::is_empty")] + #[serde( + rename = "selectedModels", + default, + skip_serializing_if = "Vec::is_empty" + )] pub selected_models: Vec, #[serde(flatten, default, skip_serializing_if = "HashMap::is_empty")] pub extra: HashMap, @@ -196,7 +256,11 @@ pub struct BubbleHeader { pub kind: u8, #[serde(default, skip_serializing_if = "Option::is_none")] pub grouping: Option, - #[serde(rename = "contentHeightHint", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "contentHeightHint", + default, + skip_serializing_if = "Option::is_none" + )] pub content_height_hint: Option, } @@ -206,11 +270,23 @@ pub struct BubbleGrouping { pub is_renderable: bool, #[serde(rename = "hasText", default, skip_serializing_if = "Option::is_none")] pub has_text: Option, - #[serde(rename = "hasThinking", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "hasThinking", + default, + skip_serializing_if = "Option::is_none" + )] pub has_thinking: Option, - #[serde(rename = "thinkingDurationMs", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "thinkingDurationMs", + default, + skip_serializing_if = "Option::is_none" + )] pub thinking_duration_ms: Option, - #[serde(rename = "capabilityType", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "capabilityType", + default, + skip_serializing_if = "Option::is_none" + )] pub capability_type: Option, #[serde(flatten, default, skip_serializing_if = "HashMap::is_empty")] pub extra: HashMap, @@ -234,23 +310,47 @@ pub struct Bubble { #[serde(rename = "richText", default, skip_serializing_if = "Option::is_none")] pub rich_text: Option, /// `15` = tool, `30` = thinking, `null` = text. - #[serde(rename = "capabilityType", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "capabilityType", + default, + skip_serializing_if = "Option::is_none" + )] pub capability_type: Option, - #[serde(rename = "conversationState", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "conversationState", + default, + skip_serializing_if = "Option::is_none" + )] pub conversation_state: Option, - #[serde(rename = "unifiedMode", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "unifiedMode", + default, + skip_serializing_if = "Option::is_none" + )] pub unified_mode: Option, #[serde(rename = "isAgentic", default)] pub is_agentic: bool, #[serde(rename = "requestId", default, skip_serializing_if = "Option::is_none")] pub request_id: Option, - #[serde(rename = "checkpointId", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "checkpointId", + default, + skip_serializing_if = "Option::is_none" + )] pub checkpoint_id: Option, - #[serde(rename = "tokenCount", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "tokenCount", + default, + skip_serializing_if = "Option::is_none" + )] pub token_count: Option, #[serde(rename = "modelInfo", default, skip_serializing_if = "Option::is_none")] pub model_info: Option, - #[serde(rename = "toolFormerData", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "toolFormerData", + default, + skip_serializing_if = "Option::is_none" + )] pub tool_former_data: Option, /// Must serialize as `[]` when empty — Cursor's renderer calls /// `Object.entries(undefined)` on the thinking-blocks indexer. @@ -266,10 +366,7 @@ pub struct Bubble { impl Bubble { pub fn created_at_utc(&self) -> Option> { - self.created_at - .as_ref()? - .parse::>() - .ok() + self.created_at.as_ref()?.parse::>().ok() } pub fn is_user(&self) -> bool { @@ -287,6 +384,10 @@ impl Bubble { pub fn is_tool(&self) -> bool { self.tool_former_data.is_some() || self.capability_type == Some(CAPABILITY_TOOL) } + + pub fn is_summarization(&self) -> bool { + self.capability_type == Some(CAPABILITY_SUMMARIZATION) + } } pub const BUBBLE_TYPE_USER: u8 = 1; @@ -294,12 +395,24 @@ pub const BUBBLE_TYPE_ASSISTANT: u8 = 2; pub const CAPABILITY_TOOL: u32 = 15; pub const CAPABILITY_THINKING: u32 = 30; +/// The `/summarize` boundary marker. The bubble has no recoverable summary or +/// kept set (those are server-side), so the provider skips it — see +/// `docs/agents/formats/cursor.md`. +pub const CAPABILITY_SUMMARIZATION: u32 = 22; #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct TokenCount { - #[serde(rename = "inputTokens", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "inputTokens", + default, + skip_serializing_if = "Option::is_none" + )] pub input_tokens: Option, - #[serde(rename = "outputTokens", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "outputTokens", + default, + skip_serializing_if = "Option::is_none" + )] pub output_tokens: Option, #[serde(flatten, default, skip_serializing_if = "HashMap::is_empty")] pub extra: HashMap, @@ -330,7 +443,11 @@ pub struct ToolFormerData { pub tool: u32, #[serde(rename = "toolIndex", default, skip_serializing_if = "Option::is_none")] pub tool_index: Option, - #[serde(rename = "modelCallId", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "modelCallId", + default, + skip_serializing_if = "Option::is_none" + )] pub model_call_id: Option, #[serde(rename = "toolCallId")] pub tool_call_id: String, @@ -343,7 +460,11 @@ pub struct ToolFormerData { pub params: String, #[serde(default, skip_serializing_if = "Option::is_none")] pub result: Option, - #[serde(rename = "additionalData", default, skip_serializing_if = "Option::is_none")] + #[serde( + rename = "additionalData", + default, + skip_serializing_if = "Option::is_none" + )] pub additional_data: Option, } @@ -583,7 +704,10 @@ mod tests { assert_eq!(cd.unified_mode.as_deref(), Some("agent")); assert_eq!(cd.default_model(), Some("default")); assert_eq!(cd.full_conversation_headers_only.len(), 1); - assert_eq!(cd.extra.get("futureField"), Some(&Value::String("kept".into()))); + assert_eq!( + cd.extra.get("futureField"), + Some(&Value::String("kept".into())) + ); } #[test] diff --git a/crates/toolpath-cursor/tests/projection_roundtrip.rs b/crates/toolpath-cursor/tests/projection_roundtrip.rs index 16c8877..60a0ee3 100644 --- a/crates/toolpath-cursor/tests/projection_roundtrip.rs +++ b/crates/toolpath-cursor/tests/projection_roundtrip.rs @@ -84,12 +84,17 @@ fn rebuilt_session_preserves_bubble_role_sequence() { let source = load_source(); let (_, rebuilt, _) = roundtrip(&source); - let kinds_of = |s: &CursorSession| -> Vec { - s.bubbles.iter().map(|b| b.kind).collect() - }; + let kinds_of = |s: &CursorSession| -> Vec { s.bubbles.iter().map(|b| b.kind).collect() }; let want = kinds_of(&source); assert_eq!(kinds_of(&rebuilt), want); - assert_eq!(want, vec![BUBBLE_TYPE_USER, BUBBLE_TYPE_ASSISTANT, BUBBLE_TYPE_ASSISTANT]); + assert_eq!( + want, + vec![ + BUBBLE_TYPE_USER, + BUBBLE_TYPE_ASSISTANT, + BUBBLE_TYPE_ASSISTANT + ] + ); } #[test] @@ -137,7 +142,9 @@ fn rebuilt_session_round_trips_file_content_via_blob_store() { // Empty before content uses Cursor's canonical SHA-256 sentinel. let before_id = result["beforeContentId"].as_str().unwrap(); - assert!(before_id.ends_with("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")); + assert!( + before_id.ends_with("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855") + ); } #[test] @@ -150,8 +157,8 @@ fn rebuilt_session_re_lifts_to_equivalent_view() { let (_, rebuilt, _) = roundtrip(&source); let view_again = session_to_view(&rebuilt); - assert_eq!(view_forward.turns.len(), view_again.turns.len()); - for (a, b) in view_forward.turns.iter().zip(view_again.turns.iter()) { + assert_eq!(view_forward.turns().count(), view_again.turns().count()); + for (a, b) in view_forward.turns().zip(view_again.turns()) { assert_eq!(a.role, b.role, "role mismatch on turn {}", a.id); assert_eq!(a.text, b.text, "text mismatch on turn {}", a.id); assert_eq!( @@ -178,10 +185,7 @@ fn rebuilt_session_re_lifts_to_equivalent_view() { fn rebuilt_session_workspace_path_preserved() { let source = load_source(); let (_, rebuilt, _) = roundtrip(&source); - assert_eq!( - rebuilt.workspace_path().unwrap().to_string_lossy(), - "/proj" - ); + assert_eq!(rebuilt.workspace_path().unwrap().to_string_lossy(), "/proj"); } #[test] @@ -210,7 +214,7 @@ fn projector_serializes_to_disk_readable_shape() { fn projector_accepts_foreign_view_shape() { use serde_json::json; use toolpath_convo::{ - EnvironmentSnapshot, FileMutation, ProducerInfo, Role, SessionBase, TokenUsage, + EnvironmentSnapshot, FileMutation, Item, ProducerInfo, Role, SessionBase, TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, }; @@ -225,8 +229,8 @@ fn projector_accepts_foreign_view_shape() { working_dir: Some("/foreign".into()), ..Default::default() }), - turns: vec![ - Turn { + items: vec![ + Item::Turn(Turn { id: "uA".into(), parent_id: None, group_id: None, @@ -245,8 +249,8 @@ fn projector_accepts_foreign_view_shape() { }), delegations: vec![], file_mutations: vec![], - }, - Turn { + }), + Item::Turn(Turn { id: "aA".into(), parent_id: Some("uA".into()), group_id: None, @@ -288,7 +292,7 @@ fn projector_accepts_foreign_view_shape() { after: Some("new\n".into()), rename_to: None, }], - }, + }), ], ..Default::default() }; diff --git a/crates/toolpath-cursor/tests/real_session_sanity.rs b/crates/toolpath-cursor/tests/real_session_sanity.rs index 8be996b..a3a0f09 100644 --- a/crates/toolpath-cursor/tests/real_session_sanity.rs +++ b/crates/toolpath-cursor/tests/real_session_sanity.rs @@ -51,7 +51,7 @@ fn real_cursor_db_round_trips_when_present() { assert_eq!(view.id, *id, "view.id should equal composer id"); assert_eq!(view.provider_id.as_deref(), Some("cursor")); assert_eq!(view.session_ids, vec![id.clone()]); - for turn in &view.turns { + for turn in view.turns() { assert!(!turn.id.is_empty(), "every turn carries a bubble id"); } let path = derive_path(&session, &DeriveConfig::default()); diff --git a/crates/toolpath-gemini/Cargo.toml b/crates/toolpath-gemini/Cargo.toml index 76fdeed..7ba036b 100644 --- a/crates/toolpath-gemini/Cargo.toml +++ b/crates/toolpath-gemini/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-gemini" -version = "0.6.0" +version = "0.7.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-gemini/src/derive.rs b/crates/toolpath-gemini/src/derive.rs index c4c7fa1..4a94d85 100644 --- a/crates/toolpath-gemini/src/derive.rs +++ b/crates/toolpath-gemini/src/derive.rs @@ -20,7 +20,10 @@ pub struct DeriveConfig { } /// Derive a single Toolpath [`Path`] from a Gemini conversation. -pub fn derive_path(conversation: &Conversation, config: &DeriveConfig) -> Path { +pub fn derive_path( + conversation: &Conversation, + config: &DeriveConfig, +) -> Path { let view = to_view(conversation); let prefix: String = view.id.chars().take(8).collect(); let base_uri = config.project_path.as_ref().map(|p| { @@ -40,7 +43,10 @@ pub fn derive_path(conversation: &Conversation, config: &DeriveConfig) -> Path { } /// Derive Toolpath Paths from multiple conversations. -pub fn derive_project(conversations: &[Conversation], config: &DeriveConfig) -> Vec { +pub fn derive_project( + conversations: &[Conversation], + config: &DeriveConfig, +) -> Vec { conversations .iter() .map(|c| derive_path(c, config)) diff --git a/crates/toolpath-gemini/src/project.rs b/crates/toolpath-gemini/src/project.rs index 39bde55..3c7c4fe 100644 --- a/crates/toolpath-gemini/src/project.rs +++ b/crates/toolpath-gemini/src/project.rs @@ -87,10 +87,10 @@ fn project_view( ) -> std::result::Result { let project_hash = cfg.project_hash.clone().unwrap_or_default(); - let mut main_messages: Vec = Vec::with_capacity(view.turns.len()); + let mut main_messages: Vec = Vec::with_capacity(view.turns().count()); let mut sub_agents: Vec = Vec::new(); - for turn in &view.turns { + for turn in view.turns() { main_messages.push(turn_to_message(turn)); for delegation in &turn.delegations { @@ -568,7 +568,10 @@ mod tests { #[test] fn tokens_from_common_unfolds_reasoning_out_of_output() { let mut breakdowns: BTreeMap> = BTreeMap::new(); - breakdowns.insert("output".into(), BTreeMap::from([("reasoning".into(), 243u32)])); + breakdowns.insert( + "output".into(), + BTreeMap::from([("reasoning".into(), 243u32)]), + ); let usage = TokenUsage { output_tokens: Some(337), breakdowns, @@ -637,12 +640,11 @@ mod tests { id: "session-uuid".into(), started_at: None, last_activity: None, - turns, + items: turns.into_iter().map(toolpath_convo::Item::Turn).collect(), total_usage: None, provider_id: Some("gemini-cli".into()), files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } diff --git a/crates/toolpath-gemini/src/provider.rs b/crates/toolpath-gemini/src/provider.rs index f39da51..38dbdcc 100644 --- a/crates/toolpath-gemini/src/provider.rs +++ b/crates/toolpath-gemini/src/provider.rs @@ -14,7 +14,7 @@ use crate::types::{ChatFile, Conversation, GeminiMessage, GeminiRole, Thought, T use serde_json::Value; use toolpath_convo::{ ConversationMeta, ConversationProvider, ConversationView, ConvoError, DelegatedWork, - EnvironmentSnapshot, Role, TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, + EnvironmentSnapshot, Item, Role, TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, }; // ── Role/tool mapping ──────────────────────────────────────────────── @@ -166,7 +166,11 @@ fn tokens_to_usage(t: &Tokens) -> TokenUsage { // Fold reasoning into output (additive in Gemini — billed as // output). None only when both output and thoughts are // absent/zero, mirroring the per-field Option semantics. - output_tokens: if generated == 0 { None } else { Some(generated) }, + output_tokens: if generated == 0 { + None + } else { + Some(generated) + }, cache_read_tokens: t.cached, cache_write_tokens: None, ..Default::default() @@ -432,6 +436,67 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { } } + // Gemini sometimes writes one assistant message across two consecutive + // lines that share a wire `id` (an empty content-only flush, then the + // same id again carrying the tool calls), each repeating the SAME + // `tokens` snapshot. Those are one message, so tag both with a shared + // `group_id` (the wire id). Downstream message-group accounting then + // counts that token total once per group instead of once per line — + // without it, summing across the split double-counts the message's + // tokens, and a wholesale-merging target (Codex) attributes the doubled + // total to the surviving turn. Only consecutive same-id ASSISTANT turns + // group; a user turn sharing an id with the next assistant does not. + // + // Compare BASE ids (the `#N` disambiguation suffix below stripped) so + // the grouping survives a project→read round-trip: after the first read + // the split's second turn carries `#1`, and on the way back through + // Gemini's wire (which has no group field) the boundary is re-detected + // only if `` and `#1` are recognized as the same message. + { + let mut i = 0; + while i < turns.len() { + if matches!(turns[i].role, Role::Assistant) { + let base = base_id(&turns[i].id).to_string(); + let mut j = i + 1; + while j < turns.len() + && matches!(turns[j].role, Role::Assistant) + && base_id(&turns[j].id) == base + { + j += 1; + } + if j - i > 1 { + for t in &mut turns[i..j] { + t.group_id = Some(base.clone()); + } + } + i = j; + } else { + i += 1; + } + } + } + + // Gemini reuses the same wire `id` across paired messages (a user + // prompt and the assistant response it triggered can share one id), + // so turn ids are not unique as-is. Disambiguate here by suffixing + // repeats with `#N` *before* the parent chain is built, so the chain + // links to the right turn. (`derive_path` also re-IDs same-id + // collisions, but only after parents are resolved on the colliding + // ids — doing it up front keeps the Gemini parent graph correct.) + let mut seen: std::collections::HashSet = std::collections::HashSet::new(); + for t in turns.iter_mut() { + if !seen.insert(t.id.clone()) { + let mut n = 1; + let mut candidate = format!("{}#{}", t.id, n); + while seen.contains(&candidate) { + n += 1; + candidate = format!("{}#{}", t.id, n); + } + t.id = candidate.clone(); + seen.insert(candidate); + } + } + // Gemini's wire format doesn't carry parent_id on messages, so link // turns sequentially. (Matches the old `derive_path_from_view`, // which used `last_step_id` as the parent for each new step.) @@ -457,12 +522,11 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { id: convo.session_uuid.clone(), started_at: convo.started_at, last_activity: convo.last_activity, - turns, + items: turns.into_iter().map(Item::Turn).collect(), total_usage, provider_id: Some("gemini-cli".into()), files_changed, session_ids: vec![], - events: vec![], base: view_base, producer: Some(toolpath_convo::ProducerInfo { name: "gemini-cli".into(), @@ -471,11 +535,39 @@ fn conversation_to_view(convo: &Conversation) -> ConversationView { } } +/// Strip a trailing `#N` disambiguation suffix added when two turns share a +/// wire id, recovering the original Gemini message id. `#1` → ``; +/// ids without the suffix pass through unchanged. +fn base_id(id: &str) -> &str { + match id.rsplit_once('#') { + Some((base, suffix)) + if !suffix.is_empty() && suffix.bytes().all(|b| b.is_ascii_digit()) => + { + base + } + _ => id, + } +} + fn sum_usage(turns: &[Turn]) -> Option { + // A split message repeats its token snapshot on every line, all sharing + // one `group_id`; count it once, on the group's last-occurring turn. + let mut group_last_idx: std::collections::HashMap<&str, usize> = + std::collections::HashMap::new(); + for (idx, turn) in turns.iter().enumerate() { + if let Some(mid) = &turn.group_id { + group_last_idx.insert(mid.as_str(), idx); + } + } + let mut total = TokenUsage::default(); let mut any = false; - for turn in turns { - if let Some(u) = &turn.token_usage { + for (idx, turn) in turns.iter().enumerate() { + let counts = turn + .group_id + .as_deref() + .is_none_or(|mid| group_last_idx.get(mid) == Some(&idx)); + if counts && let Some(u) = &turn.token_usage { any = true; total.input_tokens = Some(total.input_tokens.unwrap_or(0) + u.input_tokens.unwrap_or(0)); @@ -709,13 +801,13 @@ mod tests { ConversationProvider::load_conversation(&p, "/abs/myrepo", "session-uuid").unwrap(); assert_eq!(view.id, "session-uuid"); assert_eq!(view.provider_id.as_deref(), Some("gemini-cli")); - assert_eq!(view.turns.len(), 4); - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[0].text, "Find the bug"); - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[1].text, "I'll delegate."); + assert_eq!(view.turns().count(), 4); + assert_eq!(view.turns().next().unwrap().role, Role::User); + assert_eq!(view.turns().next().unwrap().text, "Find the bug"); + assert_eq!(view.turns().nth(1).unwrap().role, Role::Assistant); + assert_eq!(view.turns().nth(1).unwrap().text, "I'll delegate."); assert_eq!( - view.turns[1].model.as_deref(), + view.turns().nth(1).unwrap().model.as_deref(), Some("gemini-3-flash-preview") ); } @@ -725,7 +817,7 @@ mod tests { let (_t, p) = setup_provider(); let view = ConversationProvider::load_conversation(&p, "/abs/myrepo", "session-uuid").unwrap(); - let delegations = &view.turns[1].delegations; + let delegations = &view.turns().nth(1).unwrap().delegations; assert_eq!(delegations.len(), 1); let d = &delegations[0]; assert_eq!(d.agent_id, "qclszz"); @@ -741,7 +833,10 @@ mod tests { let (_t, p) = setup_provider(); let view = ConversationProvider::load_conversation(&p, "/abs/myrepo", "session-uuid").unwrap(); - let result = view.turns[1].tool_uses[0].result.as_ref().unwrap(); + let result = view.turns().nth(1).unwrap().tool_uses[0] + .result + .as_ref() + .unwrap(); assert_eq!(result.content, "Found it"); assert!(!result.is_error); } @@ -752,11 +847,11 @@ mod tests { let view = ConversationProvider::load_conversation(&p, "/abs/myrepo", "session-uuid").unwrap(); assert_eq!( - view.turns[1].tool_uses[0].category, + view.turns().nth(1).unwrap().tool_uses[0].category, Some(ToolCategory::Delegation) ); assert_eq!( - view.turns[2].tool_uses[0].category, + view.turns().nth(2).unwrap().tool_uses[0].category, Some(ToolCategory::FileWrite) ); } @@ -884,7 +979,7 @@ mod tests { let (_t, p) = setup_provider(); let view = ConversationProvider::load_conversation(&p, "/abs/myrepo", "session-uuid").unwrap(); - for turn in &view.turns { + for turn in view.turns() { let wd = turn .environment .as_ref() @@ -898,7 +993,7 @@ mod tests { let (_t, p) = setup_provider(); let view = ConversationProvider::load_conversation(&p, "/abs/myrepo", "session-uuid").unwrap(); - let sub_turn = &view.turns[1].delegations[0].turns[1]; + let sub_turn = &view.turns().nth(1).unwrap().delegations[0].turns[1]; let thinking = sub_turn.thinking.as_ref().unwrap(); assert!(thinking.contains("Searching")); assert!(thinking.contains("looking in /auth")); @@ -936,7 +1031,81 @@ mod tests { let (_t, p) = setup_provider(); let convo = p.read_conversation("/abs/myrepo", "session-uuid").unwrap(); let view = to_view(&convo); - assert_eq!(view.turns.len(), 4); + assert_eq!(view.turns().count(), 4); + } + + #[test] + fn test_to_view_uniquifies_duplicate_turn_ids() { + // Gemini reuses the same wire `id` across paired messages, so two + // turns can share an id. `to_view` must disambiguate them (else the + // unique-step-id enforcement in `derive_path` drops the collisions), + // while keeping the sequential parent chain consistent. + let chat_json = r#"{"sessionId":"s","projectHash":"","messages":[ + {"id":"dup","timestamp":"ts","type":"user","content":[{"text":"a"}]}, + {"id":"dup","timestamp":"ts","type":"gemini","content":"b"}, + {"id":"dup","timestamp":"ts","type":"user","content":[{"text":"c"}]}, + {"id":"uniq","timestamp":"ts","type":"gemini","content":"d"} +]}"#; + let chat: ChatFile = serde_json::from_str(chat_json).unwrap(); + let convo = Conversation::new("s".into(), chat); + let view = to_view(&convo); + + let ids: Vec<&str> = view.turns().map(|t| t.id.as_str()).collect(); + assert_eq!(ids, vec!["dup", "dup#1", "dup#2", "uniq"]); + + let unique: std::collections::HashSet<&str> = ids.iter().copied().collect(); + assert_eq!(unique.len(), ids.len(), "turn ids must be unique"); + + // Sequential parent chain references the uniquified ids. + let turns: Vec<&Turn> = view.turns().collect(); + assert!(turns[0].parent_id.is_none()); + assert_eq!(turns[1].parent_id.as_deref(), Some("dup")); + assert_eq!(turns[2].parent_id.as_deref(), Some("dup#1")); + assert_eq!(turns[3].parent_id.as_deref(), Some("dup#2")); + } + + #[test] + fn test_split_assistant_message_shares_group_id_and_counts_tokens_once() { + // Gemini writes one assistant message across two consecutive lines + // sharing a wire id (an empty flush, then the same id with tool + // calls), each repeating the SAME `tokens` snapshot. They are one + // message: both turns must share a `group_id`, and the session total + // must count the snapshot once, not twice. + let tokens = r#"{"input":100,"output":20,"cached":0,"thoughts":0,"tool":0,"total":120}"#; + let chat_json = format!( + r#"{{"sessionId":"s","projectHash":"","messages":[ + {{"id":"u","timestamp":"ts","type":"user","content":[{{"text":"go"}}]}}, + {{"id":"m","timestamp":"ts","type":"gemini","content":"","tokens":{tokens}}}, + {{"id":"m","timestamp":"ts","type":"gemini","content":"","tokens":{tokens},"toolCalls":[{{"id":"c0","name":"ls","args":{{}}}}]}} +]}}"# + ); + let chat: ChatFile = serde_json::from_str(&chat_json).unwrap(); + let view = to_view(&Conversation::new("s".into(), chat)); + + let asst: Vec<&Turn> = view + .turns() + .filter(|t| matches!(t.role, Role::Assistant)) + .collect(); + assert_eq!(asst.len(), 2); + assert_eq!(asst[0].group_id.as_deref(), Some("m")); + assert_eq!(asst[1].group_id.as_deref(), Some("m")); + + // Session total counts the message's tokens once. + let total = view.total_usage.as_ref().expect("total usage"); + assert_eq!(total.output_tokens, Some(20)); + assert_eq!(total.input_tokens, Some(100)); + } + + #[test] + fn test_base_id_strips_numeric_suffix_only() { + assert_eq!(base_id("abc"), "abc"); + assert_eq!(base_id("abc#1"), "abc"); + assert_eq!(base_id("abc#12"), "abc"); + // Non-numeric or empty suffix is left intact (not a dedup suffix). + assert_eq!(base_id("abc#x"), "abc#x"); + assert_eq!(base_id("abc#"), "abc#"); + // A uuid containing no '#' passes through. + assert_eq!(base_id("d1a8c61a-247c"), "d1a8c61a-247c"); } #[test] @@ -1025,7 +1194,7 @@ mod tests { let mgr = GeminiConvo::with_resolver(PathResolver::new().with_gemini_dir(&gemini)); let view = ConversationProvider::load_conversation(&mgr, "/p", "s").unwrap(); - let d = &view.turns[1].delegations[0]; + let d = &view.turns().nth(1).unwrap().delegations[0]; assert_eq!(d.agent_id, "t1"); assert_eq!(d.prompt, "go"); assert_eq!(d.result.as_deref(), Some("done")); @@ -1064,7 +1233,7 @@ mod tests { let mgr = GeminiConvo::with_resolver(PathResolver::new().with_gemini_dir(&gemini)); let view = ConversationProvider::load_conversation(&mgr, "/p", "s").unwrap(); - let delegations = &view.turns[1].delegations; + let delegations = &view.turns().nth(1).unwrap().delegations; assert_eq!(delegations.len(), 2); // a.json attaches to the task (first delegation), b.json is leftover assert_eq!(delegations[0].agent_id, "a"); diff --git a/crates/toolpath-gemini/src/watcher.rs b/crates/toolpath-gemini/src/watcher.rs index a8ac49f..e891a67 100644 --- a/crates/toolpath-gemini/src/watcher.rs +++ b/crates/toolpath-gemini/src/watcher.rs @@ -476,7 +476,7 @@ mod tests { ); let mut w = ConversationWatcher::new(mgr, "/abs/myrepo".into(), "session-uuid".into()); let (view, events) = w.poll_with_view().unwrap(); - assert_eq!(view.turns.len(), 1); + assert_eq!(view.turns().count(), 1); assert_eq!(events.len(), 1); } diff --git a/crates/toolpath-gemini/tests/fixture_roundtrip.rs b/crates/toolpath-gemini/tests/fixture_roundtrip.rs index de65fdb..e2f599a 100644 --- a/crates/toolpath-gemini/tests/fixture_roundtrip.rs +++ b/crates/toolpath-gemini/tests/fixture_roundtrip.rs @@ -45,17 +45,13 @@ fn fixture_load_via_provider() { // Provider id set assert_eq!(view.provider_id.as_deref(), Some("gemini-cli")); // User and assistant turns both present - let user_turns = view.turns.iter().filter(|t| t.role == Role::User).count(); - let assistant_turns = view - .turns - .iter() - .filter(|t| t.role == Role::Assistant) - .count(); + let user_turns = view.turns().filter(|t| t.role == Role::User).count(); + let assistant_turns = view.turns().filter(|t| t.role == Role::Assistant).count(); assert!(user_turns >= 1); assert!(assistant_turns >= 1); // At least one FileRead-categorised tool (get_internal_docs) - let has_file_read = view.turns.iter().any(|t| { + let has_file_read = view.turns().any(|t| { t.tool_uses .iter() .any(|tu| tu.category == Some(ToolCategory::FileRead)) diff --git a/crates/toolpath-gemini/tests/real_fixture_roundtrip.rs b/crates/toolpath-gemini/tests/real_fixture_roundtrip.rs index e335ecd..8094225 100644 --- a/crates/toolpath-gemini/tests/real_fixture_roundtrip.rs +++ b/crates/toolpath-gemini/tests/real_fixture_roundtrip.rs @@ -78,10 +78,7 @@ fn is_system_envelope(turn: &Turn) -> bool { } fn meaningful(view: &ConversationView) -> Vec<&Turn> { - view.turns - .iter() - .filter(|t| !is_system_envelope(t)) - .collect() + view.turns().filter(|t| !is_system_envelope(t)).collect() } fn norm(s: &str) -> String { @@ -92,7 +89,7 @@ fn norm(s: &str) -> String { fn fixture_loads() { let view = load_fixture_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "gemini fixture should produce a non-empty view" ); let m = meaningful(&view); @@ -207,14 +204,14 @@ fn roundtrip_preserves_delegations() { let original = load_fixture_view(); let after = ir_roundtrip(&original); - let total_before: usize = original.turns.iter().map(|t| t.delegations.len()).sum(); - let total_after: usize = after.turns.iter().map(|t| t.delegations.len()).sum(); + let total_before: usize = original.turns().map(|t| t.delegations.len()).sum(); + let total_after: usize = after.turns().map(|t| t.delegations.len()).sum(); assert_eq!( total_before, total_after, "total delegation count diverged: {total_before} → {total_after}" ); - for (i, (a, b)) in original.turns.iter().zip(after.turns.iter()).enumerate() { + for (i, (a, b)) in original.turns().zip(after.turns()).enumerate() { assert_eq!( a.delegations.len(), b.delegations.len(), diff --git a/crates/toolpath-opencode/Cargo.toml b/crates/toolpath-opencode/Cargo.toml index 82e90bd..9966a7d 100644 --- a/crates/toolpath-opencode/Cargo.toml +++ b/crates/toolpath-opencode/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-opencode" -version = "0.5.0" +version = "0.6.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-opencode/src/derive.rs b/crates/toolpath-opencode/src/derive.rs index 1468bd2..66921d8 100644 --- a/crates/toolpath-opencode/src/derive.rs +++ b/crates/toolpath-opencode/src/derive.rs @@ -54,7 +54,10 @@ pub fn derive_path_with_resolver( } /// Derive a `Path` from multiple sessions. -pub fn derive_project(sessions: &[Session], config: &DeriveConfig) -> Vec { +pub fn derive_project( + sessions: &[Session], + config: &DeriveConfig, +) -> Vec { sessions.iter().map(|s| derive_path(s, config)).collect() } diff --git a/crates/toolpath-opencode/src/project.rs b/crates/toolpath-opencode/src/project.rs index 0f34ae4..42fe16b 100644 --- a/crates/toolpath-opencode/src/project.rs +++ b/crates/toolpath-opencode/src/project.rs @@ -6,13 +6,15 @@ use std::path::PathBuf; use serde_json::{Map, Value}; use toolpath_convo::{ - ConversationProjector, ConversationView, ConvoError, Result, Role, ToolInvocation, Turn, + Compaction, CompactionTrigger, ConversationProjector, ConversationView, ConvoError, Result, + Role, ToolInvocation, Turn, }; use crate::types::{ - AssistantMessage, Message, MessageData, MessagePath, MessageTime, ModelRef, Part, PartData, - ReasoningPart, Session, StepFinishPart, StepStartPart, TextPart, TimeRange, Tokens, ToolPart, - ToolRunTime, ToolState, ToolStateCompleted, ToolStateError, UserMessage, + AssistantMessage, CompactionPart, Message, MessageData, MessagePath, MessageTime, ModelRef, + Part, PartData, ReasoningPart, Session, StepFinishPart, StepStartPart, TextPart, TimeRange, + Tokens, ToolPart, ToolRunTime, ToolState, ToolStateCompleted, ToolStateError, UserMessage, + UserSummary, }; const DEFAULT_AGENT: &str = "build"; @@ -85,8 +87,7 @@ fn project_view( .directory .clone() .or_else(|| { - view.turns - .iter() + view.turns() .find_map(|t| t.environment.as_ref()?.working_dir.clone()) .map(PathBuf::from) }) @@ -116,8 +117,8 @@ fn project_view( .started_at .map(|t| t.timestamp_millis()) .or_else(|| { - view.turns - .first() + view.turns() + .next() .and_then(|t| parse_timestamp_ms(&t.timestamp)) }) .unwrap_or(0); @@ -125,7 +126,7 @@ fn project_view( .last_activity .map(|t| t.timestamp_millis()) .or_else(|| { - view.turns + view.turns() .last() .and_then(|t| parse_timestamp_ms(&t.timestamp)) }) @@ -135,8 +136,7 @@ fn project_view( .title .clone() .or_else(|| { - view.turns - .iter() + view.turns() .filter(|t| matches!(t.role, Role::User)) .map(|t| t.text.as_str()) .find(|t| !t.is_empty() && !is_system_envelope(t)) @@ -149,6 +149,11 @@ fn project_view( let mut messages: Vec = Vec::new(); let mut prev_msg_id: Option = None; let mut counter: u32 = 0; + // IR turn id → the message id we re-mint for it, so a compaction's kept + // tail anchor (an IR turn id) can be rewritten to the id the projected + // session actually carries — otherwise `tailStartID` would dangle and the + // kept set would collapse to empty on re-read. + let mut id_map: HashMap = HashMap::new(); let default_provider = cfg .default_model_provider @@ -159,42 +164,81 @@ fn project_view( .clone() .unwrap_or_else(|| DEFAULT_MODEL_ID.to_string()); - for turn in &view.turns { - match turn.role { - Role::User => { - let msg = build_user_message( - turn, - &session_id, - &mut counter, - &agent, - &default_provider, - &default_model, - ); - prev_msg_id = Some(msg.id.clone()); - messages.push(msg); - } - Role::Assistant => { - let parent = prev_msg_id - .clone() - .unwrap_or_else(|| mint_message_id(&session_id, counter)); - let msg = build_assistant_message( - turn, + // Walk the ordered item stream so compaction boundaries land at their + // true position (between the turns they separate) — the inverse of the + // forward Builder, which reads `compaction` parts in message order. + for item in &view.items { + match item { + toolpath_convo::Item::Turn(turn) => match turn.role { + Role::User => { + let msg = build_user_message( + turn, + &session_id, + &mut counter, + &agent, + &default_provider, + &default_model, + ); + id_map.insert(turn.id.clone(), msg.id.clone()); + prev_msg_id = Some(msg.id.clone()); + messages.push(msg); + } + Role::Assistant => { + let parent = prev_msg_id + .clone() + .unwrap_or_else(|| mint_message_id(&session_id, counter)); + let msg = build_assistant_message( + turn, + &session_id, + &mut counter, + parent, + &directory, + &agent, + &default_provider, + &default_model, + ); + id_map.insert(turn.id.clone(), msg.id.clone()); + prev_msg_id = Some(msg.id.clone()); + messages.push(msg); + } + Role::System | Role::Other(_) => { + // opencode has no system-role message variant; fold the + // text into the next user/assistant turn's context by + // skipping. The system prompt itself rides on + // UserMessage.system if needed. + } + }, + toolpath_convo::Item::Compaction(c) => { + // Inverse of the forward Builder's compaction handling: + // emit a synthetic compaction-bearing user message so a + // re-read reproduces the `Item::Compaction`. When the + // boundary carries a summary, emit the synthetic summary + // user message the forward path reads from + // `UserMessage.summary.body`. + // Rewrite the kept-tail anchor (an IR turn id) to the message + // id we minted for that turn, so it matches a real message on + // re-read. Falls back to the raw id when the anchor wasn't a + // projected turn (e.g. a non-contiguous kept set). + let tail_anchor = c + .kept + .first() + .map(|k| id_map.get(k).cloned().unwrap_or_else(|| k.clone())); + for msg in build_compaction_messages( + c, + tail_anchor, &session_id, &mut counter, - parent, - &directory, &agent, &default_provider, &default_model, - ); - prev_msg_id = Some(msg.id.clone()); - messages.push(msg); + ) { + prev_msg_id = Some(msg.id.clone()); + messages.push(msg); + } } - Role::System | Role::Other(_) => { - // opencode has no system-role message variant; fold the - // text into the next user/assistant turn's context by - // skipping. The system prompt itself rides on - // UserMessage.system if needed. + toolpath_convo::Item::Event(_) => { + // Non-conversational events have no opencode message form; + // they're metadata that doesn't round-trip through parts. } } } @@ -292,6 +336,121 @@ fn build_user_message( } } +/// Inverse of the forward Builder's `push_compaction`: project an +/// [`Item::Compaction`] back into opencode messages so a re-read +/// reproduces it. +/// +/// opencode writes a compaction as a synthetic user message carrying a +/// single `compaction` part. We mirror that: one user message whose only +/// part is a [`CompactionPart`]. When the boundary has a `summary`, we +/// also emit, immediately after, the synthetic summary user message the +/// forward path pairs with this boundary via its `UserMessage.summary.body`. +fn build_compaction_messages( + c: &Compaction, + tail_anchor: Option, + session_id: &str, + counter: &mut u32, + agent: &str, + default_provider: &str, + default_model: &str, +) -> Vec { + let time_created = parse_timestamp_ms(&c.timestamp).unwrap_or(0); + let model = ModelRef { + provider_id: default_provider.to_string(), + model_id: default_model.to_string(), + variant: None, + }; + + let mut out = Vec::new(); + + *counter += 1; + let msg_id = mint_message_id(session_id, *counter); + let user = UserMessage { + time: MessageTime { + created: time_created, + completed: None, + }, + agent: agent.to_string(), + model: model.clone(), + format: None, + summary: Some(UserSummary { + title: None, + body: None, + diffs: vec![], + extra: HashMap::new(), + }), + system: None, + tools: None, + extra: HashMap::new(), + }; + + *counter += 1; + let compaction_part = Part { + id: mint_part_id(session_id, *counter), + message_id: msg_id.clone(), + session_id: session_id.to_string(), + time_created, + time_updated: time_created, + data: PartData::Compaction(CompactionPart { + auto: c.trigger == Some(CompactionTrigger::Auto), + overflow: None, + // The kept tail anchors on the earliest surviving turn — already + // rewritten by the caller to the message id this projection minted + // for it, so the `tailStartID` wire key resolves on re-read. + tail_start_id: tail_anchor, + extra: HashMap::new(), + }), + }; + + out.push(Message { + id: msg_id, + session_id: session_id.to_string(), + time_created, + time_updated: time_created, + data: MessageData::User(user), + parts: vec![compaction_part], + }); + + if let Some(summary) = c.summary.as_ref().filter(|s| !s.is_empty()) { + // The summary message must sort strictly AFTER the compaction message + // so the reader (which orders by `(time_created ASC, id ASC)`) sees the + // boundary first and pairs this summary with it. Sharing `time_created` + // would leave the order to the non-monotonic minted ids — dropping the + // summary whenever it happened to sort first. + let summary_time = time_created + 1; + *counter += 1; + let summary_msg_id = mint_message_id(session_id, *counter); + let summary_user = UserMessage { + time: MessageTime { + created: summary_time, + completed: None, + }, + agent: agent.to_string(), + model, + format: None, + summary: Some(UserSummary { + title: None, + body: Some(summary.clone()), + diffs: vec![], + extra: HashMap::new(), + }), + system: None, + tools: None, + extra: HashMap::new(), + }; + out.push(Message { + id: summary_msg_id, + session_id: session_id.to_string(), + time_created: summary_time, + time_updated: summary_time, + data: MessageData::User(summary_user), + parts: Vec::new(), + }); + } + + out +} + #[allow(clippy::too_many_arguments)] fn build_assistant_message( turn: &Turn, @@ -795,12 +954,11 @@ mod tests { id: "session-uuid".into(), started_at: None, last_activity: None, - turns, + items: turns.into_iter().map(toolpath_convo::Item::Turn).collect(), total_usage: None, provider_id: Some("opencode".into()), files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } @@ -814,6 +972,75 @@ mod tests { assert!(s.messages.is_empty()); } + #[test] + fn compaction_summary_and_kept_anchor_survive_projection() { + use toolpath_convo::Item; + // A compaction with a summary and a kept tail anchored on the + // assistant turn `a1`. The projector must (a) rewrite the anchor to + // the message id it mints for `a1` (not the raw IR id, which no + // projected message carries), and (b) give the summary message a + // strictly-later timestamp so the reader — which orders by + // (time_created, id) — pairs it with the boundary instead of dropping + // it on a hash-id tie. + let mut view = view_with(vec![user_turn("first"), assistant_turn("ans")]); + view.items.push(Item::Compaction(Compaction { + id: "c".into(), + parent_id: Some("a1".into()), + timestamp: "2026-04-21T12:00:02.000Z".into(), + trigger: Some(CompactionTrigger::Auto), + summary: Some("condensed".into()), + pre_tokens: None, + kept: vec!["a1".into()], + })); + + let s = OpencodeProjector::default().project(&view).unwrap(); + + let assistant_id = s + .messages + .iter() + .find(|m| matches!(m.data, MessageData::Assistant(_))) + .map(|m| m.id.clone()) + .expect("assistant message"); + + let comp_msg = s + .messages + .iter() + .find(|m| { + m.parts + .iter() + .any(|p| matches!(p.data, PartData::Compaction(_))) + }) + .expect("compaction-bearing message"); + let cp = comp_msg + .parts + .iter() + .find_map(|p| match &p.data { + PartData::Compaction(cp) => Some(cp), + _ => None, + }) + .unwrap(); + assert_eq!( + cp.tail_start_id.as_deref(), + Some(assistant_id.as_str()), + "kept anchor rewritten to the minted message id" + ); + + let summary_msg = s + .messages + .iter() + .find(|m| match &m.data { + MessageData::User(u) => { + u.summary.as_ref().and_then(|sm| sm.body.as_ref()).is_some() + } + _ => false, + }) + .expect("summary message"); + assert!( + summary_msg.time_created > comp_msg.time_created, + "summary must sort after the compaction message" + ); + } + #[test] fn user_turn_becomes_user_message_with_text_part() { let s = OpencodeProjector::default() diff --git a/crates/toolpath-opencode/src/provider.rs b/crates/toolpath-opencode/src/provider.rs index 3215868..184d762 100644 --- a/crates/toolpath-opencode/src/provider.rs +++ b/crates/toolpath-opencode/src/provider.rs @@ -21,9 +21,13 @@ //! by the derive layer to fetch file diffs. //! - `extra["opencode"]["patches"]` ← any `patch` parts (their //! `{hash, files}` records). -//! 3. Non-turn parts land in `ConversationView.events`: -//! `compaction`, `retry`, unknown types. -//! 4. `subtask` parts are captured on the turn's `delegations` +//! 3. A `compaction` part (on either a user or assistant message) emits +//! an [`Item::Compaction`] at its position in the ordered item stream, +//! parented on the last turn emitted before it. `derive_path` projects +//! it to a `conversation.compact` step. +//! 4. Other non-turn parts land in `ConversationView.events`: +//! `retry`, `file`, `agent`, unknown types. +//! 5. `subtask` parts are captured on the turn's `delegations` //! (empty-turn list — the sub-agent's own session lives under //! its own id, linked by `session.parent_id`). @@ -35,13 +39,14 @@ use crate::error::Result; use crate::io::ConvoIO; use crate::paths::PathResolver; use crate::types::{ - AssistantMessage, Message, MessageData, Part, PartData, Session, SessionMetadata, Tokens, - ToolState, UserMessage, + AssistantMessage, CompactionPart, Message, MessageData, Part, PartData, Session, + SessionMetadata, Tokens, ToolState, UserMessage, }; use toolpath_convo::{ - ConversationEvent, ConversationMeta, ConversationProvider, ConversationView, - ConvoError as ConvoTraitError, DelegatedWork, EnvironmentSnapshot, FileMutation, ProducerInfo, - Role, SessionBase, TokenUsage, ToolCategory, ToolInvocation, ToolResult, Turn, + Compaction, CompactionTrigger, ConversationEvent, ConversationMeta, ConversationProvider, + ConversationView, ConvoError as ConvoTraitError, DelegatedWork, EnvironmentSnapshot, + FileMutation, Item, ProducerInfo, Role, SessionBase, TokenUsage, ToolCategory, ToolInvocation, + ToolResult, Turn, }; /// Provider for opencode sessions. @@ -164,8 +169,13 @@ pub fn to_view_with_resolver(session: &Session, resolver: &PathResolver) -> Conv struct Builder<'a> { session: &'a Session, - turns: Vec, - events: Vec, + /// The ordered conversation stream — turns, events, and compaction + /// boundaries interleaved in real order, so a compaction lands at its + /// true position rather than after all turns. + items: Vec, + /// Id of the most recent turn pushed to `items`. A compaction boundary + /// parents on this (the last turn before it). + last_turn_id: Option, files_changed_order: Vec, files_changed_seen: std::collections::HashSet, total_usage: TokenUsage, @@ -178,20 +188,90 @@ struct Builder<'a> { /// `before` of the next turn's snapshot pair so intermediate state /// captures correctly. prev_snapshot_after: Option, + /// Items index of the most recently emitted compaction boundary that + /// is still awaiting its summary. opencode condenses each compaction's + /// pre-boundary prefix into a synthetic summary-bearing user message at + /// the boundary; the next such message's `summary.body` is *that* + /// compaction's summary. Tracked per-boundary (not session-global) so a + /// session with several compactions keeps each one's distinct summary. + pending_compaction_idx: Option, } impl<'a> Builder<'a> { fn new(session: &'a Session) -> Self { Self { session, - turns: Vec::new(), - events: Vec::new(), + items: Vec::new(), + last_turn_id: None, files_changed_order: Vec::new(), files_changed_seen: std::collections::HashSet::new(), total_usage: TokenUsage::default(), total_usage_set: false, snapshot_repo: None, prev_snapshot_after: None, + pending_compaction_idx: None, + } + } + + /// Record a turn, tracking its id for any subsequent compaction's + /// `parent_id`. + fn push_turn(&mut self, turn: Turn) { + self.last_turn_id = Some(turn.id.clone()); + self.items.push(Item::Turn(turn)); + } + + fn push_event(&mut self, event: ConversationEvent) { + self.items.push(Item::Event(event)); + } + + /// Map an opencode `compaction` part to an [`Item::Compaction`], + /// parented on the last turn emitted so far. + fn push_compaction(&mut self, part: &Part, c: &CompactionPart) { + let trigger = Some(if c.auto { + CompactionTrigger::Auto + } else { + CompactionTrigger::Manual + }); + // `tail_start_id` anchors the kept tail. When present, the kept + // tail is every turn from it through the last turn before the + // boundary; absent means the whole prior context was condensed. + let kept = match &c.tail_start_id { + Some(anchor) => self + .items + .iter() + .filter_map(|item| match item { + Item::Turn(t) => Some(&t.id), + _ => None, + }) + .skip_while(|id| *id != anchor) + .cloned() + .collect(), + None => Vec::new(), + }; + self.items.push(Item::Compaction(Compaction { + id: part.id.clone(), + parent_id: self.last_turn_id.clone(), + timestamp: millis_to_iso(part.time_created), + trigger, + // Filled in by `attach_pending_summary` from this boundary's own + // summary message (the next summary-bearing user message), not a + // single session-global summary shared across boundaries. + summary: None, + pre_tokens: None, + kept, + })); + self.pending_compaction_idx = Some(self.items.len() - 1); + } + + /// Attach `body` to the most recent compaction boundary still awaiting a + /// summary, then clear the pending marker so the next boundary claims its + /// own. An orphan summary (no boundary pending) is ignored, matching the + /// prior behavior of never surfacing a summary message as a turn. + fn attach_pending_summary(&mut self, body: String) { + if let Some(idx) = self.pending_compaction_idx.take() + && let Some(Item::Compaction(c)) = self.items.get_mut(idx) + { + c.summary = Some(body); } } @@ -229,7 +309,7 @@ impl<'a> Builder<'a> { // Refresh files_changed so it matches what landed on turns. let mut seen = std::collections::HashSet::new(); let mut ordered = Vec::new(); - for turn in &view.turns { + for turn in view.turns() { for fm in &turn.file_mutations { if seen.insert(fm.path.clone()) { ordered.push(fm.path.clone()); @@ -246,7 +326,7 @@ impl<'a> Builder<'a> { MessageData::User(u) => self.handle_user_message(msg, u), MessageData::Assistant(a) => self.handle_assistant_message(msg, a), MessageData::Other => { - self.events.push(ConversationEvent { + self.push_event(ConversationEvent { id: format!("msg-other-{}", msg.id), timestamp: millis_to_iso(msg.time_created), parent_id: None, @@ -261,7 +341,7 @@ impl<'a> Builder<'a> { id: self.session.id.clone(), started_at: Utc.timestamp_millis_opt(self.session.time_created).single(), last_activity: Utc.timestamp_millis_opt(self.session.time_updated).single(), - turns: self.turns, + items: self.items, total_usage: if self.total_usage_set { Some(self.total_usage) } else { @@ -270,20 +350,49 @@ impl<'a> Builder<'a> { provider_id: Some("opencode".into()), files_changed: self.files_changed_order, session_ids: vec![self.session.id.clone()], - events: self.events, ..Default::default() } } - fn handle_user_message(&mut self, msg: &Message, _u: &UserMessage) { + fn handle_user_message(&mut self, msg: &Message, u: &UserMessage) { let text = concat_text_parts(&msg.parts); + + // A compaction marker can ride on a user message (opencode writes a + // synthetic compaction-bearing user message at the boundary). Emit + // the boundary in place; it parents on the last turn so far. + for p in &msg.parts { + if let PartData::Compaction(c) = &p.data { + self.push_compaction(p, c); + } + } + + // opencode condenses each compaction's pre-boundary prefix into a + // synthetic user message whose `summary.body` holds the summary text. + // Pair it with the boundary still awaiting one — whether the body + // rides on the compaction-bearing message itself or the immediately + // following synthetic message. + if let Some(body) = u + .summary + .as_ref() + .and_then(|s| s.body.as_deref()) + .filter(|b| !b.is_empty()) + { + self.attach_pending_summary(body.to_string()); + } + + // Skip an empty user turn when the message carried only a + // compaction marker (the common synthetic-boundary case). + if text.is_empty() { + return; + } + let environment = Some(EnvironmentSnapshot { working_dir: Some(self.session.directory.to_string_lossy().to_string()), vcs_branch: None, vcs_revision: None, }); - self.turns.push(Turn { + self.push_turn(Turn { id: msg.id.clone(), parent_id: None, group_id: None, @@ -369,7 +478,7 @@ impl<'a> Builder<'a> { }); } PartData::File(f) => { - self.events.push(ConversationEvent { + self.push_event(ConversationEvent { id: format!("file-{}", p.id), timestamp: millis_to_iso(p.time_created), parent_id: Some(msg.id.clone()), @@ -378,7 +487,7 @@ impl<'a> Builder<'a> { }); } PartData::Agent(ag) => { - self.events.push(ConversationEvent { + self.push_event(ConversationEvent { id: format!("agent-{}", p.id), timestamp: millis_to_iso(p.time_created), parent_id: Some(msg.id.clone()), @@ -387,7 +496,7 @@ impl<'a> Builder<'a> { }); } PartData::Retry(r) => { - self.events.push(ConversationEvent { + self.push_event(ConversationEvent { id: format!("retry-{}", p.id), timestamp: millis_to_iso(p.time_created), parent_id: Some(msg.id.clone()), @@ -396,16 +505,13 @@ impl<'a> Builder<'a> { }); } PartData::Compaction(c) => { - self.events.push(ConversationEvent { - id: format!("compaction-{}", p.id), - timestamp: millis_to_iso(p.time_created), - parent_id: Some(msg.id.clone()), - event_type: "part.compaction".into(), - data: to_data_map(&serde_json::to_value(c).unwrap_or(Value::Null)), - }); + // A compaction marker on an assistant message: emit the + // boundary in place, parented on the turn before this + // one (this turn hasn't been pushed yet). + self.push_compaction(p, c); } PartData::Unknown => { - self.events.push(ConversationEvent { + self.push_event(ConversationEvent { id: format!("unknown-{}", p.id), timestamp: millis_to_iso(p.time_created), parent_id: Some(msg.id.clone()), @@ -448,7 +554,7 @@ impl<'a> Builder<'a> { // (catches gitignored paths and the no-repo case). let file_mutations = self.compute_turn_mutations(&snapshots, &tool_uses); - self.turns.push(Turn { + self.push_turn(Turn { id: msg.id.clone(), parent_id: if a.parent_id.is_empty() { None @@ -604,7 +710,10 @@ fn accumulate_tokens(total: &mut TokenUsage, step: &Tokens) { // where reasoning is already inside `output`. Fold it into output_tokens // so the IR's `output` means "all generated tokens" consistently and the // session total isn't under-counted. - add_u32(&mut total.output_tokens, (step.output + step.reasoning) as u32); + add_u32( + &mut total.output_tokens, + (step.output + step.reasoning) as u32, + ); add_u32(&mut total.cache_read_tokens, step.cache.read as u32); add_u32(&mut total.cache_write_tokens, step.cache.write as u32); // Memoize the reasoning slice we just folded into output. It's @@ -960,13 +1069,14 @@ mod tests { assert_eq!(view.id, "ses_x"); assert_eq!(view.provider_id.as_deref(), Some("opencode")); - assert_eq!(view.turns.len(), 2); - assert_eq!(view.turns[0].role, Role::User); - assert_eq!(view.turns[0].text, "make a pickle"); - assert_eq!(view.turns[1].role, Role::Assistant); - assert_eq!(view.turns[1].text, "done!"); + let turns: Vec<_> = view.turns().collect(); + assert_eq!(turns.len(), 2); + assert_eq!(turns[0].role, Role::User); + assert_eq!(turns[0].text, "make a pickle"); + assert_eq!(turns[1].role, Role::Assistant); + assert_eq!(turns[1].text, "done!"); assert_eq!( - view.turns[1].thinking.as_deref(), + turns[1].thinking.as_deref(), Some("I should write main.cpp") ); } @@ -975,7 +1085,7 @@ mod tests { fn tool_invocations_paired() { let (_t, mgr) = setup(BASIC_SQL); let view = to_view(&mgr.read_session("ses_x").unwrap()); - let assistant = &view.turns[1]; + let assistant = view.turns().nth(1).unwrap(); assert_eq!(assistant.tool_uses.len(), 2); let bash = &assistant.tool_uses[0]; assert_eq!(bash.name, "bash"); @@ -997,7 +1107,7 @@ mod tests { fn step_finish_drives_token_usage() { let (_t, mgr) = setup(BASIC_SQL); let view = to_view(&mgr.read_session("ses_x").unwrap()); - let u = view.turns[1].token_usage.as_ref().unwrap(); + let u = view.turns().nth(1).unwrap().token_usage.as_ref().unwrap(); assert_eq!(u.input_tokens, Some(100)); // output (20) + reasoning (5): opencode reports reasoning as a // separate additive category, folded into output here. @@ -1007,7 +1117,10 @@ mod tests { // The reasoning slice (5) is also memoized under // breakdowns["output"]["reasoning"] — it's the SAME number folded // into output, so Σ(inner) = 5 ≤ output (25). - assert_eq!(u.breakdowns.get("output").and_then(|m| m.get("reasoning")), Some(&5u32)); + assert_eq!( + u.breakdowns.get("output").and_then(|m| m.get("reasoning")), + Some(&5u32) + ); let total = view.total_usage.as_ref().unwrap(); assert_eq!(total.input_tokens, Some(100)); @@ -1030,7 +1143,7 @@ mod tests { "#; let (_t, mgr) = setup(body); let view = to_view(&mgr.read_session("s").unwrap()); - let u = view.turns[0].token_usage.as_ref().unwrap(); + let u = view.turns().next().unwrap().token_usage.as_ref().unwrap(); assert_eq!(u.output_tokens, Some(20)); assert!(u.breakdowns.is_empty()); } @@ -1052,10 +1165,13 @@ mod tests { "#; let (_t, mgr) = setup(body); let view = to_view(&mgr.read_session("s").unwrap()); - let u = view.turns[0].token_usage.as_ref().unwrap(); + let u = view.turns().next().unwrap().token_usage.as_ref().unwrap(); // output total: (20+5) + (4+7) = 36; reasoning slice: 5+7 = 12. assert_eq!(u.output_tokens, Some(36)); - assert_eq!(u.breakdowns.get("output").and_then(|m| m.get("reasoning")), Some(&12u32)); + assert_eq!( + u.breakdowns.get("output").and_then(|m| m.get("reasoning")), + Some(&12u32) + ); } #[test] @@ -1072,7 +1188,7 @@ mod tests { "#; let (_t, mgr) = setup(body); let view = to_view(&mgr.read_session("s").unwrap()); - assert!(view.turns[0].token_usage.is_none()); + assert!(view.turns().next().unwrap().token_usage.is_none()); } #[test] @@ -1089,30 +1205,108 @@ mod tests { "#; let (_t, mgr) = setup(body); let view = to_view(&mgr.read_session("s").unwrap()); - let tool = &view.turns[0].tool_uses[0]; + let tool = &view.turns().next().unwrap().tool_uses[0]; let r = tool.result.as_ref().unwrap(); assert!(r.is_error); assert_eq!(r.content, "exit 1"); } #[test] - fn compaction_becomes_event() { + fn compaction_becomes_compaction_item() { + // A compaction part on a user message (the synthetic-boundary + // case) emits an `Item::Compaction`, parented on the prior turn, + // not a generic `ConversationEvent`. let body = r#" INSERT INTO project (id, worktree, time_created, time_updated, sandboxes) - VALUES ('p','/p',1,2,'[]'); + VALUES ('p','/p',1,4,'[]'); INSERT INTO session (id, project_id, slug, directory, title, version, time_created, time_updated) - VALUES ('s','p','slug','/p','T','1.0.0',1,2); + VALUES ('s','p','slug','/p','T','1.0.0',1,4); INSERT INTO message (id, session_id, time_created, time_updated, data) VALUES - ('m','s',1,1,'{"parentID":"","role":"assistant","mode":"b","agent":"b","path":{"cwd":"/p","root":"/p"},"cost":0,"tokens":{"input":0,"output":0,"reasoning":0,"cache":{"read":0,"write":0}},"modelID":"m","providerID":"p","time":{"created":1}}'); + ('mu','s',1,1,'{"role":"user","time":{"created":1},"agent":"b","model":{"providerID":"p","modelID":"m"}}'), + ('ma','s',2,2,'{"parentID":"mu","role":"assistant","mode":"b","agent":"b","path":{"cwd":"/p","root":"/p"},"cost":0,"tokens":{"input":0,"output":0,"reasoning":0,"cache":{"read":0,"write":0}},"modelID":"m","providerID":"p","time":{"created":2}}'), + ('mc','s',3,3,'{"role":"user","time":{"created":3},"agent":"b","model":{"providerID":"p","modelID":"m"}}'); INSERT INTO part (id, message_id, session_id, time_created, time_updated, data) VALUES - ('p1','m','s',1,1,'{"type":"compaction","auto":true,"overflow":false}'); + ('pu','mu','s',1,1,'{"type":"text","text":"do the thing"}'), + ('pa','ma','s',2,2,'{"type":"text","text":"did it"}'), + ('pc','mc','s',3,3,'{"type":"compaction","auto":true,"overflow":false,"tailStartId":"mu"}'); "#; let (_t, mgr) = setup(body); let view = to_view(&mgr.read_session("s").unwrap()); + assert!( - view.events - .iter() - .any(|e| e.event_type == "part.compaction") + !view.events().any(|e| e.event_type == "part.compaction"), + "compaction should no longer surface as a generic event" + ); + + let compactions: Vec<_> = view.compactions().collect(); + assert_eq!( + compactions.len(), + 1, + "expected exactly one Item::Compaction" + ); + let c = compactions[0]; + assert_eq!(c.id, "pc"); + assert_eq!(c.trigger, Some(CompactionTrigger::Auto), "auto=true ⇒ Auto"); + assert_eq!( + c.parent_id.as_deref(), + Some("ma"), + "compaction parents on the last turn before it (the assistant turn)" + ); + assert_eq!( + c.kept, + vec!["mu".to_string(), "ma".to_string()], + "tailStartId present ⇒ surviving turn ids from anchor to last pre-compaction turn" + ); + + // Item order: user turn, assistant turn, then the compaction. + assert!(matches!(view.items[0], Item::Turn(ref t) if t.role == Role::User)); + assert!(matches!(view.items[1], Item::Turn(ref t) if t.role == Role::Assistant)); + assert!(matches!(view.items[2], Item::Compaction(_))); + } + + #[test] + fn multiple_compactions_keep_distinct_summaries() { + // Two boundaries, each followed by its own synthetic summary + // message. Each compaction must carry ITS OWN summary, not a single + // session-global one stamped onto every boundary — the regression + // this guards (a multi-compaction session collapsing both summaries + // onto the first). + let body = r#" + INSERT INTO project (id, worktree, time_created, time_updated, sandboxes) + VALUES ('p','/p',1,9,'[]'); + INSERT INTO session (id, project_id, slug, directory, title, version, time_created, time_updated) + VALUES ('s','p','slug','/p','T','1.0.0',1,9); + INSERT INTO message (id, session_id, time_created, time_updated, data) VALUES + ('mu','s',1,1,'{"role":"user","time":{"created":1},"agent":"b","model":{"providerID":"p","modelID":"m"}}'), + ('ma','s',2,2,'{"parentID":"mu","role":"assistant","mode":"b","agent":"b","path":{"cwd":"/p","root":"/p"},"cost":0,"tokens":{"input":0,"output":0,"reasoning":0,"cache":{"read":0,"write":0}},"modelID":"m","providerID":"p","time":{"created":2}}'), + ('mc1','s',3,3,'{"role":"user","time":{"created":3},"agent":"b","model":{"providerID":"p","modelID":"m"}}'), + ('ms1','s',4,4,'{"role":"user","time":{"created":4},"agent":"b","model":{"providerID":"p","modelID":"m"},"summary":{"body":"FIRST compaction summary"}}'), + ('mu2','s',5,5,'{"role":"user","time":{"created":5},"agent":"b","model":{"providerID":"p","modelID":"m"}}'), + ('ma2','s',6,6,'{"parentID":"mu2","role":"assistant","mode":"b","agent":"b","path":{"cwd":"/p","root":"/p"},"cost":0,"tokens":{"input":0,"output":0,"reasoning":0,"cache":{"read":0,"write":0}},"modelID":"m","providerID":"p","time":{"created":6}}'), + ('mc2','s',7,7,'{"role":"user","time":{"created":7},"agent":"b","model":{"providerID":"p","modelID":"m"}}'), + ('ms2','s',8,8,'{"role":"user","time":{"created":8},"agent":"b","model":{"providerID":"p","modelID":"m"},"summary":{"body":"SECOND compaction summary — distinct and longer"}}'); + INSERT INTO part (id, message_id, session_id, time_created, time_updated, data) VALUES + ('pu','mu','s',1,1,'{"type":"text","text":"do the thing"}'), + ('pa','ma','s',2,2,'{"type":"text","text":"did it"}'), + ('pc1','mc1','s',3,3,'{"type":"compaction","auto":true,"overflow":false,"tailStartId":"mu"}'), + ('pu2','mu2','s',5,5,'{"type":"text","text":"do another thing"}'), + ('pa2','ma2','s',6,6,'{"type":"text","text":"did it again"}'), + ('pc2','mc2','s',7,7,'{"type":"compaction","auto":false,"overflow":false,"tailStartId":"mu2"}'); + "#; + let (_t, mgr) = setup(body); + let view = to_view(&mgr.read_session("s").unwrap()); + + let compactions: Vec<_> = view.compactions().collect(); + assert_eq!(compactions.len(), 2, "expected two boundaries"); + assert_eq!( + compactions[0].summary.as_deref(), + Some("FIRST compaction summary"), + "first boundary keeps its own summary" + ); + assert_eq!( + compactions[1].summary.as_deref(), + Some("SECOND compaction summary — distinct and longer"), + "second boundary keeps ITS OWN summary, not the first's" ); } @@ -1129,7 +1323,7 @@ mod tests { "#; let (_t, mgr) = setup(body); let view = to_view(&mgr.read_session("s").unwrap()); - assert!(view.events.iter().any(|e| e.event_type == "part.unknown")); + assert!(view.events().any(|e| e.event_type == "part.unknown")); } #[test] @@ -1150,6 +1344,6 @@ mod tests { let ids = ConversationProvider::list_conversations(&mgr, "").unwrap(); assert_eq!(ids, vec!["ses_x".to_string()]); let v = ConversationProvider::load_conversation(&mgr, "", "ses_x").unwrap(); - assert_eq!(v.turns.len(), 2); + assert_eq!(v.turns().count(), 2); } } diff --git a/crates/toolpath-opencode/src/types.rs b/crates/toolpath-opencode/src/types.rs index fa6162e..45522c3 100644 --- a/crates/toolpath-opencode/src/types.rs +++ b/crates/toolpath-opencode/src/types.rs @@ -508,7 +508,16 @@ pub struct CompactionPart { pub auto: bool, #[serde(default, skip_serializing_if = "Option::is_none")] pub overflow: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] + /// First message id of the post-compaction tail. opencode writes this + /// as `tailStartID` (matching its `parentID`/`sessionID` convention); + /// the snake_case alias accepts the form used in older docs/fixtures. + #[serde( + rename = "tailStartID", + alias = "tailStartId", + alias = "tail_start_id", + default, + skip_serializing_if = "Option::is_none" + )] pub tail_start_id: Option, #[serde(flatten, skip_serializing_if = "HashMap::is_empty", default)] pub extra: HashMap, diff --git a/crates/toolpath-opencode/tests/compaction_roundtrip.rs b/crates/toolpath-opencode/tests/compaction_roundtrip.rs index e3a046a..f9ef32c 100644 --- a/crates/toolpath-opencode/tests/compaction_roundtrip.rs +++ b/crates/toolpath-opencode/tests/compaction_roundtrip.rs @@ -10,33 +10,28 @@ //! What this test asserts (and why): //! //! - A compacted session loads via the SQLite reader without crashing. -//! - `to_view` surfaces the compaction part as a `ConversationEvent` -//! in `view.events` (this is the documented contract). -//! - User/assistant content surrounding the compaction part survives -//! the IR derive/extract round-trip and the projector emits a +//! - `to_view` surfaces the compaction part as an `Item::Compaction` +//! at its position in `view.items` (this is the documented contract), +//! not a generic `ConversationEvent`. +//! - The compaction boundary survives the IR derive/extract round-trip +//! as a `conversation.compact` step, and the user/assistant content +//! surrounding it survives too, with the projector emitting a //! functionally equivalent `Session`. -//! -//! Known limitation (documented, not asserted as fully preserved): the -//! `ConversationEvent` carrying the compaction metadata does not -//! survive the `derive → extract` round-trip today — `derive_path` does -//! not emit `conversation.event` steps for `view.events`, and the -//! opencode projector does not consume `view.events`. The compaction -//! marker is purely structural metadata (the surrounding messages -//! carry the actual content), so for "good UX" today this is an -//! acceptable loss; if/when we close the gap, this test gets -//! tightened. use std::fs; +use std::path::{Path, PathBuf}; -use rusqlite::Connection; +use rusqlite::{Connection, params}; +use serde_json::Value; use tempfile::TempDir; use toolpath::v1::Graph; use toolpath_convo::{ - ConversationProjector, ConversationView, DeriveConfig, derive_path, extract_conversation, + Compaction, CompactionTrigger, ConversationProjector, ConversationView, DeriveConfig, Item, + Role, Turn, derive_path, extract_conversation, }; use toolpath_opencode::project::OpencodeProjector; -use toolpath_opencode::types::{MessageData, PartData}; -use toolpath_opencode::{OpencodeConvo, PathResolver, Session, to_view}; +use toolpath_opencode::types::{Message, MessageData, Part, PartData, Session}; +use toolpath_opencode::{OpencodeConvo, PathResolver, to_view}; /// Mid-session compaction. Schema mirrors `tests/projection_roundtrip.rs` /// but adds a `compaction` part in the middle of the assistant flow. @@ -85,7 +80,7 @@ const COMPACTION_SQL: &str = r#" ('prt_u1_1','msg_u1','ses_compact',1001,1001,'{"type":"text","text":"refactor the auth module"}'), ('prt_a1_1','msg_a1','ses_compact',1002,1002,'{"type":"step-start"}'), ('prt_a1_2','msg_a1','ses_compact',1100,1100,'{"type":"text","text":"reading the current auth code"}'), - ('prt_a1_3','msg_a1','ses_compact',1500,1500,'{"type":"compaction","auto":true,"overflow":true,"tailStartId":"prt_a1_3"}'), + ('prt_a1_3','msg_a1','ses_compact',1500,1500,'{"type":"compaction","auto":true,"overflow":true,"tailStartId":"msg_u1"}'), ('prt_u2_1','msg_u2','ses_compact',1600,1600,'{"type":"text","text":"now add session validation"}'), ('prt_a2_1','msg_a2','ses_compact',1700,1700,'{"type":"step-start"}'), ('prt_a2_2','msg_a2','ses_compact',1900,1900,'{"type":"text","text":"added session validation to login()"}'), @@ -133,23 +128,55 @@ fn fixture_loads_with_compaction_part() { } #[test] -fn to_view_surfaces_compaction_as_event() { +fn to_view_surfaces_compaction_as_compaction_item() { let (_temp, session) = setup_session(); let view = to_view(&session); - let event = view - .events - .iter() - .find(|e| e.event_type == "part.compaction"); + assert!( - event.is_some(), - "expected a `part.compaction` ConversationEvent in view.events; got: {:?}", - view.events - .iter() - .map(|e| &e.event_type) - .collect::>() + !view.events().any(|e| e.event_type == "part.compaction"), + "compaction should no longer surface as a generic event" + ); + + let compactions: Vec<_> = view.compactions().collect(); + assert_eq!( + compactions.len(), + 1, + "expected exactly one Item::Compaction; got {}", + compactions.len() + ); + let c = compactions[0]; + // The synthetic SQL fixture's compaction part has `auto: true`. + assert_eq!(c.trigger, Some(CompactionTrigger::Auto)); + assert!( + c.parent_id.is_some(), + "compaction should parent on the prior turn" + ); + // `tailStartId` anchors on `msg_u1`. The compaction part lives inside + // `msg_a1`, so that assistant turn isn't emitted yet when the boundary + // is recorded — the kept tail is just the turns emitted so far from the + // anchor onward, i.e. `[msg_u1]`. + assert_eq!( + c.kept, + vec!["msg_u1".to_string()], + "tailStartId present ⇒ surviving turn ids from anchor to last emitted turn" ); } +#[test] +fn compaction_item_survives_derive_extract() { + let (_temp, session) = setup_session(); + let view = to_view(&session); + let after = ir_roundtrip(&view); + + let before_count = view.compactions().count(); + let after_count = after.compactions().count(); + assert_eq!( + before_count, after_count, + "compaction count changed across round-trip: {before_count} → {after_count}" + ); + assert_eq!(after_count, 1, "the compaction boundary should survive"); +} + #[test] fn pre_compact_user_turn_survives_roundtrip() { let (_temp, session) = setup_session(); @@ -158,11 +185,11 @@ fn pre_compact_user_turn_survives_roundtrip() { let needle = "refactor the auth module"; assert!( - view.turns.iter().any(|t| t.text.contains(needle)), + view.turns().any(|t| t.text.contains(needle)), "pre-compact prompt missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(needle)), + after.turns().any(|t| t.text.contains(needle)), "pre-compact prompt dropped after roundtrip" ); } @@ -178,11 +205,11 @@ fn post_compact_user_and_assistant_turns_survive_roundtrip() { "added session validation to login()", ] { assert!( - view.turns.iter().any(|t| t.text.contains(needle)), + view.turns().any(|t| t.text.contains(needle)), "post-compact text {needle:?} missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(needle)), + after.turns().any(|t| t.text.contains(needle)), "post-compact text {needle:?} dropped after roundtrip" ); } @@ -260,3 +287,526 @@ fn projected_session_serdes_symmetrically() { let json = serde_json::to_string(&projected).expect("serialize"); let _: Session = serde_json::from_str(&json).expect("re-parse"); } + +// ── Real-fixture assertions ──────────────────────────────────────────── +// +// `test-fixtures/opencode/convo-compacted.json` is a captured opencode +// session with a real manual `/compact` boundary (a synthetic +// compaction-bearing user message, `auto: false`, no `tailStartId`). +// It exercises the user-message compaction path that the synthetic SQL +// fixture above (an assistant-message compaction) doesn't. + +fn compacted_fixture_path() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("..") + .join("..") + .join("test-fixtures") + .join("opencode") + .join("convo-compacted.json") +} + +/// Translate opencode's `path export` wrapper (camelCase + nested `info`) +/// into the flat snake-case `Session` shape `to_view` expects. Mirrors the +/// helper in `tests/real_fixture_roundtrip.rs`. +fn parse_opencode_export(json: &str) -> Session { + let v: Value = serde_json::from_str(json).expect("opencode wrapper parse"); + let info = &v["info"]; + let msgs_in = v["messages"].as_array().cloned().unwrap_or_default(); + + let str_or = |key: &str, fallback: &str| -> String { + info.get(key) + .and_then(Value::as_str) + .unwrap_or(fallback) + .to_string() + }; + let i64_at = |path: &[&str]| -> Option { + let mut cur = info; + for k in path { + cur = cur.get(*k)?; + } + cur.as_i64() + }; + + let mut messages: Vec = Vec::with_capacity(msgs_in.len()); + for m in msgs_in { + let mi = m.get("info").cloned().unwrap_or(Value::Null); + let mi_obj = mi.as_object().cloned().unwrap_or_default(); + let id = mi_obj + .get("id") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let session_id = mi_obj + .get("sessionID") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let time_created = mi_obj + .get("time") + .and_then(|t| t.get("created")) + .and_then(Value::as_i64) + .unwrap_or(0); + + let mut data_obj = mi_obj.clone(); + data_obj.remove("id"); + data_obj.remove("sessionID"); + let data: MessageData = + serde_json::from_value(Value::Object(data_obj)).unwrap_or(MessageData::Other); + + let mut parts: Vec = Vec::new(); + if let Some(parts_in) = m.get("parts").and_then(Value::as_array) { + for p in parts_in { + let p_obj = p.as_object().cloned().unwrap_or_default(); + let pid = p_obj + .get("id") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let pmsg = p_obj + .get("messageID") + .and_then(Value::as_str) + .unwrap_or(&id) + .to_string(); + let psess = p_obj + .get("sessionID") + .and_then(Value::as_str) + .unwrap_or(&session_id) + .to_string(); + let mut data_obj = p_obj.clone(); + data_obj.remove("id"); + data_obj.remove("messageID"); + data_obj.remove("sessionID"); + let part_data: PartData = + serde_json::from_value(Value::Object(data_obj)).unwrap_or(PartData::Unknown); + parts.push(Part { + id: pid, + message_id: pmsg, + session_id: psess, + time_created, + time_updated: time_created, + data: part_data, + }); + } + } + + messages.push(Message { + id, + session_id, + time_created, + time_updated: time_created, + data, + parts, + }); + } + + Session { + id: str_or("id", ""), + project_id: str_or("projectID", ""), + workspace_id: info + .get("workspaceID") + .and_then(Value::as_str) + .map(str::to_string), + parent_id: info + .get("parentID") + .and_then(Value::as_str) + .map(str::to_string), + slug: str_or("slug", ""), + directory: PathBuf::from(str_or("directory", "/")), + title: str_or("title", ""), + version: str_or("version", "0.0.0"), + share_url: info + .get("shareURL") + .and_then(Value::as_str) + .map(str::to_string), + summary_additions: i64_at(&["summary", "additions"]), + summary_deletions: i64_at(&["summary", "deletions"]), + summary_files: i64_at(&["summary", "files"]), + time_created: i64_at(&["time", "created"]).unwrap_or(0), + time_updated: i64_at(&["time", "updated"]) + .or_else(|| i64_at(&["time", "created"])) + .unwrap_or(0), + time_compacting: i64_at(&["time", "compacting"]), + time_archived: i64_at(&["time", "archived"]), + messages, + } +} + +fn load_compacted_fixture_session() -> Session { + let json = std::fs::read_to_string(compacted_fixture_path()).expect("read compacted fixture"); + parse_opencode_export(&json) +} + +#[test] +fn real_fixture_emits_one_manual_compaction_item() { + let session = load_compacted_fixture_session(); + let view = to_view(&session); + + let compactions: Vec<_> = view.compactions().collect(); + assert_eq!( + compactions.len(), + 1, + "expected exactly one Item::Compaction in the real fixture; got {}", + compactions.len() + ); + let c = compactions[0]; + // The fixture's `/compact` boundary has `auto: false` ⇒ Manual. + assert_eq!(c.trigger, Some(CompactionTrigger::Manual)); + // No `tailStartId` and no synthetic summary message in this fixture. + assert!( + c.kept.is_empty(), + "no tailStartId ⇒ empty kept range; got {:?}", + c.kept + ); + assert!( + c.parent_id.is_some(), + "compaction should parent on the turn before it" + ); + + // The compaction is positioned mid-stream, with turns on both sides. + let compaction_idx = view + .items + .iter() + .position(|i| matches!(i, Item::Compaction(_))) + .expect("a Compaction item"); + assert!( + view.items[..compaction_idx] + .iter() + .any(|i| matches!(i, Item::Turn(_))), + "expected turns before the compaction" + ); + assert!( + view.items[compaction_idx + 1..] + .iter() + .any(|i| matches!(i, Item::Turn(_))), + "expected turns after the compaction" + ); +} + +#[test] +fn projector_reproduces_compaction_item_through_to_view() { + // Projection round-trip: source Session → view → project → Session → + // re-read view. Exactly one `Item::Compaction` must survive, carrying + // the fixture's manual trigger, and it must land between turns — i.e. + // the projector's inverse of the forward `compaction`-part mapping. + let source = load_compacted_fixture_session(); + let view = to_view(&source); + assert_eq!( + view.compactions().count(), + 1, + "source view should have exactly one Item::Compaction" + ); + + let projector = OpencodeProjector::new() + .with_directory(source.directory.clone()) + .with_project_id(source.project_id.clone()) + .with_version(source.version.clone()); + let projected: Session = projector.project(&view).expect("project"); + + // The projected Session must carry a `compaction` part so a re-read + // reproduces the boundary. + let has_compaction_part = projected.messages.iter().any(|m| { + m.parts + .iter() + .any(|p| matches!(p.data, PartData::Compaction(_))) + }); + assert!( + has_compaction_part, + "projected session should carry a compaction part" + ); + + let reread = to_view(&projected); + let compactions: Vec<_> = reread.compactions().collect(); + assert_eq!( + compactions.len(), + 1, + "exactly one Item::Compaction should survive the projection round-trip; got {}", + compactions.len() + ); + assert_eq!( + compactions[0].trigger, + Some(CompactionTrigger::Manual), + "manual trigger (auto=false) should survive the projection round-trip" + ); + + // Positioned between turns: turns on both sides of the boundary. + let idx = reread + .items + .iter() + .position(|i| matches!(i, Item::Compaction(_))) + .expect("a Compaction item in the re-read view"); + assert!( + reread.items[..idx] + .iter() + .any(|i| matches!(i, Item::Turn(_))), + "expected turns before the compaction in the re-read view" + ); + assert!( + reread.items[idx + 1..] + .iter() + .any(|i| matches!(i, Item::Turn(_))), + "expected turns after the compaction in the re-read view" + ); +} + +#[test] +fn real_fixture_compaction_and_surrounding_turns_survive_roundtrip() { + let session = load_compacted_fixture_session(); + let view = to_view(&session); + let after = ir_roundtrip(&view); + + assert_eq!( + view.compactions().count(), + after.compactions().count(), + "compaction count diverged across round-trip" + ); + assert_eq!( + after.compactions().count(), + 1, + "the manual compaction boundary should survive the round-trip" + ); + assert_eq!( + after.compactions().next().unwrap().trigger, + Some(CompactionTrigger::Manual), + "trigger should survive as Manual" + ); + + // Surrounding turns (pre- and post-compaction) survive intact. + let before_turns = view.turns().count(); + let after_turns = after.turns().count(); + assert_eq!( + before_turns, after_turns, + "turn count diverged across round-trip: {before_turns} → {after_turns}" + ); + assert!(before_turns >= 2, "fixture should have multiple turns"); +} + +// ── True SQLite wire round-trip ───────────────────────────────────────── +// +// The tests above stop at `to_view(&projected)` — an IN-MEMORY re-read that +// walks `Session.messages` in insertion order. opencode's real reader instead +// loads rows from SQLite with `ORDER BY time_created ASC, id ASC` (see +// reader.rs), and turn ids are re-minted on projection. The test below closes +// that gap: it projects a compaction carrying BOTH a summary and a kept tail, +// writes the projected `Session` into a real temp `.db`, and reads it back +// through the actual reader — exercising the SQL ordering and id re-minting +// that the in-memory tests cannot. + +const SCHEMA_SQL: &str = r#" + CREATE TABLE project ( + id text PRIMARY KEY, worktree text NOT NULL, vcs text, name text, + icon_url text, icon_color text, + time_created integer NOT NULL, time_updated integer NOT NULL, + time_initialized integer, sandboxes text NOT NULL, commands text + ); + CREATE TABLE session ( + id text PRIMARY KEY, project_id text NOT NULL, parent_id text, + slug text NOT NULL, directory text NOT NULL, title text NOT NULL, + version text NOT NULL, share_url text, + summary_additions integer, summary_deletions integer, + summary_files integer, summary_diffs text, revert text, permission text, + time_created integer NOT NULL, time_updated integer NOT NULL, + time_compacting integer, time_archived integer, workspace_id text + ); + CREATE TABLE message ( + id text PRIMARY KEY, session_id text NOT NULL, + time_created integer NOT NULL, time_updated integer NOT NULL, + data text NOT NULL + ); + CREATE TABLE part ( + id text PRIMARY KEY, message_id text NOT NULL, session_id text NOT NULL, + time_created integer NOT NULL, time_updated integer NOT NULL, + data text NOT NULL + ); +"#; + +fn mk_turn(id: &str, role: Role, text: &str, ts: &str, model: Option<&str>) -> Turn { + Turn { + id: id.into(), + parent_id: None, + group_id: None, + role, + timestamp: ts.into(), + text: text.into(), + thinking: None, + tool_uses: vec![], + model: model.map(str::to_string), + stop_reason: None, + token_usage: None, + attributed_token_usage: None, + environment: None, + delegations: vec![], + file_mutations: vec![], + } +} + +/// Persist a projected `Session` into a real SQLite `opencode.db` and read it +/// back through the actual reader (which applies `ORDER BY time_created, id`), +/// then to the IR. This is the wire round-trip the in-memory tests skip. +fn persist_and_reread(projected: &Session) -> ConversationView { + let temp = TempDir::new().unwrap(); + let data = temp.path().join(".local/share/opencode"); + fs::create_dir_all(&data).unwrap(); + let conn = Connection::open(data.join("opencode.db")).unwrap(); + conn.execute_batch(SCHEMA_SQL).unwrap(); + + let dir = projected.directory.to_string_lossy().to_string(); + conn.execute( + "INSERT INTO project (id, worktree, time_created, time_updated, sandboxes) + VALUES (?1, ?2, ?3, ?4, '[]')", + params![ + projected.project_id, + dir, + projected.time_created, + projected.time_updated + ], + ) + .unwrap(); + conn.execute( + "INSERT INTO session + (id, project_id, slug, directory, title, version, time_created, time_updated) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)", + params![ + projected.id, + projected.project_id, + projected.slug, + dir, + projected.title, + projected.version, + projected.time_created, + projected.time_updated, + ], + ) + .unwrap(); + + for m in &projected.messages { + let mdata = serde_json::to_string(&m.data).unwrap(); + conn.execute( + "INSERT INTO message (id, session_id, time_created, time_updated, data) + VALUES (?1, ?2, ?3, ?4, ?5)", + params![m.id, m.session_id, m.time_created, m.time_updated, mdata], + ) + .unwrap(); + for p in &m.parts { + let pdata = serde_json::to_string(&p.data).unwrap(); + conn.execute( + "INSERT INTO part (id, message_id, session_id, time_created, time_updated, data) + VALUES (?1, ?2, ?3, ?4, ?5, ?6)", + params![ + p.id, + p.message_id, + p.session_id, + p.time_created, + p.time_updated, + pdata + ], + ) + .unwrap(); + } + } + drop(conn); + + let resolver = PathResolver::new() + .with_home(temp.path()) + .with_data_dir(&data); + let mgr = OpencodeConvo::with_resolver(resolver); + let session = mgr.read_session(&projected.id).expect("read_session"); + to_view(&session) +} + +#[test] +fn compaction_summary_and_kept_survive_real_sqlite_wire_roundtrip() { + // Source: u1 → a1 → [compaction: summary + kept=[a1]] → u2. + let source = ConversationView { + id: "src-sess".into(), + items: vec![ + Item::Turn(mk_turn( + "u1", + Role::User, + "refactor the auth module", + "2026-01-01T00:00:00.000Z", + None, + )), + Item::Turn(mk_turn( + "a1", + Role::Assistant, + "reading the current auth code", + "2026-01-01T00:00:01.000Z", + Some("claude-sonnet-4-6"), + )), + Item::Compaction(Compaction { + id: "c1".into(), + parent_id: Some("a1".into()), + timestamp: "2026-01-01T00:00:02.000Z".into(), + trigger: Some(CompactionTrigger::Auto), + summary: Some("condensed everything up to the auth refactor".into()), + pre_tokens: None, + kept: vec!["a1".into()], + }), + Item::Turn(mk_turn( + "u2", + Role::User, + "now add session validation", + "2026-01-01T00:00:03.000Z", + None, + )), + ], + provider_id: Some("opencode".into()), + ..Default::default() + }; + + let projector = OpencodeProjector::new() + .with_directory(PathBuf::from("/tmp/proj")) + .with_project_id("proj-test"); + let projected: Session = projector.project(&source).expect("project"); + + // Re-read through the actual SQLite reader (ORDER BY time_created, id). + let reread = persist_and_reread(&projected); + + let compactions: Vec<_> = reread.compactions().collect(); + assert_eq!( + compactions.len(), + 1, + "exactly one compaction must survive the SQLite wire round-trip" + ); + let c = compactions[0]; + + // #3: the summary must survive. It survives only because the projector + // gives the summary message a strictly-later timestamp than the boundary + // message; on a shared timestamp the SQL `id ASC` tiebreak would sometimes + // sort the summary first and the reader would drop it. + assert_eq!( + c.summary.as_deref(), + Some("condensed everything up to the auth refactor"), + "compaction summary lost across the SQLite wire round-trip" + ); + + // #4: the kept tail must survive and resolve to a real re-read turn id. + // It survives only because the projector rewrote the kept anchor to the + // re-minted message id; with the raw source id it would match no message + // and the reader would yield an empty kept set. + assert_eq!( + c.kept.len(), + 1, + "kept tail collapsed across the SQLite wire round-trip: {:?}", + c.kept + ); + let kept_id = &c.kept[0]; + assert!( + reread.turns().any(|t| &t.id == kept_id), + "kept anchor {kept_id:?} does not resolve to any re-read turn" + ); + + // Sanity: surrounding turns survived too. + assert!( + reread + .turns() + .any(|t| t.text.contains("refactor the auth module")), + "pre-compaction turn lost" + ); + assert!( + reread + .turns() + .any(|t| t.text.contains("now add session validation")), + "post-compaction turn lost" + ); +} diff --git a/crates/toolpath-opencode/tests/real_fixture_roundtrip.rs b/crates/toolpath-opencode/tests/real_fixture_roundtrip.rs index c4d2007..1ac49a3 100644 --- a/crates/toolpath-opencode/tests/real_fixture_roundtrip.rs +++ b/crates/toolpath-opencode/tests/real_fixture_roundtrip.rs @@ -192,10 +192,7 @@ fn is_system_envelope(turn: &Turn) -> bool { } fn meaningful(view: &ConversationView) -> Vec<&Turn> { - view.turns - .iter() - .filter(|t| !is_system_envelope(t)) - .collect() + view.turns().filter(|t| !is_system_envelope(t)).collect() } fn norm(s: &str) -> String { @@ -206,7 +203,7 @@ fn norm(s: &str) -> String { fn fixture_loads() { let view = load_fixture_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "opencode fixture should produce a non-empty view" ); let m = meaningful(&view); @@ -320,14 +317,14 @@ fn roundtrip_preserves_delegations() { let original = load_fixture_view(); let after = ir_roundtrip(&original); - let total_before: usize = original.turns.iter().map(|t| t.delegations.len()).sum(); - let total_after: usize = after.turns.iter().map(|t| t.delegations.len()).sum(); + let total_before: usize = original.turns().map(|t| t.delegations.len()).sum(); + let total_after: usize = after.turns().map(|t| t.delegations.len()).sum(); assert_eq!( total_before, total_after, "total delegation count diverged: {total_before} → {total_after}" ); - for (i, (a, b)) in original.turns.iter().zip(after.turns.iter()).enumerate() { + for (i, (a, b)) in original.turns().zip(after.turns()).enumerate() { assert_eq!( a.delegations.len(), b.delegations.len(), diff --git a/crates/toolpath-pi/Cargo.toml b/crates/toolpath-pi/Cargo.toml index 8ce2234..f8ff620 100644 --- a/crates/toolpath-pi/Cargo.toml +++ b/crates/toolpath-pi/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath-pi" -version = "0.6.0" +version = "0.7.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath-pi/src/derive.rs b/crates/toolpath-pi/src/derive.rs index 7b04deb..e97ad6b 100644 --- a/crates/toolpath-pi/src/derive.rs +++ b/crates/toolpath-pi/src/derive.rs @@ -134,7 +134,8 @@ mod tests { #[test] fn test_derive_graph_single_session() { let s = make_session("sess-alpha"); - let g = derive_graph(std::slice::from_ref(&s), None, &DeriveConfig::default()); + let g = + derive_graph(std::slice::from_ref(&s), None, &DeriveConfig::default()); assert_eq!(g.paths.len(), 1); assert!(matches!(&g.paths[0], PathOrRef::Path(_))); } diff --git a/crates/toolpath-pi/src/project.rs b/crates/toolpath-pi/src/project.rs index a82b15d..f9d0262 100644 --- a/crates/toolpath-pi/src/project.rs +++ b/crates/toolpath-pi/src/project.rs @@ -9,8 +9,10 @@ //! The projector consumes provider-specific data the forward path //! stashed under `Turn.extra["pi"]` — `api`/`provider`, `stopReason`, //! `toolCallId`, bash-execution metadata, custom-message markers, and -//! synthetic-turn structures (`compaction`, `branchSummary`, `custom`, -//! `customMessage`). For `ConversationView`s from non-Pi sources, the +//! synthetic-turn structures (`branchSummary`, `custom`, +//! `customMessage`). Compaction boundaries are first-class +//! `Item::Compaction`s in the view and project straight back to +//! `Entry::Compaction`. For `ConversationView`s from non-Pi sources, the //! projector synthesizes sensible defaults (api: "anthropic", //! stop_reason: "stop", etc.). //! @@ -22,7 +24,8 @@ use std::collections::HashMap; use serde_json::{Map, Value, json}; use toolpath_convo::{ - ConversationProjector, ConversationView, ConvoError, Result, Role, ToolInvocation, Turn, + Compaction, ConversationProjector, ConversationView, ConvoError, Item, Result, Role, + ToolInvocation, Turn, }; use crate::reader::PiSession; @@ -108,8 +111,7 @@ fn project_view( .cwd .clone() .or_else(|| { - view.turns - .iter() + view.turns() .find_map(|t| t.environment.as_ref()?.working_dir.clone()) }) .unwrap_or_else(|| "/".to_string()); @@ -117,15 +119,15 @@ fn project_view( let timestamp = view .started_at .map(|t| t.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)) - .or_else(|| view.turns.first().map(|t| t.timestamp.clone())) + .or_else(|| view.turns().next().map(|t| t.timestamp.clone())) .unwrap_or_default(); // Pi's session header optionally carries `parentSession` — the // forward path stashed it on the first turn's extras. Round-trip // it when present. let parent_session = view - .turns - .first() + .turns() + .next() .and_then(|t| pi_extras(t)) .and_then(|pi| pi.get("parentSession").and_then(Value::as_str)) .map(str::to_string); @@ -151,8 +153,7 @@ fn project_view( // both populates `tool_uses[i].result` AND keeps the original // tool-result message as a separate turn). let covered: std::collections::HashSet = view - .turns - .iter() + .turns() .filter(|t| matches!(t.role, Role::Other(ref s) if s == "tool")) .filter_map(|t| { pi_extras(t) @@ -162,10 +163,18 @@ fn project_view( }) .collect(); - for turn in &view.turns { - let pi = pi_extras(turn).cloned().unwrap_or_default(); - emit_pending_meta(&mut entries, turn, &pi); - emit_turn_entries(cfg, turn, &pi, &covered, &mut entries); + // Walk `view.items` in order so compaction boundaries land at their + // true position in the entry stream (between the surrounding turns). + for item in &view.items { + match item { + Item::Turn(turn) => { + let pi = pi_extras(turn).cloned().unwrap_or_default(); + emit_pending_meta(&mut entries, turn, &pi); + emit_turn_entries(cfg, turn, &pi, &covered, &mut entries); + } + Item::Compaction(comp) => emit_compaction(comp, &mut entries), + Item::Event(_) => {} + } } Ok(PiSession { @@ -176,9 +185,8 @@ fn project_view( }) } -/// Used to return `Turn.extra["pi"]`; the IR no longer carries -/// provider-namespaced extras. Always `None`. Callers fall back to -/// reconstructing source-format details from typed IR fields and +/// The IR carries no provider-namespaced extras, so this is always `None`; +/// callers reconstruct source-format details from typed IR fields and /// reasonable defaults. fn pi_extras(_turn: &Turn) -> Option<&'static Map> { None @@ -282,12 +290,9 @@ fn emit_turn_entries( covered_tool_ids: &std::collections::HashSet, entries: &mut Vec, ) { - // Synthetic compaction / branch_summary / custom turns map to - // their own Entry variants rather than `Entry::Message`. - if let Some(comp) = pi.get("compaction").and_then(Value::as_object) { - emit_compaction(turn, comp, entries); - return; - } + // Synthetic branch_summary / custom turns map to their own Entry + // variants rather than `Entry::Message`. (Compaction boundaries are + // `Item::Compaction`, not turns, and are handled in `project_view`.) if let Some(bs) = pi.get("branchSummary").and_then(Value::as_object) { emit_branch_summary(turn, bs, entries); return; @@ -547,37 +552,27 @@ fn emit_bash_execution(turn: &Turn, pi: &Map, entries: &mut Vec, entries: &mut Vec) { - let summary = comp - .get("summary") - .and_then(Value::as_str) - .map(str::to_string) - .unwrap_or_else(|| { - // Fall back to extracting from the text the forward path - // wrote ("Compacted (summary): X"). - turn.text - .strip_prefix("Compacted (summary): ") - .unwrap_or(&turn.text) - .to_string() - }); - let first_kept_entry_id = comp - .get("firstKeptEntryId") - .and_then(Value::as_str) - .map(str::to_string) - .unwrap_or_default(); - let tokens_before = comp - .get("tokensBefore") - .and_then(Value::as_u64) - .unwrap_or(0); - let details = comp.get("details").cloned(); - let from_hook = comp.get("fromHook").and_then(Value::as_bool); +/// Reconstruct a Pi `Entry::Compaction` from an `Item::Compaction`. +/// This is the inverse of the forward path's `Item::Compaction` mapping +/// in [`crate::provider::session_to_view`]. +fn emit_compaction(comp: &Compaction, entries: &mut Vec) { + let summary = comp.summary.clone().unwrap_or_default(); + // Pi's format requires a single `firstKeptEntryId` anchor; the forward + // path expands it into a flat list of surviving turn ids, the earliest + // of which is the anchor. Recover it as the first kept id (empty string + // when nothing survived). + let first_kept_entry_id = comp.kept.first().cloned().unwrap_or_default(); entries.push(Entry::Compaction { - base: base_for(turn), + base: EntryBase { + id: comp.id.clone(), + parent_id: comp.parent_id.clone(), + timestamp: comp.timestamp.clone(), + }, summary, first_kept_entry_id, - tokens_before, - details, - from_hook, + tokens_before: comp.pre_tokens.unwrap_or(0), + details: None, + from_hook: None, extra: HashMap::new(), }); } @@ -807,12 +802,11 @@ mod tests { id: "session-uuid".into(), started_at: None, last_activity: None, - turns, + items: turns.into_iter().map(toolpath_convo::Item::Turn).collect(), total_usage: None, provider_id: Some("pi".into()), files_changed: vec![], session_ids: vec![], - events: vec![], ..Default::default() } } diff --git a/crates/toolpath-pi/src/provider.rs b/crates/toolpath-pi/src/provider.rs index 6f091c2..5a8a1d2 100644 --- a/crates/toolpath-pi/src/provider.rs +++ b/crates/toolpath-pi/src/provider.rs @@ -2,9 +2,10 @@ //! //! Walks `PiSession.entries` in file order. Each `Entry::Message` becomes a //! `Turn`; metadata-only entries like `ModelChange` / `ThinkingLevelChange` / -//! `Label` buffer and attach to the next message's `extra["pi"]`. `Compaction`, -//! `BranchSummary`, `Custom`, and `CustomMessage` emit synthetic turns with -//! appropriate roles. +//! `Label` buffer and attach to the next message's `extra["pi"]`. +//! `Entry::Compaction` becomes an `Item::Compaction` at its position in the +//! stream; `BranchSummary`, `Custom`, and `CustomMessage` emit synthetic turns +//! with appropriate roles. //! //! Tool-result correlation is a two-pass process: we record tool-call ids as //! assistant turns are built, then in a second pass populate matching tool @@ -20,9 +21,9 @@ use chrono::{DateTime, Utc}; use serde_json::{Value, json}; use std::collections::HashMap; use toolpath_convo::{ - ConversationMeta, ConversationProvider, ConversationView, ConvoError, DelegatedWork, - EnvironmentSnapshot, Role, SessionBase, TokenUsage, ToolCategory, ToolInvocation, ToolResult, - Turn, + Compaction, ConversationMeta, ConversationProvider, ConversationView, ConvoError, + DelegatedWork, EnvironmentSnapshot, Item, Role, SessionBase, TokenUsage, ToolCategory, + ToolInvocation, ToolResult, Turn, }; // ── Classification helpers ─────────────────────────────────────────── @@ -213,6 +214,28 @@ fn truncate_output(output: &str, max: usize) -> String { } } +/// Expand Pi's single `firstKeptEntryId` anchor into the flat list of +/// surviving turn ids for a `Compaction.kept`. +/// +/// Pi records only the *first* retained entry; the surviving prefix runs +/// from there through the last turn before the compaction boundary. We +/// recover the full set by scanning the turns already emitted into +/// `items` for the anchor, then taking every turn id from that point to +/// the end. If the anchor doesn't match an emitted turn (it can point at +/// a metadata entry we discard, like a `model_change`), we fall back to +/// the bare anchor id so the list is never silently empty. +fn kept_ids_from(items: &[Item], first_kept_entry_id: &str) -> Vec { + let turn_ids: Vec<&str> = items + .iter() + .filter_map(Item::as_turn) + .map(|t| t.id.as_str()) + .collect(); + match turn_ids.iter().position(|id| *id == first_kept_entry_id) { + Some(start) => turn_ids[start..].iter().map(|s| s.to_string()).collect(), + None => vec![first_kept_entry_id.to_string()], + } +} + // ── Main conversion ────────────────────────────────────────────────── /// Convert a PiSession into a provider-agnostic ConversationView. @@ -220,15 +243,16 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { let env = environment_for(session); // Two-pass strategy: - // Pass 1: walk entries, emit turns. Track tool-call invocation locations - // (turn_idx, tool_idx) by id for later correlation. - // Pass 2: walk turns again for tool-result roles; find the matching + // Pass 1: walk entries, emit items (turns + compaction boundaries in + // place). Track tool-call invocation locations (item_idx, + // tool_idx) by id for later correlation. + // Pass 2: walk items again for tool-result roles; find the matching // invocation by id and populate `.result` (and any delegation // result). - let mut turns: Vec = Vec::new(); - // Map tool-call id → (turn_idx, tool_idx). + let mut items: Vec = Vec::new(); + // Map tool-call id → (item_idx, tool_idx); item_idx indexes an `Item::Turn`. let mut tool_call_locs: HashMap = HashMap::new(); - // Map tool-call id → delegation index within the turn (if any). + // Map tool-call id → (item_idx, delegation index) within the turn (if any). let mut delegation_locs: HashMap = HashMap::new(); // Per-turn tool-result info: (tool_call_id, content, is_error). let mut tool_result_payloads: Vec<(usize, String, String, bool)> = Vec::new(); @@ -242,28 +266,35 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { // a cross-harness IR field. } - Entry::Compaction { base, summary, .. } => { - turns.push(Turn { + Entry::Compaction { + base, + summary, + first_kept_entry_id, + tokens_before, + .. + } => { + // Expand Pi's `firstKeptEntryId` anchor into the flat list of + // turn ids that survive verbatim: every turn we've already + // emitted from `firstKeptEntryId` onward through the last turn + // before this compaction. If the anchor doesn't line up with an + // emitted turn (e.g. it points at a discarded metadata entry), + // fall back to the bare anchor id so `kept` is never empty. + let kept = kept_ids_from(&items, first_kept_entry_id); + items.push(Item::Compaction(Compaction { id: base.id.clone(), parent_id: base.parent_id.clone(), - group_id: None, - role: Role::System, timestamp: base.timestamp.clone(), - text: format!("Compacted (summary): {}", summary), - thinking: None, - tool_uses: vec![], - model: None, - stop_reason: None, - token_usage: None, - attributed_token_usage: None, - environment: Some(env.clone()), - delegations: vec![], - file_mutations: Vec::new(), - }); + // Pi's `fromHook` is extension-vs-default provenance, not + // auto-vs-manual, so there's no trigger to record. + trigger: None, + summary: Some(summary.clone()), + pre_tokens: Some(*tokens_before), + kept, + })); } Entry::BranchSummary { base, summary, .. } => { - turns.push(Turn { + items.push(Item::Turn(Turn { id: base.id.clone(), parent_id: base.parent_id.clone(), group_id: None, @@ -279,11 +310,11 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { environment: Some(env.clone()), delegations: vec![], file_mutations: Vec::new(), - }); + })); } Entry::Custom { base, .. } => { - turns.push(Turn { + items.push(Item::Turn(Turn { id: base.id.clone(), parent_id: base.parent_id.clone(), group_id: None, @@ -299,7 +330,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { environment: Some(env.clone()), delegations: vec![], file_mutations: Vec::new(), - }); + })); } Entry::CustomMessage { @@ -308,7 +339,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { content, .. } => { - turns.push(Turn { + items.push(Item::Turn(Turn { id: base.id.clone(), parent_id: base.parent_id.clone(), group_id: None, @@ -324,7 +355,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { environment: Some(env.clone()), delegations: vec![], file_mutations: Vec::new(), - }); + })); } Entry::Message { base, message, .. } => { @@ -357,7 +388,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { stop_reason_s = Some(stop_reason_to_string(stop_reason)); token_usage = usage_to_token_usage(usage); - let turn_idx = turns.len(); + let turn_idx = items.len(); for block in content { if let ContentBlock::ToolCall { id, @@ -448,7 +479,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { } } - turns.push(Turn { + items.push(Item::Turn(Turn { id: base.id.clone(), parent_id: base.parent_id.clone(), group_id: None, @@ -464,15 +495,15 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { environment: Some(env.clone()), delegations, file_mutations: Vec::new(), - }); + })); } } } - // Pass 2: tool-result correlation. + // Pass 2: tool-result correlation. Indices reference `Item::Turn` slots. for (_tr_turn_idx, tool_call_id, content, is_error) in &tool_result_payloads { if let Some((turn_idx, tool_idx)) = tool_call_locs.get(tool_call_id) - && let Some(turn) = turns.get_mut(*turn_idx) + && let Some(Item::Turn(turn)) = items.get_mut(*turn_idx) && let Some(inv) = turn.tool_uses.get_mut(*tool_idx) { inv.result = Some(ToolResult { @@ -481,7 +512,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { }); } if let Some((turn_idx, deleg_idx)) = delegation_locs.get(tool_call_id) - && let Some(turn) = turns.get_mut(*turn_idx) + && let Some(Item::Turn(turn)) = items.get_mut(*turn_idx) && let Some(d) = turn.delegations.get_mut(*deleg_idx) { d.result = Some(content.clone()); @@ -491,7 +522,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { // Aggregate token usage from Assistant turns. let mut have_any_usage = false; let mut total = TokenUsage::default(); - for turn in &turns { + for turn in items.iter().filter_map(Item::as_turn) { if let Some(u) = &turn.token_usage { have_any_usage = true; total.input_tokens = @@ -511,7 +542,7 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { // files_changed: dedup-in-order from FileWrite tool inputs. let mut files_changed: Vec = Vec::new(); let mut seen_files: std::collections::HashSet = std::collections::HashSet::new(); - for turn in &turns { + for turn in items.iter().filter_map(Item::as_turn) { for inv in &turn.tool_uses { if inv.category == Some(ToolCategory::FileWrite) && let Some(p) = extract_file_path(&inv.input) @@ -533,7 +564,11 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { walk_parents(session, &mut session_ids); let started_at = parse_ts(&session.header.timestamp); - let last_activity = turns.last().and_then(|t| parse_ts(&t.timestamp)); + let last_activity = items + .iter() + .filter_map(Item::as_turn) + .next_back() + .and_then(|t| parse_ts(&t.timestamp)); let base = if session.header.cwd.is_empty() { None @@ -548,12 +583,11 @@ pub fn session_to_view(session: &PiSession) -> ConversationView { id: session.header.id.clone(), started_at, last_activity, - turns, + items, total_usage, provider_id: Some("pi".to_string()), files_changed, session_ids, - events: vec![], base, ..Default::default() } @@ -744,7 +778,7 @@ mod tests { fn test_empty_session_produces_view() { let session = session_from(vec![], "/tmp/p"); let v = session_to_view(&session); - assert_eq!(v.turns.len(), 0); + assert_eq!(v.turns().count(), 0); assert_eq!(v.provider_id.as_deref(), Some("pi")); assert_eq!(v.id, "sess-1"); } @@ -753,9 +787,9 @@ mod tests { fn test_user_message_becomes_user_turn() { let session = session_from(vec![user_text_entry("a", None, "hello")], "/tmp/p"); let v = session_to_view(&session); - assert_eq!(v.turns.len(), 1); - assert_eq!(v.turns[0].role, Role::User); - assert_eq!(v.turns[0].text, "hello"); + assert_eq!(v.turns().count(), 1); + assert_eq!(v.turns().next().unwrap().role, Role::User); + assert_eq!(v.turns().next().unwrap().text, "hello"); } #[test] @@ -785,7 +819,7 @@ mod tests { }; let session = session_from(vec![entry], "/tmp/p"); let v = session_to_view(&session); - assert_eq!(v.turns[0].text, "first\nsecond"); + assert_eq!(v.turns().next().unwrap().text, "first\nsecond"); } #[test] @@ -802,10 +836,16 @@ mod tests { "claude-opus", ); let v = session_to_view(&session_from(vec![entry], "/tmp/p")); - assert_eq!(v.turns[0].role, Role::Assistant); - assert_eq!(v.turns[0].model.as_deref(), Some("claude-opus")); - assert_eq!(v.turns[0].stop_reason.as_deref(), Some("stop")); - let u = v.turns[0].token_usage.as_ref().unwrap(); + assert_eq!(v.turns().next().unwrap().role, Role::Assistant); + assert_eq!( + v.turns().next().unwrap().model.as_deref(), + Some("claude-opus") + ); + assert_eq!( + v.turns().next().unwrap().stop_reason.as_deref(), + Some("stop") + ); + let u = v.turns().next().unwrap().token_usage.as_ref().unwrap(); assert_eq!(u.input_tokens, Some(10)); assert_eq!(u.output_tokens, Some(20)); } @@ -834,8 +874,8 @@ mod tests { "m", ); let v = session_to_view(&session_from(vec![entry], "/tmp/p")); - assert_eq!(v.turns[0].text, "one\ntwo"); - assert_eq!(v.turns[0].thinking.as_deref(), Some("mmm")); + assert_eq!(v.turns().next().unwrap().text, "one\ntwo"); + assert_eq!(v.turns().next().unwrap().thinking.as_deref(), Some("mmm")); } #[test] @@ -854,8 +894,8 @@ mod tests { "m", ); let v = session_to_view(&session_from(vec![entry], "/tmp/p")); - assert_eq!(v.turns[0].tool_uses.len(), 1); - let inv = &v.turns[0].tool_uses[0]; + assert_eq!(v.turns().next().unwrap().tool_uses.len(), 1); + let inv = &v.turns().next().unwrap().tool_uses[0]; assert_eq!(inv.id, "tc1"); assert_eq!(inv.name, "Read"); assert_eq!(inv.category, Some(ToolCategory::FileRead)); @@ -908,7 +948,7 @@ mod tests { extra: HashMap::new(), }; let v = session_to_view(&session_from(vec![assistant, tr], "/tmp/p")); - let inv = &v.turns[0].tool_uses[0]; + let inv = &v.turns().next().unwrap().tool_uses[0]; let res = inv.result.as_ref().unwrap(); assert_eq!(res.content, "result"); assert!(!res.is_error); @@ -935,7 +975,7 @@ mod tests { extra: HashMap::new(), }; let v = session_to_view(&session_from(vec![tr], "/tmp/p")); - assert_eq!(v.turns.len(), 0); + assert_eq!(v.turns().count(), 0); } #[test] @@ -956,10 +996,16 @@ mod tests { extra: HashMap::new(), }; let v = session_to_view(&session_from(vec![e], "/tmp/p")); - assert_eq!(v.turns[0].role, Role::Other("bash".to_string())); - assert!(v.turns[0].text.starts_with("$ ls")); - assert_eq!(v.turns[0].tool_uses.len(), 1); - assert_eq!(v.turns[0].tool_uses[0].category, Some(ToolCategory::Shell)); + assert_eq!( + v.turns().next().unwrap().role, + Role::Other("bash".to_string()) + ); + assert!(v.turns().next().unwrap().text.starts_with("$ ls")); + assert_eq!(v.turns().next().unwrap().tool_uses.len(), 1); + assert_eq!( + v.turns().next().unwrap().tool_uses[0].category, + Some(ToolCategory::Shell) + ); } #[test] @@ -971,13 +1017,13 @@ mod tests { ], "/tmp/p", )); - assert_eq!(v.turns[1].parent_id.as_deref(), Some("a")); + assert_eq!(v.turns().nth(1).unwrap().parent_id.as_deref(), Some("a")); } #[test] - fn test_compaction_produces_system_turn() { + fn test_compaction_produces_compaction_item() { let c = Entry::Compaction { - base: base("c", None, "t"), + base: base("c", Some("u1"), "t"), summary: "sum".into(), first_kept_entry_id: "x".into(), tokens_before: 100, @@ -986,8 +1032,87 @@ mod tests { extra: HashMap::new(), }; let v = session_to_view(&session_from(vec![c], "/tmp/p")); - assert_eq!(v.turns[0].role, Role::System); - assert!(v.turns[0].text.starts_with("Compacted")); + // No synthetic turn for the compaction. + assert_eq!(v.turns().count(), 0); + assert_eq!(v.items.len(), 1); + let comp = v.items[0].as_compaction().expect("compaction item"); + assert_eq!(comp.id, "c"); + assert_eq!(comp.parent_id.as_deref(), Some("u1")); + assert_eq!(comp.summary.as_deref(), Some("sum")); + assert_eq!(comp.pre_tokens, Some(100)); + assert_eq!(comp.trigger, None); + // `firstKeptEntryId` ("x") matches no emitted turn here, so `kept` + // falls back to the bare anchor id. + assert_eq!(comp.kept, vec!["x".to_string()]); + } + + #[test] + fn test_compaction_anchor_without_matching_turn_falls_back_to_bare_id() { + let c = Entry::Compaction { + base: base("c", None, "t"), + summary: "sum".into(), + first_kept_entry_id: "x".into(), + tokens_before: 100, + details: None, + from_hook: None, + extra: HashMap::new(), + }; + let v = session_to_view(&session_from(vec![c], "/tmp/p")); + let comp = v.items[0].as_compaction().expect("compaction item"); + assert_eq!(comp.parent_id, None); + // No emitted turn matches the anchor, so `kept` is the bare anchor id + // rather than silently empty. + assert_eq!(comp.kept, vec!["x".to_string()]); + } + + #[test] + fn test_compaction_kept_expands_anchor_into_surviving_turn_ids() { + // Two emitted turns precede the compaction; the anchor points at the + // first of them, so `kept` expands to both turn ids (anchor through + // the last turn before the boundary). + let v = session_to_view(&session_from( + vec![ + user_text_entry("u1", None, "first"), + user_text_entry("u2", Some("u1"), "second"), + Entry::Compaction { + base: base("c", Some("u2"), "t"), + summary: "sum".into(), + first_kept_entry_id: "u1".into(), + tokens_before: 50, + details: None, + from_hook: None, + extra: HashMap::new(), + }, + ], + "/tmp/p", + )); + let comp = v.items[2].as_compaction().expect("compaction item"); + assert_eq!(comp.kept, vec!["u1".to_string(), "u2".to_string()]); + } + + #[test] + fn test_compaction_preserves_position_between_turns() { + let v = session_to_view(&session_from( + vec![ + user_text_entry("u1", None, "before"), + Entry::Compaction { + base: base("c", Some("u1"), "t"), + summary: "sum".into(), + first_kept_entry_id: "u1".into(), + tokens_before: 50, + details: None, + from_hook: None, + extra: HashMap::new(), + }, + user_text_entry("u2", Some("c"), "after"), + ], + "/tmp/p", + )); + assert_eq!(v.items.len(), 3); + assert!(v.items[0].as_turn().is_some()); + assert!(v.items[1].as_compaction().is_some()); + assert!(v.items[2].as_turn().is_some()); + assert_eq!(v.items[2].as_turn().unwrap().text, "after"); } #[test] @@ -1001,8 +1126,8 @@ mod tests { extra: HashMap::new(), }; let v = session_to_view(&session_from(vec![bs], "/tmp/p")); - assert_eq!(v.turns[0].role, Role::System); - assert!(v.turns[0].text.starts_with("Branch summary")); + assert_eq!(v.turns().next().unwrap().role, Role::System); + assert!(v.turns().next().unwrap().text.starts_with("Branch summary")); } #[test] @@ -1015,7 +1140,7 @@ mod tests { }; let msg = user_text_entry("u", None, "hi"); let v = session_to_view(&session_from(vec![mc, msg], "/tmp/p")); - assert_eq!(v.turns.len(), 1); + assert_eq!(v.turns().count(), 1); } #[test] @@ -1027,7 +1152,7 @@ mod tests { ], "/Users/alex/p", )); - for t in &v.turns { + for t in v.turns() { assert_eq!( t.environment.as_ref().unwrap().working_dir.as_deref(), Some("/Users/alex/p") @@ -1195,8 +1320,8 @@ mod tests { let pi = PiConvo::with_resolver(resolver); let v = ConversationProvider::load_conversation(&pi, "/tmp/p", "s1").unwrap(); assert_eq!(v.id, "s1"); - assert_eq!(v.turns.len(), 1); - assert_eq!(v.turns[0].role, Role::User); + assert_eq!(v.turns().count(), 1); + assert_eq!(v.turns().next().unwrap().role, Role::User); } #[test] @@ -1251,9 +1376,12 @@ mod tests { "m", ); let v = session_to_view(&session_from(vec![a], "/tmp/p")); - assert_eq!(v.turns[0].delegations.len(), 1); - assert_eq!(v.turns[0].delegations[0].prompt, "do the thing"); - assert_eq!(v.turns[0].delegations[0].agent_id, "d1"); + assert_eq!(v.turns().next().unwrap().delegations.len(), 1); + assert_eq!( + v.turns().next().unwrap().delegations[0].prompt, + "do the thing" + ); + assert_eq!(v.turns().next().unwrap().delegations[0].agent_id, "d1"); } #[test] @@ -1267,7 +1395,7 @@ mod tests { "m", ); let v = session_to_view(&session_from(vec![a], "/tmp/p")); - let sr = v.turns[0].stop_reason.as_deref().unwrap(); + let sr = v.turns().next().unwrap().stop_reason.as_deref().unwrap(); assert!(sr.to_lowercase().contains("tool"), "got: {}", sr); } @@ -1282,7 +1410,10 @@ mod tests { extra: HashMap::new(), }; let v = session_to_view(&session_from(vec![cm], "/tmp/p")); - assert_eq!(v.turns[0].role, Role::Other("custom:foo".to_string())); - assert_eq!(v.turns[0].text, "body"); + assert_eq!( + v.turns().next().unwrap().role, + Role::Other("custom:foo".to_string()) + ); + assert_eq!(v.turns().next().unwrap().text, "body"); } } diff --git a/crates/toolpath-pi/tests/compaction_roundtrip.rs b/crates/toolpath-pi/tests/compaction_roundtrip.rs index ea5d006..2e906e5 100644 --- a/crates/toolpath-pi/tests/compaction_roundtrip.rs +++ b/crates/toolpath-pi/tests/compaction_roundtrip.rs @@ -14,23 +14,21 @@ //! //! - The fixture loads via `reader::read_session_from_file` without //! crashing on the `Entry::Compaction` line. +//! - Each `Entry::Compaction` becomes an `Item::Compaction` at its +//! position in the stream (not a synthetic `System` turn). //! - Pre-compact user/assistant content survives the round-trip. //! - Post-compact user/assistant content survives the round-trip. +//! - The compaction items round-trip through +//! `derive_path → extract_conversation` (a `conversation.compact` +//! step in between), carrying `summary` and `pre_tokens`. //! - The conversation projects back to JSONL that re-parses through //! the Pi reader. -//! -//! Known limitation (documented, not asserted): the compaction marker -//! itself (with its `summary` text and `tokensBefore` metadata) lands -//! in `Turn.extra["pi"]["compaction"]` per the format docs, but the -//! full structural preservation through `derive → extract → project` -//! is not asserted here. Acceptable loss for "good UX" — the real -//! conversation content lives in the surrounding messages. use std::path::{Path, PathBuf}; use toolpath::v1::Graph; use toolpath_convo::{ - ConversationProjector, ConversationView, DeriveConfig, derive_path, extract_conversation, + ConversationProjector, ConversationView, DeriveConfig, Item, derive_path, extract_conversation, }; use toolpath_pi::project::PiProjector; use toolpath_pi::{reader, session_to_view}; @@ -42,11 +40,26 @@ fn fixture_path() -> PathBuf { .join("compacted_session.jsonl") } +/// The real captured Pi session with two compaction boundaries. +fn real_fixture_path() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("..") + .join("..") + .join("test-fixtures") + .join("pi") + .join("convo-compacted.jsonl") +} + fn load_view() -> ConversationView { let session = reader::read_session_from_file(&fixture_path()).expect("read fixture"); session_to_view(&session) } +fn load_real_view() -> ConversationView { + let session = reader::read_session_from_file(&real_fixture_path()).expect("read real fixture"); + session_to_view(&session) +} + fn ir_roundtrip(view: &ConversationView) -> ConversationView { let path = derive_path(view, &DeriveConfig::default()); let graph = Graph::from_path(path); @@ -60,7 +73,7 @@ fn ir_roundtrip(view: &ConversationView) -> ConversationView { fn fixture_loads_without_panic() { let view = load_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "compaction fixture should produce turns" ); } @@ -73,11 +86,11 @@ fn pre_compact_content_survives_roundtrip() { let needles = ["refactor the auth module", "reading the current auth code"]; for n in needles { assert!( - original.turns.iter().any(|t| t.text.contains(n)), + original.turns().any(|t| t.text.contains(n)), "pre-compact text {n:?} missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(n)), + after.turns().any(|t| t.text.contains(n)), "pre-compact text {n:?} dropped after roundtrip" ); } @@ -94,16 +107,73 @@ fn post_compact_content_survives_roundtrip() { ]; for n in needles { assert!( - original.turns.iter().any(|t| t.text.contains(n)), + original.turns().any(|t| t.text.contains(n)), "post-compact text {n:?} missing from initial view" ); assert!( - after.turns.iter().any(|t| t.text.contains(n)), + after.turns().any(|t| t.text.contains(n)), "post-compact text {n:?} dropped after roundtrip" ); } } +#[test] +fn compaction_entry_becomes_compaction_item() { + let view = load_view(); + let comps: Vec<&toolpath_convo::Compaction> = + view.items.iter().filter_map(Item::as_compaction).collect(); + assert_eq!(comps.len(), 1, "synthetic fixture has one compaction"); + // No synthetic System turn stands in for the compaction. + assert!( + !view.turns().any(|t| t.text.starts_with("Compacted")), + "compaction should not emit a synthetic turn" + ); +} + +#[test] +fn real_fixture_has_two_compaction_items() { + let view = load_real_view(); + let comps: Vec<&toolpath_convo::Compaction> = + view.items.iter().filter_map(Item::as_compaction).collect(); + assert_eq!(comps.len(), 2, "real fixture has two compactions"); + for c in &comps { + assert!(c.summary.is_some(), "summary should be carried"); + assert!(c.pre_tokens.is_some(), "pre_tokens should be carried"); + assert_eq!(c.trigger, None, "Pi doesn't persist auto-vs-manual"); + // `kept` is now a flat list of surviving turn ids, never empty. + assert!(!c.kept.is_empty(), "kept should carry surviving turn ids"); + } + // The first boundary's anchor is a discarded `model_change` entry, so it + // falls back to the bare anchor id; the second anchors at an emitted + // assistant turn that is the last turn before the boundary. + assert_eq!(comps[0].kept, vec!["4cc7b46c".to_string()]); + assert_eq!(comps[1].kept, vec!["3a47185e".to_string()]); +} + +#[test] +fn real_fixture_compactions_and_turns_survive_roundtrip() { + let original = load_real_view(); + let after = ir_roundtrip(&original); + + let comps_after = after.items.iter().filter_map(Item::as_compaction).count(); + assert_eq!(comps_after, 2, "both compactions survive derive → extract"); + for c in after.items.iter().filter_map(Item::as_compaction) { + assert!(c.summary.is_some(), "summary survives roundtrip"); + assert!(c.pre_tokens.is_some(), "pre_tokens survives roundtrip"); + } + + // Surrounding turns (pre- and post-compaction) survive too. + for needle in [ + "walk through a small set of tasks", + "Now print the single word: done.", + ] { + assert!( + after.turns().any(|t| t.text.contains(needle)), + "turn text {needle:?} dropped after roundtrip" + ); + } +} + #[test] fn projector_output_is_re_parseable_by_reader() { let view = load_view(); @@ -123,3 +193,80 @@ fn projector_output_is_re_parseable_by_reader() { std::fs::write(tmp.path(), lines.join("\n")).expect("write tempfile"); reader::read_session_from_file(tmp.path()).expect("re-read projected JSONL"); } + +/// Direct projection round-trip on the real two-compaction fixture: +/// view → `PiProjector` → JSONL → reader → `session_to_view`. Both +/// `Item::Compaction`s must survive the projector reconstructing +/// `Entry::Compaction` from the `Compaction` fields, and they must stay +/// positioned between the surrounding turns. +#[test] +fn projector_reconstructs_compaction_entries() { + let view = load_real_view(); + + let session = PiProjector::new().project(&view).expect("project"); + + // The projector must emit a real `Entry::Compaction` per + // `Item::Compaction` (not fold them into turns). + let emitted_compactions = session + .entries + .iter() + .filter(|e| matches!(e, toolpath_pi::Entry::Compaction { .. })) + .count(); + assert_eq!( + emitted_compactions, 2, + "projector should emit two compaction entries" + ); + + // Re-read the projected JSONL through the Pi reader and back into a view. + let lines: Vec = session + .entries + .iter() + .map(|e| serde_json::to_string(e).expect("serialize pi entry")) + .collect(); + let tmp = tempfile::Builder::new() + .suffix(".jsonl") + .tempfile() + .expect("tempfile"); + std::fs::write(tmp.path(), lines.join("\n")).expect("write tempfile"); + let reread = reader::read_session_from_file(tmp.path()).expect("re-read projected JSONL"); + let after = session_to_view(&reread); + + let comps: Vec<&toolpath_convo::Compaction> = + after.items.iter().filter_map(Item::as_compaction).collect(); + assert_eq!(comps.len(), 2, "both compactions survive projection"); + for c in &comps { + assert!(c.summary.is_some(), "summary survives projection"); + assert!(c.pre_tokens.is_some(), "pre_tokens survives projection"); + assert_eq!(c.trigger, None, "Pi doesn't persist auto-vs-manual"); + assert!(!c.kept.is_empty(), "kept survives projection"); + } + // The anchor (`kept.first()`) round-trips through projection: it's + // written back as `firstKeptEntryId` and recovered on re-read. + assert_eq!(comps[0].kept, vec!["4cc7b46c".to_string()]); + assert_eq!(comps[1].kept, vec!["3a47185e".to_string()]); + + // Each compaction is positioned in the entry stream after the turns + // it summarizes — never the first item, always preceded by a turn. + let comp_indices: Vec = after + .items + .iter() + .enumerate() + .filter(|(_, i)| i.as_compaction().is_some()) + .map(|(idx, _)| idx) + .collect(); + for &idx in &comp_indices { + assert!(idx > 0, "compaction should not be the first item"); + assert!( + after.items[..idx].iter().any(|i| i.as_turn().is_some()), + "a turn precedes the compaction" + ); + } + // And at least one compaction sits strictly between two turns (the + // first boundary in this fixture is followed by more conversation). + assert!( + comp_indices + .iter() + .any(|&idx| after.items[idx + 1..].iter().any(|i| i.as_turn().is_some())), + "at least one compaction is followed by a turn" + ); +} diff --git a/crates/toolpath-pi/tests/end_to_end.rs b/crates/toolpath-pi/tests/end_to_end.rs index 0ed9b44..03a37b4 100644 --- a/crates/toolpath-pi/tests/end_to_end.rs +++ b/crates/toolpath-pi/tests/end_to_end.rs @@ -78,7 +78,7 @@ fn test_to_view_produces_expected_turns() { // Turn count: user + assistant + assistant = 3 // (tool-result entries fold into the assistant's tool_uses[i].result; // they no longer surface as standalone turns.) - assert_eq!(view.turns.len(), 3); + assert_eq!(view.turns().count(), 3); assert_eq!(view.provider_id.as_deref(), Some("pi")); // files_changed should include "hello.rs" assert!(view.files_changed.iter().any(|f| f == "hello.rs")); diff --git a/crates/toolpath-pi/tests/real_fixture_roundtrip.rs b/crates/toolpath-pi/tests/real_fixture_roundtrip.rs index e82c67b..194ebee 100644 --- a/crates/toolpath-pi/tests/real_fixture_roundtrip.rs +++ b/crates/toolpath-pi/tests/real_fixture_roundtrip.rs @@ -54,10 +54,7 @@ fn is_system_envelope(turn: &Turn) -> bool { } fn meaningful(view: &ConversationView) -> Vec<&Turn> { - view.turns - .iter() - .filter(|t| !is_system_envelope(t)) - .collect() + view.turns().filter(|t| !is_system_envelope(t)).collect() } fn norm(s: &str) -> String { @@ -68,7 +65,7 @@ fn norm(s: &str) -> String { fn fixture_loads() { let view = load_fixture_view(); assert!( - !view.turns.is_empty(), + view.turns().next().is_some(), "pi fixture should produce a non-empty view" ); let m = meaningful(&view); @@ -182,14 +179,14 @@ fn roundtrip_preserves_delegations() { let original = load_fixture_view(); let after = ir_roundtrip(&original); - let total_before: usize = original.turns.iter().map(|t| t.delegations.len()).sum(); - let total_after: usize = after.turns.iter().map(|t| t.delegations.len()).sum(); + let total_before: usize = original.turns().map(|t| t.delegations.len()).sum(); + let total_after: usize = after.turns().map(|t| t.delegations.len()).sum(); assert_eq!( total_before, total_after, "total delegation count diverged: {total_before} → {total_after}" ); - for (i, (a, b)) in original.turns.iter().zip(after.turns.iter()).enumerate() { + for (i, (a, b)) in original.turns().zip(after.turns()).enumerate() { assert_eq!( a.delegations.len(), b.delegations.len(), diff --git a/crates/toolpath/Cargo.toml b/crates/toolpath/Cargo.toml index 77f3849..edfcb39 100644 --- a/crates/toolpath/Cargo.toml +++ b/crates/toolpath/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "toolpath" -version = "0.7.0" +version = "0.8.0" edition.workspace = true license.workspace = true repository = "https://github.com/empathic/toolpath" diff --git a/crates/toolpath/schema/toolpath.schema.json b/crates/toolpath/schema/toolpath.schema.json index 59efd48..9545440 100644 --- a/crates/toolpath/schema/toolpath.schema.json +++ b/crates/toolpath/schema/toolpath.schema.json @@ -340,7 +340,7 @@ "type": "string", "format": "uri", "description": "URI naming a kind specification this path conforms to. Defined kinds are listed at https://toolpath.net/kinds/. Kind URIs are immutable; revisions ship at a new version URI. Consumers should treat an absent or unrecognized URI as a generic path.", - "examples": ["https://toolpath.net/kinds/agent-coding-session/v1.0.0"] + "examples": ["https://toolpath.net/kinds/agent-coding-session/v1.2.0"] }, "source": { "type": "string", diff --git a/crates/toolpath/src/jsonl.rs b/crates/toolpath/src/jsonl.rs index 17139da..6391932 100644 --- a/crates/toolpath/src/jsonl.rs +++ b/crates/toolpath/src/jsonl.rs @@ -1250,7 +1250,7 @@ mod tests { }; let jsonl = p.to_jsonl_string().unwrap(); assert!( - jsonl.contains(r#""kind":"https://toolpath.net/kinds/agent-coding-session/v1.1.0""#) + jsonl.contains(r#""kind":"https://toolpath.net/kinds/agent-coding-session/v1.2.0""#) ); let back = Path::from_jsonl_str(&jsonl).unwrap(); assert_eq!(canonical_json(&p), canonical_json(&back)); diff --git a/crates/toolpath/src/types.rs b/crates/toolpath/src/types.rs index d1be613..ac027b8 100644 --- a/crates/toolpath/src/types.rs +++ b/crates/toolpath/src/types.rs @@ -141,14 +141,16 @@ pub struct Base { } /// [`PathMeta::kind`] URI for a path derived from an AI coding conversation. -/// Spec at . +/// Spec at . /// -/// v1.1.0 specifies message-level token accounting: steps derived from one +/// v1.2.0 adds the `conversation.compact` step type — a context-compaction +/// boundary recorded as its own step between the turns it separates — on top +/// of v1.1.0's message-level token accounting: steps derived from one /// provider message share a `message_id`, and the message's `token_usage` /// appears on exactly one of them (the group's last step in document /// order), so summing usage over a path's steps yields session totals. pub const PATH_KIND_AGENT_CODING_SESSION: &str = - "https://toolpath.net/kinds/agent-coding-session/v1.1.0"; + "https://toolpath.net/kinds/agent-coding-session/v1.2.0"; /// The previous version URI. Documents produced before the v1.1.0 /// accounting rule carry this kind; consumers summing their `token_usage` @@ -836,12 +838,12 @@ mod tests { }; let json = serde_json::to_string(&meta).unwrap(); assert!( - json.contains(r#""kind":"https://toolpath.net/kinds/agent-coding-session/v1.1.0""#) + json.contains(r#""kind":"https://toolpath.net/kinds/agent-coding-session/v1.2.0""#) ); let parsed: PathMeta = serde_json::from_str(&json).unwrap(); assert_eq!( parsed.kind.as_deref(), - Some("https://toolpath.net/kinds/agent-coding-session/v1.1.0") + Some("https://toolpath.net/kinds/agent-coding-session/v1.2.0") ); } diff --git a/docs/agents/formats/README.md b/docs/agents/formats/README.md index cc01167..2f73963 100644 --- a/docs/agents/formats/README.md +++ b/docs/agents/formats/README.md @@ -49,6 +49,30 @@ loader's own undocumented strictness on what it will accept). The other three sit in single files because their formats are either simpler or sufficiently covered there. +## Compaction at a glance + +How each harness records context compaction (summarizing older turns to +stay under the window). Detail lives in each harness's doc. + +| Harness | Marker | New session/file? | Kept set | Duplicate-id hazard | +|---|---|---|---|---| +| Claude Code | `compact_boundary` + synthetic `isCompactSummary` user msg (inline) | no | `compactMetadata` enumerates a contiguous recent tail — **but** an early block is *also* re-emitted, so realized retention is non-contiguous | **yes** — the re-emitted block reuses UUIDs; dedupe keeping the first occurrence | +| Codex | `compacted` rollout line | no | opaque `summary` (wholesale replace) | no | +| opencode | `compaction` part (sets `time_compacting`) | no | contiguous tail via `tail_start_id` | no | +| Pi | `Compaction` entry on the id/parentId tree | no | contiguous tail via `firstKeptEntryId` | no | +| Gemini | — (compresses in-memory; nothing persisted) | — | — | no | + +Manual vs. automatic compaction produce the **same record** in every +harness; only the trigger's visibility differs (persisted by Claude and +opencode; analytics-only/absent for Codex and Pi; Gemini persists +nothing). Two takeaways that drive our derivation: (1) **no harness +starts a new session/entity for compaction** — it's always an inline +marker (session *rotation* is a separate mechanism); and (2) **only +Claude Code reuses ids at the boundary**, so it's the only harness that +needs a dedupe pass — everyone else marks compaction cleanly with +unique ids, or (Gemini) compresses only in-memory with nothing +persisted. + ## Conventions used in these docs - **"In the wild"** = observed in real JSONL files on disk, not just in types diff --git a/docs/agents/formats/claude-code/entry-types.md b/docs/agents/formats/claude-code/entry-types.md index b139aac..5757245 100644 --- a/docs/agents/formats/claude-code/entry-types.md +++ b/docs/agents/formats/claude-code/entry-types.md @@ -330,17 +330,23 @@ equivalent. "logicalParentUuid": "...", // the real prior message UUID "compactMetadata": { "trigger": "auto", // or "manual" - "preTokens": 180000 + "preTokens": 1008497, + "postTokens": 11528, // 2.1.x + "preservedSegment": { "headUuid": "...", "anchorUuid": "...", "tailUuid": "..." }, + "preservedMessages": { "anchorUuid": "...", "uuids": ["..."] } }, ... } ``` -Immediately followed by a synthetic `user`-role message with -`isCompactSummary: true` and `isVisibleInTranscriptOnly: true` -carrying the compacted summary as its content. See -[session-chains.md](session-chains.md) for how this interacts with -file rotation. +`compactMetadata` in current 2.1.x enumerates the verbatim-kept recent +tail (`preservedSegment` / `preservedMessages`); older versions carry +only `{trigger, preTokens}`. Immediately followed by a synthetic +`user`-role message with `isCompactSummary: true` and +`isVisibleInTranscriptOnly: true` carrying the compacted summary as its +content. See [session-chains.md](session-chains.md) for how this +interacts with file rotation, and for the **duplicate-UUID +re-emission** that can accompany a compaction. --- diff --git a/docs/agents/formats/claude-code/jsonl-envelope.md b/docs/agents/formats/claude-code/jsonl-envelope.md index 0a04996..533acb2 100644 --- a/docs/agents/formats/claude-code/jsonl-envelope.md +++ b/docs/agents/formats/claude-code/jsonl-envelope.md @@ -84,7 +84,7 @@ Every envelope field we have observed, in rough order of prominence: | `isCompactSummary` | bool | synthetic user message after `compact_boundary` | Always paired with `isVisibleInTranscriptOnly: true`. | | `isVisibleInTranscriptOnly` | bool | see above | Entry is visible in the UI but not replayed to the model. | | `logicalParentUuid` | UUID | `compact_boundary` | Points at the pre-compact last message. `parentUuid` is `null` on these. | -| `compactMetadata` | object | `compact_boundary` | `{trigger: "auto"|"manual", preTokens: number}`. | +| `compactMetadata` | object | `compact_boundary` | `{trigger: "auto"|"manual", preTokens}`; 2.1.x also carries `postTokens`, `durationMs`, and `preservedSegment`/`preservedMessages` enumerating the verbatim-kept recent tail. | | `thinkingMetadata` | object | some user entries | `{level, disabled, triggers[]}`. Indicates extended-thinking configuration. | ### Hook-injected fields diff --git a/docs/agents/formats/claude-code/known-issues.md b/docs/agents/formats/claude-code/known-issues.md index bc12f86..08430d7 100644 --- a/docs/agents/formats/claude-code/known-issues.md +++ b/docs/agents/formats/claude-code/known-issues.md @@ -49,6 +49,24 @@ snapshot. **Defense:** if continuity matters, checkpoint before compaction. +### Duplicate UUIDs at compaction boundaries + +In long `[1m]`-context sessions, an auto-compaction can re-emit an +earlier block of the conversation as fresh entries that **reuse the +original `uuid`s** (with re-threaded `parentUuid`s) in the run just +before the `compact_boundary`. The result is the **same `uuid` +appearing twice in one file** — violating the usual within-file +uniqueness assumption. This is what makes a naively-derived path carry +duplicate `step.id`s, which then fails any store with a +`(path, step_id)` primary key. + +**Defense:** dedupe by `uuid`, keeping the **first** occurrence — it +carries the true `parentUuid` lineage; the re-emitted copy is +re-parented into a synthetic linear chain and should be dropped. The +re-emitted block is *not* the same as `compactMetadata.preservedMessages` +(which enumerates only the recent kept tail). See +[session-chains.md §Re-emitted messages with duplicate UUIDs](session-chains.md#re-emitted-messages-with-duplicate-uuids). + ## Race conditions ### Multi-terminal writes to the same project diff --git a/docs/agents/formats/claude-code/session-chains.md b/docs/agents/formats/claude-code/session-chains.md index b14406f..1bba4a8 100644 --- a/docs/agents/formats/claude-code/session-chains.md +++ b/docs/agents/formats/claude-code/session-chains.md @@ -134,10 +134,21 @@ The JSONL records this inline with a `compact_boundary` entry. "type": "compact_boundary", // or "system" with "subtype": "compact_boundary" "uuid": "...", "parentUuid": null, // always null on the boundary - "logicalParentUuid": "...", // points at the real prior message + "logicalParentUuid": "...", // == preservedSegment.tailUuid (the kept tail's last message) "compactMetadata": { "trigger": "auto", // "auto" or "manual" (user ran /compact) - "preTokens": 180000 // conversation size before compaction + "preTokens": 1008497, // conversation size before compaction + "postTokens": 11528, // size after the summary replaced the middle + "durationMs": 106367, // how long the compaction pass took + "preservedSegment": { // the recent tail kept verbatim + "headUuid": "...", // first kept message + "anchorUuid": "...", // == the synthetic summary's uuid (below) + "tailUuid": "..." // last kept message (== logicalParentUuid) + }, + "preservedMessages": { // explicit enumeration of the kept tail + "anchorUuid": "...", + "uuids": ["...", "..."] // every uuid in [headUuid .. tailUuid] + } }, "sessionId": "...", "timestamp": "..." @@ -148,6 +159,14 @@ Key property: `parentUuid` is `null`, resetting the DAG. The actual prior message is referenced via `logicalParentUuid` so UIs can still render the pre-compact history. +`compactMetadata` is richer than just `{trigger, preTokens}` — current +2.1.x boundaries enumerate **exactly which messages survived**: +`preservedSegment` gives the contiguous recent tail (`headUuid` → +`tailUuid`), and `preservedMessages.uuids` lists every uuid in it. The +`anchorUuid` is the synthetic summary message's own uuid. Older +versions emit only `{trigger, preTokens}` with no preserved-set +detail, so treat the segment fields as optional. + ### The synthetic summary Immediately after the boundary, a synthetic `user`-role message carries @@ -177,6 +196,33 @@ When rendering a transcript, skip these synthetic summary entries or mark them specially; treating them as real user messages will confuse consumers. +### Re-emitted messages with duplicate UUIDs + +Observed in long `[1m]`-context 2.1.x sessions that auto-compact a +>1M-token conversation: in the run of entries **immediately before** +the `compact_boundary`, Claude re-materializes an earlier block of the +conversation (e.g. the original task and setup turns) as fresh entries +that **reuse the original `uuid`s**, with `parentUuid` chains +re-threaded into a synthetic linear sequence. So a single file can +contain the same `uuid` twice — once in its original position early in +the file, once in this re-emitted block. + +These re-emitted entries: +- are **not** listed in `compactMetadata.preservedMessages` (that lists + only the recent tail) — so they are a separate phenomenon from the + marked preserved segment; +- carry the same `uuid` but a **different** `parentUuid` than their + earlier counterpart (the original has the true lineage; the copy is + re-parented linearly); +- sit between the preserved tail and the boundary marker. + +This breaks the usual "uuid is unique within a file" assumption. A +consumer that keys on `uuid` (or stores steps under a `(file, uuid)` +primary key) **must dedupe** — keep the **first** occurrence, which +carries the real `parentUuid` lineage; the re-emitted copies are the +ones to drop. See [known-issues.md §Duplicate UUIDs at compaction +boundaries](known-issues.md#duplicate-uuids-at-compaction-boundaries). + ### Compaction strategies Several compaction strategies exist internally: diff --git a/docs/agents/formats/codex.md b/docs/agents/formats/codex.md index 9734c35..a3bd725 100644 --- a/docs/agents/formats/codex.md +++ b/docs/agents/formats/codex.md @@ -175,6 +175,46 @@ pub enum RolloutItem { | `session_state` | Mid-session state updates (e.g. model switch) | 0 | | `compacted` | Inserted when Codex compacts history mid-session | 0 | +### `compacted` — context compaction + +When Codex compacts mid-session it appends a single `compacted` line to +the **same rollout file** — no new file, no new session id: + +```json +{"type":"compacted","payload":{"message":"…summary text…","replacement_history":[…],"window_id":1}} +``` + +Per current Codex `main` (`codex-rs/protocol/src/protocol.rs`, +`CompactedItem`), `payload` is `{message, replacement_history?, +window_id?}`: `message` is the summary text, `replacement_history` is +the new condensed history that replaces the old, `window_id` is the +auto-compact window counter. **There is no `trigger`, `preTokens`, or +`summary` field** — manual `/compact` and automatic (overflow) +compaction write an **identical** record; the manual/auto distinction +(`CompactionTrigger`) is analytics-only and never persisted to the +rollout. (A separate field-less `event_msg` `ContextCompacted` is also +written — "either automatically or manually".) + +The turns on either side keep their original ids — Codex does **not** +replay or re-id messages across the boundary, so there's no +duplicate-id hazard. `toolpath-codex` maps the marker to an +`Item::Compaction` positioned between the turns it separates (see +`tests/compaction_roundtrip.rs`), which the shared `derive_path` +projects to a `conversation.compact` step; the surrounding turns +survive intact. Only `message` is consumed (as `Compaction.summary`). +Codex never persists the manual-vs-auto trigger or the pre-compaction +token count, and `replacement_history` is a wholesale replacement we +don't fold in, so `trigger`/`pre_tokens` are `None` and `kept` is +empty. The marker carries no id of its own, so a stable `compact-` +is synthesized; `parent_id` links to the last turn before the boundary. + +> Note: the repo fixture `tests/fixtures/compacted_session.jsonl` is +> synthetic (real compaction needs a full context window to trigger), +> but its `compacted` line uses the real +> `{message, replacement_history}` shape. The captured fixture +> `test-fixtures/codex/convo-compacted.jsonl` is a production rollout +> that actually compacted (with an empty `message`). + ## `session_meta` — first line of every file ```json @@ -862,6 +902,7 @@ The mapping below is what the provider actually emits. Source: | `event_msg.token_count.info.total_token_usage` | cumulative; differenced per step → `Turn.attributed_token_usage`, summed per round → `Turn.token_usage` (round's final turn) + `ConversationView.total_usage` | | `event_msg.token_count.info.total_token_usage.reasoning_output_tokens` (⊆ output, cumulative) | differenced per step → `breakdowns["output"]["reasoning"]` on `attributed_token_usage`; summed per round onto `token_usage` (informational, never summed into the total) | | `event_msg` non-turn types (`task_started`, `task_complete`, `user_message`, `agent_message`, etc.) | `ConversationView.events` as typed `ConversationEvent`s | +| `compacted` (`payload.message`) | `Item::Compaction` slotted between the surrounding turns (`summary = message`; `trigger`/`pre_tokens` `None`, `kept` empty); projects to a `conversation.compact` step | | unknown `response_item` / `event_msg` kinds | preserved verbatim in `events` and round-trip via `RolloutItem::Unknown` / `ResponseItem::Other` / `EventMsg::Other` | ### Fidelity guarantees diff --git a/docs/agents/formats/cursor.md b/docs/agents/formats/cursor.md index 20d1212..a55ecc5 100644 --- a/docs/agents/formats/cursor.md +++ b/docs/agents/formats/cursor.md @@ -462,6 +462,43 @@ For `run_terminal_command_v2`, the structure is richer: Pair tool-call bubbles to their downstream effect via `toolCallId` and `checkpointId` (see below). +## Compaction (`/summarize`) — boundary marker on disk, summary server-side + +Cursor compacts context three ways: automatically as a conversation nears +the context-window limit, manually via the **`/summarize`** slash command +(Cursor 1.6+), and — for the Composer model family — via RL-trained +"self-summarization" that compresses the model's own context in-loop. + +On disk a compaction leaves **one durable marker**: a bubble with +`type: 2` and **`capabilityType: 22`**, empty `text`, written into the +stream at the point compaction occurred. All original bubbles are +retained (compaction changes what the model sees, not what's stored), so +the pre-compaction history stays intact. + +But the **summary text and the kept-message set are not recoverable from +the local store** — verified against a live `/summarize`d session: + +- The marker bubble carries no summary (empty `text`; its + `conversationState` is just `"~"`). +- There is **no `latestConversationSummary`** field on `composerData` + (the schema some older reverse-engineering describes is absent in + current Cursor). +- The composer's `conversationState` (a `~`-prefixed base64 **protobuf** + blob) holds the recompacted *context* — system prompt, tool/agent/skill + definitions — but not the summary prose and no kept-bubble references. +- The `speculativeSummarizationEncryptionKey` on each composer gates a + summarization payload that **isn't stored locally** (it's server-side); + no encrypted local blob is tied to the composer. + +Net for us: like Gemini, there is **no reconstructable compaction to +model** — but for a different reason. Gemini persists nothing; Cursor +persists a boundary *marker* whose content lives server-side. A +content-less `conversation.compact` (no summary, empty `kept`) would be +misleading, so `toolpath-cursor` recognizes the `capabilityType: 22` +bubble (`Bubble::is_summarization`) and **skips it** — emitting neither a +turn nor a compaction. The marker is documented here for whoever later +gains access to Cursor's server-side summary payload. + ## Tool catalogue Cursor's wire-level tool inventory is defined by the diff --git a/docs/agents/formats/gemini.md b/docs/agents/formats/gemini.md index 3fa8cfc..3e7f0e1 100644 --- a/docs/agents/formats/gemini.md +++ b/docs/agents/formats/gemini.md @@ -505,6 +505,27 @@ overflow or plan-mode transitions. Each session is exactly one starts, a new `session-*.json` is created. There is no chain index to follow. +## Compaction — in-memory only, never persisted + +Gemini CLI **does** compress context — automatically when token usage +crosses a configurable threshold, and manually via `/compress` (aliases +`summarize`/`compact`). Both go through the same +`tryCompressChat` / `ChatCompressionService.compress` path, differing +only by a `force` flag (manual forces; auto gates on the threshold). + +But compression is **purely in-memory**. No summary, boundary, or +marker is ever written to the session file — and, per a known +gemini-cli bug (issues #20803 / #21335), the on-disk file isn't even +updated to the compressed state: it retains the **full pre-compression +history**. So a derivation reading the session file always sees the +complete, uncompressed conversation with no compaction event in it. + +Net for us: still **no compaction provenance to model** and no +duplicate-id hazard — but the reason is "compresses but persists +nothing," not "no compaction mechanism." (The `summary` field on the +format remains a sub-agent's reported result — see +[§Sub-agents](#sub-agents) — not a context summary.) + ## Timestamps and encoding - All timestamps are ISO-8601 UTC with millisecond precision and a diff --git a/docs/agents/formats/opencode.md b/docs/agents/formats/opencode.md index f4c7f84..c03cbbd 100644 --- a/docs/agents/formats/opencode.md +++ b/docs/agents/formats/opencode.md @@ -516,6 +516,24 @@ context window. Messages before `tail_start_id` are summarized into a single synthetic user message; the history above the marker is kept in the DB for reverts. +Compaction stays **within one session** — it sets the session row's +`time_compacting` timestamp but does not create a new session row. +(`session.parent_id` is for forked sub-agent sessions, not +compaction.) `tail_start_id` is a single anchor describing a +**contiguous** kept tail — everything from it forward survives, so +there is no non-contiguous "pinned" retention here. Message and part +ids are **not** reused across the boundary, so opencode compaction +carries no duplicate-id hazard. + +**IR-projection notes.** `auto` is a plain boolean, so the IR `trigger` +round-trips as auto / not-auto only: a boundary with no trigger (or a +`manual` one) reads back as `Manual`, and only context-overflow boundaries +read back as `Auto`. Each boundary's summary is the `summary.body` of *its +own* synthetic summary message (the one opencode writes at that boundary), +so a session with several compactions keeps a distinct summary per +boundary — they are not collapsed onto the session's first summary. +opencode carries no per-boundary `pre_tokens`. + ## Tool catalogue A reader should not enumerate tool names — any agent config can @@ -698,7 +716,7 @@ Minimum viable mapping, if we follow the Pi-style approach (build a | `patch` part | file-artifact sibling `ArtifactChange.raw` from `git diff ` | | `step-finish.tokens` | `Turn.token_usage` (delta) + summed into `ConversationView.total_usage` | | `subtask` part | `Turn.delegations[]`, with sub-session linked via `session.parent_id` | -| `compaction` part | `ConversationEvent { event_type: "compaction" }` | +| `compaction` part | `Item::Compaction` in place (parented on the prior turn; `auto` ⇒ `trigger`, `tailStartID` ⇒ `kept`); projects to a `conversation.compact` step | | `retry` part | `ConversationEvent { event_type: "retry" }` | | `todo` row | `ConversationEvent { event_type: "todo" }` or top-level path meta | | unknown part `type` | `ConversationEvent` preserving the raw payload | diff --git a/docs/agents/formats/pi.md b/docs/agents/formats/pi.md index ca1d77e..9eb1f9f 100644 --- a/docs/agents/formats/pi.md +++ b/docs/agents/formats/pi.md @@ -79,6 +79,34 @@ Tagged by `type`. Every non-session entry shares an [`EntryBase`] Future entry types should round-trip via `extra` — Pi has added variants between minor versions. +#### `compaction` entry fields + +The `Compaction` entry (`crates/toolpath-pi/src/types.rs`) carries: + +| Field | Type | Notes | +|---|---|---| +| `summary` | string | The summary that replaces the discarded prefix. | +| `firstKeptEntryId` | string | First entry **not** discarded — everything before it was summarized. A single contiguous-tail anchor. | +| `tokensBefore` | u64 | Context token count before compaction. | +| `details` | object? | Optional opaque detail. | +| `fromHook` | bool? | `true` if an **extension** supplied the summary (via the `session_before_compact` hook); `false`/absent for Pi's default compaction. **Not** an auto-vs-manual flag — manual `/compact` and automatic compaction both use Pi's default path and produce the same entry. (Legacy field name.) | + +Compaction is an **in-file** marker on the existing id/parentId tree — +it does **not** start a new session or reuse entry ids, so there's no +duplicate-id hazard. (The separate `parentSession` header field links a +*forked/resumed* session to a parent file; that is unrelated to +compaction.) + +**Projecting a foreign compaction into pi is lossy by format.** pi's entry +can't carry everything the cross-harness IR holds, so a `Compaction` from +another harness is coerced on the way in: `trigger` is dropped (no +auto-vs-manual concept — see `fromHook` above), `pre_tokens` becomes `0` +when unknown (`tokensBefore` is a mandatory `u64`), and `kept` is never +empty — a "wholesale" boundary that kept nothing gains the +`firstKeptEntryId` anchor. After a round-trip through pi you therefore +can't distinguish a real `0` from an unknown pre-token count, nor a +kept-nothing boundary from one that kept a single turn. + ### Message roles `message` entries wrap an `AgentMessage` discriminated by `role`: diff --git a/scripts/capture-elicit-fixtures.sh b/scripts/capture-elicit-fixtures.sh index 8adab71..e775911 100755 --- a/scripts/capture-elicit-fixtures.sh +++ b/scripts/capture-elicit-fixtures.sh @@ -6,7 +6,9 @@ # # Drive each available harness through docs/agents/feature-elicit.prompt.txt # in a fresh scratch directory and copy the resulting session file into -# test-fixtures// at the workspace root. +# test-fixtures// at the workspace root. Then run a second pass +# that forces a context COMPACTION and captures it as +# test-fixtures//convo-compacted.. # # Run from a logged-in shell that already has each harness's CLI # installed and authenticated. Harnesses whose CLIs aren't on PATH are @@ -15,12 +17,25 @@ # Usage: # ./scripts/capture-elicit-fixtures.sh # all harnesses # ./scripts/capture-elicit-fixtures.sh claude codex # specific subset +# +# Env opt-outs: +# KEEP_SESSIONS=1 keep the scratch sessions in each agent's history +# (default: delete them after capture) +# SKIP_COMPACTION=1 capture only the base convo.* fixtures, no compaction +# +# Compaction is forced per harness (see drivers): Claude honors `/compact` +# in continue+print mode; Codex auto-compacts when resumed with a tiny +# `model_context_window`; Pi auto-compacts when `reserveTokens` is raised; +# opencode is summarized via its headless HTTP server. Gemini compresses +# only in memory and persists nothing, so it has no compaction fixture. set -euo pipefail REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" PROMPT_FILE="$REPO_ROOT/docs/agents/feature-elicit.prompt.txt" FIXTURES_ROOT="$REPO_ROOT/test-fixtures" +KEEP_SESSIONS="${KEEP_SESSIONS:-0}" +SKIP_COMPACTION="${SKIP_COMPACTION:-0}" if [[ ! -f "$PROMPT_FILE" ]]; then echo "missing prompt: $PROMPT_FILE" >&2 @@ -34,12 +49,24 @@ SELECTED=("${@:-${ALL_HARNESSES[@]}}") # Fresh scratch dir per harness so they can't see each other's files. SCRATCH_BASE="$(mktemp -d -t toolpath-elicit.XXXXXX)" echo "scratch base: $SCRATCH_BASE" +[[ "$KEEP_SESSIONS" == "1" ]] && echo "KEEP_SESSIONS=1 — scratch sessions will be left in agent histories" +[[ "$SKIP_COMPACTION" == "1" ]] && echo "SKIP_COMPACTION=1 — capturing base fixtures only" echo -# Files-newer-than marker, set just before each harness runs so we can -# diff its session storage afterwards and pick out the newly written -# session file. Only `find -newer` is portable across macOS and Linux -# without ctime tricks. +# Safety net: if the Pi driver patched settings.json and the script dies +# mid-pass, restore it on exit. Format: "::". +PI_SETTINGS_RESTORE="" +on_exit() { + if [[ -n "${PI_SETTINGS_RESTORE:-}" ]]; then + local bak="${PI_SETTINGS_RESTORE%%::*}" dst="${PI_SETTINGS_RESTORE##*::}" + [[ -f "$bak" ]] && cp "$bak" "$dst" + fi +} +trap on_exit EXIT + +# Files-newer-than marker, set just before a step runs so we can diff +# session storage afterwards and pick out the newly written file. Only +# `find -newer` is portable across macOS and Linux without ctime tricks. mark() { local marker="$1" : > "$marker" @@ -47,35 +74,41 @@ mark() { } newest_under() { - local dir="$1" - local pattern="$2" - local marker="$3" - if [[ ! -d "$dir" ]]; then - return 1 - fi - find "$dir" -type f -name "$pattern" -newer "$marker" -print 2>/dev/null \ - | head -1 + local dir="$1" pattern="$2" marker="$3" + [[ -d "$dir" ]] || return 1 + find "$dir" -type f -name "$pattern" -newer "$marker" -print 2>/dev/null | head -1 } -# Tail the last 10 lines of a captured stderr file with a leading indent -# so it's visually nested under the harness's FAIL line. dump_log() { local log="$1" - if [[ -f "$log" ]]; then - sed 's/^/ │ /' < "$log" | tail -10 - fi + [[ -f "$log" ]] && sed 's/^/ │ /' < "$log" | tail -10 +} + +# Remove scratch sessions created during this run, unless KEEP_SESSIONS=1. +# Accepts a harness label followed by paths (files or dirs) to delete. +cleanup_paths() { + local name="$1"; shift + [[ "$KEEP_SESSIONS" == "1" ]] && return 0 + local removed=0 p + for p in "$@"; do + [[ -n "$p" && -e "$p" ]] && rm -rf "$p" && removed=1 + done + [[ "$removed" == "1" ]] && echo "$name: cleaned scratch session(s)" + return 0 } # ── Harness drivers ────────────────────────────────────────────────── # # Each driver: # 1. cd's into a fresh scratch dir -# 2. snapshots its session storage location with a marker file -# 3. invokes its CLI in non-interactive prompt mode -# 4. finds the new session file and copies it into fixtures// +# 2. snapshots its session storage location +# 3. invokes its CLI in non-interactive prompt mode (base fixture) +# 4. forces a compaction and captures convo-compacted.* (unless SKIP_COMPACTION) +# 5. deletes the scratch session(s) from the agent's history (unless KEEP_SESSIONS) # -# Edit the invocation lines if your harness version uses different -# flags; the driver shape stays the same. +# Edit the invocation lines if your harness version uses different flags; +# the driver shape stays the same. Compaction passes are best-effort: a +# failure warns and continues without aborting the base capture. drive_claude() { if ! command -v claude >/dev/null; then @@ -84,27 +117,21 @@ drive_claude() { local scratch="$SCRATCH_BASE/claude"; mkdir -p "$scratch"; cd "$scratch" # Snapshot existing project dirs before the run; whichever appears # afterward is the scratch's project dir. Claude sanitizes the cwd - # path into the dir name (replacing `/`, `.`, and other punctuation - # with `-`), and the exact rule isn't documented; snapshot-diff - # avoids hard-coding it. + # path into the dir name; snapshot-diff avoids hard-coding the rule. local projs_before projs_before="$(ls -1 "$HOME/.claude/projects" 2>/dev/null | sort)" local log="$scratch/.stderr.log" echo "claude: running…" if ! claude -p "$PROMPT" >/dev/null 2> "$log"; then - echo "claude: FAIL (CLI returned non-zero)" - dump_log "$log" - return 1 + echo "claude: FAIL (CLI returned non-zero)"; dump_log "$log"; return 1 fi - local projs_after + local projs_after new_proj projs_after="$(ls -1 "$HOME/.claude/projects" 2>/dev/null | sort)" - local new_proj new_proj="$(comm -13 <(echo "$projs_before") <(echo "$projs_after") | head -1)" if [[ -z "$new_proj" ]]; then echo "claude: FAIL (no new project dir under ~/.claude/projects)"; return 1 fi - local proj_dir="$HOME/.claude/projects/$new_proj" - local session + local proj_dir="$HOME/.claude/projects/$new_proj" session session="$(ls -1t "$proj_dir"/*.jsonl 2>/dev/null | head -1)" if [[ -z "$session" ]]; then echo "claude: FAIL (no session file under $proj_dir)"; return 1 @@ -112,6 +139,25 @@ drive_claude() { mkdir -p "$FIXTURES_ROOT/claude" cp "$session" "$FIXTURES_ROOT/claude/convo.jsonl" echo "claude: OK → test-fixtures/claude/convo.jsonl" + + # ── compaction: `/compact` is honored in continue (-c) + print (-p) + # mode and appends a compact_boundary + synthetic summary to the same + # session file. ── + if [[ "$SKIP_COMPACTION" != "1" ]]; then + if claude -c -p "/compact" /dev/null 2>&1; then + local cfile + cfile="$(grep -l compact_boundary "$proj_dir"/*.jsonl 2>/dev/null | head -1)" + if [[ -n "$cfile" ]]; then + cp "$cfile" "$FIXTURES_ROOT/claude/convo-compacted.jsonl" + echo "claude: OK → test-fixtures/claude/convo-compacted.jsonl" + else + echo "claude: WARN (no compact_boundary after /compact; skipped compacted fixture)" + fi + else + echo "claude: WARN (/compact failed; skipped compacted fixture)" + fi + fi + cleanup_paths claude "$proj_dir" } drive_codex() { @@ -121,16 +167,10 @@ drive_codex() { local scratch="$SCRATCH_BASE/codex"; mkdir -p "$scratch"; cd "$scratch" local marker="$scratch/.marker"; mark "$marker" echo "codex: running…" - # `--cd` pins workdir (without it codex resets to the nearest git ancestor). - # `--skip-git-repo-check` avoids the "trusted directory" gate. - # `-s workspace-write` lets the model write files in cwd. - # `/dev/null 2> "$log"; then - echo "codex: FAIL (CLI returned non-zero)" - dump_log "$log" - return 1 + echo "codex: FAIL (CLI returned non-zero)"; dump_log "$log"; return 1 fi local session session="$(newest_under "$HOME/.codex/sessions" "rollout-*.jsonl" "$marker")" @@ -140,6 +180,29 @@ drive_codex() { mkdir -p "$FIXTURES_ROOT/codex" cp "$session" "$FIXTURES_ROOT/codex/convo.jsonl" echo "codex: OK → test-fixtures/codex/convo.jsonl" + + # ── compaction: resuming with a tiny `model_context_window` makes the + # next turn exceed it and auto-compact. The resume forks a new rollout + # that carries the full history plus a `compacted` item. ── + local compacted="" + if [[ "$SKIP_COMPACTION" != "1" ]]; then + local mk2="$scratch/.marker2"; mark "$mk2" + if codex exec --skip-git-repo-check --cd "$scratch" -s workspace-write \ + -c model_context_window=4000 resume --last "Reply with the single word: ok." \ + /dev/null 2>&1; then + compacted="$(newest_under "$HOME/.codex/sessions" "rollout-*.jsonl" "$mk2")" + if [[ -n "$compacted" ]] && grep -q '"type":"compacted"' "$compacted"; then + cp "$compacted" "$FIXTURES_ROOT/codex/convo-compacted.jsonl" + echo "codex: OK → test-fixtures/codex/convo-compacted.jsonl" + else + echo "codex: WARN (no compacted item after resume; skipped compacted fixture)" + compacted="" + fi + else + echo "codex: WARN (resume failed; skipped compacted fixture)" + fi + fi + cleanup_paths codex "$session" "$compacted" } drive_gemini() { @@ -147,45 +210,37 @@ drive_gemini() { echo "gemini: SKIP (not on PATH)"; return 0 fi local scratch="$SCRATCH_BASE/gemini"; mkdir -p "$scratch"; cd "$scratch" - # Snapshot existing slot dirs before the run; whichever appears - # afterward is the slot for this scratch (gemini disambiguates by - # appending `-N` when a basename collides, so we can't predict it). local slots_before slots_before="$(ls -1 "$HOME/.gemini/tmp" 2>/dev/null | sort)" echo "gemini: running…" local log="$scratch/.stderr.log" if ! gemini --skip-trust --yolo -p "$PROMPT" /dev/null 2> "$log"; then - echo "gemini: FAIL (CLI returned non-zero)" - dump_log "$log" - return 1 + echo "gemini: FAIL (CLI returned non-zero)"; dump_log "$log"; return 1 fi - local slots_after + local slots_after new_slot slots_after="$(ls -1 "$HOME/.gemini/tmp" 2>/dev/null | sort)" - local new_slot new_slot="$(comm -13 <(echo "$slots_before") <(echo "$slots_after") | head -1)" if [[ -z "$new_slot" ]]; then echo "gemini: FAIL (no new slot under ~/.gemini/tmp)"; return 1 fi - local chats_dir="$HOME/.gemini/tmp/$new_slot/chats" - local session + local chats_dir="$HOME/.gemini/tmp/$new_slot/chats" session session="$(ls -1 "$chats_dir"/session-*.jsonl 2>/dev/null | head -1)" - if [[ -z "$session" ]]; then - session="$(ls -1 "$chats_dir"/session-*.json 2>/dev/null | head -1)" - fi + [[ -z "$session" ]] && session="$(ls -1 "$chats_dir"/session-*.json 2>/dev/null | head -1)" if [[ -z "$session" ]]; then echo "gemini: FAIL (no session file under $chats_dir)"; return 1 fi mkdir -p "$FIXTURES_ROOT/gemini" local out_ext="${session##*.}" cp "$session" "$FIXTURES_ROOT/gemini/convo.${out_ext}" - # Sub-agent sibling dir lands next to the main file; copy it too if present. - local stem - stem="$(basename "$session" ".${out_ext}")" + local stem; stem="$(basename "$session" ".${out_ext}")" local sub_dir="$chats_dir/${stem#session-}" - if [[ -d "$sub_dir" ]]; then - cp -r "$sub_dir" "$FIXTURES_ROOT/gemini/" - fi + [[ -d "$sub_dir" ]] && cp -r "$sub_dir" "$FIXTURES_ROOT/gemini/" echo "gemini: OK → test-fixtures/gemini/convo.${out_ext}" + + # Gemini compresses context only in memory and never writes a marker + # to disk, so there is no compaction fixture to capture. + [[ "$SKIP_COMPACTION" != "1" ]] && echo "gemini: (no compaction persisted on disk — none to capture)" + cleanup_paths gemini "$HOME/.gemini/tmp/$new_slot" } drive_pi() { @@ -196,25 +251,19 @@ drive_pi() { local projs_before projs_before="$(ls -1 "$HOME/.pi/agent/sessions" 2>/dev/null | sort)" echo "pi: running…" - # Pi's non-interactive flag varies by version; `-p` is the - # best-guess default. If your version uses something different, - # edit this line. local log="$scratch/.stderr.log" # pi prints provider/auth errors to stdout, not stderr; capture both. if ! pi -p "$PROMPT" "$log" 2>&1; then echo "pi: FAIL (CLI returned non-zero — see docs/agents/feature-elicit.md for the manual workflow)" - dump_log "$log" - return 1 + dump_log "$log"; return 1 fi - local projs_after + local projs_after new_proj projs_after="$(ls -1 "$HOME/.pi/agent/sessions" 2>/dev/null | sort)" - local new_proj new_proj="$(comm -13 <(echo "$projs_before") <(echo "$projs_after") | head -1)" if [[ -z "$new_proj" ]]; then echo "pi: FAIL (no new project dir under ~/.pi/agent/sessions)"; return 1 fi - local proj_dir="$HOME/.pi/agent/sessions/$new_proj" - local session + local proj_dir="$HOME/.pi/agent/sessions/$new_proj" session session="$(ls -1t "$proj_dir"/*.jsonl 2>/dev/null | head -1)" if [[ -z "$session" ]]; then echo "pi: FAIL (no session file under $proj_dir)"; return 1 @@ -222,6 +271,39 @@ drive_pi() { mkdir -p "$FIXTURES_ROOT/pi" cp "$session" "$FIXTURES_ROOT/pi/convo.jsonl" echo "pi: OK → test-fixtures/pi/convo.jsonl" + + # ── compaction: Pi auto-compacts when context exceeds + # `contextWindow - compaction.reserveTokens`. Temporarily raise + # reserveTokens so the next turn compacts; restore settings after + # (the EXIT trap is a safety net if we die mid-pass). ── + if [[ "$SKIP_COMPACTION" != "1" ]]; then + local settings="$HOME/.pi/agent/settings.json" bak + if [[ -f "$settings" ]]; then + bak="$(mktemp)"; cp "$settings" "$bak" + PI_SETTINGS_RESTORE="$bak::$settings" + python3 - "$settings" <<'PY' || true +import json,sys +p=sys.argv[1] +try: d=json.load(open(p)) +except Exception: d={} +d["compaction"]={"enabled":True,"reserveTokens":10000000,"keepRecentTokens":400} +json.dump(d,open(p,"w"),indent=2) +PY + pi -c -p "Now print the single word: done." /dev/null 2>&1 || true + cp "$bak" "$settings"; rm -f "$bak"; PI_SETTINGS_RESTORE="" # restore now + local cfile + cfile="$(grep -l '"type":"compaction"' "$proj_dir"/*.jsonl 2>/dev/null | head -1)" + if [[ -n "$cfile" ]]; then + cp "$cfile" "$FIXTURES_ROOT/pi/convo-compacted.jsonl" + echo "pi: OK → test-fixtures/pi/convo-compacted.jsonl" + else + echo "pi: WARN (no compaction entry after forced auto-compact; skipped compacted fixture)" + fi + else + echo "pi: WARN (no settings.json to tune; skipped compacted fixture)" + fi + fi + cleanup_paths pi "$proj_dir" } drive_opencode() { @@ -229,29 +311,77 @@ drive_opencode() { echo "opencode: SKIP (not on PATH)"; return 0 fi local scratch="$SCRATCH_BASE/opencode"; mkdir -p "$scratch"; cd "$scratch" + local db="$HOME/.local/share/opencode/opencode.db" echo "opencode: running…" - # `--format json` makes opencode emit structured events with explicit - # `sessionID` fields, easy to extract with grep. Sessions live in - # SQLite, so snapshot-diff doesn't apply; we get the id, then dump - # via `opencode export ` which writes JSON to stdout (with a - # chatty header on stderr we discard). - local log="$scratch/.stderr.log" - local run_out + local log="$scratch/.stderr.log" run_out if ! run_out="$(opencode run --format json "$PROMPT" "$log")"; then - echo "opencode: FAIL (opencode run returned non-zero)" - dump_log "$log" - return 1 + echo "opencode: FAIL (opencode run returned non-zero)"; dump_log "$log"; return 1 fi local session_id session_id="$(printf '%s\n' "$run_out" | grep -oE 'ses_[A-Za-z0-9]+' | head -1)" + # Fallback: the newest session row in the DB (run output format varies by version). + [[ -z "$session_id" ]] && session_id="$(sqlite3 "$db" "select id from session order by time_created desc limit 1" 2>/dev/null)" if [[ -z "$session_id" ]]; then - echo "opencode: FAIL (no ses_… id found in run output)"; return 1 + echo "opencode: FAIL (could not determine ses_… id)"; return 1 fi mkdir -p "$FIXTURES_ROOT/opencode" if ! opencode export "$session_id" 2>/dev/null > "$FIXTURES_ROOT/opencode/convo.json"; then echo "opencode: FAIL (opencode export $session_id failed)"; return 1 fi echo "opencode: OK → test-fixtures/opencode/convo.json" + + # ── compaction: opencode's headless server exposes the same summarize + # the TUI `/compact` calls. Start a server, POST summarize with the + # provider/model the session used, then re-export. ── + if [[ "$SKIP_COMPACTION" != "1" ]]; then + local pm prov modl + pm="$(sqlite3 "$db" "select data from message where session_id='$session_id' order by time_created desc limit 8" 2>/dev/null | python3 -c " +import sys,json +prov=modl='' +for line in sys.stdin: + line=line.strip() + if not line: continue + try: d=json.loads(line) + except Exception: continue + def w(o): + global prov,modl + if isinstance(o,dict): + for k,v in o.items(): + if k=='providerID' and isinstance(v,str) and not prov: prov=v + if k=='modelID' and isinstance(v,str) and not modl: modl=v + w(v) + elif isinstance(o,list): + [w(x) for x in o] + w(d) +print(prov+'|'+modl) +" 2>/dev/null)" + prov="${pm%%|*}"; modl="${pm##*|}" + if [[ -n "$prov" && -n "$modl" ]]; then + local port=7771 + opencode serve --port "$port" >/dev/null 2>&1 & + local svpid=$! + local up=0 + for _ in $(seq 1 40); do curl -s "http://127.0.0.1:$port/app" >/dev/null 2>&1 && { up=1; break; }; sleep 0.5; done + if [[ "$up" == "1" ]]; then + curl -s -X POST "http://127.0.0.1:$port/session/$session_id/summarize" \ + -H 'content-type: application/json' \ + -d "{\"providerID\":\"$prov\",\"modelID\":\"$modl\"}" >/dev/null 2>&1 || true + sleep 8 + fi + kill "$svpid" 2>/dev/null || true + if opencode export "$session_id" 2>/dev/null | grep -qE '"type": *"compaction"'; then + opencode export "$session_id" 2>/dev/null > "$FIXTURES_ROOT/opencode/convo-compacted.json" + echo "opencode: OK → test-fixtures/opencode/convo-compacted.json" + else + echo "opencode: WARN (summarize produced no compaction part; skipped compacted fixture)" + fi + else + echo "opencode: WARN (could not resolve provider/model; skipped compacted fixture)" + fi + fi + if [[ "$KEEP_SESSIONS" != "1" ]]; then + opencode session delete "$session_id" >/dev/null 2>&1 && echo "opencode: cleaned scratch session(s)" + fi } # ── Driver dispatch ────────────────────────────────────────────────── diff --git a/site/_data/crates.json b/site/_data/crates.json index abf8029..fbb0030 100644 --- a/site/_data/crates.json +++ b/site/_data/crates.json @@ -1,7 +1,7 @@ [ { "name": "toolpath", - "version": "0.7.0", + "version": "0.8.0", "description": "Core types, builders, and query API", "docs": "https://docs.rs/toolpath", "crate": "https://crates.io/crates/toolpath", @@ -9,7 +9,7 @@ }, { "name": "toolpath-convo", - "version": "0.11.0", + "version": "0.12.0", "description": "Provider-agnostic conversation types, traits, and Toolpath-Path derivation", "docs": "https://docs.rs/toolpath-convo", "crate": "https://crates.io/crates/toolpath-convo", @@ -33,7 +33,7 @@ }, { "name": "toolpath-claude", - "version": "0.12.0", + "version": "0.13.0", "description": "Derive from Claude conversation logs", "docs": "https://docs.rs/toolpath-claude", "crate": "https://crates.io/crates/toolpath-claude", @@ -41,7 +41,7 @@ }, { "name": "toolpath-gemini", - "version": "0.6.0", + "version": "0.7.0", "description": "Derive from Gemini CLI conversation logs", "docs": "https://docs.rs/toolpath-gemini", "crate": "https://crates.io/crates/toolpath-gemini", @@ -49,7 +49,7 @@ }, { "name": "toolpath-codex", - "version": "0.6.0", + "version": "0.7.0", "description": "Derive from Codex CLI rollout files", "docs": "https://docs.rs/toolpath-codex", "crate": "https://crates.io/crates/toolpath-codex", @@ -57,7 +57,7 @@ }, { "name": "toolpath-opencode", - "version": "0.5.0", + "version": "0.6.0", "description": "Derive from opencode SQLite databases", "docs": "https://docs.rs/toolpath-opencode", "crate": "https://crates.io/crates/toolpath-opencode", @@ -65,7 +65,7 @@ }, { "name": "toolpath-pi", - "version": "0.6.0", + "version": "0.7.0", "description": "Derive Toolpath provenance documents from Pi (pi.dev) agent session logs", "docs": "https://docs.rs/toolpath-pi", "crate": "https://crates.io/crates/toolpath-pi", @@ -73,7 +73,7 @@ }, { "name": "toolpath-cursor", - "version": "0.2.0", + "version": "0.3.0", "description": "Derive Toolpath provenance documents from Cursor (IDE) composers", "docs": "https://docs.rs/toolpath-cursor", "crate": "https://crates.io/crates/toolpath-cursor", @@ -105,7 +105,7 @@ }, { "name": "path-cli", - "version": "0.14.0", + "version": "0.15.0", "description": "Unified CLI (binary: path)", "docs": "https://docs.rs/path-cli", "crate": "https://crates.io/crates/path-cli", @@ -113,7 +113,7 @@ }, { "name": "toolpath-cli", - "version": "0.14.0", + "version": "0.15.0", "description": "Deprecated alias for path-cli", "docs": "https://docs.rs/toolpath-cli", "crate": "https://crates.io/crates/toolpath-cli", diff --git a/site/kinds/agent-coding-session/index.md b/site/kinds/agent-coding-session/index.md index 5bf48e6..6ced562 100644 --- a/site/kinds/agent-coding-session/index.md +++ b/site/kinds/agent-coding-session/index.md @@ -12,5 +12,6 @@ Documents reference a specific version URI. They do not depend on this landing p ## Versions -- [**v1.1.0**](/kinds/agent-coding-session/v1.1.0/): `https://toolpath.net/kinds/agent-coding-session/v1.1.0` _(current)_ — adds `group_id` and specifies message-level token accounting (a message's usage appears on exactly one step, so per-step sums equal session totals) +- [**v1.2.0**](/kinds/agent-coding-session/v1.2.0/): `https://toolpath.net/kinds/agent-coding-session/v1.2.0` _(current)_ — adds the `conversation.compact` compaction-boundary step type +- [**v1.1.0**](/kinds/agent-coding-session/v1.1.0/): `https://toolpath.net/kinds/agent-coding-session/v1.1.0` — adds `group_id` and specifies message-level token accounting (a message's usage appears on exactly one step, so per-step sums equal session totals) - [**v1.0.0**](/kinds/agent-coding-session/v1.0.0/): `https://toolpath.net/kinds/agent-coding-session/v1.0.0` — superseded; see its erratum on token accounting diff --git a/site/kinds/agent-coding-session/v1.2.0/index.md b/site/kinds/agent-coding-session/v1.2.0/index.md new file mode 100644 index 0000000..5db28eb --- /dev/null +++ b/site/kinds/agent-coding-session/v1.2.0/index.md @@ -0,0 +1,173 @@ +--- +layout: base.njk +title: "Kind: agent-coding-session v1.2.0" +permalink: /kinds/agent-coding-session/v1.2.0/ +--- + +# Kind: `agent-coding-session` v1.2.0 + +
+
URI
+
https://toolpath.net/kinds/agent-coding-session/v1.2.0
+
Schema
+
schema.json
+
+ +A Toolpath path whose `meta.kind` is this URI records an AI coding conversation. It is an ordinary path with the extra structure described here. `head`-ancestry, dead ends, signatures, and `base` all behave as in the [base format](/format/). + +Every such path comes from one place: the shared `ConversationView → Path` derivation in `toolpath-convo` (`derive_path`), which the provider crates (`toolpath-claude`, `toolpath-gemini`, `toolpath-codex`, `toolpath-opencode`, `toolpath-cursor`, `toolpath-pi`) all call. The field shapes below are therefore exact. The only producer-specific parts are the contents of a tool's `input`, the diff text in a change's `raw`, and the value (not the meaning) of `group_id`. + +Constraints apply by structural `type`, not by artifact key: a `change` entry is checked only when its `structural.type` is one named here, and extra properties never make a path invalid. [`schema.json`](./schema.json) encodes the rules; apply it alongside the base schema. The URI is immutable. Later revisions ship under a new version URI. + +**Changed from [v1.1.0](/kinds/agent-coding-session/v1.1.0/):** adds the [**compaction boundary**](#compaction-boundary) step type (`conversation.compact`). v1.1.0's `group_id` and [group accounting](#group-accounting) carry forward unchanged. v1.2.0 documents are structurally valid v1.1.0 documents; the new version exists so consumers can rely on the compaction step type. + +## The turn payload + +One entry in a turn's `change` map has `structural.type` of `"conversation.append"`. Find it by that type: the artifact key is producer-specific, formed as `://` from the harness in `meta.source` (e.g. `claude-code://…`, `gemini-cli://…`, `codex://…`, `opencode://…`, `cursor://…`, `pi://…`). + +Its `structural` object always carries: + +| Field | Type | Meaning | +| ------ | ------ | ------------------------------------------------------------------ | +| `type` | string | the literal `"conversation.append"` | +| `role` | string | `"user"`, `"assistant"`, `"system"`, or a producer-specific string | +| `text` | string | the visible prose; present even when empty (`""`) | + +It may also carry any of the following, present only when the turn has them: + +| Field | Type | Meaning | +| ------------- | ------ | ------------------------------------------------------------- | +| `thinking` | string | the model's reasoning text | +| `group_id` | string | groups the steps derived from one source accounting unit (see below) | +| `tool_uses` | array | tools the agent invoked (shape below) | +| `token_usage` | object | the group's token counts (shape and rule below) | +| `attributed_token_usage` | object | this step's own attributed spend, when known (see below) | +| `stop_reason` | string | why the model stopped (`end_turn`, `tool_use`, …) | +| `delegations` | array | sub-agent work spawned from this turn (shape below) | +| `environment` | object | working environment at this turn (shape below) | + +The model identifier is not on the change. It lives in `step.actor` (`agent:`) and `meta.actors`. There is no provider-specific blob: every field the derivation captures is one of those listed above. + +### `group_id` + +The provider's identifier for the **source accounting unit** these steps were derived from — Claude Code's `message.id` (`msg_…`) for one split message, Codex's round `turn_id` for one round (which may itself contain several messages). It is a **grouping key, not a step identifier**: when a producer derives several steps from one accounting unit (Claude Code writes one JSONL line per content block; a Codex round emits a commentary turn plus a final turn), every sibling step carries the same `group_id`. A step without a `group_id` is its own group of one. The stored value is the provider's verbatim id; only its *meaning* (which unit it names) is provider-specific. + +### Group accounting + +How `token_usage` on steps relates to the source's accounting units: + +1. `token_usage` records a group's spend — a **per-group amount, never a cumulative session counter**. +2. Within a run of consecutive steps sharing a `group_id` (document order), the run's **last step carries the group's total `token_usage`, verbatim from the source**. In this version, the run's other steps carry none. +3. A step without a `group_id` is its own group and carries its own `token_usage` (when the source records one). + +Consequence: **summing `token_usage` over a v1.2.0 path's steps yields the session totals.** Consumers need no dedup heuristics. (JSON Schema cannot express the once-per-run rule, so it is normative prose, enforced by producer test suites.) + +`token_usage` has **one meaning everywhere it appears: the total for a group**. A step without a `group_id` is a one-step group, so its `token_usage` is that group's total (which is also its own spend — the two coincide for a group of one). Within a multi-step group, the total sits on the final step. Interpreting a value never requires reading the rest of its group: the key tells you it is a total, and `group_id` on the same payload tells you which group it totals. Per-step spend, when the source has it, rides a separate [`attributed_token_usage`](#per-step-attribution-attributed_token_usage) key — never `token_usage`. When a source format offers both a group total and a finer breakdown (Claude's `usage.iterations`, opencode's per-part `step-finish` tokens), `token_usage` carries the total; the breakdown is subordinate detail and does not ride `token_usage`. + +### Per-step attribution: `attributed_token_usage` + +Some sources expose, per step, the spend attributable to that step alone — distinct from the group total. Where a producer has it, the step carries an **`attributed_token_usage`** object (same shape as [`token_usage`](#token_usage)) holding *this step's own share*. It is **optional and orthogonal to `token_usage`**: whether a number is a group total or a step share is structural — the key it sits under — never positional. This is the rule that lets per-step accounting be added by any producer at any time without a new kind version. + +How it relates to the group total: + +- Within a `group_id` group, `Σ attributed_token_usage` over the group's steps is the group's attributed spend. The **unattributed remainder** — anything the source could not pin to a step — is *computed* by a consumer as `group's token_usage − Σ group's attributed_token_usage`; it is never recorded, so stored values stay verbatim source observations and source inconsistencies stay visible. +- For a group where the source attributes everything (e.g. Codex, where each step is a separate API call and the per-call delta is reported directly), the remainder is zero and `Σ attributed_token_usage == token_usage`. +- A group with no per-step data carries no `attributed_token_usage` at all — only the group total. Producers must not fabricate a split. + +A producer populates `attributed_token_usage` only when the source genuinely reports per-step spend. Among current producers, **Codex does** (its `token_count` events carry a per-call delta). **Claude does not**: its per-content-block `usage` values are cumulative streaming snapshots stamped at flush time, not per-block costs, so deriving a split from them would be fabrication — Claude-derived steps carry the group total only. + +`Σ token_usage` over a path's steps is unaffected by `attributed_token_usage` (they are separate keys), so the session-total guarantee above always holds. A consumer wanting per-step cost reads `attributed_token_usage` where present and falls back to the group total otherwise. + +### `tool_uses` + +Each element is an object: + +| Field | Type | Notes | +| ---------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| `id` | string | provider-assigned invocation ID | +| `name` | string | provider tool name (`Read`, `Bash`, `edit`, …) | +| `input` | any | tool arguments; shape is producer-specific | +| `category` | string \| null | Toolpath's classification: `file_read`, `file_write`, `file_search`, `shell`, `network`, `delegation`, or `null` when unrecognized | +| `result` | object | `{ "content": string, "is_error": boolean }`, when the result landed in the same turn | + +`id`, `name`, `input`, and `category` are always present (`category` may be `null`); `result` is optional. + +### `token_usage` + +| Field | Type | Notes | +| -------------------- | --------------- | ------------------------------- | +| `input_tokens` | integer \| null | always present | +| `output_tokens` | integer \| null | always present | +| `cache_read_tokens` | integer | only when the source records it | +| `cache_write_tokens` | integer | only when the source records it | +| `breakdowns` | object | only when the source itemizes a class (see below) | + +Values follow the [group accounting](#group-accounting) rule above. + +`breakdowns` is an **optional, informational** decomposition of a top-level class into named sub-classes. It is keyed by the class being broken down (e.g. `"output"`); each value is a map of sub-class → tokens (e.g. `{ "output": { "reasoning": 450 } }`). Breakdowns are **never summed into any total** — the parent class already counts these tokens; a breakdown only says *how* that class divides. Invariant: **`Σ(inner) ≤` the parent class's value**. The field is omitted entirely when empty. The same shape and rule apply on `attributed_token_usage`. Among current producers, Gemini, OpenCode, and Codex record `output → { reasoning }` (their reasoning/thoughts tokens are part of `output_tokens`); Claude records none (its JSONL `usage` does not itemize thinking tokens). + +### `environment` + +`{ "working_dir"?: string, "vcs_branch"?: string, "vcs_revision"?: string }`; every field optional. + +### `delegations` + +Each element is `{ "agent_id": string, "prompt": string, "turns"?: array, "result"?: string }`. `turns` holds the sub-agent's own turns when the producer inlines them. + +## File changes + +When a turn writes files, its step carries sibling `change` entries keyed by file path, each with `structural.type` of `"file.write"`. The unified diff, when available, is on the change's `raw`, not inside `structural`. The `structural` object holds, all optional: + +| Field | Meaning | +| ------------------ | ------------------------------------------------------------------ | +| `tool_id` | the `tool_uses[].id` that produced the mutation, when attributable | +| `tool` | that tool's `name` | +| `operation` | `"add"`, `"update"`, `"delete"`, or a producer-specific tag | +| `before` / `after` | file contents before / after, when known | +| `rename_to` | the new path, for a rename | + +## Compaction boundary + +When a harness compacts its context, the derivation emits one step whose `change` entry has `structural.type` of `"conversation.compact"`. It uses the same `://` artifact key as the turn payload. The step sits between the turns it separates: the turns after the boundary parent on it, so the `head`-ancestry walk crosses the compaction in order. + +Only `type` is always present. Every other field appears only when the source records it: + +| Field | Type | Meaning | +| ------------ | ---------------- | -------------------------------------------------------------------------------- | +| `type` | string | the literal `"conversation.compact"` | +| `trigger` | string | `"auto"` (context overflow) or `"manual"` (user-invoked), when known | +| `summary` | string | the compaction summary text the harness produced, when one was recorded | +| `pre_tokens` | number | the context token count immediately before the boundary, when known | +| `kept` | array | ids of the prior turns that survive verbatim into the post-compaction window (may be non-contiguous; empty = wholesale) | + +A compaction step has no `text`, `role`, or `tool_uses` — it is not a turn. Consumers that only care about the transcript can skip it; consumers reconstructing the source format use it to place the boundary. The `kept` ids are the harness-agnostic payload: each harness's projector renders that set in its own form (Claude re-emits those turns on-chain before the boundary; opencode/Pi anchor a kept tail at the earliest id; Codex keeps none). + +## Non-turn entries + +Entries that aren't turns (attachments, preamble lines, snapshots, hook results) become steps with `structural.type` of `"conversation.event"`, carrying `entry_type` and sometimes `event_source_id` plus the producer's event data. They exist so a document round-trips back to the source format. They are not part of the transcript. + +## Actors + +`step.actor` follows the `type:name` convention, assigned by role: + +| Actor | Turn | +| ----------------- | ---------------------------------------------------------------------------------------------- | +| `human:user` | a user message | +| `agent:` | a model reply, named by the recorded model, or `agent:unknown` when none was recorded | +| `tool:` | a system turn (session init, system prompt), a compaction boundary, any other producer role, or a non-turn event step | + +`meta.actors` defines each actor the steps reference; `agent:` entries carry `provider` and `model`. A turn's original role is always in its `role` field, so collapsing system and other roles onto `tool:` loses nothing. Walk steps in `head`-ancestry order for the linear transcript. + +## Path metadata + +| Field | Meaning | +| -------------------- | -------------------------------------------------------------------------------- | +| `meta.kind` | this URI | +| `meta.source` | the producing harness: `claude-code`, `gemini-cli`, `codex`, `opencode`, `cursor`, or `pi` | +| `meta.title` | session title | +| `meta.actors` | the actor definitions the steps reference | +| `meta.files_changed` | file paths touched across the session | +| `meta.vcs_remote` | repository URL, when known | +| `meta.producer` | `{ "name": string, "version"?: string }`, the software that produced the session | + +`files_changed`, `vcs_remote`, and `producer` sit directly under `meta` (they ride `PathMeta`'s flattened `extra`), not under a nested `meta.extra`. diff --git a/site/kinds/agent-coding-session/v1.2.0/schema.json b/site/kinds/agent-coding-session/v1.2.0/schema.json new file mode 120000 index 0000000..4282dd2 --- /dev/null +++ b/site/kinds/agent-coding-session/v1.2.0/schema.json @@ -0,0 +1 @@ +../../../../crates/path-cli/kinds/agent-coding-session/v1.2.0/schema.json \ No newline at end of file diff --git a/site/kinds/index.md b/site/kinds/index.md index 91164e5..1270708 100644 --- a/site/kinds/index.md +++ b/site/kinds/index.md @@ -14,4 +14,4 @@ Kind URIs are immutable: revisions ship at a new version URI, and old URIs keep | Kind | Current URI | Spec | | ------------------------------------------------------ | -------------------------------------------------------- | --------------------------------------------- | -| [`agent-coding-session`](/kinds/agent-coding-session/) | `https://toolpath.net/kinds/agent-coding-session/v1.1.0` | [v1.1.0](/kinds/agent-coding-session/v1.1.0/) | +| [`agent-coding-session`](/kinds/agent-coding-session/) | `https://toolpath.net/kinds/agent-coding-session/v1.2.0` | [v1.2.0](/kinds/agent-coding-session/v1.2.0/), [v1.1.0](/kinds/agent-coding-session/v1.1.0/), [v1.0.0](/kinds/agent-coding-session/v1.0.0/) | diff --git a/test-fixtures/claude/convo-compacted.jsonl b/test-fixtures/claude/convo-compacted.jsonl new file mode 100644 index 0000000..556da1a --- /dev/null +++ b/test-fixtures/claude/convo-compacted.jsonl @@ -0,0 +1,71 @@ +{"type":"queue-operation","operation":"enqueue","timestamp":"2026-06-10T18:40:50.720Z","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","content":"You're going to walk through a small set of tasks designed to exercise every common tool category. Stay inside the current working directory — do not touch anything outside it. Take the steps in order. Briefly think out loud before each tool call so the reasoning is captured. Tool-selection rule: if a step mentions a dedicated tool *and* that tool is available to you, use it; otherwise, use whatever shell-based equivalent you have (e.g. `ls`, `cat`, `sed`, `find`, `grep`). Don't skip steps because a dedicated tool is missing.\n\n1. List the files in the current directory (prefer a dedicated list/directory tool; `bash ls` is fine).\n2. Using a dedicated write/create-file tool if available (otherwise a shell heredoc), write `notes.md` with exactly these two lines:\n scratch — feature elicitation\n this file will be edited and searched.\n3. Read `notes.md` back using a dedicated read-file tool (preferred) or `bash cat`. Confirm both lines are present.\n4. Using a dedicated edit/replace/string-replace tool if available (otherwise `sed -i` or rewrite via the write tool), change the word \"scratch\" to \"fixture\" in `notes.md`.\n5. Find every file in the current directory whose name starts with \"note\". Prefer a dedicated glob/find-files tool; otherwise `bash find . -maxdepth 1 -name 'note*'`.\n6. Search the contents of files in the current directory for the word \"fixture\". Prefer a dedicated grep/content-search tool; otherwise `bash grep -r fixture .`. It should appear in `notes.md`.\n7. Read a file that does not exist: `does-not-exist.txt`. Use a read-file tool (or `cat does-not-exist.txt`) — the call MUST fail and the failure MUST be reported as a tool result, not silently swallowed. Acknowledge the error in one sentence and move on.\n8. Write a shell script `count.sh` that prints the numbers 1 through 5 on separate lines (one `echo` per line is fine). Run it with `bash count.sh` and confirm the output is exactly five lines.\n9. If you have a sub-agent / task-dispatch tool available (Claude `Task`, Gemini sub-agents, Codex sub-tasks, opencode subtask, etc.), dispatch a sub-agent with this exact instruction: \"Count the words in `notes.md` and report the number.\" Wait for the sub-agent's reply and quote its result in one sentence. If no such tool is available, skip this step and say so explicitly.\n10. Briefly summarize in 2–3 sentences what you did. Stop without asking for further work."} +{"type":"queue-operation","operation":"dequeue","timestamp":"2026-06-10T18:40:50.721Z","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"parentUuid":null,"isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":"You're going to walk through a small set of tasks designed to exercise every common tool category. Stay inside the current working directory — do not touch anything outside it. Take the steps in order. Briefly think out loud before each tool call so the reasoning is captured. Tool-selection rule: if a step mentions a dedicated tool *and* that tool is available to you, use it; otherwise, use whatever shell-based equivalent you have (e.g. `ls`, `cat`, `sed`, `find`, `grep`). Don't skip steps because a dedicated tool is missing.\n\n1. List the files in the current directory (prefer a dedicated list/directory tool; `bash ls` is fine).\n2. Using a dedicated write/create-file tool if available (otherwise a shell heredoc), write `notes.md` with exactly these two lines:\n scratch — feature elicitation\n this file will be edited and searched.\n3. Read `notes.md` back using a dedicated read-file tool (preferred) or `bash cat`. Confirm both lines are present.\n4. Using a dedicated edit/replace/string-replace tool if available (otherwise `sed -i` or rewrite via the write tool), change the word \"scratch\" to \"fixture\" in `notes.md`.\n5. Find every file in the current directory whose name starts with \"note\". Prefer a dedicated glob/find-files tool; otherwise `bash find . -maxdepth 1 -name 'note*'`.\n6. Search the contents of files in the current directory for the word \"fixture\". Prefer a dedicated grep/content-search tool; otherwise `bash grep -r fixture .`. It should appear in `notes.md`.\n7. Read a file that does not exist: `does-not-exist.txt`. Use a read-file tool (or `cat does-not-exist.txt`) — the call MUST fail and the failure MUST be reported as a tool result, not silently swallowed. Acknowledge the error in one sentence and move on.\n8. Write a shell script `count.sh` that prints the numbers 1 through 5 on separate lines (one `echo` per line is fine). Run it with `bash count.sh` and confirm the output is exactly five lines.\n9. If you have a sub-agent / task-dispatch tool available (Claude `Task`, Gemini sub-agents, Codex sub-tasks, opencode subtask, etc.), dispatch a sub-agent with this exact instruction: \"Count the words in `notes.md` and report the number.\" Wait for the sub-agent's reply and quote its result in one sentence. If no such tool is available, skip this step and say so explicitly.\n10. Briefly summarize in 2–3 sentences what you did. Stop without asking for further work."},"uuid":"f17f582a-49c2-48ab-93b6-a76b2d2c9702","timestamp":"2026-06-10T18:40:50.731Z","permissionMode":"auto","promptSource":"sdk","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"f17f582a-49c2-48ab-93b6-a76b2d2c9702","isSidechain":false,"attachment":{"type":"deferred_tools_delta","addedNames":["CronCreate","CronDelete","CronList","EnterPlanMode","EnterWorktree","ExitPlanMode","ExitWorktree","LSP","Monitor","NotebookEdit","PushNotification","RemoteTrigger","TaskCreate","TaskGet","TaskList","TaskOutput","TaskStop","TaskUpdate","WebFetch","WebSearch"],"addedLines":["CronCreate","CronDelete","CronList","EnterPlanMode","EnterWorktree","ExitPlanMode","ExitWorktree","LSP","Monitor","NotebookEdit","PushNotification","RemoteTrigger","TaskCreate","TaskGet","TaskList","TaskOutput","TaskStop","TaskUpdate","WebFetch","WebSearch"],"removedNames":[],"readdedNames":[],"pendingMcpServers":[]},"type":"attachment","uuid":"0da34f94-916d-4d39-adf2-f1ec81ccc577","timestamp":"2026-06-10T18:40:50.731Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"0da34f94-916d-4d39-adf2-f1ec81ccc577","isSidechain":false,"attachment":{"type":"agent_listing_delta","addedTypes":["claude","Explore","general-purpose","Plan","statusline-setup"],"addedLines":["- claude: Catch-all for any task that doesn't fit a more specific agent. FleetView's default when no agent name is typed. (Tools: *)","- Explore: Read-only search agent for broad fan-out searches — when answering means sweeping many files, directories, or naming conventions and you only need the conclusion, not the file dumps. It reads excerpts rather than whole files, so it locates code; it doesn't review or audit it. Specify search breadth: \"medium\" for moderate exploration, \"very thorough\" for multiple locations and naming conventions. (Tools: All tools except Agent, ExitPlanMode, Edit, Write, NotebookEdit)","- general-purpose: General-purpose agent for researching complex questions, searching for code, and executing multi-step tasks. When you are searching for a keyword or file and are not confident that you will find the right match in the first few tries use this agent to perform the search for you. (Tools: *)","- Plan: Software architect agent for designing implementation plans. Use this when you need to plan the implementation strategy for a task. Returns step-by-step plans, identifies critical files, and considers architectural trade-offs. (Tools: All tools except Agent, ExitPlanMode, Edit, Write, NotebookEdit)","- statusline-setup: Use this agent to configure the user's Claude Code status line setting. (Tools: Read, Edit)"],"removedTypes":[],"isInitial":true,"showConcurrencyNote":true},"type":"attachment","uuid":"ac803354-8520-4fad-8b93-6acdc5e5c68a","timestamp":"2026-06-10T18:40:50.731Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"ac803354-8520-4fad-8b93-6acdc5e5c68a","isSidechain":false,"attachment":{"type":"skill_listing","content":"- deep-research: Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report. - When the user wants a deep, multi-source, fact-checked research report on any topic. BEFORE invoking, check if the question is specific enough to research directly — if underspecified (e.g., \"what car to buy\" without budget/use-case/region), ask 2-3 clarifying questions to narrow scope. Then pass the refined question as args, weaving the answers in.\n- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors (\"from now on when X\", \"each time X\", \"whenever X\", \"before/after X\") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions (\"allow X\", \"add permission\", \"move permission to\"), env vars (\"set X=Y\"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: \"allow npm commands\", \"add bq permission to global settings\", \"move permission to user settings\", \"set DEBUG=true\", \"when claude stops show X\". For simple settings like theme/model, suggest the /config command.\n- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: \"rebind ctrl+s\", \"add a chord shortcut\", \"change the submit key\", \"customize keybindings\".\n- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.\n- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.\n- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.\n- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.\n- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. \"check the deploy every 5 minutes\", \"keep running /babysit-prs\"). Do NOT invoke for one-off tasks.\n- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run (\"run this once at 3pm\", \"remind me to check X tomorrow\").\n- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.\nTRIGGER — read BEFORE opening the target file; don't skip because it \"looks like a one-liner\" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).\nSKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).\n- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).\n- init: Initialize a new CLAUDE.md file with codebase documentation\n- review: Review a pull request\n- security-review: Complete a security review of the pending changes on the current branch","skillCount":14,"isInitial":true,"names":["deep-research","update-config","keybindings-help","verify","code-review","simplify","fewer-permission-prompts","loop","schedule","claude-api","run","init","review","security-review"]},"type":"attachment","uuid":"5fcf4cd0-01fa-4214-8bb4-04bc14897100","timestamp":"2026-06-10T18:40:50.731Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"type":"ai-title","aiTitle":"Execute coding tool exercise tasks","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"parentUuid":"5fcf4cd0-01fa-4214-8bb4-04bc14897100","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01WiAWgtEy3NtR8tZpGumwRT","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"Er0GCmMIDhgCKkBrDOYUfuBCYRfYI41up6GfL4WJ8bmAk0z6eLmbmXmGSOnPNShVJyMESB9k9Wiwc9rkLd4585QTPHBUZ40n9+XgMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDA8AJ5Jot2HJVypRoRoMfootb7bRZikEoUscIjCMi5a7AdanxQHz/K/qFvShSGpdgY3Xhu9QjlE+GTndWYrBVeKH+iMMgLWTkf2pb8gqhwV9IcjA3XgMjn7P+gQIUf6yIrlpMaDKeDrdQInvE3KuR+rwfo+8FMjOcMRHt9B3dWCPPbphfbh9X7nCsMCeH7+gSBlQyatY+fpyvRBBUaXz9kACooC5DxHLvGC3TAwQGlrNK+UpxOty2pY7gYDy5f9DOEX8/qguHeR/RzUaqjc3j6iQNC4z6K3POkouKBHAWIjz19b1vAHAJpTDKNIyb85ABSoZYSaOXs82asdGVBqHS3wXVjXbeh8x3FVDO3c/d0q9c4H6biUFspNBxY/C563DZvSKFozkDp9HUgZ8AJZWscavM1RO+9S8QU7eUNt7MPN9MYTRROVeTSmUbIA6U/O+QgPzy0ha9V/LCCpKGgQ1bTbVUIpz7Y+/xB91h9cwB8BcGLtBYT0+CU007RTHYscZQ1Kuc0u0qTf/zHp7EOTx+bwWZYGEQtnDtmHArd2R7e4/26tvARx4b7mL56WTlwln0HARMSGR+qQtKO8Wq+r4fuaCMCghVqStZ3gbeC822YMNyv0DT3BnrBzPYXp3lfoVsgh/RLN7oeOhf2ZJoGO8/cXukTRD804mOMDrUb9hozckKLoCSLwOeVA+VsrzPDnQdTDRGNiwH0xwoB2DeubzdG+Vqv1hsPnconK62qNFqDYSBDHR+ZORlrwY7Ooo2W63rC6KbE9J4q4N1qsnRDvMuRsJ2tjakd+LprXXjFi7iKXNP8AIGvzlkA6qJyhpmHUrsCDYj2F6SjewbsmfYnKwrbTxWZ/AXZilTuM4N4UHS1SXo4ylYqsPrmDqmYx+ut2azxVPvJZvwnIKcfnWaDQprKWxKdLsbeepGX28QKLNKmpkgTkX1yrASQRDThqqGXohx2U2juFzgBgB"}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2486,"cache_creation_input_tokens":3217,"cache_read_input_tokens":15334,"output_tokens":368,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":3217,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2486,"output_tokens":368,"cache_read_input_tokens":15334,"cache_creation_input_tokens":3217,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":3217},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4x18Wh753z3Yu2MazU","type":"assistant","uuid":"e924d226-863e-4dd3-b45e-d9be12e0e995","timestamp":"2026-06-10T18:40:56.773Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"e924d226-863e-4dd3-b45e-d9be12e0e995","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01WiAWgtEy3NtR8tZpGumwRT","type":"message","role":"assistant","content":[{"type":"text","text":"I'll work through these in order, thinking out loud before each tool call.\n\n**Step 1:** List the current directory. I don't have a dedicated list tool at the top level (only Read for files), so I'll use `bash ls`."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2486,"cache_creation_input_tokens":3217,"cache_read_input_tokens":15334,"output_tokens":368,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":3217,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2486,"output_tokens":368,"cache_read_input_tokens":15334,"cache_creation_input_tokens":3217,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":3217},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4x18Wh753z3Yu2MazU","type":"assistant","uuid":"286849be-62ea-4f80-9f76-f405298d5d42","timestamp":"2026-06-10T18:40:57.692Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"286849be-62ea-4f80-9f76-f405298d5d42","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01WiAWgtEy3NtR8tZpGumwRT","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_011vbmsMtLevCQoKAtgYwHMf","name":"Bash","input":{"command":"ls -la","description":"List files in current directory"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2486,"cache_creation_input_tokens":3217,"cache_read_input_tokens":15334,"output_tokens":368,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":3217,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2486,"output_tokens":368,"cache_read_input_tokens":15334,"cache_creation_input_tokens":3217,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":3217},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4x18Wh753z3Yu2MazU","type":"assistant","uuid":"1e074165-5e3e-4897-87b8-0b16d8ab043f","timestamp":"2026-06-10T18:40:57.923Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"1e074165-5e3e-4897-87b8-0b16d8ab043f","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_011vbmsMtLevCQoKAtgYwHMf","type":"tool_result","content":"total 0\ndrwxr-xr-x 3 ben staff 96 Jun 10 14:40 .\ndrwx------ 3 ben staff 96 Jun 10 14:40 ..\n-rw-r--r-- 1 ben staff 0 Jun 10 14:40 .stderr.log","is_error":false}]},"uuid":"e6b884d5-c6d2-40a0-bed8-b58c02e649e1","timestamp":"2026-06-10T18:40:57.978Z","toolUseResult":{"stdout":"total 0\ndrwxr-xr-x 3 ben staff 96 Jun 10 14:40 .\ndrwx------ 3 ben staff 96 Jun 10 14:40 ..\n-rw-r--r-- 1 ben staff 0 Jun 10 14:40 .stderr.log","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false},"sourceToolAssistantUUID":"1e074165-5e3e-4897-87b8-0b16d8ab043f","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"e6b884d5-c6d2-40a0-bed8-b58c02e649e1","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_0153UE5DPd2YnWjBTuiRPeWb","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"Er0CCmMIDhgCKkA3QmK4isJ9g2rBOZo/uhh6rid3M767YWZg6ZBb62Ly4b98XaRYutkNC+N7H5AE3CcpXzrnLPx1NNfhMoGxgqieMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDDT0qEGdJtA2d/weNRoMIfa/VkZJgivM4O1fIjCIIsp99jypGjOxCAtJ2JdWzkZzW9m8km8huzCTT7voZIePyiZ7xoxfqDpAweXxBuUqhwETQGvK+Tj7xLTF2djUeUi84PTC0OxcuwqO0nxTMWjhP9g87uTBLa9Rqcq04cdQ/0T/6SzgZB8Z9lXwvDVzk6bVU2Lb+7++ORS2BdYrl24YfTd934kiibn2mRkEJUZWz9RGd36gDGnP5mO6Qp8Y/2bnfe/cKDEGFTKbYfjq1MrNDbuOop6qCvsYAQ=="}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":2955,"cache_read_input_tokens":18551,"output_tokens":253,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":2955,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":253,"cache_read_input_tokens":18551,"cache_creation_input_tokens":2955,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":2955},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4xYEn7qQQ9CXxydzfv","type":"assistant","uuid":"aecdae4e-9ab2-4add-8d7d-fef3766c7ce1","timestamp":"2026-06-10T18:41:01.620Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"aecdae4e-9ab2-4add-8d7d-fef3766c7ce1","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_0153UE5DPd2YnWjBTuiRPeWb","type":"message","role":"assistant","content":[{"type":"text","text":"The directory contains only `.stderr.log`.\n\n**Step 2:** Create `notes.md` with the two exact lines. I have a dedicated `Write` tool, so I'll use it."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":2955,"cache_read_input_tokens":18551,"output_tokens":253,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":2955,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":253,"cache_read_input_tokens":18551,"cache_creation_input_tokens":2955,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":2955},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4xYEn7qQQ9CXxydzfv","type":"assistant","uuid":"5be76eae-18ac-4c84-a92c-702de1e3bb52","timestamp":"2026-06-10T18:41:02.200Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"5be76eae-18ac-4c84-a92c-702de1e3bb52","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_0153UE5DPd2YnWjBTuiRPeWb","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_012E3VJqXjHmrW98NGLy9sTt","name":"Write","input":{"file_path":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md","content":"scratch — feature elicitation\nthis file will be edited and searched.\n"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":2955,"cache_read_input_tokens":18551,"output_tokens":253,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":2955,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":253,"cache_read_input_tokens":18551,"cache_creation_input_tokens":2955,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":2955},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4xYEn7qQQ9CXxydzfv","type":"assistant","uuid":"7e9875c5-e173-45ba-b79c-2d103900b0d3","timestamp":"2026-06-10T18:41:04.563Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"7e9875c5-e173-45ba-b79c-2d103900b0d3","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_012E3VJqXjHmrW98NGLy9sTt","type":"tool_result","content":"File created successfully at: /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md (file state is current in your context — no need to Read it back)"}]},"uuid":"4838cfe4-26a1-4106-b915-00fb2296a581","timestamp":"2026-06-10T18:41:04.574Z","toolUseResult":{"type":"create","filePath":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md","content":"scratch — feature elicitation\nthis file will be edited and searched.\n","structuredPatch":[],"originalFile":null,"userModified":false},"sourceToolAssistantUUID":"7e9875c5-e173-45ba-b79c-2d103900b0d3","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"4838cfe4-26a1-4106-b915-00fb2296a581","isSidechain":false,"attachment":{"type":"deferred_tools_delta","addedNames":["mcp__claude_ai_Gmail__authenticate","mcp__claude_ai_Gmail__complete_authentication","mcp__claude_ai_Google_Calendar__authenticate","mcp__claude_ai_Google_Calendar__complete_authentication","mcp__claude_ai_Google_Drive__authenticate","mcp__claude_ai_Google_Drive__complete_authentication"],"addedLines":["mcp__claude_ai_Gmail__authenticate","mcp__claude_ai_Gmail__complete_authentication","mcp__claude_ai_Google_Calendar__authenticate","mcp__claude_ai_Google_Calendar__complete_authentication","mcp__claude_ai_Google_Drive__authenticate","mcp__claude_ai_Google_Drive__complete_authentication"],"removedNames":[],"readdedNames":[],"pendingMcpServers":[]},"type":"attachment","uuid":"ab3f8c76-d7d2-4925-b64b-6ee6e0a3678d","timestamp":"2026-06-10T18:41:04.718Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"ab3f8c76-d7d2-4925-b64b-6ee6e0a3678d","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01TQaGq6JtusEF5c44A47x4g","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"Ev0BCmMIDhgCKkCKb/FmuVNZ6/zcWkC7FIQz8rZHTLJDOcbi4/ZLtrU2is35dypeuhPKneYsAWm1064CU37qWiOwoJZLFIGBZO0tMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDI/Xv5tcKZaXNjW9pRoMakzWh+ebGJJEcFk0IjBcUJ73AMWEBhvsQPjAYOX4eAq0i1RS1FtdOcC805M0EVS2XSv5/xshCcjrYiHwl6MqSIJ+FVCiZtxpo4AsL1foIsX/pUOGc1J3cZIgKJhRIqfTXQQDIoSfEmFjZkDeUwDgBteUA4tr5MVzAgnC5kAXKugU4ZFNUb0TcxgB"}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":223,"cache_creation_input_tokens":342,"cache_read_input_tokens":21506,"output_tokens":169,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":342,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":223,"output_tokens":169,"cache_read_input_tokens":21506,"cache_creation_input_tokens":342,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":342},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4y2kmSngGb4yZBbgXN","type":"assistant","uuid":"cffb7e83-d4eb-4393-9ff3-3f2f62ecb0e5","timestamp":"2026-06-10T18:41:07.922Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"cffb7e83-d4eb-4393-9ff3-3f2f62ecb0e5","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01TQaGq6JtusEF5c44A47x4g","type":"message","role":"assistant","content":[{"type":"text","text":"**Step 3:** Read `notes.md` back with the dedicated `Read` tool to confirm both lines."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":223,"cache_creation_input_tokens":342,"cache_read_input_tokens":21506,"output_tokens":169,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":342,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":223,"output_tokens":169,"cache_read_input_tokens":21506,"cache_creation_input_tokens":342,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":342},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4y2kmSngGb4yZBbgXN","type":"assistant","uuid":"3765115b-fc5b-4e5f-8891-eebe0e37d152","timestamp":"2026-06-10T18:41:08.447Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"3765115b-fc5b-4e5f-8891-eebe0e37d152","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01TQaGq6JtusEF5c44A47x4g","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_015XmV5A4F4S98hXAjBHG1S9","name":"Read","input":{"file_path":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":223,"cache_creation_input_tokens":342,"cache_read_input_tokens":21506,"output_tokens":169,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":342,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":223,"output_tokens":169,"cache_read_input_tokens":21506,"cache_creation_input_tokens":342,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":342},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4y2kmSngGb4yZBbgXN","type":"assistant","uuid":"42fd1210-5ea7-463b-bf37-defe8a98994e","timestamp":"2026-06-10T18:41:08.822Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"42fd1210-5ea7-463b-bf37-defe8a98994e","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_015XmV5A4F4S98hXAjBHG1S9","type":"tool_result","content":"1\tscratch — feature elicitation\n2\tthis file will be edited and searched.\n3\t"}]},"uuid":"27f9570c-6126-44c8-8bf2-bb73c040d447","timestamp":"2026-06-10T18:41:08.829Z","toolUseResult":{"type":"text","file":{"filePath":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md","content":"scratch — feature elicitation\nthis file will be edited and searched.\n","numLines":3,"startLine":1,"totalLines":3}},"sourceToolAssistantUUID":"42fd1210-5ea7-463b-bf37-defe8a98994e","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"type":"last-prompt","lastPrompt":"You're going to walk through a small set of tasks designed to exercise every common tool category. Stay inside the current working directory — do not touch anything outside it. Take the steps in order…","leafUuid":"27f9570c-6126-44c8-8bf2-bb73c040d447","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"ai-title","aiTitle":"Execute coding tool exercise tasks","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"parentUuid":"27f9570c-6126-44c8-8bf2-bb73c040d447","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_019KXzjJB7Vj6quJVgooMMC9","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"Ep0CCmMIDhgCKkAbUNGEemd0S1AmHy0CkiX+TzYIKGPSNlbhqlBhu0Abj2cTNmxcy2WrqQfCSYP6ADLU9kS9lxwen6fl/W26FyQnMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDE+PPdhECSyv0Gy4KBoMOfp40xKC4D5YIMtvIjAPTyjrvvA5g8egA+C9IKzDkYNQNwiaeOC/stzIu6BO0o3n7kjyFd31y88qg3T9ql0qaL4riMRBr5Ve3BspXleRVksbTJYD1Vn/6Um2iWnIA0WD15/xspNQOZWdJsHGMXZbr5SW16Vh2n73ZjgIizRv2M1p0hIpVxslcdcURx8oWGR2DSyPecZyWxmNaN+/xj099qmttWVuegLrGAE="}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":423,"cache_read_input_tokens":21848,"output_tokens":252,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":423,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":252,"cache_read_input_tokens":21848,"cache_creation_input_tokens":423,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":423},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yLh2JanjsEdinNQ1N","type":"assistant","uuid":"d3b2fc30-0938-48ab-a3fb-fba53b5b78ad","timestamp":"2026-06-10T18:41:12.441Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"d3b2fc30-0938-48ab-a3fb-fba53b5b78ad","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_019KXzjJB7Vj6quJVgooMMC9","type":"message","role":"assistant","content":[{"type":"text","text":"Both lines are present.\n\n**Step 4:** Change \"scratch\" to \"fixture\" using the dedicated `Edit` tool."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":423,"cache_read_input_tokens":21848,"output_tokens":252,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":423,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":252,"cache_read_input_tokens":21848,"cache_creation_input_tokens":423,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":423},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yLh2JanjsEdinNQ1N","type":"assistant","uuid":"8c62cc4b-c8a0-4e9c-80d3-e690ed99f46f","timestamp":"2026-06-10T18:41:12.952Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"8c62cc4b-c8a0-4e9c-80d3-e690ed99f46f","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_019KXzjJB7Vj6quJVgooMMC9","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01MTB1w9KLmWzjPh17yqsGqX","name":"Edit","input":{"replace_all":false,"file_path":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md","old_string":"scratch — feature elicitation","new_string":"fixture — feature elicitation"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":423,"cache_read_input_tokens":21848,"output_tokens":252,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":423,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":252,"cache_read_input_tokens":21848,"cache_creation_input_tokens":423,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":423},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yLh2JanjsEdinNQ1N","type":"assistant","uuid":"5fb954f4-884c-49b6-8e22-e7febccaa7e1","timestamp":"2026-06-10T18:41:14.060Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"5fb954f4-884c-49b6-8e22-e7febccaa7e1","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01MTB1w9KLmWzjPh17yqsGqX","type":"tool_result","content":"The file /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md has been updated successfully. (file state is current in your context — no need to Read it back)"}]},"uuid":"d52f5bda-01ae-4197-8d95-fefaecf304ba","timestamp":"2026-06-10T18:41:14.068Z","toolUseResult":{"filePath":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md","oldString":"scratch — feature elicitation","newString":"fixture — feature elicitation","originalFile":"scratch — feature elicitation\nthis file will be edited and searched.\n","structuredPatch":[{"oldStart":1,"oldLines":2,"newStart":1,"newLines":2,"lines":["-scratch — feature elicitation","+fixture — feature elicitation"," this file will be edited and searched."]}],"userModified":false,"replaceAll":false},"sourceToolAssistantUUID":"5fb954f4-884c-49b6-8e22-e7febccaa7e1","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"d52f5bda-01ae-4197-8d95-fefaecf304ba","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01HBCFjpuMV9BLgVQtVYbTP3","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"Et4ECmMIDhgCKkBaaJg6bTcmDlpX1L4F5X2GoYxyfsD3C8MjJ0gQQPWIQA3XVd2F4xsWVbGfRNtYMcuPFUnIyn36IV/SyFiwglvdMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDLp6YmvMs7IEjhsLJRoMs3wPUKJEMBTp/sqrIjA4Ek+cLwwudoQ06DAhtsdPgMkt02P9Ao6vHaX2ngelVZEo5COPatplTb4MjFqUZKUqqAPnBBgjFcOKwAYiXrB3O3dOtNeYTkqbstQ57p+tRf64nKTSMux1+n8MQgA9rE+iF8PlVojQYYSzW6LBlzIZql+Odlel8Y7T2qmdcay1HIv6kMawgq92s3d6ryUZXNU3CuE4amxSWn1Rjhc2RCOVsG1uIZEEgAgSFxy2a6Y7eoyYgr1MClwd7+PDhbi5oNehByykrEywDDeJgOb2GN+gx0QVF14McYyoBZss6+2d3pluknBvF5Tg2AZ247jTJKsA/iKlPKrQqUusxZEXB/yMLHu54h+w9TlTodZNzLGMG1Mr44ONxB4scjAhgvvmBRONvRolC3igiCCw17OJtHxi1gexVeGzm4UtMSkTbpWXDWwAfA5CPJjROSVvEIwemgUYrKmZB050d1ZvA31xuj4yYJJ9aTglVnOZSA7iJSvHzlDe3XL2jcFBVJSuxpT63/pNrjNJ06fCZ28ARR9LEtAHbLaBmC2d6df/kLHZ8IxuOo6InlqZUXEWSMxUcipvcmwO+0ctWhzSVZin879bA18hY3pbqQdnD8K2LWUpjB6DiIJvfR9tUgeNMaQNGAE="}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":368,"cache_read_input_tokens":22271,"output_tokens":410,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":368,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":410,"cache_read_input_tokens":22271,"cache_creation_input_tokens":368,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":368},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yj4g7uVKzFvvVD2Vw","type":"assistant","uuid":"e52b145a-ec79-4f53-a00b-eb0f0deb7e4f","timestamp":"2026-06-10T18:41:19.070Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"e52b145a-ec79-4f53-a00b-eb0f0deb7e4f","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01HBCFjpuMV9BLgVQtVYbTP3","type":"message","role":"assistant","content":[{"type":"text","text":"**Step 5 & 6:** No dedicated Glob or Grep tool is loaded at the top level, so I'll use `bash find` and `bash grep`. These two are independent, so I'll run them together."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":368,"cache_read_input_tokens":22271,"output_tokens":410,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":368,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":410,"cache_read_input_tokens":22271,"cache_creation_input_tokens":368,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":368},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yj4g7uVKzFvvVD2Vw","type":"assistant","uuid":"22353881-4ab4-42eb-82b4-dc079d618513","timestamp":"2026-06-10T18:41:19.544Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"22353881-4ab4-42eb-82b4-dc079d618513","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01HBCFjpuMV9BLgVQtVYbTP3","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01Hbmmt2ZzHbweps2MRmZNRi","name":"Bash","input":{"command":"find . -maxdepth 1 -name 'note*'","description":"Find files starting with note"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":368,"cache_read_input_tokens":22271,"output_tokens":410,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":368,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":410,"cache_read_input_tokens":22271,"cache_creation_input_tokens":368,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":368},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yj4g7uVKzFvvVD2Vw","type":"assistant","uuid":"1cbe6bf0-0fb3-43a6-97e4-f80b3ffbe03f","timestamp":"2026-06-10T18:41:20.494Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"1cbe6bf0-0fb3-43a6-97e4-f80b3ffbe03f","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01Hbmmt2ZzHbweps2MRmZNRi","type":"tool_result","content":"./notes.md","is_error":false}]},"uuid":"f5ab19c5-3a8f-4faa-bfac-88936bb3ed45","timestamp":"2026-06-10T18:41:20.533Z","toolUseResult":{"stdout":"./notes.md","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false},"sourceToolAssistantUUID":"1cbe6bf0-0fb3-43a6-97e4-f80b3ffbe03f","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"f5ab19c5-3a8f-4faa-bfac-88936bb3ed45","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01HBCFjpuMV9BLgVQtVYbTP3","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_014JqqqcyKQtpSCcCNH42znb","name":"Bash","input":{"command":"grep -rn fixture .","description":"Search file contents for fixture"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":368,"cache_read_input_tokens":22271,"output_tokens":410,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":368,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":410,"cache_read_input_tokens":22271,"cache_creation_input_tokens":368,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":368},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4yj4g7uVKzFvvVD2Vw","type":"assistant","uuid":"db5079de-82cc-4c4f-8ece-9190ed2529a3","timestamp":"2026-06-10T18:41:20.864Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"db5079de-82cc-4c4f-8ece-9190ed2529a3","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_014JqqqcyKQtpSCcCNH42znb","type":"tool_result","content":"notes.md:1:fixture — feature elicitation","is_error":false}]},"uuid":"6d9249b5-08b8-4d04-b075-595c6bbcd899","timestamp":"2026-06-10T18:41:20.894Z","toolUseResult":{"stdout":"notes.md:1:fixture — feature elicitation","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false},"sourceToolAssistantUUID":"db5079de-82cc-4c4f-8ece-9190ed2529a3","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"6d9249b5-08b8-4d04-b075-595c6bbcd899","isSidechain":false,"attachment":{"type":"task_reminder","content":[],"itemCount":0},"type":"attachment","uuid":"c6a924aa-1d70-4fac-b53c-c5c2b3e22bca","timestamp":"2026-06-10T18:41:20.945Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"c6a924aa-1d70-4fac-b53c-c5c2b3e22bca","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01X3WmZzCwANDrFGcS3eGNRB","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"EvcCCmMIDhgCKkBtfv1QQnAn6WukPTRW5wMAW4VFNufm/VruulgOo0ejHUJFqjDZFwWst8T9GfdrJZVwnCFKf22B3SNFiInlKfKJMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDBMSX2MkrwyfTE23eBoMK3XGRIFsllzmGvICIjBWqdO+ylrPBer7wsJ+JzKHIOGuabaDlIxm0jYlhRThSjckrQO3Ip1S+0CqEeKRghgqwQH4JhTpW62mLY3xrDB0tDtNdjX5t7SnS6LY7g1aXWPJsUDdadkFibVlEuNposcGekQBiuUYKq55bxBP+VCnp225ub5gdQQdu0Xw9pKh6AQWokairk3f+CICR2oOo6w7CwPZk8lH6+oNsg7iwxTwN/SKkwnno6R2hNMNifD6b+pltYg6yu2FDSiuapyBZgGv+6WexOIuIlGWBQBNHS7Q4pvufxFP51hcw2w3Dh/A6DdqdgRplTm5plKFBDPkDHOF8LGmGAE="}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":131,"cache_creation_input_tokens":488,"cache_read_input_tokens":22639,"output_tokens":281,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":488,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":131,"output_tokens":281,"cache_read_input_tokens":22639,"cache_creation_input_tokens":488,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":488},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4zEHLhkpN2f9m14WoB","type":"assistant","uuid":"f53f6cd0-a102-44bf-95f9-2b2888c685b1","timestamp":"2026-06-10T18:41:24.562Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"f53f6cd0-a102-44bf-95f9-2b2888c685b1","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01X3WmZzCwANDrFGcS3eGNRB","type":"message","role":"assistant","content":[{"type":"text","text":"Found `./notes.md`, and \"fixture\" appears in `notes.md` line 1 as expected.\n\n**Step 7:** Read a nonexistent file `does-not-exist.txt`. This call must fail, and I'll report it. Using the `Read` tool."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":131,"cache_creation_input_tokens":488,"cache_read_input_tokens":22639,"output_tokens":281,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":488,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":131,"output_tokens":281,"cache_read_input_tokens":22639,"cache_creation_input_tokens":488,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":488},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4zEHLhkpN2f9m14WoB","type":"assistant","uuid":"81336178-8a9b-407c-ba27-4040bbb0f03d","timestamp":"2026-06-10T18:41:25.702Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"81336178-8a9b-407c-ba27-4040bbb0f03d","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01X3WmZzCwANDrFGcS3eGNRB","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01CBVGgLJudohKLBvmSGVWkC","name":"Read","input":{"file_path":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/does-not-exist.txt"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":131,"cache_creation_input_tokens":488,"cache_read_input_tokens":22639,"output_tokens":281,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":488,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":131,"output_tokens":281,"cache_read_input_tokens":22639,"cache_creation_input_tokens":488,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":488},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4zEHLhkpN2f9m14WoB","type":"assistant","uuid":"f2099b88-c5f2-4628-badf-c76685c8badc","timestamp":"2026-06-10T18:41:26.164Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"f2099b88-c5f2-4628-badf-c76685c8badc","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"type":"tool_result","content":"File does not exist. Note: your current working directory is /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude.","is_error":true,"tool_use_id":"toolu_01CBVGgLJudohKLBvmSGVWkC"}]},"uuid":"322e3cda-9bce-44cc-9531-bdfa4de1ea5f","timestamp":"2026-06-10T18:41:26.173Z","toolUseResult":"Error: File does not exist. Note: your current working directory is /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude.","sourceToolAssistantUUID":"f2099b88-c5f2-4628-badf-c76685c8badc","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"322e3cda-9bce-44cc-9531-bdfa4de1ea5f","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_019PsXsTJEwe53m3CniwR4Me","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"EvwCCmMIDhgCKkC7FMEpYDYBfWxTnt0UhIRmT8ALCGMyKfW4Ptc9hATKGQ9GJXL+mt05cIIt/TWJAwKlgeQz5Pzrmoz6KuSG23BhMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDE5BWKDLpoyo0vTNTBoMwgEDPYIb4CmfLnZ5IjAPqgX6qaJCmeIN/n0tO30rxJ6a1jk2pf+txy4ag11FD9Rx2ffMJYIA5MQZpIk30bEqxgHIfrs/VnyZffxjlNT0OoOxJcW6YKbK910tF5sNuL9w6LglzCeKOUL3ALS9n+s4hsyJ7iRDm49U1mrabQX2HAdr5+RW8p7jiJl9Fw7saKealszIQwAi4k4Ney59a5nv9qV5NxI3AxIW0KahQ0wXj43YDsffjkOGhmZkWEze3hXAqDpEyyp3sGADANYt8dishb3h1YrI5aoXr9iwZ7Y6WmEyJZIkyGQO+fT5GhyngJd90qLNX9QaMQgwHGo+VEj9CHJVneqotukYAQ=="}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":492,"cache_read_input_tokens":23127,"output_tokens":319,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":492,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":319,"cache_read_input_tokens":23127,"cache_creation_input_tokens":492,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":492},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4zcpAgFNviwamXU2Nm","type":"assistant","uuid":"87e5c184-7340-4f24-a815-521ce6c22d54","timestamp":"2026-06-10T18:41:29.958Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"87e5c184-7340-4f24-a815-521ce6c22d54","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_019PsXsTJEwe53m3CniwR4Me","type":"message","role":"assistant","content":[{"type":"text","text":"As expected, reading `does-not-exist.txt` failed with a \"File does not exist\" error returned as a tool result.\n\n**Step 8:** Write `count.sh` that prints 1–5 on separate lines, then run it. Using the `Write` tool to create it."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":492,"cache_read_input_tokens":23127,"output_tokens":319,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":492,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":319,"cache_read_input_tokens":23127,"cache_creation_input_tokens":492,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":492},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4zcpAgFNviwamXU2Nm","type":"assistant","uuid":"220cefbd-0cd5-40e3-a154-f52e8ce5ea4d","timestamp":"2026-06-10T18:41:31.487Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"220cefbd-0cd5-40e3-a154-f52e8ce5ea4d","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_019PsXsTJEwe53m3CniwR4Me","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01AfhGtP87far2eixcTPoWBb","name":"Write","input":{"file_path":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/count.sh","content":"#!/usr/bin/env bash\necho 1\necho 2\necho 3\necho 4\necho 5\n"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":492,"cache_read_input_tokens":23127,"output_tokens":319,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":492,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":319,"cache_read_input_tokens":23127,"cache_creation_input_tokens":492,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":492},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv4zcpAgFNviwamXU2Nm","type":"assistant","uuid":"fdfc6f3f-489e-4f07-999f-e1e53e7e0982","timestamp":"2026-06-10T18:41:32.138Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"fdfc6f3f-489e-4f07-999f-e1e53e7e0982","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01AfhGtP87far2eixcTPoWBb","type":"tool_result","content":"File created successfully at: /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/count.sh (file state is current in your context — no need to Read it back)"}]},"uuid":"06ee6ca5-4e80-480b-8e8e-a5b36f47bbef","timestamp":"2026-06-10T18:41:32.146Z","toolUseResult":{"type":"create","filePath":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/count.sh","content":"#!/usr/bin/env bash\necho 1\necho 2\necho 3\necho 4\necho 5\n","structuredPatch":[],"originalFile":null,"userModified":false},"sourceToolAssistantUUID":"fdfc6f3f-489e-4f07-999f-e1e53e7e0982","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"06ee6ca5-4e80-480b-8e8e-a5b36f47bbef","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01L5iADpvDnBKfFNGwTYSTCV","type":"message","role":"assistant","content":[{"type":"text","text":"Now I'll run it and count the output lines."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":408,"cache_read_input_tokens":23619,"output_tokens":116,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":408,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":116,"cache_read_input_tokens":23619,"cache_creation_input_tokens":408,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":408},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv514Q3osdBrSKxDSBud","type":"assistant","uuid":"170de12c-3bac-4ea3-9081-da39f1383a3f","timestamp":"2026-06-10T18:41:35.870Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"170de12c-3bac-4ea3-9081-da39f1383a3f","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01L5iADpvDnBKfFNGwTYSTCV","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01GTjFYsEe9tbaWegZQxVrPs","name":"Bash","input":{"command":"bash count.sh; echo \"---\"; bash count.sh | wc -l","description":"Run count.sh and count output lines"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":408,"cache_read_input_tokens":23619,"output_tokens":116,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":408,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":116,"cache_read_input_tokens":23619,"cache_creation_input_tokens":408,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":408},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv514Q3osdBrSKxDSBud","type":"assistant","uuid":"a3984337-67ed-4f02-9557-073ed329b065","timestamp":"2026-06-10T18:41:36.561Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"a3984337-67ed-4f02-9557-073ed329b065","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01GTjFYsEe9tbaWegZQxVrPs","type":"tool_result","content":"1\n2\n3\n4\n5\n---\n5","is_error":false}]},"uuid":"056c361e-7c46-495d-b5bb-1118e6a99d81","timestamp":"2026-06-10T18:41:39.304Z","toolUseResult":{"stdout":"1\n2\n3\n4\n5\n---\n5","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false},"sourceToolAssistantUUID":"a3984337-67ed-4f02-9557-073ed329b065","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"056c361e-7c46-495d-b5bb-1118e6a99d81","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01EJdHKfRqtqCrkifu3dz1yM","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"EuAKCmMIDhgCKkCjYuLKlw4ppHTqokvhCeZE2JYjeYFSyj7/XiWaVVC4REuO7SJDduQ9+cHG4nEXGtbBA6O+SLUNzeASeMU8G6O9Mg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDPLT+JwASoLXkqwpWRoMc6sSLiWxCnRZQJAJIjD/ujZB/c84zhYKY6ZmfVhJzRrib/4RC2Y09UsdTm/8TSImNe2mbSUQCCQuHU66rZQqqgm+l0XWkJtPVkuKW5PJ5ebZ3wrNhN8BkaIU1E6gqp7SdNZwCs+U1Z/pr3eZ7CyfUt7C6oPK7/+qB0VMGL+PudPvocguwHr153kKqidggMlOrDoNqE+ggSnj0mU8WRjh66qZ5leFY5hulld7kUuxHfKkY0Ve0Grv1wK7MRyn2EvXcFykBxeVT1rBKzD0SreRW+AE66+v2mwTpb20r6eeLtXCMHjJ+UTKYkJYnw7x6jKCuVmXeAXImb1zByd9CHlIfhi+wLPA2yYIGdkgLTFucfmXHqUBnC0MyPgWvdxKZSLIQAxVXNHIA7WS0GrRdY09Rza9Aju6Z6s0UkUj1GqQvMrBdMQSiTw7tnpp3oXkaGQDoyBXtfyRAZEATsc8qCq0BfyADdxStyH64O2CqQ2bHtvat2Ox1AgnUlC6CwB3BHHDOs7vmqudb+whw6GyvBOtk9x+1GG5QnusmpyCR4tQT7PDwkkAcC1ukiwONPgJ/5FjDU3z9E8XsmW9+9EowTv+YHutlWpryqwFTmfXRRBjUOOGKL8Diwyq6cl14J+HXZxX7U57V6+AUh77Z/DuRLfJ1U4uesyp/fjl6Csi06Ld5YNUUQRAYa/Y8zam2tPDvAPm98h8r+SZP01WhHOfDHga/hVkmZg8/1K4MGwcxv7YAlQtGTlrTEoq/1H2UClurB5yRwzxhZnSRToURKcDqJydWKPL1iAMpVGnLQ2HGRRl/YEKNCQI9hrGfzGTFQo2jvkukuIcwYvXOD/mqpyrcQXhx5wym0lAFvE3gpDfs4+pBKzVoxkctO5MuMXQlXXxIHRperAJd5B44vG5B9/Izfs+ZdsWXU0q/dTz+B7IJi085YDI9dbMvlkiDisB7jAe/jly9gn53Mv7Ic8+Js06CBb4hy42kQOtM9F53w5WWkJtBNvwv8/8cMcEM00Cx5vItinfH4f8YqzHAdX4rT41HMnXu5N0QK3uA2IIc/hV5v70ZDkOrs8yjRzfdM5vur2O8VWtUEuSnoXK+GIOfowTL96gWyotSjVxtIO3mQJGWAAI0Kp5AKeEoxori39Snn5iYL02b04XCNbt8ZsnVqLH1wTe4Ly1yNPhzwUqkHSEkLbz39fn99WMaHntSEuYOMCPFpvorwoPfatF8r2lK5IcLeek58+F1AFvAjfE0RGBUTEOWH+68qpgXuiKP2oU+Ddijwa6u/ZVZ0IeUcl5g+3WIoytZRhdiecQ+DXOBR1nOe2QDgstAO+W/9q68z+FmXirvt72c6fAjKYYLg/oUsGmtQ1Qt9/QGUYzcWpn9ouar6dp4D/L8+d9vElrSBdAzw/otOY8NlqOULo3zfsL1lUQWaaQ6A7d3lZvGkzLXcKStkx1x91UQDINYXcoxGdrIZvrtSKyeyqA486Gy+L3trFf+HcJcajJomLUpxrqttySFHBz0ijeLezgH6zMDcDcEFtrTGJ9xEmApVwcf0vx5ovamIbBgqxl4eGH0WPOsNju/l23Q4bm7ti6Cmzq9TUIPJJj1AKTH3xAMzRX7eeMjuacAGMQk0L+9UeqVXqEXb7uHGJtcgvCFPF6c+17k/FyTIz//fWLFSxQZrpA1z1+aYIYAQ=="}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":134,"cache_read_input_tokens":24027,"output_tokens":670,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":134,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":670,"cache_read_input_tokens":24027,"cache_creation_input_tokens":134,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":134},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv51afDbrgwiMgKg3pcA","type":"assistant","uuid":"801a5b3b-521a-4c1e-aa88-4fb48bfaf5c7","timestamp":"2026-06-10T18:41:47.361Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"801a5b3b-521a-4c1e-aa88-4fb48bfaf5c7","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01EJdHKfRqtqCrkifu3dz1yM","type":"message","role":"assistant","content":[{"type":"text","text":"`count.sh` ran and produced exactly five lines (1–5), confirmed by `wc -l` = 5.\n\n**Step 9:** I do have a sub-agent dispatch tool (`Agent`). I'll dispatch one with the exact instruction, including the file's absolute path so it can locate `notes.md`."}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":134,"cache_read_input_tokens":24027,"output_tokens":670,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":134,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":670,"cache_read_input_tokens":24027,"cache_creation_input_tokens":134,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":134},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv51afDbrgwiMgKg3pcA","type":"assistant","uuid":"6725d1ff-1f40-41af-baec-4c15c0aa34bd","timestamp":"2026-06-10T18:41:48.225Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"6725d1ff-1f40-41af-baec-4c15c0aa34bd","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01EJdHKfRqtqCrkifu3dz1yM","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01CrV24hMMZ29fzvESWzVWw8","name":"Agent","input":{"description":"Count words in notes.md","subagent_type":"Explore","prompt":"Count the words in `notes.md` and report the number.\n\nThe file is located at: /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md"},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":134,"cache_read_input_tokens":24027,"output_tokens":670,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":134,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":670,"cache_read_input_tokens":24027,"cache_creation_input_tokens":134,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":134},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv51afDbrgwiMgKg3pcA","type":"assistant","uuid":"77e22484-6dea-46c4-ac4b-5181ca4f51ef","timestamp":"2026-06-10T18:41:49.340Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"type":"last-prompt","lastPrompt":"You're going to walk through a small set of tasks designed to exercise every common tool category. Stay inside the current working directory — do not touch anything outside it. Take the steps in order…","leafUuid":"77e22484-6dea-46c4-ac4b-5181ca4f51ef","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"ai-title","aiTitle":"Execute coding tool exercise tasks","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"parentUuid":"77e22484-6dea-46c4-ac4b-5181ca4f51ef","isSidechain":false,"promptId":"d0f56170-8c85-47cd-99c6-6302db4e0d72","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01CrV24hMMZ29fzvESWzVWw8","type":"tool_result","content":[{"type":"text","text":"The file `notes.md` contains **11 words**."}]}]},"uuid":"9f9f8402-cacb-4678-8e47-fb05660c6437","timestamp":"2026-06-10T18:41:57.708Z","toolUseResult":{"status":"completed","prompt":"Count the words in `notes.md` and report the number.\n\nThe file is located at: /private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude/notes.md","agentId":"aef08691ddf654768","agentType":"Explore","content":[{"type":"text","text":"The file `notes.md` contains **11 words**."}],"totalDurationMs":5645,"totalTokens":8185,"totalToolUseCount":2,"usage":{"input_tokens":5,"cache_creation_input_tokens":186,"cache_read_input_tokens":7978,"output_tokens":16,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":186},"inference_geo":"not_available","iterations":[{"input_tokens":5,"output_tokens":16,"cache_read_input_tokens":7978,"cache_creation_input_tokens":186,"cache_creation":{"ephemeral_5m_input_tokens":186,"ephemeral_1h_input_tokens":0},"type":"message"}],"speed":"standard"},"toolStats":{"readCount":1,"searchCount":0,"bashCount":1,"editFileCount":0,"linesAdded":0,"linesRemoved":0,"otherToolCount":0}},"sourceToolAssistantUUID":"77e22484-6dea-46c4-ac4b-5181ca4f51ef","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"9f9f8402-cacb-4678-8e47-fb05660c6437","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01429NLJe4VUi5mQKX9u1dh3","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"EtkFCmMIDhgCKkDjv+MsclYcmRCYgBcmbUioUpqtB8qwd66FjiJ3RiYr9ykIOrR7djmCTAENYUQoQEnGkYpsRcrMzL/ZYp6F6LJGMg9jbGF1ZGUtb3B1cy00LTg4AEIIdGhpbmtpbmcSDPkrVB21IVoIsapYShoMJ9CBbo0Ql7cTYK2bIjDLU3oScEnWiBf0k6mJxBljbUxFErmavyQY+DX4CsBEd0DWSotb2ANyBVqcqKY6ze8qowSAV9gVYDOpm+CFqFPowo7lat8J28rt6OnNL+LLKQQyohIwbi4E1xw9Ziv0aEaH0fqTQiVzAK4gZgGJy2cKuziM5s010RsaI8dODekSEbvlkMEsBuWHVYe3SvnkVU1dCtrJf6wjsajLBSGXMxQZU+lUvZ4ZbEOMlMpAqTIG7LsuzfhL1JOEwy8R6kTzoRP5GRzRjUIQenUa7xqHfBF8+L2o+YzjaFVXuD1HcdXXBUDBwEJOKMqvjrqWFu62TFwI6rBXcK2aquIPvcFD4ORAgLTTBzFzCwidfFjGYWsAdcmG4Xxy4RML3dZxnQ7ykCdMNLDJ9hHoTL5/j8OKICQqnS5L7XvQq+sJ0atgETAcnGs/XHAA9ZcRTE3VPD6nOABRgqAx1XcDcA9tJ7ckVqj40UmYyTDSFomBCgNzKYAHY0UCh1Ztn4sYkbdBIh+GeUVDi3ZgiX2IBZHMPtMiRCgd88XRlDAcOxDxStoAM/BnaD7FiNVi68hRJ+QOBcE0DMhKB5KsRdvn4C4+6QrcyI4LFTztb2VHi3Y/qfQz6oLffcHsZb+P7uASCwcXjCbWeSQrFWD2QzMPjZ+E1uy44mdyl5dKIOr7C0cFou4mHLUDDUpOaEALR2PaL7xTpsnZ27XczQSftGxsSGMJYzv2R6kkftUm/vwJjNaKXYMooeyt9M31kN3IpbeeZ/HDp+4ymCLtm0tOKeNVg+Wu/xj227RgRn+Cz+M0GAE="}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":691,"cache_read_input_tokens":24161,"output_tokens":438,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":691,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":438,"cache_read_input_tokens":24161,"cache_creation_input_tokens":691,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":691},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv52wPB6jHhRxkhsNKsL","type":"assistant","uuid":"8a1c3178-ba2b-43cc-a376-3ad159a03d25","timestamp":"2026-06-10T18:42:03.219Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"parentUuid":"8a1c3178-ba2b-43cc-a376-3ad159a03d25","isSidechain":false,"message":{"model":"claude-opus-4-8","id":"msg_01429NLJe4VUi5mQKX9u1dh3","type":"message","role":"assistant","content":[{"type":"text","text":"The sub-agent reported that `notes.md` contains **11 words**.\n\n**Step 10 — Summary:** I listed the directory, created `notes.md` with the two specified lines, read it back, then used the `Edit` tool to replace \"scratch\" with \"fixture\", and confirmed both a filename search (`note*` → `notes.md`) and a content search (`fixture` → line 1). I then verified error handling by reading a nonexistent file (which correctly failed as a reported tool result), wrote and ran `count.sh` to produce exactly five lines (1–5), and dispatched a sub-agent that reported `notes.md` has 11 words. All work stayed within the current working directory."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":691,"cache_read_input_tokens":24161,"output_tokens":438,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":691,"ephemeral_5m_input_tokens":0},"inference_geo":"not_available","iterations":[{"input_tokens":2,"output_tokens":438,"cache_read_input_tokens":24161,"cache_creation_input_tokens":691,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":691},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cbv52wPB6jHhRxkhsNKsL","type":"assistant","uuid":"1b85db73-91ac-4095-a45e-6feb3e495282","timestamp":"2026-06-10T18:42:05.646Z","userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD"} +{"type":"last-prompt","lastPrompt":"You're going to walk through a small set of tasks designed to exercise every common tool category. Stay inside the current working directory — do not touch anything outside it. Take the steps in order…","leafUuid":"1b85db73-91ac-4095-a45e-6feb3e495282","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"ai-title","aiTitle":"Execute coding tool exercise tasks","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"ai-title","aiTitle":"Execute coding tool exercise tasks","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"queue-operation","operation":"enqueue","timestamp":"2026-06-10T18:42:06.391Z","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","content":"/compact"} +{"type":"queue-operation","operation":"dequeue","timestamp":"2026-06-10T18:42:06.391Z","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"ai-title","aiTitle":"Execute coding tool exercise tasks","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"type":"mode","mode":"normal","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469"} +{"parentUuid":null,"logicalParentUuid":"1b85db73-91ac-4095-a45e-6feb3e495282","isSidechain":false,"type":"system","subtype":"compact_boundary","content":"Conversation compacted","isMeta":false,"timestamp":"2026-06-10T18:42:48.427Z","uuid":"c039fcfe-5f11-43de-8f6b-319572180a93","level":"info","compactMetadata":{"trigger":"manual","preTokens":25450,"durationMs":42025,"preservedSegment":{"headUuid":"8a1c3178-ba2b-43cc-a376-3ad159a03d25","anchorUuid":"0c5bb4cc-5f7a-4dca-bbcc-bb2e4c333016","tailUuid":"1b85db73-91ac-4095-a45e-6feb3e495282"},"preservedMessages":{"anchorUuid":"0c5bb4cc-5f7a-4dca-bbcc-bb2e4c333016","uuids":["8a1c3178-ba2b-43cc-a376-3ad159a03d25","1b85db73-91ac-4095-a45e-6feb3e495282"],"allUuids":["8a1c3178-ba2b-43cc-a376-3ad159a03d25","1b85db73-91ac-4095-a45e-6feb3e495282"]},"postTokens":2724},"userType":"external","entrypoint":"sdk-cli","cwd":"/private/var/folders/jk/4rycn96j1rdbj2mz275yv11m0000gn/T/toolpath-elicit.7HDqFK/claude","sessionId":"c07efcc2-c931-4e4e-bc2c-b14ded596469","version":"2.1.170","gitBranch":"HEAD","slug":"curried-whistling-parrot"} +{"parentUuid":"c039fcfe-5f11-43de-8f6b-319572180a93","isSidechain":false,"promptId":"9688f065-f66a-49bf-a2f0-1a327facbb46","type":"user","message":{"role":"user","content":"This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.\n\nSummary:\n1. Primary Request and Intent:\n The user requested a structured walkthrough of a fixed set of 10 ordered tasks designed to exercise every common tool category (list, write, read, edit, find/glob, grep/search, error-handling on a missing file, shell script execution, sub-agent dispatch, and a final summary). Explicit constraints and rules stated by the user:\n - \"Stay inside the current working directory — do not touch anything outside it.\"\n - \"Take the steps in order.\"\n - \"Briefly think out loud before each tool call so the reasoning is captured.\"\n - Tool-selection rule (verbatim intent): \"if a step mentions a dedicated tool *and* that tool is available to you, use it; otherwise, use whatever shell-based equivalent you have (e.g. `ls`, `cat`, `sed`, `find`, `grep`). Don't skip steps because a dedicated tool is missing.\"\n The 10 steps: (1) list files; (2) write `notes.md` with two exact lines; (3) read `notes.md` back; (4) change \"scratch\" to \"fixture\"; (5) find files starting with \"note\"; (6) search contents for \"fixture\"; (7) attempt to read nonexistent `does-not-exist.txt` (must fail and be reported, not swallowed); (8) write `count.sh` printing 1–5 and run it confirming 5 lines; (9) dispatch a sub-agent with the exact instruction \"Count the words in `notes.md` and report the number.\" and quote its reply; (10) summarize in 2–3 sentences what was done, then stop without asking for further work.\n\n2. Key Technical Concepts:\n - Dedicated-tool-vs-shell-fallback selection logic.\n - Claude Code tool set: Write, Read, Edit, Bash, Agent (sub-agent dispatch), ToolSearch (for deferred tools), Skill, Workflow.\n - Sub-agent dispatch via the Agent tool with subagent_type (used \"Explore\", a read-only search agent).\n - Bash utilities as fallbacks: `ls`, `find -maxdepth 1 -name`, `grep -rn`, `wc -l`, `bash