[asubbarao]

read_conversations left about 20 unified columns entirely NULL for Claude, even though the transcripts carry the data. It also had several reader defects that the same coverage measurement exposed. This PR fills those columns and fixes the defects. One Codex bug is fixed too: a forked rollout was relabelled with its parent's session id.

Column → source (Claude)

column	source	scope
response_id	message.id (msg_…), else requestId	record (assistant/thinking)
reasoning_effort	perTurnEffort, else effort	record
reasoning_tokens	message.usage.output_tokens_details.thinking_tokens	rows that carry usage
usage_scope / usage_source_line	'response' / line number, on rows that carry usage	record
event_type	record type; a system record's subtype; an attachment's kind	record
parent_uuid	parentUuid, else logicalParentUuid (compaction boundary), else leafUuid (last-prompt)	record
record_id	claude: + FNV-1a of session, project dir, file, line, sub-row	record
model_provider	'anthropic'	all rows
originator	entrypoint, same as client (Codex derives client from originator)	carried
turn_id	promptId, carried to every row of its turn	turn
root_turn_id	= turn_id for main sessions; for a subagent, the parent turn that issued its spawning tool call (meta.json toolUseId)	turn
thread_source	turnOrigin, else origin.kind, for the turn	turn
agent_nickname	agent-name record; subagent: meta.json name	session
agent_role	agent-setting record; subagent: meta.json agentType (Explore, general-purpose, …)	session
repository	latest pr-link prRepository (all links in metadata.session.prLinks)	session
channel	bridge-session bridgeSessionId	session
forked_from_session_id	the session whose continued-in record names this one	session
session_created_at / updated_at	first / last timestamp in the session (main + subagent files)	session
cwd, slug, git_branch, version, client	carried onto session-state rows that lack them (latest value, else the file's first)	session
metadata	{"session": titles, mode, permissionMode, sessionKind, prLinks, worktreeSession, subagent meta.json, …, "record": every top-level field not already in a column}	record

Session-level records (agent-name, agent-setting, pr-link, bridge-session, continued-in, custom-title, ai-title, mode, permission-mode, worktree-state) have no uuid; NULL uuid on those rows is correct. Their facts are now applied to every row of the session, not only to the record itself.

Reader defects fixed

• Thinking-only lines. Claude writes each content block as its own line. A line whose only block is thinking used to produce an empty assistant row: 59,973 rows on the author's data, from redacted (signature-only) thinking. When the thinking had text, the line produced an empty assistant row beside the thinking row. It is now one thinking row that keeps the line's usage, effort and response_id. Redacted thinking has NULL content.
• Attachments. Attachment rows put their kind (hook_additional_context, total_tokens_reminder, …) in tool_name. Now the kind is in event_type and message_role = 'system'. tool_name/tool_use_id are set only when the attachment's toolUseID cites a real tool_use in the same session.
• System rows. event_type = subtype. The payload (hookInfos, durationMs, compactMetadata, …) is in metadata.record. Empty content is NULL instead of ''.
• Resumed turns. Queued prompts interleave, so a promptId can resume after another turn. thread_source keeps the origin that turn already had.
• Duplicated tool calls. A rewound or compacted session replays earlier tool calls later in the file. A subagent's root turn is the first, issuing occurrence.

Codex

• A spawned or forked rollout replays its parent's session_meta as its second line. The reader adopted it, so the child's rows took the parent's session id and lost forked_from_session_id. The first session_meta now keeps the thread's identity, and a later one with a different id is treated as replay. On the author's data, forked_from_session_id goes from 100% NULL to filled for every forked thread.
• channel falls back to phase (commentary / final_answer), the name current rollouts use.

Columns that stay NULL, and why

• Claude git_commit: transcripts record only the branch, never a commit.
• Claude archived: Claude has no archive state in its files.
• parse_error: NULL whenever the transcript is clean.
• Codex parent_uuid: current rollouts write no parent item ids. The reader still reads parent_id when present.
• Codex project_dir: this is Claude's encoded projects/ directory, and Codex has no equivalent.
• Codex agent_role: the reader fills it from thread_spawn.agent_role. Every local rollout writes it as null, and the fixture proves the mapping.
• Codex cache_creation_tokens: Codex usage reports only cached input, not cache creation.

Tests

• test/sql/unified_columns.test uses the new fixtures test/data_claude_unified (main session + subagent with meta.json + a continued-from session) and test/data_codex_unified (spawned rollout with a replayed parent session_meta, phase, index). It asserts each mapped column on fixture lines that carry the field, plus:

  • the session-wide propagation
  • thinking-only and redacted-thinking rows
  • attachments citing a real tool call
  • a whole-schema check per source in one statement: SUMMARIZE (SELECT nullif(trim(COLUMNS(*)::VARCHAR), '') …). No column may be entirely NULL/blank except the allowlist above.

• test/mapping/claude_raw_mapping.sql (new CI job Claude raw-to-column coverage) runs the following checks:

  • duck_diff schema_diff, table_diff, and table_diff_summary compare the mapped columns with the same columns derived again from raw_event (and the subagent meta.json) in SQL, keyed by file_path, line_number, record_id. The run fails unless n_total = n_identical.
  • finetype ft_unpack(raw_event) inventories the raw top-level fields. Any field that is neither read into a column nor kept in metadata.record fails the run. ft_profile checks that the session span columns are the same ISO-8601 type as timestamp.
  • A per-message_type blank report (COLUMNS(*) + UNPIVOT, 0 < pct < 100) for review. There is also a failing check that no assistant row is an empty reply, and that no attachment names a tool outside its session's tool calls.

  These checks run as their own job with the DuckDB CLI on the linux_amd64 artifact. The sqllogictest suite runs on every distribution platform, where community builds of duck_diff and finetype cannot be relied on.

• The full suite passes locally against the freshly built release binary: 37/37 sqllogictest files and 16 Rust unit tests. The runner loads build/release/agent_data.duckdb_extension via --external-extension. Two existing expectations changed on purpose: claude_attachments.test (tool_name → event_type) and claude_open_types.test (attachment role is system).

• The mapping check was also run once against the author's real Claude home (595,329 rows). Results:

  • schema_diff ok on all 27 columns.
  • table_diff_summary n_identical = n_total = 595,329.
  • No empty assistant rows, and no attachment names a tool outside its session.

  The ft_unpack inventory step does not finish at that scale. The CI job runs it on the fixture. Running it on a full home is not supported.

Does not overlap #4 (modified_after).

🤖 Generated with Claude Code