diff --git a/SPEC.md b/SPEC.md index 7f97f2e..e9cbc56 100644 --- a/SPEC.md +++ b/SPEC.md @@ -704,6 +704,7 @@ tool call's text is still stored, so older purged requests keep `file_path` NULL | `GET` | `/api/sessions` | List all sessions (newest first). Each session carries `status` (`active`\|`ended`\|`archived`) and `archived_at`. | | `GET` | `/api/sessions/{id}` | Get session detail + aggregated token stats for that session. 404 if missing. | | `GET` | `/api/sessions/{id}/lineage` | Complete session lineage graph. Returns exact/inferred continuation edges, diagnostic root-to-leaf paths, conservative conversation groups and membership, uncertainty diagnostics, timing, and per-edge context-delta summaries. Exact parents captured outside the selected session are external nodes. Each node also carries `conversation_code` (`C` / `AUX`, as on request cards; `null` for external nodes) and `conversation_previous_request_id` / `conversation_next_request_id`, the neighbouring requests in its conversation in session order (`null` for auxiliary and external nodes; neighbours are not proven parent/child links). | +| `GET` | `/api/sessions/{id}/trend` | Per-request metric values for the session trend chart: a `metrics` registry (`id`, `label`, `unit`, `estimated`, `empty_hint`) and one `series` per conversation (plus auxiliary), each with `points` (`request_id`, `session_seq`, `ordinal`, `time` = `started_at` else `timestamp`, `context_fidelity`, `purpose`, `values`). Metrics: `context_estimated`, `cache_hit_pct` (`cache_read / provider_input`, as request detail), `ttft_ms`, `duration_ms`; a value is `null` when unavailable. A request appears in one series only; ids from other sessions are dropped. 404 if missing. | | `PATCH` | `/api/sessions/{id}` | Rename a session. Body: `{ "name": "string" }`. 422 if blank, 404 if missing. | | `POST` | `/api/sessions/{id}/end` | End a session. 404 if missing. | | `POST` | `/api/sessions/{id}/archive` | Archive an ended session (one-way; repeatable): removes raw payloads and unreferenced block text, keeps all analysis data. Returns `{ session, freed: { requests, request_body_bytes, content_rows, content_bytes }, space: { auto_vacuum: "incremental"\|"none", reclaimed_bytes, free_bytes_remaining, note }, already_archived }`. 404 if missing, 409 if the session is active or an archive of it is already running. Broadcasts `session_archived` over the WebSocket. | diff --git a/contextspy/analysis/accounting.py b/contextspy/analysis/accounting.py new file mode 100644 index 0000000..1ef48ec --- /dev/null +++ b/contextspy/analysis/accounting.py @@ -0,0 +1,26 @@ +# Copyright 2026 Rimantas Zukaitis +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Small shared derivations over provider-reported usage.""" +from __future__ import annotations + + +def cached_share_pct(cache_read_tokens: int | None, provider_input_tokens: int | None) -> float | None: + """Share of provider-reported input tokens served from the prompt cache, as a percentage. + + ``None`` when the provider did not report input tokens (or reported zero). Single definition + used by the request detail ("Cached share") and the session trend chart. + """ + if provider_input_tokens is None or provider_input_tokens <= 0: + return None + return (cache_read_tokens or 0) / provider_input_tokens * 100 diff --git a/contextspy/analysis/activity.py b/contextspy/analysis/activity.py index c3b4e12..29989dc 100644 --- a/contextspy/analysis/activity.py +++ b/contextspy/analysis/activity.py @@ -6,7 +6,7 @@ Derived at read time from the persisted ``source_key`` (nothing is stored), so the tables below can be refined without a migration. Vocabulary: read, search, edit, vcs, test, command, web, -orchestration, mcp, other. See plans/wi0-data-foundation.md section 10. +orchestration, mcp, other. See plans/archive/wi0-data-foundation.md section 10. """ from __future__ import annotations diff --git a/contextspy/analysis/block_hotspots.py b/contextspy/analysis/block_hotspots.py index 173170c..6bfb050 100644 --- a/contextspy/analysis/block_hotspots.py +++ b/contextspy/analysis/block_hotspots.py @@ -7,7 +7,7 @@ Hot spots rank the blocks of a scope (a conversation or a whole session) by the visible tokens they add up to over all the requests that carry them. Nothing here needs block content: labels come from stored structure (type, tool, source key, file path, provider item type), so everything works after a -session was archived. See plans/hot-spots.md. +session was archived. See plans/archive/hot-spots.md. """ from __future__ import annotations diff --git a/contextspy/analysis/block_occurrences.py b/contextspy/analysis/block_occurrences.py index 9d19022..8c7370b 100644 --- a/contextspy/analysis/block_occurrences.py +++ b/contextspy/analysis/block_occurrences.py @@ -8,7 +8,7 @@ an ordered list of requests (a whole session, one conversation, or a single request). Presence is reported as runs of consecutive positions in that order, so a block that is in every request of a conversation is one run even when other conversations' requests are interleaved in session order. -Token figures are visible-block tokens, never provider totals. See plans/info-panel.md. +Token figures are visible-block tokens, never provider totals. See plans/archive/info-panel.md. """ from __future__ import annotations diff --git a/contextspy/analysis/paths.py b/contextspy/analysis/paths.py index 20cfaec..e6866af 100644 --- a/contextspy/analysis/paths.py +++ b/contextspy/analysis/paths.py @@ -8,7 +8,7 @@ by any extractor (structured tool arguments, shell commands, patch headers) goes through it. Today it validates and trims. If paths ever become a privacy problem (shared databases, exports), obfuscation (basename only, a salted hash, a setting) is a change inside that function plus a -re-derivation migration, nothing else. See plans/file-paths.md. +re-derivation migration, nothing else. See plans/archive/file-paths.md. Only a path that a known read/edit tool targets is read; no other argument (command text, URLs, patterns, patch bodies, secrets) is ever returned from this module. diff --git a/contextspy/analysis/purpose.py b/contextspy/analysis/purpose.py index 4f95c35..16d3ac9 100644 --- a/contextspy/analysis/purpose.py +++ b/contextspy/analysis/purpose.py @@ -7,7 +7,7 @@ The baseline is structural and works for every provider: it looks at the last input message (trailing user text vs trailing tool results) and at what the response contained. Agent-specific detectors (compaction, housekeeping side calls, ...) can be registered; none ships yet because -they need captures from each agent. See plans/wi0-data-foundation.md section 5. +they need captures from each agent. See plans/archive/wi0-data-foundation.md section 5. """ from __future__ import annotations diff --git a/contextspy/analysis/sources.py b/contextspy/analysis/sources.py index 8a022eb..b4ca1d5 100644 --- a/contextspy/analysis/sources.py +++ b/contextspy/analysis/sources.py @@ -5,7 +5,7 @@ """What produced a block: a compact, provider-neutral ``source_key``. Keys are persisted at capture time (``blocks.source_key``) because tool-call arguments are -purged by archive/retention. See plans/wi0-data-foundation.md section 4. +purged by archive/retention. See plans/archive/wi0-data-foundation.md section 4. Grammar: bare keys ``system``, ``user``, ``assistant``, ``reasoning``, ``other``; otherwise ``:``: @@ -17,7 +17,7 @@ Parsers record program names. The only argument ever read is the file a known read/edit tool targets, stored separately in ``blocks.file_path`` (``analysis/paths.py`` is the single choke point); -command text, URLs, patterns and patch bodies are never stored. See plans/file-paths.md. +command text, URLs, patterns and patch bodies are never stored. See plans/archive/file-paths.md. """ from __future__ import annotations diff --git a/contextspy/api/routers/sessions.py b/contextspy/api/routers/sessions.py index 4f72012..eb9e47e 100644 --- a/contextspy/api/routers/sessions.py +++ b/contextspy/api/routers/sessions.py @@ -92,6 +92,16 @@ def get_session_lineage(session_id: str): } +@router.get("/sessions/{session_id}/trend") +def get_session_trend(session_id: str): + """Per-request metric values (context size, cache hit, TTFT, latency) per conversation.""" + with get_db() as db: + if not crud.get_session(db, session_id): + raise HTTPException(status_code=404, detail="Session not found") + with db.begin_nested(): + return crud.get_session_trend(db, session_id) + + @router.get("/sessions/{session_id}/lineage/revision") def get_session_lineage_revision(session_id: str): """Small polling response so diagnostics reloads only when evidence changes.""" diff --git a/contextspy/db/compaction.py b/contextspy/db/compaction.py index 3495e99..9dfc6f3 100644 --- a/contextspy/db/compaction.py +++ b/contextspy/db/compaction.py @@ -7,7 +7,7 @@ SQLite never returns freed pages to the filesystem on its own: deleting block contents or nulling request bodies (retention, session archive) only grows the file's free list. ``VACUUM`` rebuilds the file without it, and switching the database to *incremental* auto-vacuum lets later deletions shrink -the file online (``PRAGMA incremental_vacuum``). See plans/db-compact.md. +the file online (``PRAGMA incremental_vacuum``). See plans/archive/db-compact.md. Everything here uses the stdlib ``sqlite3`` module directly and runs with the maintenance lock held, so the server cannot be running. ``VACUUM`` is atomic: an interruption leaves the original file intact. diff --git a/contextspy/db/crud.py b/contextspy/db/crud.py index 7ed91a0..bec2737 100644 --- a/contextspy/db/crud.py +++ b/contextspy/db/crud.py @@ -25,7 +25,7 @@ from sqlalchemy.orm import Session as OrmSession, load_only from contextspy.analysis.blocks import BlockType, Direction -from contextspy.db import session_lineage_service +from contextspy.db import session_lineage_service, trend_service from contextspy.db.models import BlockContent, BlockRecord, Request, Session, ToolStat if TYPE_CHECKING: @@ -766,6 +766,12 @@ def get_timeline( return sorted(buckets.values(), key=lambda x: x["bucket"]) +def get_session_trend(db: OrmSession, session_id: str) -> dict: + """Per-request metric series grouped by conversation for the session trend chart.""" + graph = get_session_lineage_graph(db, session_id) + return trend_service.build_session_trend(db, session_id, graph) + + # --------------------------------------------------------------------------- # Sessions summary (for dashboard timeline table) # --------------------------------------------------------------------------- diff --git a/contextspy/db/hotspots_service.py b/contextspy/db/hotspots_service.py index 0e0487d..36f987c 100644 --- a/contextspy/db/hotspots_service.py +++ b/contextspy/db/hotspots_service.py @@ -12,7 +12,7 @@ (``FROM hs_scope s CROSS JOIN blocks b ON b.request_id = s.id``): left to itself SQLite scans ``idx_blocks_content_hash`` for a ``GROUP BY content_hash`` and is 15-100x slower. A test asserts the plan. Block identity is the content hash; input blocks only (output blocks reappear as input in the next request, -so counting both would double count). See plans/hot-spots.md. +so counting both would double count). See plans/archive/hot-spots.md. """ from __future__ import annotations @@ -155,7 +155,7 @@ def get_session_hotspots( block_type: str | None = None, source: str | None = None, in_context: str = "all", limit: int = 25, offset: int = 0, ) -> dict | None: - """The ranking described in plans/hot-spots.md; None when the session does not exist.""" + """The ranking described in plans/archive/hot-spots.md; None when the session does not exist.""" if db.get(Session, session_id) is None: return None resolved = scope_for_session( diff --git a/contextspy/db/migrations.py b/contextspy/db/migrations.py index 32ab114..d60ea05 100644 --- a/contextspy/db/migrations.py +++ b/contextspy/db/migrations.py @@ -988,7 +988,7 @@ def _v9_match_json_paths(stored: list, parsed: list) -> dict[int, str]: def _migrate_to_v9(db: OrmSession) -> None: - """Backfill request purpose, block ``source_key`` and block ``json_path`` (plans/wi0-data-foundation.md section 8).""" + """Backfill request purpose, block ``source_key`` and block ``json_path`` (plans/archive/wi0-data-foundation.md section 8).""" _backfill_classification(db, "v9", json_paths=True) @@ -997,7 +997,7 @@ def _migrate_to_v10(db: OrmSession) -> None: Same phase A as v9 (a database upgraded straight from before v9 has already done it, so this finds nothing to do). File paths come from retained tool-call content only; purged tool calls keep - ``file_path`` NULL, so historical coverage is partial by nature (plans/file-paths.md). + ``file_path`` NULL, so historical coverage is partial by nature (plans/archive/file-paths.md). """ _backfill_classification(db, "v10", json_paths=False) diff --git a/contextspy/db/models.py b/contextspy/db/models.py index addef83..bacbc4f 100644 --- a/contextspy/db/models.py +++ b/contextspy/db/models.py @@ -28,6 +28,7 @@ ) from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship +from contextspy.analysis.accounting import cached_share_pct from contextspy.analysis.activity import activity_for from contextspy.analysis.tokenizer import TOKENIZER_ID @@ -232,11 +233,7 @@ def to_dict(self, include_raw: bool = True) -> dict: if provider_input is not None and provider_input > 0 else None ) - cached_share = ( - (self.cache_read_tokens or 0) / provider_input * 100 - if provider_input is not None and provider_input > 0 - else None - ) + cached_share = cached_share_pct(self.cache_read_tokens, provider_input) d["context_accounting"] = { "visible_input_tokens": visible_input, "provider_input_tokens": provider_input, @@ -355,7 +352,7 @@ class BlockRecord(Base): # compact JSON array (e.g. ["messages",3,"content",1]); NULL when unknown. json_path: Mapped[Optional[str]] = mapped_column(Text, nullable=True) # The file a read/edit tool call (and its result) targets; written only via analysis/paths.py. - # Kept after archive. See plans/file-paths.md. + # Kept after archive. See plans/archive/file-paths.md. file_path: Mapped[Optional[str]] = mapped_column(Text, nullable=True) def to_dict( diff --git a/contextspy/db/session_archive.py b/contextspy/db/session_archive.py index 13a6e8c..2f5b639 100644 --- a/contextspy/db/session_archive.py +++ b/contextspy/db/session_archive.py @@ -7,7 +7,7 @@ Archiving removes what is large and re-derivable only from the original traffic (the five request/response body columns and the ``block_contents`` text no other session needs) and keeps everything the analysis views use: block rows (hash, type, category, tokens, source key, JSON path), lineage and conversation data. -See plans/session-archive.md. +See plans/archive/session-archive.md. Concurrency: SQLite serialises writers, and ContextSpy's capture path keeps writing while an archive runs. Every write here is therefore a short transaction. The content cleanup is the delicate part: it must never delete content diff --git a/contextspy/db/trend_service.py b/contextspy/db/trend_service.py new file mode 100644 index 0000000..d4e07d6 --- /dev/null +++ b/contextspy/db/trend_service.py @@ -0,0 +1,127 @@ +# Copyright 2026 Rimantas Zukaitis +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Per-request metric series for the session trend chart. + +Every number the chart plots is computed here (AGENTS.md: analysis lives in Python); the UI only +formats and lays out. A metric is one entry in ``METRICS``: adding one needs no client change +beyond unit formatting. +""" +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Callable + +from sqlalchemy import select +from sqlalchemy.orm import Session as OrmSession, load_only + +from contextspy.analysis.accounting import cached_share_pct +from contextspy.db.models import Request + + +@dataclass(frozen=True) +class Metric: + id: str + label: str + unit: str # "tokens" | "percent" | "ms" + estimated: bool + empty_hint: str | None + compute: Callable[[Request], float | int | None] + + +METRICS: tuple[Metric, ...] = ( + Metric("context_estimated", "Context size (estimated)", "tokens", True, None, + lambda r: r.tokens_total_input), + Metric("cache_hit_pct", "Cache hit %", "percent", False, + "Provider did not report cache usage", + lambda r: cached_share_pct(r.cache_read_tokens, r.provider_input_tokens)), + Metric("ttft_ms", "TTFT", "ms", False, + "No streamed responses (TTFT needs streaming)", lambda r: r.ttft_ms), + Metric("duration_ms", "Latency", "ms", False, + "No completed responses", lambda r: r.duration_ms), +) + +_COLUMNS = ( + Request.id, Request.session_id, Request.session_seq, Request.timestamp, Request.started_at, + Request.purpose, Request.context_fidelity, Request.tokens_total_input, + Request.provider_input_tokens, Request.cache_read_tokens, Request.ttft_ms, Request.duration_ms, +) + + +def _point(r: Request, ordinal: int) -> dict[str, Any]: + when = r.started_at or r.timestamp + return { + "request_id": r.id, + "session_seq": r.session_seq, + "ordinal": ordinal, # 1-based position in the session; Request-axis fallback for null seq + "time": when.isoformat(), + "context_fidelity": r.context_fidelity, + "purpose": r.purpose, + "values": {m.id: m.compute(r) for m in METRICS}, + } + + +def _sort_key(r: Request) -> tuple: + return (r.session_seq is None, r.session_seq or 0, r.started_at or r.timestamp, r.id) + + +def build_session_trend(db: OrmSession, session_id: str, graph: dict) -> dict[str, Any]: + """One series per conversation group (plus auxiliary) of ``graph``, one point per request. + + Ids that are not rows of this session (external lineage parents) are dropped; a request listed + in several groups is plotted only in the first one; a session request in no group falls into + the auxiliary series. + """ + rows = list(db.execute( + select(Request).options(load_only(*_COLUMNS, raiseload=True)) + .where(Request.session_id == session_id) + ).scalars().all()) + ordered = sorted(rows, key=_sort_key) + by_id = {r.id: r for r in ordered} + position = {r.id: i for i, r in enumerate(ordered)} + + def members(group: dict, claimed: set[str]) -> list[Request]: + ids = [rid for rid in group["request_ids"] if rid in by_id and rid not in claimed] + claimed.update(ids) + return [by_id[rid] for rid in sorted(ids, key=position.__getitem__)] + + claimed: set[str] = set() + series: list[tuple[dict, bool, list[Request]]] = [] + for group in graph["conversations"]: + series.append((group, False, members(group, claimed))) + aux_group = graph.get("auxiliary") + aux_members = members(aux_group, claimed) if aux_group else [] + leftover = [by_id[rid] for rid in sorted(by_id.keys() - claimed, key=position.__getitem__)] + aux_all = sorted(aux_members + leftover, key=lambda r: position[r.id]) + + out = [ + {"key": g["key"], "label": g["label"], "auxiliary": aux, "request_count": len(reqs), + "points": [_point(r, position[r.id] + 1) for r in reqs]} + for g, aux, reqs in series + ] + if aux_all: + out.append({ + "key": aux_group["key"] if aux_group else f"session:{session_id}:auxiliary", + "label": aux_group["label"] if aux_group else "Auxiliary requests", + "auxiliary": True, "request_count": len(aux_all), + "points": [_point(r, position[r.id] + 1) for r in aux_all], + }) + return { + "session_id": session_id, + "metrics": [ + {"id": m.id, "label": m.label, "unit": m.unit, "estimated": m.estimated, + "empty_hint": m.empty_hint} + for m in METRICS + ], + "series": out, + } diff --git a/docs/changelog.md b/docs/changelog.md index 30fef37..9f5d95a 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -11,6 +11,16 @@ - Two defaults changed: payloads are **no longer deleted after 7 days** (see *Retention and archive*), and new databases use incremental auto-vacuum. +### Session trend chart + +- The session Summary gets a second chart next to *Token usage over time*. It plots a metric **per request**. Pick **Context size + (estimated)**, **Cache hit %** (same figure as *Cached share* in request detail), **TTFT** or **Latency**, plotted by + request number. The chart shows the latest 100 requests; pan back with the slider, the arrow buttons or by dragging, + and use **Latest** to return (it follows new requests while you stay at the end). +- One line per conversation, with a checkbox to show or hide each; the auxiliary group starts hidden. The tooltip flags requests whose + context is only partly captured (their context size is under-reported). Click a point to open the request. +- The selected metric is kept in the page URL. The data comes from `GET /api/sessions/{id}/trend`. + ### Hot spots - **New session view: Hot spots.** Ranks what the context window keeps carrying: the blocks (tool definitions, big diff --git a/plans/ANALYSIS_ROADMAP.md b/plans/ANALYSIS_ROADMAP.md index 4335fcc..21a6b14 100644 --- a/plans/ANALYSIS_ROADMAP.md +++ b/plans/ANALYSIS_ROADMAP.md @@ -24,27 +24,27 @@ serves one of those four questions. | D1 | **Cost = tokens only.** No monetary values now or in the near future. | The tool's purpose is comparing token cost. | | D2 | **No cache-aware cost weighting.** Cache read/write tokens are already stored per request (`requests.cache_read_tokens`, `cache_creation_tokens`) and stay request-level facts. Cache reporting in the new views is **postponed until after the initial implementation**. | Nothing in these plans may allocate cache tokens to blocks. Adding cache columns later is purely additive. | | D3 | **Analysis must keep working when content is purged.** Block rows (hash, type, tokens) are kept forever; only content/raw bodies go. Purged blocks render as greyed nodes with token counts, never hidden. | | -| D4 | **Retention becomes explicit.** Replace implicit "N days at startup" purge with a session lifecycle `active → ended → archived`. Archive is a user action on the session screen, **one-way, with confirmation**. No auto-archive rule in the first version. | See `session-archive.md` (Plan 3b) and `db-compact.md` (Plan 3a). | +| D4 | **Retention becomes explicit.** Replace implicit "N days at startup" purge with a session lifecycle `active → ended → archived`. Archive is a user action on the session screen, **one-way, with confirmation**. No auto-archive rule in the first version. | See `archive/session-archive.md` (Plan 3b) and `archive/db-compact.md` (Plan 3a). | | D5 | **Scope of analysis = per conversation by default, with a switch to whole session**, mirroring the dashboard. Conversation membership **must reuse the existing dashboard/lineage definition** (`db/session_lineage_service.py`, `crud._node_group_key`, `annotated_lineage_nodes`), not a new one. | Lineage breaks (compaction, client restart, uncaptured request) are *events the user wants to see*, not things to hide. | | D6 | **Drop `SESSION_ANALYSIS_PLAN.md`** (deleted from `plans/postponed/`). It predates lineage, conversations and many other changes. Ideas worth keeping from it are folded into the individual plans (typed `json_path` per block, one bulk query instead of N+1, lazy tokenization of the selected block only, controlled JSON viewer that can reveal a path). | `REQUEST_LINEAGE_PLAN.md`'s links to the deleted file were replaced by plain text on 2026-10-06. | -| D7 | **Classification must be extensible**: future work will differentiate tools, MCP servers, subagents, etc. Use coarse stable enums + free-form JSON detail + `classifier_version`, never a growing enum. | See `request-purpose.md`. | -| D8 | **Info panel is the shared surface for actions/insights** on blocks and requests (cost/"present in", compare, etc.). Existing panels (block inspector in Request detail, context-change panel in the conversation view) are enough for now; the tree page gets the panel as its right-hand pane, reacting to node selection. No drawer variant, no new panel framework. | See `info-panel.md`. | +| D7 | **Classification must be extensible**: future work will differentiate tools, MCP servers, subagents, etc. Use coarse stable enums + free-form JSON detail + `classifier_version`, never a growing enum. | See `archive/request-purpose.md`. | +| D8 | **Info panel is the shared surface for actions/insights** on blocks and requests (cost/"present in", compare, etc.). Existing panels (block inspector in Request detail, context-change panel in the conversation view) are enough for now; the tree page gets the panel as its right-hand pane, reacting to node selection. No drawer variant, no new panel framework. | See `archive/info-panel.md`. | | D9 | **Tree/compare pages must visually match the Request detail page.** Earlier attempts looked bad. Focus: tree representation, node icons, collapse/expand animation, lightweight minimal node labels; details live in the info panel. | See `unconfirmed_drafts/context-tree.md`. | -| D16 | **Provider neutrality (user, 2026-10-05):** requests come from Anthropic, OpenAI, Copilot, Ollama, llama.cpp, vLLM and any OpenAI-compatible API. **No decision may be justified by the contents of the author's local DB** (heavily Codex). Agent/provider-specific logic is a pluggable enhancement over a generic baseline; fixtures must cover every adapter. | `analysis-architecture.md` §0. | -| D11 | **Fidelity:** include `opaque`/`partial` requests in all analysis views, show a per-request fidelity badge, label totals "visible tokens". | See `analysis-architecture.md` §4. | -| D12 | **Source/purpose at request and block level.** Persist `blocks.source_key` computed at capture via a generic baseline plus a pluggable parser registry (parsers shipped: Codex `exec`/`js` and JSON-argument `Bash`; everything else name-based, expand with data/feedback from other agents); derive block `activity` from it via a mapping table. | Revises the "derive on read" idea in `request-purpose.md`: args are purged on archive. | +| D16 | **Provider neutrality (user, 2026-10-05):** requests come from Anthropic, OpenAI, Copilot, Ollama, llama.cpp, vLLM and any OpenAI-compatible API. **No decision may be justified by the contents of the author's local DB** (heavily Codex). Agent/provider-specific logic is a pluggable enhancement over a generic baseline; fixtures must cover every adapter. | `archive/analysis-architecture.md` §0. | +| D11 | **Fidelity:** include `opaque`/`partial` requests in all analysis views, show a per-request fidelity badge, label totals "visible tokens". | See `archive/analysis-architecture.md` §4. | +| D12 | **Source/purpose at request and block level.** Persist `blocks.source_key` computed at capture via a generic baseline plus a pluggable parser registry (parsers shipped: Codex `exec`/`js` and JSON-argument `Bash`; everything else name-based, expand with data/feedback from other agents); derive block `activity` from it via a mapping table. | Revises the "derive on read" idea in `archive/request-purpose.md`: args are purged on archive. | | D13 | **`json_path` implemented for all adapters in one change.** | | | D14 | **Retention:** time-based purge defaults to off (`0`), setting kept as legacy; explicit archive is the main path. | Refined by D17: also log a startup notice when a config explicitly enables it. | -| D15 | **Batched schema migration v9** for all new columns (requests purpose fields, blocks `source_key`/`json_path`, sessions `archived_at`). | **Done** (columns, backfill, tests). See `analysis-architecture.md` §5. | -| D17 | **Compaction before archive; incremental auto-vacuum** (user, 2026-10-05). Deleting rows never shrinks a SQLite file (the author's 6.6 GB DB is 65% free pages). `db-compact` (offline VACUUM + converts to `auto_vacuum=INCREMENTAL`) ships first; new databases start incremental; archive then shrinks the file online. | `db-compact.md`. D14 refined: default `0` **plus a startup notice** when a config explicitly enables the purge. | -| D18 | **File paths are stored** (user, 2026-10-06; "the retention policy was a mistake anyway"). Read/edit tool calls get their target file in `blocks.file_path`, persisting after archive; every other argument stays unstored. This reverses WI-0's "program names only" rule for that one field. A single function (`analysis/paths.py: normalize_file_path`) is the only writer, so obfuscation (basename/hash/setting) can be added later if paths become a problem. | `file-paths.md`. | +| D15 | **Batched schema migration v9** for all new columns (requests purpose fields, blocks `source_key`/`json_path`, sessions `archived_at`). | **Done** (columns, backfill, tests). See `archive/analysis-architecture.md` §5. | +| D17 | **Compaction before archive; incremental auto-vacuum** (user, 2026-10-05). Deleting rows never shrinks a SQLite file (the author's 6.6 GB DB is 65% free pages). `db-compact` (offline VACUUM + converts to `auto_vacuum=INCREMENTAL`) ships first; new databases start incremental; archive then shrinks the file online. | `archive/db-compact.md`. D14 refined: default `0` **plus a startup notice** when a config explicitly enables the purge. | +| D18 | **File paths are stored** (user, 2026-10-06; "the retention policy was a mistake anyway"). Read/edit tool calls get their target file in `blocks.file_path`, persisting after archive; every other argument stays unstored. This reverses WI-0's "program names only" rule for that one field. A single function (`analysis/paths.py: normalize_file_path`) is the only writer, so obfuscation (basename/hash/setting) can be added later if paths become a problem. | `archive/file-paths.md`. | | D19 | **Hot-spots cache is postponed** (user, 2026-10-06). 4b ships without it; the grouping is one function (`aggregate_select`) so a cache can wrap it later. | `postponed/hot-spots-cache.md`, GitHub issue #67 (`perf:` prefix for performance issues). | | D20 | **Similarity grouping (changed-and-reloaded blocks, version timelines) is v2** (user, 2026-10-06). Hot spots group by **exact content hash** only. | Plan 8 draft. | -| D21 | **Hot-spots semantics** (architect, agreed): identity = `content_hash`; input blocks only; tokens are *visible* tokens; file rows split `result_tokens` (reads) from `call_tokens` (edits/patch text) because patches are carried too; relative and absolute spellings of one file are **not** unified (the proxy does not see the working directory). | `hot-spots.md`. | +| D21 | **Hot-spots semantics** (architect, agreed): identity = `content_hash`; input blocks only; tokens are *visible* tokens; file rows split `result_tokens` (reads) from `call_tokens` (edits/patch text) because patches are carried too; relative and absolute spellings of one file are **not** unified (the proxy does not see the working directory). | `archive/hot-spots.md`. | | D22 | **Category filter is in the hot-spots UI** (user, 2026-10-06), using the labels the dashboard already has (`ContextBar.tsx`). My first reason for leaving it out (UI must not know the vocabulary) was wrong. | | | D23 | **Sequence after this work** (user, 2026-10-06): merge `analysis_revamp`, produce a release for users to test, then continue the draft plans (5-8) and UI styling fixes (the Hot spots page has styling remarks pending) once the plans are sorted out. | See "Release readiness". | -| D24 | **`compaction_trigger` purpose rule stays** (user, 2026-10-06): a request whose last conversational message contains a provider `compaction_trigger` item (OpenAI Responses) is `compaction`. It is a wire-format signal, not an agent heuristic. Other agents' compaction (e.g. Claude Code's summarisation prompt looks like `user_turn`) is not detected until captures justify an agent detector. | `wi0-data-foundation.md` §17. | -| D25 | **Legacy leaf-form `blocks.json_path` rows stay as they are** (user, 2026-10-06): 6,794 blocks from an earlier prototype in the author's own database; valid paths, only more specific, never overwritten. No release impact. Revisit only if the context tree needs one convention. | `wi0-data-foundation.md` §17. | +| D24 | **`compaction_trigger` purpose rule stays** (user, 2026-10-06): a request whose last conversational message contains a provider `compaction_trigger` item (OpenAI Responses) is `compaction`. It is a wire-format signal, not an agent heuristic. Other agents' compaction (e.g. Claude Code's summarisation prompt looks like `user_turn`) is not detected until captures justify an agent detector. | `archive/wi0-data-foundation.md` §17. | +| D25 | **Legacy leaf-form `blocks.json_path` rows stay as they are** (user, 2026-10-06): 6,794 blocks from an earlier prototype in the author's own database; valid paths, only more specific, never overwritten. No release impact. Revisit only if the context tree needs one convention. | `archive/wi0-data-foundation.md` §17. | | D10 | One plan file per feature. Plans that are confirmed live in `plans/`; plans still being refined live in `plans/unconfirmed_drafts/`. | | ## Repository policies that constrain every plan (from `AGENTS.md`) @@ -57,11 +57,11 @@ serves one of those four questions. - After changing `ui/src/`, rebuild with `make ui`; run `pytest` and `cd ui && npm test`. - Lineage/conversation membership is **derived at read time and never persisted** (`session_lineage_service.py` header). Do not add persisted columns that encode conversation - membership (this is why `turn_id` is *derived*, not stored — see `request-purpose.md`). + membership (this is why `turn_id` is *derived*, not stored — see `archive/request-purpose.md`). ## Architecture review -[`analysis-architecture.md`](analysis-architecture.md) holds measured data facts (sample-only measurements: a +[`archive/analysis-architecture.md`](archive/analysis-architecture.md) holds measured data facts (sample-only measurements: a 4,032-request / 791k-block session shows the scale that can occur), the data-model decisions, API contracts (bounded, run-length occurrences, revision-cached, fidelity-aware) and the batched migration. **Read it before implementing any plan.** Where it conflicts with an individual plan, it wins and the plan must be updated. @@ -99,11 +99,11 @@ Schema is **v10**. Tests at this point: 685 backend, 199 frontend, `npm run chec Facts worth knowing about the data, measured on a copy of the author's database (samples, not rules, D16): upgrading 7.2k requests / 1.5M blocks from schema 8 takes ~2-3 min (v9 117-174 s, v10 alone 104 s); 8.4% of blocks have a `file_path`; hot-spots aggregation ~1.1 s on a 4,032-request session; **the existing conversation (lineage) analysis costs 65 s cold on that session** and every conversation-scoped feature inherits it. -**Not implemented:** Plans 5, 6, 7, 8 (context tree, compare, hints, similarity); `housekeeping` purpose detection and any agent purpose detectors; turn grouping (`turn_index`; specified in `request-purpose.md`, needed by the tree's "by turn" mode); source/file parsers beyond Codex `exec`/`js`, `Bash` and the generic structured tools (no MCP file tools); Ollama tool-call blocks (its adapter emits none, so no sources/paths there); purpose in the request list and conversation cards; jumping from a block's JSON location to the raw JSON viewer; cache reporting (D2); the hot-spots cache (D19). +**Not implemented:** Plans 5, 6, 7, 8 (context tree, compare, hints, similarity); `housekeeping` purpose detection and any agent purpose detectors; turn grouping (`turn_index`; specified in `archive/request-purpose.md`, needed by the tree's "by turn" mode); source/file parsers beyond Codex `exec`/`js`, `Bash` and the generic structured tools (no MCP file tools); Ollama tool-call blocks (its adapter emits none, so no sources/paths there); purpose in the request list and conversation cards; jumping from a block's JSON location to the raw JSON viewer; cache reporting (D2); the hot-spots cache (D19). ## Plans and order -> **Naming:** *Plans 1–7* are the feature plans in the table below. *WI-0* is the data-foundation work item that precedes them; its own sub-steps are called *slices 1–5* inside `wi0-data-foundation.md`. "Slice" never refers to a numbered plan. +> **Naming:** *Plans 1–7* are the feature plans in the table below. *WI-0* is the data-foundation work item that precedes them; its own sub-steps are called *slices 1–5* inside `archive/wi0-data-foundation.md`. "Slice" never refers to a numbered plan. ``` ┌────────────────────┐ @@ -119,13 +119,13 @@ Facts worth knowing about the data, measured on a copy of the author's database | # | Plan | File | Status | Depends on | |---|------|------|--------|-----------| -| 0 | Data foundation: migration v9, capture-time classification, `json_path` | [`wi0-data-foundation.md`](wi0-data-foundation.md) | **implemented and committed** (`528f3db`, `88bd6e0`); not released; no browser check or real-DB `db-upgrade` yet | — | -| 1 | Info panel: "present in" / totals for a block | [`info-panel.md`](info-panel.md) | **implemented and committed** (`153a9d0`); not released, not checked in a browser; shared-pieces extraction was dropped in review | WI-0 | -| 2 | Request purpose & extensible classification | [`request-purpose.md`](request-purpose.md) → implemented by WI-0 | **baseline implemented in WI-0** (`user_turn`, `tool_continuation`, `compaction`, `unknown`; `housekeeping` and agent detectors NOT implemented; UI shows it in Request detail only) | — | -| 3a | `contextspy db-compact` (reclaim free pages, enable incremental auto-vacuum) | [`db-compact.md`](db-compact.md) | **implemented and committed** (`f34fa29`); not released; the author's live DB not yet compacted | — | -| 3b | Session lifecycle & explicit archive | [`session-archive.md`](session-archive.md) | **implemented and committed** (`a2a4b8c`); not released, not checked in a browser; only the `sessions.archived_at` column exists (WI-0), nothing sets it | 3a | -| 4a | Capture the file a block is about (`blocks.file_path`, schema v10) | [`file-paths.md`](file-paths.md) | **implemented and committed** (`d608561`); not released; not seen in a browser | WI-0 | -| 4b | Hot spots (per conversation / per session; by block, source, file) | [`hot-spots.md`](hot-spots.md) | **implemented and committed** (`e6c4396`, category filter `4cb4c84`); not released; profiler-style layout (bars, expandable rows, `layout=name|bar` comparison toggle) implemented, uncommitted, see [`hot-spots-layout.md`](hot-spots-layout.md); cache postponed ([`postponed/hot-spots-cache.md`](postponed/hot-spots-cache.md), issue #67) | 1, 3b, 4a | +| 0 | Data foundation: migration v9, capture-time classification, `json_path` | [`archive/wi0-data-foundation.md`](archive/wi0-data-foundation.md) | **implemented and committed** (`528f3db`, `88bd6e0`); not released; no browser check or real-DB `db-upgrade` yet | — | +| 1 | Info panel: "present in" / totals for a block | [`archive/info-panel.md`](archive/info-panel.md) | **implemented and committed** (`153a9d0`); not released, not checked in a browser; shared-pieces extraction was dropped in review | WI-0 | +| 2 | Request purpose & extensible classification | [`archive/request-purpose.md`](archive/request-purpose.md) → implemented by WI-0 | **baseline implemented in WI-0** (`user_turn`, `tool_continuation`, `compaction`, `unknown`; `housekeeping` and agent detectors NOT implemented; UI shows it in Request detail only) | — | +| 3a | `contextspy db-compact` (reclaim free pages, enable incremental auto-vacuum) | [`archive/db-compact.md`](archive/db-compact.md) | **implemented and committed** (`f34fa29`); not released; the author's live DB not yet compacted | — | +| 3b | Session lifecycle & explicit archive | [`archive/session-archive.md`](archive/session-archive.md) | **implemented and committed** (`a2a4b8c`); not released, not checked in a browser; only the `sessions.archived_at` column exists (WI-0), nothing sets it | 3a | +| 4a | Capture the file a block is about (`blocks.file_path`, schema v10) | [`archive/file-paths.md`](archive/file-paths.md) | **implemented and committed** (`d608561`); not released; not seen in a browser | WI-0 | +| 4b | Hot spots (per conversation / per session; by block, source, file) | [`archive/hot-spots.md`](archive/hot-spots.md) | **implemented and committed** (`e6c4396`, category filter `4cb4c84`); not released; profiler-style layout (bars, expandable rows, `layout=name|bar` comparison toggle) implemented, uncommitted, see [`archive/hot-spots-layout.md`](archive/hot-spots-layout.md); cache postponed ([`postponed/hot-spots-cache.md`](postponed/hot-spots-cache.md), issue #67) | 1, 3b, 4a | | 5 | Context tree page | [`unconfirmed_drafts/context-tree.md`](unconfirmed_drafts/context-tree.md) | **draft** (updated 2026-10-06 with what the code now offers; open questions unanswered) | 1, 2 (needs turn grouping), 4a/4b helpers | | 6 | Request compare | [`unconfirmed_drafts/request-compare.md`](unconfirmed_drafts/request-compare.md) | **draft** (updated 2026-10-06) | 5 | | 7 | Optimisation hints ("carried but dead") | [`unconfirmed_drafts/optimisation-hints.md`](unconfirmed_drafts/optimisation-hints.md) | **idea only** (inputs from 4b now exist; see the draft) | 4b | @@ -165,7 +165,7 @@ Pitfalls found the hard way (each cost real time): - **SQLAlchemy cannot step `PRAGMA incremental_vacuum`** (it closes the result after one step); use `engine.raw_connection()` and `fetchall()`. `auto_vacuum` must be set *before* `journal_mode=WAL` on a new file. - SQLite plans: an aggregate `GROUP BY content_hash` over a join can make the planner scan `idx_blocks_content_hash` (15-100x slower); drive from the scope table with `CROSS JOIN`. Cheap-looking per-click lineage calls cost 1-7 s on long sessions (revision check hashes every block row); cache membership (see `db/block_occurrence_service.py`). - Pysqlite begins transactions lazily at the first DML; to make "measure then delete" consistent, take the write lock first with a harmless write. -- The server refuses to start while a data migration is pending (`contextspy db-upgrade` first); the live database is **still not compacted/archived/upgraded by the agent**, and the author's `blocks.json_path` column already holds 6,794 legacy leaf-form values from an earlier prototype (see `wi0-data-foundation.md` §17). +- The server refuses to start while a data migration is pending (`contextspy db-upgrade` first); the live database is **still not compacted/archived/upgraded by the agent**, and the author's `blocks.json_path` column already holds 6,794 legacy leaf-form values from an earlier prototype (see `archive/wi0-data-foundation.md` §17). Where things stand at the end of the 2026-10-06 session: WI-0 and Plans 1, 3a, 3b, 4a, 4b are implemented and committed (not released, not seen in a browser by the author); Plans 5, 6, 7, 8 are drafts/ideas. Open decisions waiting for the user: answers to Plan 5's seven open questions (deliberately deferred until work on that plan starts); a decision on the hot-spots groupings idea (Plan 9: **the user asked to be prompted for it when work on that plan starts**; the schema-timing question before the release is also still open); whether to do something about the 65 s cold conversation analysis (issue #68, draft plan) and the slow request-listing query (issue #69, a small covering-index fix). The user's plan (D23): merge the branch, release for testing, then continue the drafts and UI styling. diff --git a/plans/analysis-architecture.md b/plans/archive/analysis-architecture.md similarity index 99% rename from plans/analysis-architecture.md rename to plans/archive/analysis-architecture.md index 4ab9d41..d8a5b25 100644 --- a/plans/analysis-architecture.md +++ b/plans/archive/analysis-architecture.md @@ -1,6 +1,6 @@ # Analysis features: architecture review and data-model decisions -Status: architect review, 2026-10-05; **its decisions are implemented** (WI-0, Plans 1, 3, 4; see "What changed after implementation" at the end). Companion to [ANALYSIS_ROADMAP.md](ANALYSIS_ROADMAP.md). The measurements below are 2026-10-05 samples. +Status: architect review, 2026-10-05; **its decisions are implemented** (WI-0, Plans 1, 3, 4; see "What changed after implementation" at the end). Companion to [../ANALYSIS_ROADMAP.md](../ANALYSIS_ROADMAP.md). The measurements below are 2026-10-05 samples. Items marked **[PROPOSED]** await user confirmation; items marked **[DECIDED]** follow from roadmap decisions D1–D14 or are forced by measured data. Section 5 lists the schema changes as one batched migration. User answers of 2026-10-05 (fidelity, tool source, json_path order, retention) are folded in below. diff --git a/plans/db-compact.md b/plans/archive/db-compact.md similarity index 99% rename from plans/db-compact.md rename to plans/archive/db-compact.md index 6328f51..ba170b4 100644 --- a/plans/db-compact.md +++ b/plans/archive/db-compact.md @@ -1,7 +1,7 @@ # Plan 3a: `contextspy db-compact` — reclaim free space, enable automatic shrinking Status: **implemented 2026-10-06, committed in `f34fa29`; not released; the author's live database has NOT been compacted.** See "Implementation status" at the end. First half of the former "Plan 3: session archive"; the second half is -[session-archive.md](session-archive.md) (Plan 3b), which depends on this. Part of [ANALYSIS_ROADMAP.md](ANALYSIS_ROADMAP.md). +[session-archive.md](session-archive.md) (Plan 3b), which depends on this. Part of [../ANALYSIS_ROADMAP.md](../ANALYSIS_ROADMAP.md). ## Why (measured, author's database, 2026-10-05; a sample, not a rule) diff --git a/plans/file-paths.md b/plans/archive/file-paths.md similarity index 99% rename from plans/file-paths.md rename to plans/archive/file-paths.md index 452f5ef..af48e29 100644 --- a/plans/file-paths.md +++ b/plans/archive/file-paths.md @@ -1,6 +1,6 @@ # Plan 4a: Capture the file a block is about (`blocks.file_path`) -Status: **implemented 2026-10-06, committed in `d608561`; not released; not seen in a browser. See "Implementation status" at the end.** Prerequisite of [hot-spots.md](hot-spots.md) (Plan 4b). Part of [ANALYSIS_ROADMAP.md](ANALYSIS_ROADMAP.md). +Status: **implemented 2026-10-06, committed in `d608561`; not released; not seen in a browser. See "Implementation status" at the end.** Prerequisite of [hot-spots.md](hot-spots.md) (Plan 4b). Part of [../ANALYSIS_ROADMAP.md](../ANALYSIS_ROADMAP.md). ## Decision (user, 2026-10-06) and its consequence Hot spots should be able to say "this file's contents are carried N times and cost T tokens". That needs the file a read/edit tool targeted. The user decided to **store full file paths** diff --git a/plans/hot-spots-layout-handoff.md b/plans/archive/hot-spots-layout-handoff.md similarity index 98% rename from plans/hot-spots-layout-handoff.md rename to plans/archive/hot-spots-layout-handoff.md index 5d7e2cc..8f4f33b 100644 --- a/plans/hot-spots-layout-handoff.md +++ b/plans/archive/hot-spots-layout-handoff.md @@ -2,7 +2,7 @@ # Hand-off: Hot spots page layout work (written 2026-10-06, before a context compaction) -Read this first when picking up the **layout/styling of the Hot spots page**. It is working notes, not a plan; the plan and its implementation record are [hot-spots.md](hot-spots.md), the overall state is [ANALYSIS_ROADMAP.md](ANALYSIS_ROADMAP.md) ("Maintenance notes", "Release readiness"). +Read this first when picking up the **layout/styling of the Hot spots page**. It is working notes, not a plan; the plan and its implementation record are [hot-spots.md](hot-spots.md), the overall state is [../ANALYSIS_ROADMAP.md](../ANALYSIS_ROADMAP.md) ("Maintenance notes", "Release readiness"). ## What you are doing and the ground rules - The author will give **styling remarks** for the Hot spots page; the page is functionally complete and committed (`e6c4396`, category filter `4cb4c84`) but **the author's first look at it in a browser produced the remarks**, so treat the current look as a first draft. Ask for / wait for the remarks; do not invent a redesign. diff --git a/plans/hot-spots-layout.md b/plans/archive/hot-spots-layout.md similarity index 95% rename from plans/hot-spots-layout.md rename to plans/archive/hot-spots-layout.md index 7540ec2..3d4a518 100644 --- a/plans/hot-spots-layout.md +++ b/plans/archive/hot-spots-layout.md @@ -1,6 +1,6 @@ # Plan: Hot spots page layout (profiler-style bars) — DRAFT for review -Status: **implemented, uncommitted, awaiting the author's browser review** (2026-10-06). Decisions below were confirmed: `summary.occurrences_total` added (all block instances of the filtered scope, the same population as the token total), header click toggles the sort, `dropped` badge stays on the collapsed row, 2 px minimum bar. Both label variants are built behind `layout=name|bar`; removing the loser is planned in [postponed/hot-spots-label-layout-removal.md](postponed/hot-spots-label-layout-removal.md); the release ships with both. Follows [hot-spots-layout-handoff.md](hot-spots-layout-handoff.md); the data/API plan is [hot-spots.md](hot-spots.md). +Status: **implemented, uncommitted, awaiting the author's browser review** (2026-10-06). Decisions below were confirmed: `summary.occurrences_total` added (all block instances of the filtered scope, the same population as the token total), header click toggles the sort, `dropped` badge stays on the collapsed row, 2 px minimum bar. Both label variants are built behind `layout=name|bar`; removing the loser is planned in [../postponed/hot-spots-label-layout-removal.md](../postponed/hot-spots-label-layout-removal.md); the release ships with both. Follows [hot-spots-layout-handoff.md](hot-spots-layout-handoff.md); the data/API plan is [hot-spots.md](hot-spots.md). Rules from the hand-off still apply: D9 (same tokens/classes as Request detail), current colours and styling kept, presentation only, the user commits. Everything in "Earlier observations" of the hand-off (repeated %, icons, path truncation, mobile) is **deferred** until this change is in; re-check which still apply afterwards. @@ -65,7 +65,7 @@ Only one row open at a time? **No**: independent, local state per row (`Set` of - `ui/src/api/client.ts`: `summary.occurrences_total` (if accepted). - Backend (only if accepted): `db/hotspots_service.py`, `tests/test_hotspots.py`, `SPEC.md`. - `HotSpots.test.tsx`: update (see below). `SessionDetail.test.tsx` should not change. -- Docs: `plans/hot-spots.md` (layout section), roadmap status line, handoff file superseded note, `docs/changelog.md`. +- Docs: `plans/archive/hot-spots.md` (layout section), roadmap status line, handoff file superseded note, `docs/changelog.md`. ## Tests that change deliberately - Row buttons named `Open the latest request carrying