Local agent: latency, context isolation, reliable lists

docs/design/agent/review-2026-09.md

@@ -0,0 +1,224 @@
+# Agent review, September 2026
+
+Status: assessment. No code changed.
+
+A review of the agent stacklet against its own targets (plan.md: median < 8 s,
+p95 < 20 s), triggered by observed 3-5 minute responses to simple questions,
+unreliable todo handling, and context pollution across threads and speakers.
+Covers: root causes in the current code, the 2025/26 state of the art it should
+follow, and a staged first iteration. Two experiments are scoped at the end:
+a containerised pi agent and an established OSS memory layer (Hindsight).
+
+## 1. Why responses take minutes
+
+No single bug. Five multipliers stack, all contending for one 35B MoE
+(Qwen3.6-35B-A3B-UD-MLX-4bit) on one Mac:
+
+| # | Multiplier | Where | Effect |
+|---|---|---|---|
+| 1 | Nested LLM call inside retrieval | `stacklets/memory/cli/search.py:182-215` -> `rewrite.py` | Every multi-word `memory_search` runs a second chat completion (keyword rewrite) on the same GPU before the search even walks the vault. `grep` under `vault/` is silently rerouted into the same path (`runtime/grep_tool.py:16-85`), and `AGENTS.md:4` tells the model grep is the primary lookup. |
+| 2 | KV cache self-invalidation | `runtime/lean_state.py:64-103` | Prior tool results are retroactively rewritten to `[prior result of ...]` placeholders. The message list at turn N+1 diverges from turn N at the first prior tool message, so oMLX re-prefills from that point on every turn that follows a tool-using turn. The late-brief injection work (`sitecustomize.py:112-124`, measured 4096/4746 tokens cached) is largely given back here. |
+| 3 | Iteration count | `config.json:17` | `max_tool_iterations: 12`, each iteration re-sends the ~10K-token prefix plus history. No total-turn budget exists. |
+| 4 | Tool transport | container -> TCP socket -> famstack-api -> fresh `./stack` interpreter -> `docker exec` into bot-runner | Five hops, two cold Python starts, per tool call. Timeout ladder: 60 s (rewrite) < 120 s (host CLI) < 125 s (socket) < 130 s (tool). |
+| 5 | Event-loop blocking | `runtime/vault_write.py:87-95` (sync `subprocess.run`, up to 120 s), `runtime/brief.py:64-74` (git log per turn, 5 s) | Blocks Matrix sync and the agent loop while running. |
+
+Prefill economics make #2 and #3 fatal on their own: prefill on Apple Silicon
+is compute-bound, roughly 7.6 s per 1K tokens at the small end of our own
+measurements (`lib/stack/ai/client.py:169-213` comment), and oMLX's paged
+prefix cache only helps when the prompt is byte-stable and append-only.
+"Failed tool calls" are a real but secondary contributor: the honest-exit-code
+work already fixed the worst retry loops; the transcripts now show flailing
+(15 `read_file` probes for one path) caused by four competing retrieval
+routes, not by protocol errors.
+
+## 2. Why context pollutes
+
+1. **Session key is the room id.** All threads and all speakers in a family
+   room share one transcript (`addressing.md:68-70` flags it; the designed fix,
+   folding the thread root into the session key, never shipped). DMs are clean
+   only because a DM is its own room.
+2. **Model-writable prefix.** nanobot's `USER.md` and `memory/MEMORY.md` sit
+   inside the cached prefix and the model can rewrite them. Any self-edit both
+   pollutes every future conversation and permanently invalidates the cache.
+3. **Partial decay.** `lean_state` placeholders only cover tool results. An
+   answer recited from the injected brief has no tool call to anchor decay, so
+   stale data is indistinguishable from conversation (post-mortem in
+   `pointer-memory.md:63-70`).
+
+The Chroma context-rot data (all 18 tested models degrade monotonically with
+input length, distractors actively mislead) makes isolation a quality fix,
+not just hygiene.
+
+## 3. Why we cannot see any of this
+
+- The only prompt record is `llm-state.log`: one line per message, clipped at
+  320 chars, arguments dropped. Shape and ordering only, never the bytes sent.
+- `resp.usage` is read nowhere in production code. We have no token counts, no
+  cached-token counts, no per-call latency, no iteration index. The KV cache
+  hit rate, the single most important production-agent metric (Manus, 2025),
+  is invisible.
+
+## 4. What the field says (research summary, Sept 2026)
+
+- **Context engineering consensus** (Manus 07/2025, Anthropic 09/2025,
+  unchanged into 2026): byte-stable prefix, fixed tool list, append-only
+  history, no timestamps or dynamic content early in the prompt, compaction
+  only at natural boundaries. baibot's trick applies to us: inject
+  conversation *start* time once, never "now" per request.
+- **oMLX already has the machinery**: vLLM-style paged prefix cache with SSD
+  tier. Reported TTFT 1-3 s on long contexts when prompts are append-only.
+  Nothing on our side configures or measures it.
+- **Memory**: markdown files + grep + a nightly consolidation pass is the
+  proven pragmatic tier (Claude Code pattern; Letta's sleep-time-compute
+  validates background consolidation, +18% accuracy in their study). Heavy
+  graph systems (Zep CE is discontinued; Graphiti needs a graph DB) are
+  overkill for a family. The one established system worth a trial here is
+  **Hindsight** (see section 7).
+- **Two-model router**: evidence leans against it as a first move. Docker's
+  2025 eval (21 models, 3570 cases) found generalist Qwen3 8B/14B beat
+  dedicated tool-calling models (Qwen3-14B F1 0.971 vs Watt-Tool-8B 0.484).
+  A second resident model costs RAM and splits the KV cache on one Mac. Our
+  latency is a cache/prefill problem, not a model-size problem. Revisit only
+  if triage latency still hurts after the cache fixes (candidates then:
+  xLAM-2, Hammer, Qwen3.5-4B).
+- **Minimal harnesses**: the field moved to tiny loops with owned context
+  (pi, smolagents, nanobot itself). Our nine monkeypatches over pinned
+  nanobot 0.2.2 internals are the opposite of owning the loop; ADR-012
+  records the failure mode (shims fail soft, tests stay green while
+  production breaks).
+
+## 5. First iteration
+
+Ordered so each step is measurable before the next. Steps 1-3 are small
+changes to the existing stacklet; no architecture swap yet.
+
+### Step 1: see the context (observability)
+
+- Dump the exact request: full `messages` JSON per LLM call to a per-room
+  JSONL under `{data_dir}/agent/prompts/`, with turn id, iteration index, and
+  wall time. Replaces guessing with bytes.
+- Read `resp.usage` on every call and log prompt tokens, completion tokens,
+  and cached tokens if oMLX reports them. Log TTFT.
+- Success check: for two consecutive turns in one room, the second request's
+  prompt is a byte-prefix of the first plus appended messages, and cached
+  tokens ~= previous prompt tokens.
+
+### Step 2: stop breaking the cache, stop stacking calls
+
+- Delete the `lean_state` retroactive rewrite. History becomes append-only.
+  Accept larger transcripts; compact only at a natural boundary (size
+  threshold), which costs one deliberate cache reset instead of one per turn.
+- Drop the NL-rewrite hop from the agent's search path: `memory_search` runs
+  the literal/regex search directly; the model itself already chooses
+  keywords. Keep `--nl` for human CLI use.
+- One retrieval route. Un-instruct grep-then-read in `workspace/AGENTS.md`,
+  align `SKILL.md`, keep `memory_search`/`memory_person`/`memory_history` as
+  the only documented path. Fixes the flailing transcripts and the doc drift
+  in one move.
+- `max_tool_iterations` 12 -> 6, plus a total turn budget (fail loud into the
+  room rather than grind for five minutes).
+- Async the blocking calls (`vault_write` subprocess, brief git log; cache the
+  git log line for a few minutes).
+- Freeze the prefix: seed `USER.md`/`MEMORY.md` read-only or route nanobot's
+  self-memory writes outside the prompt prefix.
+
+### Step 3: isolate conversations
+
+- Session key: `(room_id, thread_root)` for threads, `(room_id, sender)` in
+  multi-user rooms outside threads, `room_id` for DMs. This is the already
+  designed but unshipped step from addressing.md layer 2.
+- Cross-conversation knowledge flows only through the vault (write in one
+  thread, `memory_search` in another), never through a shared transcript.
+
+### Step 4: measure
+
+Re-run the plan.md targets with the step-1 logging in place: median < 8 s,
+p95 < 20 s for no-tool turns; one-tool turns < 30 s. Only if this fails does
+the router idea (section 4) come back on the table.
+
+## 6. Experiment A: containerised pi agent
+
+Assessment: worth running. pi (earendil-works, ex badlogic/pi-mono, MIT,
+v0.85.1 as of 2026-09-05) is a maintained minimal harness that gives us,
+out of the box, most of what the shim stack fights for:
+
+- **Headless operation**: `pi -p` one-shot, `--mode json` (JSONL events), and
+  `--mode rpc` (long-lived process, JSONL over stdin/stdout, `prompt` /
+  `steer` / `abort` / `switch_session`). No TTY needed. RPC fits a
+  per-conversation resident process better than spawn-per-message.
+- **Sessions are files**: JSONL trees under a mountable dir
+  (`PI_CODING_AGENT_SESSION_DIR`). One session per (room, thread) key gives
+  the step-3 isolation for free, inspectable with `jq`.
+- **oMLX is a config entry**: `models.json` provider with
+  `baseUrl: http://host.docker.internal:8888/v1`, `api: openai-completions`,
+  explicit `contextWindow`. No code.
+- **Cache-friendly by design**: system prompt < 1K tokens, append-only
+  sessions, compaction is the only prefix-invalidating event.
+- **Extensible without monkeypatching**: `pi.registerTool()` for
+  `memory_search` etc. (TypeScript extensions), AGENTS.md and the Skills
+  standard supported natively, `.pi/SYSTEM.md` replaces the system prompt.
+- **Containerisation**: documented pattern, `node:24-bookworm-slim` base,
+  named volume for `/root/.pi/agent`, `PI_OFFLINE=1`. No official image, so
+  we build and pin our own line per house rules.
+
+Experiment shape (test rig, not production): a thin Python bridge keeps one
+`pi --mode rpc` process per conversation key, forwards Matrix messages as
+`prompt` commands, posts `message_end` events back. Vault mounted read-only;
+the write path stays `stack memory write` via exec. Compare against the
+nanobot stacklet on the same prompts: TTFT, cached tokens, turn latency,
+retrieval sanity.
+
+Risks: Node runtime in the stack, tool extensions are TypeScript (new
+language surface), Matrix bridging is ours to write (pi has no channel
+concept), project is young even if very active.
+
+Decision point: if steps 1-3 get nanobot to target latency, pi remains a
+bench experiment. If the shim stack keeps fighting us (ADR-012), pi becomes
+the migration target instead of a nanobot fork.
+
+## 7. Experiment B: established OSS memory layer (Hindsight)
+
+Candidates reviewed: Mem0 OSS, Letta, Zep CE/Graphiti, LangMem, Hindsight,
+cognee, Memvid. Verdicts:
+
+| System | Verdict |
+|---|---|
+| Hindsight (vectorize-io, MIT, v0.10) | Trial first. One container + pgvector. The only candidate with published open-model benchmarks (83.2% LoCoMo with a 20B open model). In-process local embeddings, so oMLX only serves chat. Banks map to per-person/per-room isolation. Markdown projection matches the vault philosophy. |
+| Mem0 OSS (Apache-2.0) | Fallback. Cleanest `add/search(user_id)` API, but independently measured OSS quality (32.4% LongMemEval) trails its marketing, and small-model JSON extraction is its documented failure mode. Default constructor silently phones OpenAI; must be configured away. |
+| Graphiti (Zep CE is discontinued) | Only if temporal graph semantics prove necessary. Needs a graph DB; small-model JSON extraction is fragile. |
+| Letta | Skip. It is an agent runtime, not a memory sidecar; wrong shape next to nanobot/pi. |
+| LangMem | Skip. Pre-1.0 library that drags the LangChain tree in; closer to hand-rolled than established. |
+| cognee, Memvid | Wrong shape (document ETL / storage format). |
+
+Trial shape (shadow mode, test rig): run Hindsight (pinned 0.10 line) +
+pgvector alongside the vault. Feed the same room transcripts into banks
+(`person:<name>`, `room:<id>`, `family:shared`) via `retain()` in the
+background; at question time call `recall()` in parallel with the existing
+`memory_search` and log both result sets. Judge after two weeks: recall
+quality on German family content (the default bge-small-en-v1.5 embedder is
+English-only; configure `intfloat/multilingual-e5-small`), retain throughput
+on the oMLX model, and whether the markdown projection is usable next to the
+vault. The vault stays the source of truth either way; Hindsight would be
+the retrieval/consolidation layer, not the store.
+
+## 8. Deferred and cleanup
+
+Deferred: two-model router (decision gate in step 4); nanobot 0.3 migration
+(the 0.2.2 pin is the current safety mechanism; the handover doc predates the
+pin and was never executed).
+
+Doc drift found during review, worth a cleanup pass:
+
+- `docs/design/handover/nanobot-0.3-shim-migration.md`: untracked, stale,
+  superseded by the pin eight minutes after it was written. Reads as
+  current-and-broken.
+- `docs/design/agent/plan.md`: specifies MicroBot bridge, tools-server
+  endpoint, agent-notes repo, and the invariant "the agent never mutates
+  family/memory", all overtaken by what shipped.
+- `workspace/AGENTS.md` and `SKILL.md` disagree on the primary retrieval
+  route and neither names the `memory_search` tool (fixed by step 2).
+- `runtime/README.md` says "two independent shims"; sitecustomize carries
+  nine.
+- `todo-strike-flaky.md` reads as an open proposal but its alternative was
+  implemented.
+- `runtime/__pycache__/` and `hooks/__pycache__/` are present in the tree.

lib/stack/links.py

@@ -108,6 +108,38 @@ def go_person(slug: str, leaf: str | None = None) -> str:
     return _entity_path("person", slug, leaf)
 
 
+def go_page(repo_path: str) -> str:
+    """Logical link for a vault page path, or "" when no stable route exists.
+
+    Edit verbs answer with the page they touched. The reply carries this
+    link so the family can open the page in the wiki. Only pages with a
+    resolver route get a link; anything else returns "" rather than a
+    path-keyed link that breaks on the next rename.
+
+    >>> go_page("family/camping/todos.md")
+    '/topic/family/camping/todo'
+    >>> go_page("family/camping/about.md")
+    '/topic/family/camping'
+    >>> go_page("homer/about.md")
+    '/person/homer'
+    >>> go_page("homer/notes/2026/09/errands.md")
+    ''
+    """
+    parts = [p for p in repo_path.strip("/").split("/") if p]
+    if len(parts) < 2:
+        return ""
+    if parts[-1] == "todos.md":
+        leaf = "todo"
+    elif parts[-1] == "about.md":
+        leaf = None
+    else:
+        return ""
+    scope = "/".join(parts[:-1])
+    if len(parts) == 2 and parts[0] != "family":
+        return go_person(parts[0], leaf)
+    return go_topic(scope, leaf)
+
+
 def _entity_path(kind: str, scope: str, leaf: str | None) -> str:
     """Join `kind`, the scope segments, and an optional leaf into a path.
 

stacklets/agent/config.json

@@ -14,7 +14,9 @@
   "agents": {
     "defaults": {
       "model_preset": "primary",
-      "max_tool_iterations": 12
+      "max_tool_iterations": 12,
+      "max_messages": 40,
+      "disabled_skills": ["memory", "my"]
     }
   },
   "tools": {

stacklets/agent/entrypoint.sh

@@ -19,6 +19,11 @@ for f in SOUL.md AGENTS.md; do
     [ -f "$NB/workspace/$f" ] || sed "s/__AGENT_NAME__/$NAME/g" "/app/workspace-seed/$f" > "$NB/workspace/$f"
 done
 
+# Seed a minimal USER.md so nanobot does not install its default profile
+# template. The template costs ~1K prompt characters per request and holds
+# no data: the runtime brief carries the per-turn speaker instead.
+[ -f "$NB/workspace/USER.md" ] || printf '# User Profile\n\nThe runtime brief names the current speaker each turn. No data here.\n' > "$NB/workspace/USER.md"
+
 # Product-defined skills always refresh from the image seed (not user-editable).
 if [ -d /app/workspace-seed/skills ]; then
     mkdir -p "$NB/workspace/skills"