feat(hook-context-intelligence): self-healing event replay for undelivered events

[Diego Colombo (colombod)]

Summary

Fixes microsoft/amplifier#403. Against a remote destination the telemetry hook silently
failed to deliver the large majority of a session's events while reporting itself healthy.
Measured on one workstation over 25 days (232 shutdown_undelivered records, each matched
to its session's events.jsonl): 239,690 events produced, 54,145 never delivered
(22.6%); the median session lost 55.5% of its events; 128 of 232 sessions lost half or
more; the worst lost 719 of 720. breaker_open=False on 232 of 232 — the destination was
healthy every single time. _DestinationDispatcher posts serially, so its ceiling is one
event per round-trip; a busy session out-produces it all day, the queue hits
dispatch_queue_capacity and sheds.

Worse than the raw rate suggests: the queue drops the newest event on overflow, so what
goes missing is each session's tail — session:end, final tool calls, outcomes. The
graph ended up full of sessions that started and never finished.

This adds a per-session, per-destination byte watermark into the already-durable
events.jsonl, plus a bounded, continuous catch-up sweep that advances it. Data loss
becomes delivery latency: what the live path drops is swept later, automatically.

Builds on #111 (close_drain_timeout 10→20), which addressed short tails only and said so.

Scope / guardrails

• Seam crossed — deliberately. Client↔server boundary, networking, and hook config
  wiring. Proven with a real DTU run (below), not mocks.
• Fan-out routing untouched. The sweep sends only where the live path already would,
  reusing fanout.destination_is_active. No new routing decisions.
• No import of the upload tool. The dependency arrow is tool → hook; the reverse would
  be circular. The sweep is self-contained because build_payload already lives in the hook.
• sweep_concurrency ships at 1 — exactly today's behaviour. The knob exists; raising
  it is a separate, measured change that must first settle breaker-window ownership.
• context-intelligence-recover (new on main) is untouched and unreferenced. See Notes.

Verification

• Module tests pass — modules/hook-context-intelligence: 762 passed
  (the module this change lives in)
• Top-level tests pass — tests/: 854 passed
• ruff check — All checks passed! · ruff format --check — 183 files already formatted
• pyright — 0 errors, 0 warnings, 0 informations
• Full bundle validation — scripts/validate-full.sh: 12/12 bundles load clean,
  0 errors across discovery/hygiene/structure/placement/packaging. The lone
  unadvertised_but_referenced mode ERROR is the documented FALSE POSITIVE per
  AGENTS.md — not acted on.

Real evidence on seams (not mock-only)

• Seam proven against real behaviour.

New DTU profile context-intelligence-self-healing-replay-validation.yaml, run against a
real context-intelligence-server 6.7.0 + real Neo4j 5.26.22 (this repo's own
context-intelligence-backend.yaml), with a latency-injecting proxy (250 ms/request,
support/self-healing-replay/latency_proxy.py) in front, driving real amplifier
sessions with the kernel-mounted hook, installed from the branch via Gitea. Readiness
7/7, including sweep-defaults-on verified live.

Scenario	Result
S1 reproduce	26 records, only 22 :Event on the server; watermark 127983/129234 — short of EOF, tail undelivered
S2 heal	a later session's sweep took the watermark to EOF, no user action
S3 duplicate-free	session nodes 25 → 25 after a full forced replay of all 26 records
S4 outage stays loud	dead port: delivered=0, watermark stayed 0, last_outcome=no_progress, real ConnectError recorded
S5 age bound audible	7-day-old session not sent, reported stranded at 168h

The duplicate check is asserted on Neo4j node counts scoped per session, not on
{"status":"duplicate"} responses: idempotency_key is an in-memory 7-day/100k LRU that
?replay=true bypasses and a restart clears. The durable contract is the deterministic
node_id + (node_id, workspace) uniqueness with MERGE (verified in the server source at
739022c). The sweep therefore deliberately does not set replay=true.

Exit cost, measured in both states — the constraint this change amends:

no sweep mid-delivery (normal, incl. nothing to do) : 0.000s
a sweep IS mid-delivery                             : 2.002s   (= sweep_close_grace_seconds)
sweep_close_grace_seconds: 0.0                      : 0.000s

Docs & diagrams

• bundle.dot / bundle.png — N/A — no bundle structure change (two new handler
  modules inside an existing module; the validator reported no BUNDLE_DOT_STALE).
• README updated — seven sweep_* config keys, and a Self-healing backlog sweep
  section covering the watermark, the bounds, and the exit budget.
• docs/remote-server-troubleshooting.md — the shutdown: N undelivered entry's
  Case 2 now describes automatic recovery and how to confirm it from the watermark
  file; new entries for the stranded-events warning and for "why did exit take a
  couple of seconds longer".
• AGENTS.md — new section on scheduling defects, because this work is a worked
  example of why the seam gate exists (see Notes).

Notes / follow-ups

1. Three defects shipped past a fully green 740-test suite and were caught only by the
   DTU, all the same shape — work performed but not recorded, or a wait bounded on the
   wrong thing: (a) the sweep never ran, cancelled at teardown before its first round-trip;
   (b) cancellation discarded a pass's completed deliveries; (c) the fix for (a) waited on a
   task that never completes, adding a flat 2.003 s to every exit. A fourth was a test
   defect of the same family — asserting duplicate-freedom on a workspace-wide node count
   while the test itself created new sessions, a confident false FAIL. The lasting lesson is
   captured in AGENTS.md.

2. Relationship to context-intelligence-recover (landed on main during this work).
   Complementary, not duplicative, and nothing in the hook references it: the sweep is
   automatic, in-process, bounded to a 48h window and posts to /events; recover is an
   operator CLI with a stronger SQLite checkpoint that posts to /recovery/events behind a
   native-recovery capability. The natural division — the sweep stops the bleeding,
   recover cleans up history — is borne out by the same workstation: 38 sessions /
   2,686 events sit inside the sweep's window, and 191 sessions of historical loss sit
   outside it and will only ever be recovered by the CLI. Three follow-ups deserve their
   own issue: should the sweep prefer /recovery/events when advertised; adopt recover's
   stronger truncation detection (prefix hash + inode, vs this PR's size-only guard); and
   name an owner for the two byte-offset cursors now tracking the same file.

3. Not yet shipped from the design: the cross-session backlog-growth warning (needs
   state outliving one session), raising sweep_concurrency above 1 (needs breaker-window
   ownership settled first), and deleting the in-memory queue's role as source of truth —
   the subtraction this work makes possible.

4. Untested seam: set_ingestion_filters mid-session swaps cancel and restart sweeps
   correctly by construction, but no DTU scenario exercises that path. Most significant
   gap remaining.

Generated with Amplifier

[Diego Colombo (colombod)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts by merging origin/main and preserving both the updated shutdown/backlog guidance from main and this PR’s self-healing sweep docs. Addressed in merge commit 3644011.

[Diego Colombo (colombod)]

DTU evidence: all six scenarios green against real infrastructure

Addressing the "not proven end-to-end" gap. Run against a real
context-intelligence-server 6.7.0 + real Neo4j 5.26.22 (this repo's own
context-intelligence-backend.yaml), with a latency-injecting proxy at 250 ms/request
in front, driving real amplifier sessions with the kernel-mounted hook, branch
installed through the CLI from a Gitea mirror at 2b9b8b7. No mock on the seam.

Scenario	Result
S1 reproduce the shortfall	27 local records, only 17 :Event on the server; watermark 131301/135477 — short of EOF, tail undelivered
S2 a later session heals it	watermark reached EOF (135477), no user action, no warning
S3 replay is duplicate-free	session nodes 26 → 26 after a full forced replay; watermark accumulated 0 → 129925 → 135241 → 135477 across sessions
S4 a real outage stays loud	dead port: delivered=0, watermark stayed 0, last_outcome=no_progress, real ConnectError recorded
S5 the age bound is audible	7-day-old session not sent, reported stranded at 168h
S6 the sweep obeys include/exclude	excluded workspace: 0 nodes. Permitted: 16. No watermark written for the excluded session — it was never opened

S6 output verbatim:

=== S6: the sweep MUST obey each session's own include/exclude ===
planted /work/public (allowed) and /work/client-x (EXCLUDED)
delivered=15 failed=0 filtered=1 blocked=0 err=None
--- server truth: the EXCLUDED workspace must be empty ---
{"nodes": 0, "events": 0, "workspace": "heal-133218-secret"}
{"nodes": 16, "events": 15, "workspace": "heal-133218-allowed"}
PASS: excluded workspace has 0 nodes; permitted has 16
PASS: no watermark written for the excluded session -- never opened

Both sessions sit in one project directory — the collision that makes ignoring the
filter a leak rather than an inefficiency.

Harness defects found and fixed while doing this (ace29f0)

S1–S3 passed first time; S4–S6 did not, and every failure was in the harness, not the
code. Two are worth calling out because one of them was silently dangerous:

• S5 would have passed for the wrong reason. It carried a hardcoded bearer token from
  an earlier run, and it asserts zero delivered to prove the age bound — a bad token
  produces zero as well. It was green-by-accident and proving nothing. Both S5 and S6 now
  read SERVER_TOKEN, and S6 asserts events_failed == 0 before asserting the routing
  result, so a delivery fault can never again masquerade as a filter decision.
• S6 planted into a fixed project directory, so a re-run found the permitted session
  already swept and reported delivered=0. Unique per invocation now — the same
  "reused namespace manufactures a false result" trap already fixed for the workspace name.

Also: S4/S5 constructed BacklogSweeper without the now-required spec; _common.sh
sourced its env without exporting, so python children never saw it; and the
new-handlers-importable readiness check used a bare PYTHONPATH import that cannot
satisfy pathspec (amplifier resolves module deps at session load, not into its own
interpreter). That check now runs under uv run --with and additionally asserts the
routing gate exists and that spec is a required argument — so a sweeper that cannot
evaluate include/exclude fails readiness rather than shipping.

Gates

780 module tests · 857 root tests · ruff clean · pyright 0 errors from both the repo root
and the module directory · validate-full.sh 12/12 bundles clean (the lone
unadvertised_but_referenced mode ERROR is the documented false positive per AGENTS.md).

Resource hygiene

Both DTUs (ci-s6, ci-s6-backend) destroyed; host returned to its pre-run baseline.

[Diego Colombo (colombod)]

Update pushed — head 04cf76af8a501154245b044f82399f7f4eef2440, rebased onto a80ce69, all checks green.

What this PR contributes

Against a remote destination, the telemetry hook delivered events serially — one per round-trip — so a busy session out-produced it, the in-memory queue hit capacity, and it shed the newest event. What went missing was each session's tail: session:end, final tool calls, outcomes. The graph filled with sessions that started and never finished, while the destination reported itself healthy throughout.

This adds a per-session, per-destination byte watermark into the already-durable events.jsonl, plus a bounded, continuous catch-up sweep that advances it. Data loss becomes delivery latency: what the live path sheds is swept later, automatically, with no user action.

What changed in this update

Permanent rejections no longer stall recovery. The sweep now uses the live dispatcher's own HTTP outcome classifier instead of treating every non-2xx as a stop. A record the server will never accept (403, 400, 404, 410, 413, 422, 3xx) is stepped over and recorded; a retryable failure (5xx, 429, 408, 401) still halts the pass.
Improves: previously one permanently-rejected record parked the cursor in front of it, so every later pass re-read it and advanced nothing — leaving the rest of that session's backlog unrecovered until it aged out of the window. Importing the classifier rather than re-deriving it also means the two delivery paths cannot disagree about what is retryable.

Per-session progress is now reported, not just per-pass. A new sessions_no_progress counter and an accompanying warning fire even when the pass as a whole delivered.
Improves: progress was summed across the whole sweep, so a session retiring nothing stayed hidden behind any other session that delivered. That is the same "reports healthy while losing data" shape this change exists to remove.

Sweep outcomes now reach the durable diagnostics log. sweep_permanent_reject and sweep_no_progress records are written to the same per-day forwarding-*.jsonl the dispatcher uses, tagged "source": "backlog_sweep".
Improves: sweep health was console-only, which is nowhere in a non-interactive session. docs/remote-server-troubleshooting.md documents both record kinds and includes a jq recipe for listing sessions that are not retiring backlog.

Destination watermarks are bound to the destination that owns them. The watermark filename is derived from the destination name, and that mapping is many-to-one — two differently-spelled names can resolve to the same file. The stored name is now checked on load, and a mismatch rewinds that destination with a note.
Improves: previously a second destination could inherit the first one's cursor and read itself as already caught up.

Budget accounting and type checking. sweep_max_sessions is charged only against sessions the destination is actually permitted to receive, so ineligible sessions cannot consume the budget ahead of permitted ones. The module's own pyright invocation is now clean.

Verification

root test suite                          933 passed
modules/hook-context-intelligence        785 passed
ruff check . / ruff format --check .     clean, 190 files
pyright (repo root)                      0 errors
pyright (module directory)               0 errors
git diff --check                         clean
CI at 04cf76a                            all checks green

Five tests were added for the behaviours above, each confirmed failing against the prior code before being committed. The end-to-end DTU profile that accompanies this PR exercises the same paths against a real server and Neo4j.

Scope

Delivery itself remains serial — sweep_concurrency ships at 1, matching current behaviour. Making the dispatcher concurrent is a separate change: this client is effectively the rate limiter for a single-process server, and circuit-breaker window ownership would need to be settled first.

Two behaviour notes for reviewers. The sweep does not exclude the session currently being written, and the live watermark is debounced, so in practice the sweep is a second delivery path working the same log alongside the live one rather than a strictly idle-when-healthy pass; re-delivery is absorbed by the server's MERGE on a deterministic node_id. And a duplicate receipt reflects the server's deduplication cache, not a proof of durable acceptance.