Slice 090: observability, the cost ledger, and a turn as a trace

.github/workflows/gate.yml

@@ -82,6 +82,11 @@ jobs:
       DATABASE_URL: postgres://trinity:trinity@localhost:5432/trinity_test
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+        with:
+          # `test/changelog_test.exs` reads `git tag -l 'slice/*'` and refuses to pass without
+          # tags, because a test that finds nothing to check is indistinguishable from one that
+          # found everything in order. The default depth of one fetches no tags.
+          fetch-depth: 0
 
       - uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
         with:

CHANGELOG.md

@@ -22,8 +22,55 @@ evidence for each increment is retained by the maintainers and summarised here.
   tree, engineering rules consolidated into `docs/03-conventions.md`, and this changelog added as
   the public record of delivered work.
 
+## 2026-09-24
+
+### `slice/090.1` — Observability and the cost ledger
+Every LLM call, tool call, approval, session transition, gateway message and budget refusal emits a
+`:telemetry` event from a single catalogue (`docs/telemetry.md`), written before the emitters rather
+than documented after them, under one rule over the whole set: no prompt text, no completion text,
+no tool arguments and no key material. A turn carries a trace identifier into the task that runs it,
+so a turn reads as a tree without a tracing library and without a new dependency. A primary Logger
+filter redacts credentials by shape rather than by name, so a secret nobody thought to list is still
+caught, and it rewrites a message rather than dropping it. The cost ledger reads the usage rows the
+tree already writes, by day, model, session and persona, against budgets that warn and, when
+configured to, refuse. `/activity` shows the live event stream with the spend beside it, and a
+LiveDashboard page lists running sessions by asking each process its state rather than reading a row.
+OpenTelemetry was assessed and deferred against a measured need rather than added because the
+criterion named it; the reason is recorded with the slice.
+
+**Tagged `slice/090.1`, not `slice/090`.** The `slice/090` tag was pushed before its merge had
+landed and points at slice 080's merge commit. This repository's ruleset forbids moving or deleting
+a tag, so the error is permanent and public; the correct tag carries the correction in its own
+annotation rather than leaving a reader to reconcile the two.
+
+### `slice/080` — Subagents and delegation
+The assistant can hand a bounded piece of work to a child session through a `delegate` tool. The
+child has its own context and its own history, and hands back a result as text: the parent's
+conversation never contains the child's messages, which is the property that makes delegation worth
+having. Several children run at once under a concurrency cap, each with a budget in turns, tokens
+and wall clock; exceeding it stops the child and says so. A child that dies does not take its parent
+with it, and the parent is told; cancelling a parent cancels the tree beneath it. Approvals a child
+raises appear on the same permissions page as any other, named with the child that raised them, and
+a `/subagents` panel shows what is running with a stop control for each.
+
 ## 2026-09-23
 
+### `slice/025` — Encryption at rest, and the key-custody seam
+The database, the receipt chain and the key material can be sealed at rest, and the seam that holds
+the keys is named rather than assumed. Sealing is a deployment decision with the trade-off stated in
+both directions: a key lost is data lost, so the custody options are documented with their recovery
+paths, and the default is the one whose failure mode an operator can survive. What the seam does
+*not* cover is written down beside what it does.
+
+### `slice/002` — Supply chain, early
+A CycloneDX bill of materials is generated by the quality gate on every commit and ships beside
+every packaged binary, so whoever holds an artifact holds the list of what is in it; the bill states
+its own coverage inside the document, including what it does not cover. Every packaged binary
+carries build provenance attested through the workflow's own identity and recorded in a public
+transparency log, verified in the same run that produced it. A minimum TLS version is enforced for
+outbound connections. Brought forward from M9 because a supply-chain claim made late is a claim
+made about work already done.
+
 ### `slice/070` — Gateway core
 Trinity becomes reachable from channels other than the desktop. An adapter behaviour carries text
 and nothing else; a router admits the sender, applies a rate limit, dispatches commands and binds

README.md

@@ -39,10 +39,14 @@ desktop application. Apache-2.0, developed in the open from the first commit.
 
 Pre-alpha, and usable from source. Milestones M0 to M4 are approved, **M5a Automates** with them
 (050, the scheduler; 059, the MCP measurement; 060, the MCP client; 061, the MCP server; 062, MCP
-authorization), and the first slice of M5b (070, the gateway core), with 002 (the supply chain
-slice) beside them: 26 slices, each merged with a merge commit and tagged `slice/NNN`
-(`git tag -l 'slice/*' | wc -l` → 26, on 2026-09-23). What
-that means in practice:
+authorization), **M5b Reaches** in part (070, the gateway core; 080, subagents) and the first slice
+of **M6 Ships** (090, observability and the cost ledger), with 002 (the supply chain slice) beside
+them: 29 slices, each merged with a merge commit and tagged `slice/NNN`
+(`git tag -l 'slice/*' | wc -l` → 30, on 2026-09-24: one more tag than slices, because
+`slice/090` was pushed at the wrong commit before its merge landed and this repository's ruleset
+forbids moving or deleting a tag; the correct one is `slice/090.1`, and `git tag -n20 slice/090.1`
+says so in its own annotation rather than leaving a reader to work it out). What that means in
+practice:
 
 **Assessed against a recognised baseline.** Trinity holds the
 [OpenSSF Best Practices **silver** badge](https://www.bestpractices.dev/projects/14772), awarded
@@ -110,6 +114,26 @@ enforces it.
   conversation. What a channel may approve is capped below what the desktop may: a `write` at
   most by default, so an `exec` or a `destructive` request is decided at the machine, and the
   refusal is receipted. The console adapter ships (`mix trinity.console`); the platforms are next.
+- **Delegates.** The assistant can hand a bounded piece of work to a child session with a
+  `delegate` tool: its own context, its own history, and a result handed back as text. The
+  parent's conversation never contains the child's messages, which is the point:
+  delegation that leaks the child's transcript back into the parent buys nothing. Several children run at
+  once under a cap, each with a budget in turns, tokens and wall clock; exceeding it stops the
+  child and says so. A child dies without taking its parent with it, and the parent is told;
+  cancelling a parent cancels the tree beneath it. Approvals a child raises surface on the same
+  permissions page as any other, named with the child they came from, and the `/subagents` panel
+  shows what is running with a stop button per child.
+- **Shows what it costs.** Every LLM call, tool call, approval, session transition and gateway
+  message emits a `:telemetry` event from one catalogue (`docs/telemetry.md`), written before the
+  emitters rather than after them, with one rule over the whole set: no prompt text, no completion
+  text, no tool arguments, no key material. A turn carries a trace id down into the task that runs
+  it, so a turn reads as a tree without a tracing library or a new dependency. A primary Logger
+  filter redacts credentials by **shape** (bearer and basic headers, `sk-` and `pk-` prefixed
+  keys, JWTs), so a secret nobody thought to name is still caught, and the filter never drops a
+  message, only rewrites it. The cost ledger reads the usage rows the tree already writes: by day,
+  by model, by session, by persona, against budgets that warn and, when you ask them to, refuse.
+  `/activity` shows the live stream with the spend beside it; a LiveDashboard page lists the
+  running sessions, asking each process its state rather than trusting a row.
 - **Runs on a schedule.** Tasks on the `/tasks` page: a prompt, a persona, the skills to hint,
   and when (a cron expression, a one-shot time, or a phrase like "every weekday at 9am" the
   model turns into cron). Each run is a fresh conversation you can open, its result waits on
@@ -118,9 +142,10 @@ enforces it.
   curator that marks old memories stale and archives the untouched ones (never deleting)
   run on the same queues; `/oban` shows the jobs.
 
-Not there yet: the messaging platforms themselves and subagents (M5b),
-the native desktop shell and signed releases (M6), executable skills in a sandbox (M7). The [Milestones](#milestones) section below sets out the order the
-remaining work is being built in.
+Not there yet: the messaging platforms themselves (071 Telegram and 072 Discord, which need a bot
+token the project does not yet hold), the native desktop shell and signed releases (the rest of M6),
+executable skills in a sandbox (M7). The [Milestones](#milestones) section below sets out the order
+the remaining work is being built in.
 
 The interface is a local web page; the desktop shell exists as a packaging spike, not a product.
 If you want to follow along, watch the roadmap and the tags.
@@ -240,7 +265,7 @@ rather than described by it.
 | M7 Sandboxed | Executable skills in an in-VM sandbox | 110 approved |
 | M9 Donatable | Open-source hygiene audited, supply chain signed, shared libraries extracted | 002 and 120 to 123 approved |
 
-M0 to M5a are approved as of 2026-09-22, and 070 of M5b on 2026-09-23. Slice numbers have gaps on purpose (000, 001, 010, 011 and so on) so that a slice can be inserted
+M0 to M5a are approved as of 2026-09-22. Of M5b, 070 and 080 are approved as of 2026-09-23; 071 and 072 wait on a messaging bot token the project does not yet hold. Of M6, 090 is approved as of 2026-09-24. Slice numbers have gaps on purpose (000, 001, 010, 011 and so on) so that a slice can be inserted
 later without renumbering anything.
 
 **[`ROADMAP.md`](ROADMAP.md)** sets out what the project intends to do and what it intends not to
@@ -254,6 +279,7 @@ do over the next year, and why the order is what it is.
 | `VERSIONS.md` | The verified dependency versions, generated from `lib/trinity/versions.ex` |
 | `docs/` | Vision, architecture, tech stack, conventions, data model, risks, security model, standards register; packaging, the FIPS leg, backup and restore, performance measurements |
 | `docs/mcp-server.md` | Connecting a client to Trinity's MCP server (Claude Code, VS Code, Codex, goose), stdio, approvals over the wire, the headless profile |
+| `docs/telemetry.md` | The telemetry event catalogue: every event this tree emits, its measurements and metadata, and the rule about what never appears in one |
 | `docs/adr/` | Architecture decision records. One is added whenever a decision changes |
 | `docs/10-assurance-case.md` | The structured argument that the security claims hold, with the evidence for each and the assumptions and limits named |
 | `docs/09-standards-register.md` | One row per control a regulated deployment may ask about, with its evidence path and status |

config/test.exs

@@ -191,6 +191,12 @@ config :trinity, TrinityWeb.Endpoint,
   secret_key_base: "PAlioLgquSvnIrD4YjwqUt4LEP1x1E5d56Z7KY/QhgmyRUFR9ynpk3oz/hITAYwN",
   server: true
 
+# Slice 090: the LiveDashboard routes are compiled in the test environment too. They are
+# `compile_env`-gated and were dev-only, which meant the custom sessions page had no test and no
+# way to be screenshotted without running a dev server against a dev database. A route that exists
+# only in the environment nothing tests is a route that breaks quietly.
+config :trinity, dev_routes: true
+
 # Print only warnings and errors during test
 config :logger, level: :warning
 

coverage.tsv

@@ -25,3 +25,4 @@ slice_id	percent	sha	date
 002	80.31	d65c48c	2026-09-23
 025	80.31	2c4fe1f	2026-09-23
 080	80.31	a7630ba	2026-09-24
+090	80.31	3e4185a	2026-09-24

docs/01-architecture.md

@@ -96,7 +96,7 @@ without anything failing.
 | `Trinity.Subagents` | Delegation, result collection (as built at 080: `delegate/3` and `delegate_many/3` creating child sessions with `origin: "subagent"` and `parent_id` set, one turn each under a monitor, the awaiting shape reused from slice 050's `RunTask` including its drain of the start `:idle`; budgets of turns, tokens and wall clock; `restarts` defaulting to **0**, because replaying a turn replays arbitrary tool calls and a `:session` or `:always` approval would cover the replay silently; `cancel_subtree/1` depth first; `running?/1` asked of the process rather than of `SessionRow.status`, which is the session lifecycle and stays `active` after a subagent finishes; `short_id/1` taking the **tail** of a UUIDv7, whose head is shared between children created in the same second) | Sessions |
 | `Trinity.Sandbox` | Luerl runners, resource limits | none |
 | `Trinity.Desktop` | ex_tauri bridge | PubSub |
-| `Trinity.Telemetry` | events, cost ledger, metrics | Repo |
+| `Trinity.Telemetry` | The event catalogue and the only module that names an event (as built at 090: `span/3` and `emit/3` routing everything through one place; emitters for the LLM, tool, approval, session, gateway and budget events listed in `docs/telemetry.md`, which was written **before** them because an event name is an interface and a handler attached to a name that no longer fires simply never runs; `trace/2` giving a turn one trace id so its events nest, with no tracing library and no dependency; `Telemetry.Costs` reading `usage_events` schemalessly, because `Trinity.LLM` depends on `Trinity` and naming its schema would be a cycle; `Telemetry.Activity`, a bounded buffer whose handler does one `send` because a telemetry handler runs in the emitting process; `Telemetry.Redaction`, a primary Logger filter that masks shapes rather than names and never drops a line) | Repo |
 | `TrinityWeb` | LiveViews, components, API | all `Trinity.*` public APIs |
 
 The six rows in bold were corrected on 2026-09-05. As written, the table forbade the effect path drawn in the

docs/06-risk-register.md

@@ -28,3 +28,4 @@
 | R25 | `glib` 0.18.5 carries GHSA-wrw7-89jp-8q8g (medium, unsoundness in `Iterator` and `DoubleEndedIterator` impls); first patched in 0.20.0 | Low | Med | The advisory is open against `src-tauri/Cargo.lock`; measured 2026-09-23 | It is transitive through the GTK 0.18 stack (`gtk` 0.18.2, `webkit2gtk` 2.0.2) that Tauri 2.11.5 requires on Linux, so it is fixed by a Tauri release built on gtk-rs 0.20 and not by a direct bump. **Scope:** the desktop shell only. The `headless` release is `steps: [:assemble]` with no Burrito and no `src-tauri`, so a server deployment does not ship this dependency. Owner: the maintainer. Lift condition: a Tauri release whose Linux stack resolves `glib` >= 0.20, adopted by the desktop shell work | 100 |
 | R26 | An intermittent `Exqlite.Error: Database busy` fails the test suite on loaded CI runners, in a different test each time, always on a write to the main repository | Med | Low (test-only; no production path) | Failed the gate on pull requests 67, 70 and again on the commit that attempted to fix it; the same commit passed on its other run, so it is seed- and load-dependent. Not once reproduced locally across repeated runs | Ruled out: `busy_timeout`, raised from the shipped 5 s to 30 s in the suite, did not stop it. SQLite returns BUSY immediately, without the busy handler, when a read transaction upgrades to a write while another connection holds the write lock; no timeout affects that path. The suite runs two connections (one for the sandbox owner, one for Oban's boot check) and deliberately queues fifty concurrent writers at slice 070 AC7, which is the load that provokes it. Candidate fixes, not yet taken: drain the stress test's sessions before the suite continues; give that test its own tag and CI step; or remove the second connection's need. Owner: the maintainer. Lift condition: ten consecutive green CI runs of the full suite with the stress test in the default run | 070 |
 | R26 | Occurrence recorded 2026-09-23: the same class of failure appeared on the **FIPS leg** rather than the ordinary gate, which the row above does not cover. `Trinity.Gateways.ApprovalsTest` AC4 failed in its setup, with the crash in `Trinity.Receipts.ChainWriter.terminate/2` calling `insert_checkpoint/2` after the sandbox owner had gone: an ownership error at termination rather than a `Database busy` on a write. **Re-running the identical commit passed all four checks**, which is the evidence that matters: the code did not change between the red run and the green one, so the failure is in the environment or the ordering and not in the change under test. What this adds to the row above: the mechanism is not only write contention, it also reaches process termination racing sandbox teardown, and it is not confined to the ordinary gate's runners. A candidate fix specific to this shape: `ChainWriter.terminate/2` writes a checkpoint on the way out, and a terminate callback that touches the database after the test that owns the connection has finished is a race by construction; it should skip the write when the repo is in sandbox mode and has no owner, rather than attempting it and crashing. **Second occurrence the same day, and it changes the severity.** The very next pull request, which changed *only this file*, failed the FIPS leg again: `Trinity.Gateways.RouterTest` AC6, `Exqlite.Error: Database busy` on an `INSERT INTO gateway_identities` - the row's original signature, on the leg the row did not cover. A change that cannot affect behaviour failing a required check twice in a row means this is no longer an intermittent annoyance: **it blocks work**, and the honest reading is that the FIPS leg provokes it more reliably than the ordinary gate does, most likely because that leg is slower (163 s against about 110 s locally) and every contention window is correspondingly wider. The row above stands as written | **High** (was Med: it now blocks merges) | Low (test-only; no production path) | The lift condition in the row above, still unmet; two failures on the FIPS leg on 2026-09-23, one of them on a documentation-only change | docs/06-risk-register.md R26 | 070 |
+| R26 | Counter-example recorded 2026-09-24, in slice 090, and it corrects a heuristic the rows above state. At commit `b1bca1f` the `pull_request` run passed all four checks and the `push` run of the **same commit** failed `postgres` (runs 36001064067 and 36001057879). By the reading the 2026-09-23 entry offers - the code did not change between the green run and the red one, so the fault is the environment - this was R26 again. **It was not.** It was a real defect in slice 080's code, on a genuinely racy path: every call into a child session is a `gen_statem.call`, and a call into a process that dies exits the caller, so killing a child killed whoever called `delegate/3`. Whether the kill landed before or after the call returned decided the outcome, which is why the same commit could go either way. Fixed at `1f33f12`, after which both runs of the same commit passed. **What this changes:** "the same commit passed on its other run" is evidence that the failure is *timing-dependent*, which is not the same as evidence that it is environmental. A real race in library code produces exactly the signature R26 is recognised by, so the signature alone cannot be the diagnosis - the stack trace has to be read before a failure is filed here. This slice nearly filed one. The rows above stand as written | none (not an occurrence) | none | none | none | 080, 090 |