|
| 1 | +# Slice 011: NOTES |
| 2 | + |
| 3 | +## G1 plan, 2026-09-20 |
| 4 | + |
| 5 | +Tree at `6af7ee6` on `main` (010 approved); branch `slice/011-llm-provider-layer`; ROADMAP row 011 set to |
| 6 | +`in_progress` in this commit. req_llm 1.24.0 was read from its Hex tarball before this plan was written (public |
| 7 | +entry points `generate_text/3`, `stream_text/3`, `generate_object/4`, `embed/3`; stream chunk types `:content`, |
| 8 | +`:thinking`, `:tool_call`, `:meta`; usage keys `input_tokens`, `output_tokens`, `total_tokens`, `cached_tokens`, |
| 9 | +`reasoning_tokens`, `total_cost`; model spec `"provider:model"`; keys from a per-request `:api_key` option, the |
| 10 | +`:req_llm` application env, or the provider's env var). Each line names its test; the order is the build order. |
| 11 | + |
| 12 | +1. Dependencies: `req_llm` at 1.24.0 inside the `~> 1.22` pin, `mox` for tests. What enters `mix.lock` with |
| 13 | + req_llm is counted and its licences listed in NOTES.md, because the desktop binary carries it. VERSIONS rows |
| 14 | + flipped by `mix versions.gen`. Test: `mix hex.audit` and `mix deps.audit` green; `versions.verify` OK. |
| 15 | +2. `Trinity.LLM.Event`: the seven event shapes from SLICE.md as a typespec plus `valid?/1`, the single source of |
| 16 | + truth 012 and 013 consume. `Trinity.LLM.Request` struct (system, messages, tools, model, params) with a |
| 17 | + changeset-free `new/1` that validates roles and tool schemas. Tests: every shape accepted, a foreign shape |
| 18 | + refused. |
| 19 | +3. `Trinity.LLM.Provider` behaviour: `stream/3`, `generate/2`, `generate_object/3`, `embed/2`, `models/0`, |
| 20 | + `capabilities/1`. `Trinity.LLM.Registry` from `config :trinity, :llm` (models as `{id, provider, model, caps, |
| 21 | + price}`, `default_model`), `models/0`, `default_model/0`, `lookup/1`. Test: a registry entry resolves to its |
| 22 | + provider module; an unknown id is refused by name. |
| 23 | +4. `Trinity.LLM` public API, resolving the provider from the registry entry for the request's model; `stream/3` |
| 24 | + (function) and `stream_to/3` (pid, events as messages); `Trinity.LLM.ProviderMock` (Mox) as the test provider |
| 25 | + for the registry's `fake` provider. Boundary `Trinity.LLM` with `deps: [Trinity]`, exporting the API, |
| 26 | + `Request` and `Event`. Test (AC7): switching `default_model` in config changes the provider used, no code |
| 27 | + change. |
| 28 | +5. `Trinity.LLM.Providers.Fake` in `test/support`: streams a scripted response (deltas, one tool call in three |
| 29 | + chunks, usage, done), scriptable to raise transient or non-transient errors N times. Test (AC1): the full |
| 30 | + event sequence, in order. |
| 31 | +6. `Trinity.LLM.Providers.ReqLLM`: `Request` to req_llm messages and `ReqLLM.Tool`; `stream_text` chunks to |
| 32 | + events with tool calls assembled (start on the first chunk naming a call, deltas while arguments grow, end |
| 33 | + when the call is complete or the stream ends); `:meta` to `{:usage, _}` and `{:done, reason}`; `generate`, |
| 34 | + `generate_object` (req_llm's JSON schema path), `embed`. Provider mapping for `anthropic`, `openai`, |
| 35 | + `openrouter`, `google`, `openai_compatible` (the openai provider with `base_url` and a per-request key). |
| 36 | + Errors classified: timeouts, 429, 5xx, connection refused are transient; 4xx otherwise are not. Unit tests |
| 37 | + over recorded chunk sequences; live tests in line 9. |
| 38 | +7. `Trinity.LLM.Retry`: attempts and base backoff from config; transient errors retried, non-transient returned |
| 39 | + at once; optional fallback model list tried in order after the attempts. Test (AC4): the fake raises a |
| 40 | + transient error twice then succeeds; raises N+1 times then `{:error, _}`; a non-transient error returns |
| 41 | + immediately with one attempt. |
| 42 | +8. `usage_events` migration and `Trinity.LLM.Usage`: one row per completed call with model, provider, tokens, |
| 43 | + and cost computed from the registry price; req_llm's own `total_cost` kept in `provider_meta` and never |
| 44 | + used. Test (AC5): a row exists after a fake call with the expected cost; the Postgres job applies the |
| 45 | + migration. |
| 46 | +9. Live tests, `@tag :live`, excluded by default (NetworkGuard's opt-in path): OpenRouter with |
| 47 | + `TRINITY_LIVE_MODEL` and NVIDIA through `openai_compatible` with `NEMOTRON_BASE_URL`, `NEMOTRON_MODEL` and the |
| 48 | + key as `:api_key`; a preflight that the model id is on `GET /models` and names a 410 as end of life. Cases: |
| 49 | + a streamed completion with deltas and done; a tool call; `generate_object` against a schema (AC2); |
| 50 | + `embed` against `nvidia/nemotron-3-embed-1b` with the declared dimension (AC3). Output pasted, keys |
| 51 | + redacted (AC6). |
| 52 | +10. `Trinity.Config.secret/1`: environment now, keychain at slice 100; the adapter reads keys only through it. |
| 53 | + Test: a missing key is a named error, not a nil passed to the provider. |
| 54 | +11. Anthropic prompt-cache hint: a `cache: true` param maps to req_llm's cache-control option for the anthropic |
| 55 | + provider and is dropped for others. Unit test on the mapping; not measured live (no Anthropic key here). |
| 56 | +12. Gate, coverage row, PROOF.md, ROADMAP to `done`, through a pull request; tag `slice/011` after the merge. |
| 57 | + |
| 58 | +Manual verification queue, as SLICE.md tags them: AC2 (live `generate_object`) and AC6 (`mix test --only live`, |
| 59 | +output pasted with keys redacted). Both run on this machine with the owner's keys in `.env`; the owner reads |
| 60 | +the pasted output at G4. The developer machine, the model ids and the probe outputs behind the choice: |
| 61 | + |
| 62 | +- OpenRouter, `inclusionai/ling-3.0-flash-vl:free` (owner's pick): tool call `get_weather` with |
| 63 | + `{"city": "Paris"}`, 298 tokens, served by Novita; streamed completion 59 chunks. |
| 64 | +- NVIDIA endpoint, `nvidia/nemotron-3.5-lightning-30b-a3b`: tool call, 317 tokens; streamed completion 20 |
| 65 | + chunks. The id first set, `nvidia/llama-3.3-nemotron-super-49b-v1.5`, answered `410 Gone` (end of life |
| 66 | + 2026-08-26); that is why line 9 carries a preflight. |
| 67 | + |
| 68 | +Deviations from SLICE.md, stated before building: the five provider mappings are written; only OpenRouter and |
| 69 | +the NVIDIA endpoint are measured live, and PROOF.md says so per provider. `Trinity.LLM.Supervisor` from docs/01 |
| 70 | +(rate limiters) is not built here: nothing in this slice needs a process, and a supervisor with no children |
| 71 | +would be a claim; 012 adds it when the Session needs one. Recorded as a follow-up. |
0 commit comments