diff --git a/CHANGELOG.md b/CHANGELOG.md index 47816ce..2645ad5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,35 @@ missing or wrong declaration, and full-text recall keeps working through every r The evaluation that decides the default model is built and waiting: a generated corpus, 300 queries, and labels that the scored run will not read until 100 of them have been audited and frozen. +### `slice/134`: a stronger embedding model from a service the operator runs, held to a pin + +Semantic memory can now use an embedding model served by an Ollama the operator runs, without any +neural runtime inside Trinity. The service is admitted on the operator's pin and nothing else: the +model's digest as the service lists it, the weights' and tokenizer's SHA-256, the context length, +the prompt templates and the service's version are all part of the embedding space. If the model +behind the name is replaced, the version changes, the model disappears or the service stops +answering, semantic recall turns off with that reason within one check interval, full-text recall +keeps answering, and no other model or space is chosen. + +Every request asks the service to refuse rather than truncate an input that is too long, which is +not its default. Trinity also counts every input's tokens itself, with the tokenizer read out of the +signed weights (it gives the reference library's token ids on 5,000 natural sentences and 1,221 +hostile inputs), refuses an over-length input before sending it, and treats a service that reports +fewer tokens than it was sent as cutting inputs short: no vector, and the embedder off until restart. + +Weights reach the service only through `mix trinity.tier3.import`: an OCI artifact whose cosign +signature and OMS signature over the model files are verified offline against keys the operator +holds, every blob's digest checked as it is unpacked, the model created through the service's API, +and the resulting digest recorded. Trinity ships no Tier 3 weights. + +Four gates decide whether a model is admitted on a given image, and they were run on the upstream +Ollama 0.40.0 image with Qwen3-Embedding-0.6B (16-bit weights): every one of 500 fixtures within a +cosine of 0.999 of the sentence-transformers reference (lowest 0.99969); an input one token too long +refused, never embedded; a running node off 160 ms after the model behind its tag was swapped; and +the service answering with networking disabled. The 8-bit build of the same model missed the +parity bar (lowest 0.99854) and is not offered. The gates have yet to be run on the hardened image +this is meant for. `docs/tier3-embedder.md` is the operator's page and `docs/perf.md` has the figures. + ## 2026-09-26 ### `slice/128`: the checks that guard attribution now say what they check diff --git a/README.md b/README.md index a891e8d..5a223d5 100644 --- a/README.md +++ b/README.md @@ -226,7 +226,11 @@ say-so, never on its own) or the static floor (a 7.8 MB weights file, pure Elixi under the data directory; not yet distributed with Trinity while its licence is reviewed). Without one the tier is off and full-text search still works. Every stored vector belongs to one named embedding space, and a store answers only from the space it is pinned to: changing models is an explicit re-embed (`mix trinity.space.retier`), -never a silent switch. `docs/perf.md` has what it costs. +never a silent switch. A stronger model can come from an Ollama service you run instead +(`docs/tier3-embedder.md`): its weights are imported only with their signatures verified offline +(`mix trinity.tier3.import`), Trinity holds the service to the imported model's digest and version, and +if either changes semantic recall turns off with the reason rather than answering from another model. +`docs/perf.md` has what it costs. `mix gate` runs the full quality gate and has to pass before every commit: format check, compile with warnings as errors, Credo, Sobelow, dependency audits, version verification, the naming and diff --git a/docs/01-architecture.md b/docs/01-architecture.md index 63a2314..35f068b 100644 --- a/docs/01-architecture.md +++ b/docs/01-architecture.md @@ -88,7 +88,7 @@ without anything failing. | `Trinity.Effects` | The membrane; the runner in force (`Effects.Runner`, the executor `Tools.Runner` takes as a function); decision and query receipts; the boot receipt | **Tools**, Permissions, Authority, Receipts, Repo | | `Trinity.Authority` | Behaviour; `Local` implementation (the one caller of `execute/2` for effectful tools); selection at boot; `Staged` | Receipts, Repo | | `Trinity.Receipts` | Local chain (one supervised writer per scope, ADR-0013), the signer seam (Ed25519, P-384, ML-DSA-87), key custody and the registry, checkpoints, the verifier, the alarm | Repo (`Repo.Receipts`) | -| `Trinity.Memory` | Always-on tiers with their budget and consolidator (030), search (031), semantic store and retrieval (032), compaction (023); at 133 the embedding spaces (`Space`, the fourteen-field identity; `Spaces`, the one active space, re-tier, drop), the static floor (`Embedders.Static`, `WordPiece`, `StaticArtifact`), the int8 `Scorer`, the embedder configuration's faults (`EmbedderConfig`) and the operator's `ReEmbedWorker` | LLM (summaries/embeddings), Repo | +| `Trinity.Memory` | Always-on tiers with their budget and consolidator (030), search (031), semantic store and retrieval (032), compaction (023); at 133 the embedding spaces (`Space`, the fourteen-field identity; `Spaces`, the one active space, re-tier, drop), the static floor (`Embedders.Static`, `WordPiece`, `StaticArtifact`), the int8 `Scorer`, the embedder configuration's faults (`EmbedderConfig`) and the operator's `ReEmbedWorker`; at 134 the Tier 3 client (`Embedders.Ollama` with its pin watcher `Embedders.Ollama.Watch`), its token counter (`BPE`), the GGUF header reader (`GGUF`) and the signed import path (`Tier3.Import`, `Tier3.Verifier`, `mix trinity.tier3.import`) | LLM (summaries/embeddings), Repo | | `Trinity.Skills` | SKILL.md parsing, registry, loader, manager, scanner (as built at 040: parser, sources, registry, index, the three tools; at 041: staging, promotion, manager, scanner, diff, learn, `skill_manage` and `learn`) | Repo, Permissions, **Effects**, **Receipts**, Sandbox (as built at 041: Tools, Memory, Permissions, Receipts and LLM; Effects is not a dependency: the promotion is not a tool call, it writes its own effect receipt; Tools never depends on Skills) | | `Trinity.Scheduler` | Oban workers for agent tasks, delivery (as built at 050: the `tasks` and `task_runs` rows, `next_run_at` computed from a cron expression or a one-shot datetime, `Workers.Tick` (the one Cron plugin entry, every minute, enqueuing `Workers.RunTask` for what is due), `RunTask` (one turn in a fresh `origin: "cron"` session, the summary, the delivery), `Delivery` with `Delivery.Desktop`, `Parse.human/2` (a phrase to cron through the model, checked by Oban's parser); Oban itself on the app's repo, Lite on SQLite and Basic on Postgres, queues `agent_tasks`, `memory`, `maintenance`; the observer as `Trinity.Memory.ObserverWorker` and the curator as `Trinity.Memory.Curator` live in Memory) | Sessions, LLM, **Repo** (Gateways at 070) | | `Trinity.MCP` | Client manager, tool bridge, server (as built at 059: the boundary alone, holding the core's version, its JSON depth and its telemetry event names; at 060: `Servers` and `ServerConfig` (the rows), `Supervisor` and `Boot` (one `Client` per enabled row), `Client` with its `Wire` (the outbound request, the headers, the core's decoder and validator) and two transports (`Transport.Stdio`, a child on a Port; `Transport.HTTP`, one POST per request), `Bridge` (the one tool module every MCP tool runs through) and `Client.Auth` (062's seam, a static token at 060); at 061: `Server` (the module above the core, `@behaviour BeamMCP.Server`, handed to both transports through `:server`; `tools/call` answered through the membrane, everything else the core's), `Server.Catalog` (the exported tools, sorted), `Server.Exports` (the configured names, `:catalog` refused), `Server.Session` (the `origin: "mcp"` session with its own persona), `Server.Envelope` and `Server.Replay` (the sealed `requestState` and the nonce table), `Server.Auth.Local` (the static bearer, 062's seam on the server side), `Server.Plug` (the `POST /mcp` endpoint, mounted in `TrinityWeb.Endpoint` ahead of the parsers) and `Server.Stdio` with `mix trinity.mcp.stdio`; at 062: `Trinity.MCP.Auth`, a boundary of its own under `Trinity.MCP` with `deps: []` (the behaviour `authorize/2` answering a `Principal`, never the token; `Config`, `Token`, `JWKS`, `Discovery`, `Scopes`; the profiles `Local`, `External` and `Embedded` with `Embedded.Keys`; the client role `Client` with `Client.Store`), and `AuthHost` in `Trinity.MCP` (the configuration from the tree's paths and the authority selection, the personal profile's boot, the refusal receipts, the client role's flow for the page); `Server.Auth.Local` is gone, `Server.Plug` authorizes ahead of the transport and answers `401` with a challenge, `Server` checks scope before the gate) | Tools, **Effects**, **Permissions**, Memory (as built at 059: a top-level boundary, like `Trinity.Smoke`, with `deps: [Trinity, BeamMCP.JSON]`; the boundary compiler checks every call into the `beam_mcp` application and this boundary alone lists its modules; at 060 the deps are `[Trinity, BeamMCP.JSON, BeamMCP.Schema]`, Trinity's tools, permissions and receipts reached through `Trinity`'s exports, and `Trinity.Application` and `TrinityWeb` list `Trinity.MCP`; at 061 the core's `Server`, `Catalog`, `ToolSpec` and both transports are listed too, and `Trinity` exports `Tools.Registry` and `Effects.Runner` to it) | @@ -115,7 +115,7 @@ through those modules only. `boundary` `exports:` lists enforce this. | `Trinity.Tools.Tool` | `name/0`, `description/0`, `schema/0`, `risk/0`, `execute/2` | config `:tools` list + MCP dynamic registration | | `Trinity.Gateways.Adapter` | `child_spec/1`, `deliver/2`, `capabilities/0` | config `:gateways` list | | `Trinity.Memory.VectorStore` | `upsert/3`, `search/3`, `delete/1`, `count/1` (as built at 032: the scope filter is `search/3`'s required argument; as built at 133: `search/3` and `count/1` over one named space, `{:error, {:mixed_space, n}}` rather than a ranking when a row does not fit it, writes through `Spaces.put_vector/5`) | the database adapter, at compile time (`Brute` on SQLite, `Pgvector` on Postgres; `Hnswlib` when a slice measures past 10^5 rows) | -| `Trinity.Memory.Embedder` | `embed/1`, `dim/0`, `model_id/0`, `availability/0`; at 133 `space/0` and `thresholds/0` | `config :trinity, :memory, embedder:` (`:local` default, `:static` the floor, `:hosted` opt-in only with a declared `locality:`, `:fake` in the suite, or a module; a list names several, and the store is served by the one whose space is pinned) | +| `Trinity.Memory.Embedder` | `embed/1`, `dim/0`, `model_id/0`, `availability/0`; at 133 `space/0` and `thresholds/0`; at 134 the optional `embed_query/1` (a query template), `endpoint/2` (where text is sent, for the locality check), `check_config/1` (the embedder's own configuration faults) and `children/0` (processes `Trinity.Memory.Supervisor` starts) | `config :trinity, :memory, embedder:` (`:local` default, `:static` the floor, `:hosted` opt-in only with a declared `locality:`, `:ollama` the Tier 3 service on the operator's pin with a declared `locality:`, `:fake` in the suite, or a module; a list names several, and the store is served by the one whose space is pinned) | | `Trinity.Skills.Loader` | `load/1`, `validate/1` | config `:skill_loaders` (not built at 040: the three roots are fixed and `Trinity.Skills.Sources` scans them; a loader behaviour waits for a fourth kind of source) | | `Trinity.Permissions.Policy` | `decide/3` → `:allow | :deny | {:ask, prompt}` | config | @@ -138,6 +138,16 @@ trinity.space.drop --confirm`). `Semantic.status/0` serves the store throug space is active and answers OFF with a named reason otherwise: a configuration fault (`EmbedderConfig`; under `:regulated` the boot is refused instead), a space mismatch, the embedder's availability, or the last runtime fault (an unreachable endpoint, a store row that does not fit its space). Full-text recall goes on in every case. + +**The Tier 3 embedder (Slice 134, as built):** `Embedders.Ollama` reaches an operator-run Ollama over HTTP and serves +only the model the operator pinned: the `/api/tags` digest, the GGUF's SHA-256, the tokenizer's SHA-256, `num_ctx`, +both prompt templates and the runtime version are fields of its space, all from configuration. Every request carries +`truncate: false`; Trinity counts each input with `BPE` (the tokenizer read out of the signed weights) and refuses one +over its limit before sending, and an answer whose `prompt_eval_count` is below that count turns the embedder off. +`Embedders.Ollama.Watch`, started by `Trinity.Memory.Supervisor` through `children/0`, compares the service with the +pin at boot and on an interval, and a changed digest or version is OFF with its reason. `Tier3.Import` is the only way +weights reach the service: cosign and OMS verified offline by the reference programs, every blob's digest checked, +the model created through the service's API and its digest recorded. `docs/tier3-embedder.md` is the operator's page. ``` Every state transition is persisted before it is broadcast. A crash between persist and broadcast is safe diff --git a/docs/perf.md b/docs/perf.md index 226534e..5f41076 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -151,3 +151,34 @@ seconds, and rebuilding gave the same digest. `scripts/static_floor_release.sh` (`TRINITY_WITHOUT_ML=1`): the headless release assembled from a build that never had nx, exla, xla, axon, bumblebee or tokenizers holds none of them in `lib/`, boots, writes two memories and recalls one through the static space (slice 133, AC8). Slice 130 measured exla at 443 MB of a 522 MB release. + +## The Tier 3 embedder and its gates (slice 134) + +MEASURED on 2026-10-08 on the same machine as slice 133's figures above (AMD RYZEN AI MAX+ 395, 32 logical CPUs, +125 GiB, Linux 6.14.0-37), with other work on it (load average 9 to 16 during the throughput run). The service: +`ollama/ollama:0.40.0` (index digest `sha256:1bef639749741b375e9a1eb2c1346fb57ce52f5432de1f74846e44ccc18e1687`), +the upstream image at the version the hardened government image pinned that day, on the CPU, limited to 16 GiB. +**It is not that hardened image**, which these gates have yet to be run on. The model: `Qwen3-Embedding-0.6B-f16.gguf` +(SHA-256 `421a27e58d165478cc7acb984a688c2aa41404968b0203e7cd743ece44c54340`), imported through +`mix trinity.tier3.import`; the reference: sentence-transformers 6.1.0 over the same model's safetensors, float32. + +| gate | result | +|---|---| +| 1, parity, 500 fixtures at `num_ctx` 2048 | PASS: minimum cosine 0.99968977, median 0.999875; every `prompt_eval_count` equal to the reference tokenizer's count | +| 1, red: `num_ctx` 256 with the service's default truncation | FAIL as intended: 60 of 500 under 0.999, minimum 0.95426396, all of them inputs over 255 tokens | +| 1, the same model quantized to Q8_0 | FAIL: minimum 0.99853698, 60 of 500 under 0.999 (not offered) | +| 2, truncation at `num_ctx` 512 and 2048 | PASS: 511 (2047) tokens answered, 512 (2048) refused with `truncate: false`; with `truncate: true` or omitted, a vector of the first 511 (2047) | +| 3, the digest pin, interval 2 s | PASS: OFF with `:model_digest_changed` 160 ms after the blob behind the tag was replaced; with the comparison removed, still on after 7.4 s | +| 4, offline | PASS: `--network none`, only `lo`; `/api/embed` answered from the verified directory; a pull failed for want of a network | + +Throughput through Trinity's client (tokenizing, the request and the checks), 301 sentences of 7 to 200 tokens +(median 28) cycled, no pass mark: + +| batch | calls | p50 | p95 | texts per second at p50 | +|---|---|---|---|---| +| 1 | 200 | 34.8 ms | 57.7 ms | 28.7 | +| 32 | 40 | 1227.8 ms | 1382.5 ms | 26.1 | + +Batching does not raise throughput on this CPU path. The service held 5.31 GiB of its 16 GiB afterwards. Trinity's +own token count matches the reference tokenizer on 5,000 natural sentences and 1,221 hostile inputs, and reading +the tokenizer out of the 1.2 GB GGUF's header takes 337 ms. diff --git a/docs/regulated/data-flow.md b/docs/regulated/data-flow.md index cba7c41..828ab6e 100644 --- a/docs/regulated/data-flow.md +++ b/docs/regulated/data-flow.md @@ -70,6 +70,28 @@ What this means for an accreditation: it is in the conversation it is in the prompt on the next turn, because the conversation tier is sent verbatim. +## What leaves for an embedding service + +Semantic memory needs an embedding model. Two configurations send text off the Trinity process to +get one: `:hosted` (a provider's embedding endpoint) and `:ollama` (an Ollama the operator runs, +`docs/tier3-embedder.md`). In both, **the text of every memory the observer writes and of every +query a turn recalls with leaves in the clear**, the latter prefixed by the configured query +template. Nothing is hashed or redacted on the way. + +| What crosses | Raw or hashed | To | Path | +|---|---|---|---| +| Memory text and query text | Raw | The configured embedding endpoint | `lib/trinity/memory/embedders/hosted.ex`, `lib/trinity/memory/embedders/ollama.ex` | +| The model name, `num_ctx`, `truncate: false` | Raw | The same endpoint, `:ollama` only | `lib/trinity/memory/embedders/ollama.ex` | +| The model's weights, at import only (`mix trinity.tier3.import`) | Raw | The operator's own service | `lib/trinity/memory/tier3/import.ex` | +| No text: the pin check is two reads, `GET /api/tags` and `GET /api/version` | Not applicable | The same endpoint, `:ollama` only | `lib/trinity/memory/embedders/ollama/watch.ex` | + +What constrains it: the endpoint's locality is declared by the operator (`:within_boundary` or +`:external`), never inferred from the address; `:external` needs `external_opt_in: true`; and under +`TRINITY_PROFILE=regulated` the endpoint must be on `TRINITY_REGULATED_LLM_ENDPOINTS` or the node does +not boot (`lib/trinity/memory/embedder_config.ex`). Whether CUI or PHI may go to that endpoint is the +same policy question as for the model endpoint above. An embedding service the operator runs inside +the boundary may still reach out on its own: see the operating notes in `docs/tier3-embedder.md`. + ## Where the data rests | Store | Contents | Encrypted by Trinity? | Path | diff --git a/docs/tier3-embedder.md b/docs/tier3-embedder.md new file mode 100644 index 0000000..3811f60 --- /dev/null +++ b/docs/tier3-embedder.md @@ -0,0 +1,118 @@ + + +# The Tier 3 embedder: an operator-run embedding service + +Trinity's semantic memory can use an embedding model served by an Ollama the operator runs, +instead of a model inside Trinity's own process. The service holds the weights; Trinity holds a pin +to them and refuses to serve from anything else. This page is for the operator who sets it up. + +The short version: import signed weights with `mix trinity.tier3.import`, paste the configuration +block it prints, declare where the service sits, and re-tier the store onto the new space. If the +service changes underneath Trinity (another model behind the tag, another Ollama version, the model +gone, the service down, the service cutting inputs short), semantic recall turns **off** with the +reason, full-text recall keeps answering, and nothing is re-embedded or switched until you act. + +## What Trinity checks, and when + +| When | What | If it fails | +|---|---|---| +| Every request | `truncate: false` and the pinned `num_ctx` are sent | the service refuses an over-length input instead of embedding its first part | +| Before a request | Trinity counts each input's tokens itself, with the tokenizer taken from the signed weights | an input over `max_input_tokens` is `{:error, :input_too_long}`, nothing is sent, and that one input goes without a vector | +| After a request | the service's `prompt_eval_count` is not below Trinity's count; each vector has the declared width and, for `l2`, unit length | a cut input: no vector, and the embedder is OFF with `{:service_truncated, ...}` until Trinity restarts | +| At boot, then every `check_interval_ms` (5 minutes by default) | `/api/tags` lists the model with the pinned digest; `/api/version` is the pinned version | OFF with `:model_digest_changed`, `:runtime_version_changed`, `:model_not_served` or `:endpoint_unreachable`; until the first check has answered, `:digest_unchecked` | +| Configuration | the locality is declared, the pin is complete, `max_input_tokens` is below `num_ctx` | under `TRINITY_PROFILE=regulated` the boot is refused; otherwise semantic recall is OFF with `{:config, reason}` | + +The digest check is not sticky: if the pinned digest comes back, the same weights are back, and +the next check turns recall on again. A service caught cutting inputs short is sticky until restart, +because a later short input succeeding says nothing about whether it still cuts. + +## What crosses to the service + +The text of every memory the observer writes and of every query a turn recalls with, prefixed by +the configured prompt template, in the clear, over HTTP to the configured `base_url`. Nothing is +hashed or redacted. That is why the endpoint's locality must be declared (`:within_boundary` or +`:external`, never inferred from the address), why `:external` needs `external_opt_in: true`, and why +under `TRINITY_PROFILE=regulated` the endpoint must be on `TRINITY_REGULATED_LLM_ENDPOINTS`. + +The service itself may reach out: Ollama 0.40.0 fetched model recommendations and a cloud model +list from its vendor at start in our measurement. Deny the service egress, and set +`OLLAMA_NO_CLOUD=1`, which in our measurement stopped the calls at start (a refresh was still +scheduled hours later and was not observed). + +## Importing signed weights + +Trinity never ships Tier 3 weights. They arrive as an OCI image layout, the form `cosign save` +writes, holding one artifact whose layers are files named by the `org.opencontainers.image.title` +annotation: one `.gguf` and `model.sig`, an OMS (OpenSSF Model Signing) signature over the model +files. The artifact itself carries a cosign signature. Both are verified offline against public +keys you hold. + +``` +mix trinity.tier3.import --layout --cosign-key cosign.pub --oms-key oms.pub \ + --base-url http://embed.internal:11434 --model qwen3-embedding-0.6b \ + [--cosign ] [--model-signing ] [--out ] \ + [--model-id ] [--num-ctx 8192] [--max-input-tokens 8191] [--query-prompt ] +``` + +In order, any step refusing ends the import: + +1. The layout is copied into a private staging directory; a symbolic link in it is refused. +2. `cosign verify --key --offline=true --local-image --insecure-ignore-tlog=true`. There + is no transparency log on an air-gapped host; your key is the trust. +3. The one image manifest is unpacked; every blob's SHA-256 and size must match its descriptor, and + every title must be a plain file name used once. +4. `model_signing verify key --signature model.sig --public_key ` over the unpacked files. +5. The GGUF's SHA-256 must be the one the OMS statement lists for it. +6. The tokenizer is read out of the GGUF and written under its own SHA-256. +7. The GGUF is sent to the service with `/api/blobs` and the model made with `/api/create`; + `/api/show` must name that same blob; the `/api/tags` digest and `/api/version` are read. +8. A JSON record of every digest is written, and the configuration block is printed. + +The two programs are yours: `cosign` and `model_signing`, from their own releases, on the import +host. Trinity runs them and reads their exit status; it does not carry a signature verifier of its +own. + +The import never changes Trinity's configuration. You set the printed block, then move the store: +`mix trinity.space.retier ollama` builds the new space beside the old one and moves the pointer when +it is complete. + +## The pin is the space + +The embedding space (see `docs/01-architecture.md`) is built from the configuration only. The +`/api/tags` digest is its `revision`, the GGUF's SHA-256 its `weights_digest`, the tokenizer file's +SHA-256 its `tokenizer_digest`, and `num_ctx`, both prompt templates, `max_input_tokens`, the +Ollama version and the declared locality are fields of their own. Changing any of them is a new +space, so a store pinned to the old one turns off until you re-tier: vectors from two +configurations are never ranked together. + +## Operating notes + +- **Replacing a model behind a tag deletes the old blob.** Ollama removes a blob no manifest refers + to. To return to a pinned model after a swap, import its signed delivery again; the same content + gives the same digest. +- **The models directory is content-addressed.** Each blob's file name is its SHA-256, and the + `/api/tags` digest is the SHA-256 of the model's manifest, which is itself a blob naming the GGUF's + SHA-256. A copied directory can be checked with `sha256sum` before it is served. +- **A read-only models directory does not start Ollama 0.40.0**: it removes a legacy `manifests/` + directory at start. Verify the directory, then mount it read-write. +- **Thresholds are per model.** The defaults (recall floor 0.3, dedupe 0.94) were measured on a few + hand-made pairs with Qwen3-Embedding-0.6B; set `thresholds:` for another model. + +## The four gates + +A model is admitted on this runtime only after four gates pass, and they are run again for a new +model, a new Ollama version or a new image. The scripts are in `scripts/tier3/`, and +`scripts/tier3/gates.sh` takes the image as `TIER3_IMAGE`. + +1. **Parity** (`parity.exs`): cosine 0.999 or more against the sentence-transformers reference on + every one of 500 fixtures, at the pinned `num_ctx`. Its red is a `num_ctx` too small for the long + fixtures with the service's default truncation. +2. **Truncation** (`truncation.exs`): an input one token over the most the service accepts, with + `truncate: false`, is an error and never a vector. +3. **The digest pin** (`digest_pin.exs`): a running Trinity goes OFF with `:model_digest_changed` + when the blob behind the tag is replaced. +4. **Offline** (`gates.sh offline`): the service starts with networking disabled and serves + `/api/embed` from a verified models directory. + +Throughput (`bench.exs`) is recorded with no pass mark. The results on the maintainers' machine +are in `docs/perf.md`. diff --git a/lib/mix/tasks/trinity.space.retier.ex b/lib/mix/tasks/trinity.space.retier.ex index 6db70c7..2117178 100644 --- a/lib/mix/tasks/trinity.space.retier.ex +++ b/lib/mix/tasks/trinity.space.retier.ex @@ -8,7 +8,7 @@ defmodule Mix.Tasks.Trinity.Space.Retier do (every semantic memory embedded into it, in batches), then moves the active pointer to it in one transaction. The old space's vectors stay until `mix trinity.space.drop --confirm`. - mix trinity.space.retier [static|local|hosted|fake] + mix trinity.space.retier [static|local|hosted|ollama|fake] Without a name, the first embedder `config :trinity, :memory, embedder:` names. For a store to keep answering while it builds, configure the target ahead of the current embedder diff --git a/lib/mix/tasks/trinity.tier3.import.ex b/lib/mix/tasks/trinity.tier3.import.ex new file mode 100644 index 0000000..e74b0b0 --- /dev/null +++ b/lib/mix/tasks/trinity.tier3.import.ex @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Mix.Tasks.Trinity.Tier3.Import do + @shortdoc "Verifies signed Tier 3 weights offline and creates the model on the operator's Ollama" + + @moduledoc """ + Slice 134, the Tier 3 import path (`Trinity.Memory.Tier3.Import`): an OCI layout as `cosign save` + wrote it, verified with cosign and OMS against the operator's public keys, offline; unpacked + with every blob's digest checked; the tokenizer taken out of the weights; the model created on + the operator's Ollama through its API; and the resulting `/api/tags` digest recorded. + + mix trinity.tier3.import --layout --cosign-key --oms-key \\ + --base-url http://embed.internal:11434 --model qwen3-embedding-0.6b \\ + [--cosign ] [--model-signing ] [--out ] \\ + [--model-id ] [--num-ctx 8192] [--max-input-tokens 8191] \\ + [--query-prompt ] [--document-prompt ] + + On success it prints the record's path and the configuration block that pins Trinity to the + model; setting it is the operator's action. On a refusal it exits non-zero with the reason. + Nothing here starts Trinity or touches its database. + """ + use Boundary, classify_to: Trinity + use Mix.Task + + alias Trinity.Memory.Tier3.Import + + @switches [ + layout: :string, + cosign_key: :string, + oms_key: :string, + base_url: :string, + model: :string, + cosign: :string, + model_signing: :string, + out: :string, + model_id: :string, + num_ctx: :integer, + max_input_tokens: :integer, + query_prompt: :string, + document_prompt: :string + ] + + @impl Mix.Task + def run(argv) do + {opts, _, invalid} = OptionParser.parse(argv, strict: @switches) + if invalid != [], do: Mix.raise("trinity.tier3.import: unknown options #{inspect(invalid)}") + + Mix.Task.run("app.config") + {:ok, _} = Application.ensure_all_started(:req) + + import_opts = + Keyword.take(opts, [:layout, :cosign_key, :oms_key, :base_url, :model, :out]) ++ + [verifier_opts: Keyword.take(opts, [:cosign, :model_signing])] + + case Import.run(import_opts) do + {:ok, record} -> + block = + Import.config_block( + record, + Keyword.take(opts, [ + :model_id, + :num_ctx, + :max_input_tokens, + :query_prompt, + :document_prompt + ]) + ) + + Mix.shell().info(""" + imported #{record["ollama"]["model"]}: digest #{record["ollama"]["digest"]} + weights sha256 #{record["weights_sha256"]}, tokenizer sha256 #{record["tokenizer_sha256"]} + record #{record["record_path"]} + + #{block}\ + """) + + {:error, reason} -> + Mix.raise("trinity.tier3.import refused: #{inspect(reason)}") + end + end +end diff --git a/lib/trinity.ex b/lib/trinity.ex index 36b53e6..9908e06 100644 --- a/lib/trinity.ex +++ b/lib/trinity.ex @@ -73,6 +73,10 @@ defmodule Trinity do # Slice 133: the eval (its own top-level boundary) reads these. Memory.Scorer, Memory.Embedders.Static, + # Slice 134: the eval embeds its queries through `Embedder.embed_query/2`, and the import + # task drives the Tier 3 import path. + Memory.Embedder, + Memory.Tier3.Import, Tools, # Slice 060: what the MCP bridge (a top-level boundary) implements and returns. Tools.Tool, diff --git a/lib/trinity/eval/memory_run.ex b/lib/trinity/eval/memory_run.ex index 70fca49..de5ee69 100644 --- a/lib/trinity/eval/memory_run.ex +++ b/lib/trinity/eval/memory_run.ex @@ -109,7 +109,7 @@ defmodule Trinity.Eval.MemoryRun do try do {:ok, mv} = module.embed(Enum.map(memories, & &1["text"])) - {:ok, qv} = module.embed(Enum.map(queries, & &1["text"])) + {:ok, qv} = Trinity.Memory.Embedder.embed_query(module, Enum.map(queries, & &1["text"])) quant = module.space().quantization %{ diff --git a/lib/trinity/memory.ex b/lib/trinity/memory.ex index 8fc498d..bcffa0c 100644 --- a/lib/trinity/memory.ex +++ b/lib/trinity/memory.ex @@ -31,6 +31,12 @@ defmodule Trinity.Memory do Vector, Scorer, EmbedderConfig, + # Slice 134: the Tier 3 client, its token counter, the GGUF reader and the import path. + Embedders.Ollama, + Embedders.Ollama.Watch, + BPE, + GGUF, + Tier3.Import, ReEmbedWorker, Observer, Retriever, diff --git a/lib/trinity/memory/bpe.ex b/lib/trinity/memory/bpe.ex new file mode 100644 index 0000000..68c342a --- /dev/null +++ b/lib/trinity/memory/bpe.ex @@ -0,0 +1,297 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.BPE do + @moduledoc """ + A byte-level BPE tokenizer in pure Elixir (slice 134): how the Tier 3 client counts the tokens + it is about to send, so that it can refuse an over-length input itself and catch a service + that truncates silently (`Trinity.Memory.Embedders.Ollama`). + + It reproduces the pipeline of the Qwen2 family's tokenizer (`tokenizer.ggml.pre = "qwen2"` in a + GGUF; a `ByteLevel` BPE in Hugging Face `tokenizers`): + + 1. **Added tokens first**, matched in the raw text, leftmost and longest: the control and + user-defined tokens (`<|endoftext|>`, ``, ...) stand for their own ids, and the text + between them is tokenized on its own. + 2. **NFC** each remaining piece. + 3. **Pre-tokenise** with the Qwen2 pattern (contractions, letters with one leading + non-letter, single digits, punctuation runs, newlines, whitespace). + 4. **Byte-level**: every UTF-8 byte of a piece becomes one printable character (the GPT-2 + byte map), so no input is ever unknown. + 5. **BPE**: merge the adjacent pair of lowest rank until no pair has a rank, then look each + symbol up. + + `count/2` adds the beginning and end tokens the model's metadata asks for (Qwen3-Embedding + appends `<|endoftext|>`), which is what the service reports as `prompt_eval_count`. + + The vocabulary and merges are the model's, read from its verified GGUF at import and kept in a + file of their own (`to_file/1`, `from_file/1`); none are in this tree. + """ + + @enforce_keys [:vocab, :ranks, :added, :bos, :eos] + defstruct [:vocab, :ranks, :added, :bos, :eos] + + @type t :: %__MODULE__{ + vocab: %{String.t() => non_neg_integer()}, + ranks: %{{String.t(), String.t()} => non_neg_integer()}, + added: [{String.t(), non_neg_integer()}], + bos: non_neg_integer() | nil, + eos: non_neg_integer() | nil + } + + @format "trinity-bpe-1" + + # Hugging Face's Qwen2 split pattern (the `(?i:...)` contractions, then letters with an optional + # leading non-letter, single digits, punctuation runs with trailing newlines, newline runs, + # whitespace not followed by a non-space, whitespace). `u` makes `\p{L}` and `\p{N}` Unicode. + # `\s` is spelled out as Unicode's White_Space, which is what the reference's regex engine + # means by it: PCRE's `\s` also matches U+180E (a format character since Unicode 6.3), and + # that one character split a parity fixture differently (slice 134 NOTES, decision 14). + @ws ~S"\t\n\x{0B}\f\r \x{85}\x{A0}\x{1680}\x{2000}-\x{200A}\x{2028}\x{2029}\x{202F}\x{205F}\x{3000}" + @pattern "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^#{@ws}\\p{L}\\p{N}]+[\\r\\n]*|[#{@ws}]*[\\r\\n]+|[#{@ws}]+(?![^#{@ws}])|[#{@ws}]+" + + @doc """ + A tokenizer from a model's tokens (in id order), its merges (`"a b"` strings, in rank order), + and options: `added:` the ids matched as whole tokens in the raw text, `bos:` and `eos:` the ids + `count/2` adds (nil for none). + """ + @spec new([String.t()], [String.t()], keyword()) :: t() + def new(tokens, merges, opts \\ []) do + vocab = tokens |> Enum.with_index() |> Map.new() + + ranks = + merges + |> Enum.with_index() + |> Map.new(fn {m, i} -> + [a, b] = String.split(m, " ", parts: 2) + {{a, b}, i} + end) + + by_id = List.to_tuple(tokens) + + added = + opts + |> Keyword.get(:added, []) + |> Enum.map(&{elem(by_id, &1), &1}) + |> Enum.sort_by(fn {t, _} -> -byte_size(t) end) + + %__MODULE__{ + vocab: vocab, + ranks: ranks, + added: added, + bos: Keyword.get(opts, :bos), + eos: Keyword.get(opts, :eos) + } + end + + @doc """ + The tokenizer a GGUF's metadata describes (`Trinity.Memory.GGUF.metadata/1`): a `gpt2` model + with the `qwen2` pre-tokenizer. Control (3) and user-defined (4) token types are the added + tokens. `{:error, reason}` for any other tokenizer, which this module does not reproduce. + """ + @spec from_gguf(map()) :: {:ok, t()} | {:error, term()} + def from_gguf(%{} = meta) do + with {:model, "gpt2"} <- {:model, meta["tokenizer.ggml.model"]}, + {:pre, "qwen2"} <- {:pre, meta["tokenizer.ggml.pre"]}, + tokens when is_list(tokens) <- meta["tokenizer.ggml.tokens"], + merges when is_list(merges) <- meta["tokenizer.ggml.merges"], + types when is_list(types) <- meta["tokenizer.ggml.token_type"] do + added = for {t, i} <- Enum.with_index(types), t in [3, 4], do: i + + {:ok, + new(tokens, merges, + added: added, + bos: + if(meta["tokenizer.ggml.add_bos_token"] == true, + do: meta["tokenizer.ggml.bos_token_id"] + ), + eos: + if(meta["tokenizer.ggml.add_eos_token"] == true, + do: meta["tokenizer.ggml.eos_token_id"] + ) + )} + else + {:model, other} -> {:error, {:unsupported_tokenizer, other}} + {:pre, other} -> {:error, {:unsupported_pre_tokenizer, other}} + _ -> {:error, :tokenizer_metadata_missing} + end + end + + @doc """ + The token ids of a text, without the beginning and end tokens. + """ + @spec encode(t(), String.t()) :: [non_neg_integer()] + def encode(%__MODULE__{} = t, text) when is_binary(text) do + t.added + |> split_added(text) + |> Enum.flat_map(fn + {:added, id} -> [id] + {:text, piece} -> encode_piece(t, piece) + end) + end + + @doc "How many tokens the model sees for a text: `encode/2` plus the beginning and end tokens." + @spec count(t(), String.t()) :: non_neg_integer() + def count(%__MODULE__{} = t, text) do + length(encode(t, text)) + if(t.bos, do: 1, else: 0) + if(t.eos, do: 1, else: 0) + end + + defp encode_piece(t, piece) do + piece + |> :unicode.characters_to_nfc_binary() + |> pre_tokenize() + |> Enum.flat_map(fn word -> word |> byte_symbols() |> merge(t.ranks) |> ids(t.vocab) end) + end + + @doc "The Qwen2 pre-tokenizer's pieces of a (normalised) text." + @spec pre_tokenize(String.t()) :: [String.t()] + def pre_tokenize(text) do + Regex.scan(regex(), text) |> Enum.map(&hd/1) + end + + defp regex do + case :persistent_term.get({__MODULE__, :regex}, nil) do + nil -> + r = Regex.compile!(@pattern, "u") + :persistent_term.put({__MODULE__, :regex}, r) + r + + r -> + r + end + end + + # Leftmost, longest added token; the text around it split again. + defp split_added([], text), do: [{:text, text}] + + defp split_added(added, text) do + case first_added(added, text) do + nil -> + [{:text, text}] + + {pos, tok, id} -> + before = binary_part(text, 0, pos) + rest_at = pos + byte_size(tok) + rest = binary_part(text, rest_at, byte_size(text) - rest_at) + head = if before == "", do: [], else: [{:text, before}] + head ++ [{:added, id} | split_added(added, rest)] + end + end + + defp first_added(added, text) do + added + |> Enum.flat_map(fn {tok, id} -> + case :binary.match(text, tok) do + {pos, _} -> [{pos, tok, id}] + :nomatch -> [] + end + end) + |> Enum.min_by(fn {pos, tok, _} -> {pos, -byte_size(tok)} end, fn -> nil end) + end + + ## Byte level + + @byte_map (fn -> + printable = + Enum.to_list(?!..?~) ++ Enum.to_list(0xA1..0xAC) ++ Enum.to_list(0xAE..0xFF) + + {map, _} = + Enum.reduce(0..255, {%{}, 0}, fn b, {acc, n} -> + if b in printable, + do: {Map.put(acc, b, <>), n}, + else: {Map.put(acc, b, <<256 + n::utf8>>), n + 1} + end) + + map + end).() + + @doc "The GPT-2 byte map: each byte's printable stand-in." + @spec byte_char(byte()) :: String.t() + def byte_char(b) when b in 0..255, do: Map.fetch!(@byte_map, b) + + defp byte_symbols(word), do: for(<>, do: Map.fetch!(@byte_map, b)) + + ## BPE + + defp merge([_] = symbols, _ranks), do: symbols + defp merge([], _ranks), do: [] + + defp merge(symbols, ranks) do + case best_pair(symbols, ranks) do + nil -> symbols + pair -> symbols |> apply_merge(pair, []) |> merge(ranks) + end + end + + defp best_pair(symbols, ranks) do + symbols + |> Enum.chunk_every(2, 1, :discard) + |> Enum.reduce(nil, fn [a, b], best -> + case Map.fetch(ranks, {a, b}) do + {:ok, r} when best == nil or r < elem(best, 0) -> {r, {a, b}} + _ -> best + end + end) + |> case do + nil -> nil + {_, pair} -> pair + end + end + + defp apply_merge([a, b | rest], {a, b} = pair, acc), do: apply_merge(rest, pair, [a <> b | acc]) + defp apply_merge([x | rest], pair, acc), do: apply_merge(rest, pair, [x | acc]) + defp apply_merge([], _pair, acc), do: Enum.reverse(acc) + + # A byte-level vocabulary holds every single byte's character, so a symbol it lacks is split + # back into those; a vocabulary missing a byte is not one this module can count, and raises. + defp ids(symbols, vocab) do + Enum.flat_map(symbols, fn s -> + case Map.fetch(vocab, s) do + {:ok, id} -> [id] + :error -> s |> String.graphemes() |> Enum.map(&Map.fetch!(vocab, &1)) + end + end) + end + + ## The tokenizer file + + @doc """ + The tokenizer as a file's bytes: JSON with fixed key order (`format`, `bos`, `eos`, `added`, + `tokens`, `merges`), so the same tokenizer is always the same bytes and its SHA-256 can be a + space's `tokenizer_digest`. + """ + @spec to_file(t()) :: binary() + def to_file(%__MODULE__{} = t) do + tokens = t.vocab |> Enum.sort_by(&elem(&1, 1)) |> Enum.map(&elem(&1, 0)) + + merges = + t.ranks |> Enum.sort_by(&elem(&1, 1)) |> Enum.map(fn {{a, b}, _} -> a <> " " <> b end) + + added = t.added |> Enum.map(&elem(&1, 1)) |> Enum.sort() + + Jason.OrderedObject.new([ + {"format", @format}, + {"bos", t.bos}, + {"eos", t.eos}, + {"added", added}, + {"tokens", tokens}, + {"merges", merges} + ]) + |> Jason.encode!() + end + + @doc "A tokenizer back from `to_file/1`'s bytes." + @spec from_file(binary()) :: {:ok, t()} | {:error, term()} + def from_file(bin) when is_binary(bin) do + case Jason.decode(bin) do + {:ok, %{"format" => @format, "tokens" => tokens, "merges" => merges} = m} + when is_list(tokens) and is_list(merges) -> + {:ok, new(tokens, merges, added: m["added"] || [], bos: m["bos"], eos: m["eos"])} + + {:ok, _} -> + {:error, :not_a_tokenizer_file} + + {:error, _} -> + {:error, :not_a_tokenizer_file} + end + end +end diff --git a/lib/trinity/memory/embedder.ex b/lib/trinity/memory/embedder.ex index 089dca6..77c86cb 100644 --- a/lib/trinity/memory/embedder.ex +++ b/lib/trinity/memory/embedder.ex @@ -7,7 +7,8 @@ defmodule Trinity.Memory.Embedder do all-MiniLM-L6-v2 through Bumblebee and EXLA, on this machine), `:static` (`Embedders.Static`, slice 133: the pure-Elixir floor), `:hosted` (`Embedders.Hosted`, a configured alternative only: it sends text to a provider and is never chosen for the - operator), `:fake` (the suite), or a module implementing this behaviour. + operator), `:ollama` (`Embedders.Ollama`, slice 134: an operator-run Ollama, admitted on the + operator's pin), `:fake` (the suite), or a module implementing this behaviour. Slice 133: every embedder names the embedding space it writes (`space/0`, a `Trinity.Memory.Space`), and every stored vector carries that space's ID. A store answers @@ -40,8 +41,33 @@ defmodule Trinity.Memory.Embedder do """ @callback thresholds() :: %{floor: float(), dedupe: float()} + @doc """ + Slice 134: the vectors for queries, when the model embeds a query differently from a document + (an instruction-tuned model's query template). Without it, `embed/1` serves both. + """ + @callback embed_query([String.t()]) :: {:ok, [vector()]} | {:error, term()} + + @doc """ + Slice 134: the endpoint the embedder sends text to, from `config :trinity, :memory` and the + LLM registry's models: `{:endpoint, url}` (nil when unstated), or `:in_process`. Without it the + embedder runs in this BEAM. `Trinity.Memory.EmbedderConfig` holds its locality to it. + """ + @callback endpoint(memory :: keyword(), models :: [map()]) :: + {:endpoint, String.t() | nil} | :in_process + + @doc """ + Slice 134: the embedder's own configuration faults (`:ok` or `{:error, reason}`), which refuse + a regulated boot and turn semantic memory OFF under `:default`, like 133's locality faults. + """ + @callback check_config(memory :: keyword()) :: :ok | {:error, term()} + + @doc "Slice 134: the processes the embedder needs, started by `Trinity.Memory.Supervisor`." + @callback children() :: [Supervisor.child_spec() | module() | {module(), term()}] + + @optional_callbacks embed_query: 1, endpoint: 2, check_config: 1, children: 0 + @typedoc "An embedder as configuration names it." - @type name :: :local | :static | :hosted | :fake | module() + @type name :: :local | :static | :hosted | :ollama | :fake | module() @doc """ The configured choice: `:local` unless configuration says otherwise. Slice 133: @@ -65,6 +91,7 @@ defmodule Trinity.Memory.Embedder do def module(:local), do: Trinity.Memory.Embedders.Bumblebee def module(:static), do: Trinity.Memory.Embedders.Static def module(:hosted), do: Trinity.Memory.Embedders.Hosted + def module(:ollama), do: Trinity.Memory.Embedders.Ollama def module(:fake), do: Trinity.Memory.Embedders.Fake def module(module) when is_atom(module), do: module @@ -72,6 +99,7 @@ defmodule Trinity.Memory.Embedder do "local" => :local, "static" => :static, "hosted" => :hosted, + "ollama" => :ollama, "fake" => :fake } @@ -101,6 +129,19 @@ defmodule Trinity.Memory.Embedder do @spec embed([String.t()]) :: {:ok, [vector()]} | {:error, term()} def embed(texts) when is_list(texts), do: impl().embed(texts) + @doc """ + Embeds queries through `module`: its `embed_query/1` when it has one (slice 134), else + `embed/1`. + """ + @spec embed_query(module(), [String.t()]) :: {:ok, [vector()]} | {:error, term()} + def embed_query(module, texts) when is_atom(module) and is_list(texts) do + Code.ensure_loaded(module) + + if function_exported?(module, :embed_query, 1), + do: module.embed_query(texts), + else: module.embed(texts) + end + @doc "Cosine similarity of two vectors of the same length (both L2-normalised: their dot)." @spec cosine(vector(), vector()) :: float() def cosine(a, b) when length(a) == length(b) do diff --git a/lib/trinity/memory/embedder_config.ex b/lib/trinity/memory/embedder_config.ex index c438c3f..0c7a3e5 100644 --- a/lib/trinity/memory/embedder_config.ex +++ b/lib/trinity/memory/embedder_config.ex @@ -15,6 +15,8 @@ defmodule Trinity.Memory.EmbedderConfig do `:in_process` is a fault (an endpoint is not in this process). * `:external` needs the operator's opt-in, `external_opt_in: true`, under both profiles: no memory text leaves for an external embedder unless the operator opted in for this deployment. + * Slice 134: an embedder may name its own configuration faults (`check_config/1`, the Tier 3 + client's pin and limits); they are faults like these, checked after the locality. * Under `:regulated` only, the endpoint must be stated and on `TRINITY_REGULATED_LLM_ENDPOINTS` (`:not_allow_listed` otherwise), for `:within_boundary` and `:external` alike. @@ -45,9 +47,16 @@ defmodule Trinity.Memory.EmbedderConfig do end defp check_one(profile, name, memory, models, raw) do + module = Embedder.module(name) + + with :ok <- check_locality(profile, name, module, memory, models, raw), + do: own_config(module, memory) + end + + defp check_locality(profile, name, module, memory, models, raw) do declared = Keyword.get(memory, :locality) - case endpoint(Embedder.module(name), memory, models) do + case endpoint(module, memory, models) do :in_process -> if declared in [nil, :in_process], do: :ok, @@ -93,17 +102,19 @@ defmodule Trinity.Memory.EmbedderConfig do end end - # Whether an embedder reaches an endpoint, and which. Only the hosted embedder does today; its - # URL is its registry model's `base_url` (nil when the provider's default endpoint is used, - # which `:regulated` refuses as unstated). - defp endpoint(Trinity.Memory.Embedders.Hosted, memory, models) do - model = Keyword.get(memory, :hosted_model, "nvidia:embed") + # Whether an embedder reaches an endpoint, and which: its `endpoint/2` callback (slice 134; 133 + # asked the hosted embedder by name). An embedder without one runs in this BEAM. + defp endpoint(module, memory, models) do + Code.ensure_loaded(module) - url = - Enum.find_value(models, fn m -> if Map.get(m, :id) == model, do: Map.get(m, :base_url) end) - - {:endpoint, url} + if function_exported?(module, :endpoint, 2), + do: module.endpoint(memory, models), + else: :in_process end - defp endpoint(_in_process, _memory, _models), do: :in_process + # Slice 134: the embedder's own configuration faults, after its locality. + defp own_config(module, memory) do + Code.ensure_loaded(module) + if function_exported?(module, :check_config, 1), do: module.check_config(memory), else: :ok + end end diff --git a/lib/trinity/memory/embedders/hosted.ex b/lib/trinity/memory/embedders/hosted.ex index d5c0b34..4a5c2a7 100644 --- a/lib/trinity/memory/embedders/hosted.ex +++ b/lib/trinity/memory/embedders/hosted.ex @@ -54,6 +54,21 @@ defmodule Trinity.Memory.Embedders.Hosted do } end + @doc """ + The endpoint: the registry model's `base_url` (nil when the provider's default endpoint is + used, which `:regulated` refuses as unstated). Slice 134 moved this here from + `Trinity.Memory.EmbedderConfig`, unchanged, as the `endpoint/2` callback. + """ + @impl true + def endpoint(memory, models) do + model = Keyword.get(memory, :hosted_model, "nvidia:embed") + + url = + Enum.find_value(models, fn m -> if Map.get(m, :id) == model, do: Map.get(m, :base_url) end) + + {:endpoint, url} + end + @doc "Not measured for any hosted model (slice 032 G1: nemotron's raw vectors do not separate)." @impl true def thresholds do diff --git a/lib/trinity/memory/embedders/ollama.ex b/lib/trinity/memory/embedders/ollama.ex new file mode 100644 index 0000000..9a181ef --- /dev/null +++ b/lib/trinity/memory/embedders/ollama.ex @@ -0,0 +1,496 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Embedders.Ollama do + @moduledoc """ + The Tier 3 client (slice 134): an embedding model served by an operator-run Ollama, reached + over HTTP, admitted only on the operator's pin and turned OFF, never silently wrong, when the + service changes underneath it. + + config :trinity, :memory, + embedder: :ollama, + locality: :within_boundary, # declared by the operator, never inferred (D2) + ollama: [ + base_url: "http://embed.internal:11434", + model: "qwen3-embedding-0.6b:latest", + model_id: "Qwen/Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-f16.gguf", + digest: "", + weights_sha256: "", + tokenizer_sha256: "", + runtime_version: "0.40.0", + num_ctx: 8192, + max_input_tokens: 8191, + dim: 1024, + pooling: "last", + query_prompt: "Instruct: ...\\nQuery:" + ] + + `mix trinity.tier3.import` prints this block from a verified import (`Trinity.Memory.Tier3.Import`). + + **Every request** to `/api/embed` carries `truncate: false` and `options.num_ctx`, so the + service refuses an over-length input instead of embedding its first part, which is Ollama's + default. **Trinity counts first**: each input is tokenized here (`Trinity.Memory.BPE`, over + the tokenizer taken from the verified weights), and one over `max_input_tokens` is + `{:error, :input_too_long}` with nothing sent. **And checks after**: the answer's + `prompt_eval_count` must not be below Trinity's count; when it is, the service cut the input + despite `truncate: false`, the caller gets `{:error, :input_too_long}` and no vector, and the + embedder is OFF with `{:service_truncated, ...}` until the node restarts (slice 134 NOTES, + decision 4). Every vector is checked for the declared dimension and, when `normalisation` is + `"l2"`, for unit length. + + **The pin.** The space (`space/0`) is built from configuration only: the `/api/tags` digest + is its `revision`, the GGUF's SHA-256 its `weights_digest`, the tokenizer file's its + `tokenizer_digest`, with `num_ctx`, both prompt templates and the runtime version, so changing + any of them is another space (AC3). `Embedders.Ollama.Watch` compares the service with the pin + at boot and every `check_interval_ms`; a different digest is `{:off, :model_digest_changed}`, + another version `{:off, :runtime_version_changed}` (AC2). Nothing here ever follows the + service to a new space. + + What crosses to the service: the text of each memory and each query, with its prompt template, + in the clear, to the operator's declared endpoint. Nothing is hashed or redacted, which is why + the endpoint's locality is declared and, under `:regulated`, allow-listed + (`Trinity.Memory.EmbedderConfig`). + """ + @behaviour Trinity.Memory.Embedder + + require Logger + + alias Trinity.Memory.{BPE, Space} + + Module.register_attribute(__MODULE__, :sobelow_skip, persist: true) + + @schema NimbleOptions.new!( + base_url: [type: :string, required: true, doc: "The service's base URL."], + model: [type: :string, required: true, doc: "The model's name on the service."], + model_id: [type: :string, doc: "The upstream model's id (the space's `model_id`)."], + digest: [type: :string, required: true, doc: "The pinned `/api/tags` digest."], + weights_sha256: [type: :string, required: true, doc: "The GGUF's SHA-256."], + tokenizer_sha256: [ + type: :string, + required: true, + doc: "The tokenizer file's SHA-256." + ], + tokenizer_path: [type: :string, doc: "The tokenizer file (default: data dir)."], + runtime_version: [type: :string, required: true, doc: "The pinned Ollama version."], + num_ctx: [type: :pos_integer, required: true, doc: "Sent on every request."], + max_input_tokens: [type: :pos_integer, required: true, doc: "Trinity's own limit."], + dim: [type: :pos_integer, required: true, doc: "The vectors' width."], + pooling: [type: :string, default: "last", doc: "The model's pooling, declared."], + normalisation: [ + type: {:in, ["l2", "none"]}, + default: "l2", + doc: "Checked per vector." + ], + query_prompt: [type: :string, default: "", doc: "Prefixed to every query."], + document_prompt: [type: :string, default: "", doc: "Prefixed to every document."], + check_interval_ms: [type: :pos_integer, default: 300_000, doc: "The pin's re-check."], + timeout_ms: [type: :pos_integer, default: 30_000, doc: "One request's limit."], + thresholds: [type: :map, doc: "`%{floor: f, dedupe: f}`; provisional default."] + ) + + @hex64 ~r/\A[0-9a-f]{64}\z/ + @check_key {__MODULE__, :check} + @truncated_key {__MODULE__, :service_truncated} + @norm_tolerance 1.0e-3 + + ## Configuration + + @doc "The configuration's options, documented (`NimbleOptions.docs/1`)." + @spec options_doc() :: String.t() + def options_doc, do: NimbleOptions.docs(@schema) + + @doc """ + The validated configuration, or the fault: an option missing or of the wrong type, a digest + that is not 64 lowercase hex characters, or `max_input_tokens` not below `num_ctx` (the + service accepts at most `num_ctx - 1` tokens: slice 134 NOTES). + """ + @spec config(keyword()) :: {:ok, keyword()} | {:error, term()} + def config(memory \\ Application.get_env(:trinity, :memory, [])) do + with {:ok, opts} <- validate(Keyword.get(memory, :ollama, [])), + :ok <- hex(opts, :digest), + :ok <- hex(opts, :weights_sha256), + :ok <- hex(opts, :tokenizer_sha256) do + if opts[:max_input_tokens] < opts[:num_ctx], + do: {:ok, opts}, + else: {:error, {:max_input_not_below_num_ctx, opts[:max_input_tokens], opts[:num_ctx]}} + end + end + + defp validate(opts) when is_list(opts) do + case NimbleOptions.validate(opts, @schema) do + {:ok, opts} -> + {:ok, opts} + + {:error, %NimbleOptions.ValidationError{key: key, message: m}} -> + {:error, {:option, key, m}} + end + end + + defp validate(_), do: {:error, {:option, :ollama, "must be a keyword list"}} + + defp hex(opts, key) do + if Regex.match?(@hex64, opts[key]), do: :ok, else: {:error, {:not_a_sha256, key}} + end + + @doc "A configuration fault, as `Trinity.Memory.EmbedderConfig` asks every embedder." + @impl true + def check_config(memory) do + case config(memory) do + {:ok, _} -> :ok + {:error, reason} -> {:error, {:ollama_config, reason}} + end + end + + @doc "The endpoint this embedder sends text to (D2: its locality must be declared)." + @impl true + def endpoint(memory, _models), do: {:endpoint, get_in(memory, [:ollama, :base_url])} + + ## The embedder + + @impl true + def dim, do: opt(:dim) || 0 + + @impl true + def model_id, do: "ollama:" <> to_string(opt(:model) || "unconfigured") + + @doc """ + The space, from configuration only (decision 5): `revision` is the pinned `/api/tags` + digest, `weights_digest` the GGUF's SHA-256, `tokenizer_digest` the tokenizer file's; + `num_ctx`, the prompts and the runtime version are fields of their own. A field the + configuration lacks is `"unrecorded"` (such a configuration is refused before it serves). + """ + @impl true + def space do + u = "unrecorded" + memory = Application.get_env(:trinity, :memory, []) + o = Keyword.get(memory, :ollama, []) + get = fn key, default -> Keyword.get(o, key) || default end + + %Space{ + model_id: get.(:model_id, nil) || get.(:model, u), + revision: get.(:digest, u), + weights_digest: get.(:weights_sha256, u), + tokenizer_digest: get.(:tokenizer_sha256, u), + dim: get.(:dim, 1), + pooling: get.(:pooling, "last"), + normalisation: get.(:normalisation, "l2"), + quantization: "f32", + query_prompt: get.(:query_prompt, ""), + document_prompt: get.(:document_prompt, ""), + max_input_tokens: get.(:max_input_tokens, u), + truncation: "refuse", + runtime: "ollama", + runtime_version: get.(:runtime_version, u), + locality: (Keyword.get(memory, :locality) || u) |> to_string(), + num_ctx: get.(:num_ctx, u) + } + end + + @doc """ + `config ..., ollama: [thresholds: %{floor: _, dedupe: _}]`, else floor 0.3 and dedupe 0.94: + provisional, measured on hand pairs with Qwen3-Embedding-0.6B (f16) on 2026-10-08 (slice 134 + NOTES, decision 17): restatements 0.9489 to 0.965, a decision and its negation 0.884, two facts + differing in one entity about 0.86, unrelated pairs 0.18 to 0.25, a query and its answer 0.5557 + with the query template. A swap of word order ("tea over coffee", "coffee over tea") measured + 0.9623, inside the restatements, and no threshold separates it. Another model needs its own; the + eval re-derives them. + """ + @impl true + def thresholds, do: opt(:thresholds) || %{floor: 0.3, dedupe: 0.94} + + @doc """ + `:ok`, or why not: the configuration (`{:config, reason}`), the tokenizer + (`:tokenizer_missing`, `:tokenizer_digest_mismatch`), a service caught truncating, or the + pin's last check (`:digest_unchecked` until the first has answered, `:model_digest_changed`, + `:runtime_version_changed`, `:model_not_served`, `:endpoint_unreachable`). + """ + @impl true + def availability do + with {:ok, opts} <- config_or_off(), + {:ok, _tok} <- tokenizer(opts), + :ok <- not_truncating(opts) do + last_check(opts) + end + end + + defp config_or_off do + case config() do + {:ok, opts} -> {:ok, opts} + {:error, reason} -> {:off, {:config, reason}} + end + end + + defp not_truncating(opts) do + case :persistent_term.get(@truncated_key, nil) do + {pin, detail} -> if pin == pin(opts), do: {:off, {:service_truncated, detail}}, else: :ok + nil -> :ok + end + end + + defp last_check(opts) do + case :persistent_term.get(@check_key, nil) do + {pin, result} -> if pin == pin(opts), do: result, else: {:off, :digest_unchecked} + nil -> {:off, :digest_unchecked} + end + end + + @doc "The children this embedder needs: the pin's watcher (`Trinity.Memory.Supervisor`)." + @impl true + def children, do: [__MODULE__.Watch] + + @doc "Embeds documents (the document template)." + @impl true + def embed(texts) when is_list(texts), do: embed_with(texts, :document_prompt) + + @doc "Embeds queries (the query template)." + @impl true + def embed_query(texts) when is_list(texts), do: embed_with(texts, :query_prompt) + + defp embed_with(texts, prompt_key) do + with :ok <- availability(), + {:ok, opts} <- config_or_off(), + {:ok, tok} <- tokenizer(opts) do + inputs = Enum.map(texts, &(opts[prompt_key] <> &1)) + counts = Enum.map(inputs, &BPE.count(tok, &1)) + + if Enum.any?(counts, &(&1 > opts[:max_input_tokens])), + do: {:error, :input_too_long}, + else: request(opts, inputs, Enum.sum(counts)) + else + {:off, reason} -> {:error, reason} + {:error, reason} -> {:error, reason} + end + end + + defp request(_opts, [], _counted), do: {:ok, []} + + defp request(opts, inputs, counted) do + body = %{ + model: opts[:model], + input: inputs, + truncate: false, + options: %{num_ctx: opts[:num_ctx]} + } + + case http(:post, opts, "/api/embed", json: body) do + {:ok, %{status: 200, body: %{"embeddings" => vectors} = answer}} -> + answered(opts, vectors, answer["prompt_eval_count"], counted, length(inputs)) + + {:ok, %{status: 400, body: body}} -> + if context_length?(body), + do: {:error, :input_too_long}, + else: {:error, {:service_refused, 400, error_text(body)}} + + {:ok, %{status: 404}} -> + {:error, :model_not_served} + + {:ok, %{status: status}} when status >= 500 -> + {:error, :endpoint_unreachable} + + {:ok, %{status: status, body: body}} -> + {:error, {:service_refused, status, error_text(body)}} + + {:error, _} -> + {:error, :endpoint_unreachable} + end + end + + defp answered(opts, vectors, served, counted, n) do + cond do + not is_integer(served) -> + {:error, :prompt_eval_count_missing} + + served < counted -> + detail = %{counted: counted, served: served} + :persistent_term.put(@truncated_key, {pin(opts), detail}) + Logger.error("memory: the embedding service cut the input (#{inspect(detail)}); OFF") + {:error, :input_too_long} + + length(vectors) != n -> + {:error, {:vector_count, length(vectors), n}} + + bad = Enum.find(vectors, &(not fits?(&1, opts))) -> + {:error, {:vector_shape, length(bad), opts[:dim]}} + + true -> + if served > counted, + do: Logger.warning("memory: the service counted #{served} tokens, Trinity #{counted}") + + {:ok, vectors} + end + end + + defp fits?(v, opts) when is_list(v) do + length(v) == opts[:dim] and + (opts[:normalisation] == "none" or + abs(:math.sqrt(Enum.reduce(v, 0.0, &(&1 * &1 + &2))) - 1.0) < @norm_tolerance) + end + + defp fits?(_, _), do: false + + defp context_length?(%{"error" => e}) when is_binary(e), do: e =~ "context length" + defp context_length?(_), do: false + + defp error_text(%{"error" => e}) when is_binary(e), do: e + defp error_text(_), do: nil + + ## The pin's check (the watcher calls it; a test may too) + + @doc """ + Compares the service with the pin now: `/api/tags` must list the model with the pinned + digest, `/api/version` must be the pinned version. Records and returns the result. + """ + @spec check() :: :ok | {:off, term()} + def check do + case config() do + {:ok, opts} -> + result = check_service(opts) + record(opts, result) + result + + {:error, reason} -> + {:off, {:config, reason}} + end + end + + defp check_service(opts) do + with {:ok, digest} <- served_digest(opts), + :ok <- same(digest, opts[:digest], :model_digest_changed), + {:ok, version} <- served_version(opts) do + same(version, opts[:runtime_version], :runtime_version_changed) + end + end + + defp same(a, a, _reason), do: :ok + defp same(_, _, reason), do: {:off, reason} + + @doc "The digest the service lists for the configured model, or why not." + @spec served_digest(keyword()) :: {:ok, String.t()} | {:off, term()} + def served_digest(opts) do + case http(:get, opts, "/api/tags", []) do + {:ok, %{status: 200, body: %{"models" => models}}} when is_list(models) -> + name = full_name(opts[:model]) + + case Enum.find(models, &(full_name(&1["name"] || &1["model"]) == name)) do + %{"digest" => d} when is_binary(d) -> {:ok, String.replace_prefix(d, "sha256:", "")} + _ -> {:off, :model_not_served} + end + + _ -> + {:off, :endpoint_unreachable} + end + end + + defp served_version(opts) do + case http(:get, opts, "/api/version", []) do + {:ok, %{status: 200, body: %{"version" => v}}} when is_binary(v) -> {:ok, v} + _ -> {:off, :endpoint_unreachable} + end + end + + @doc "A model name with Ollama's default tag added (`name` is `name:latest`)." + @spec full_name(String.t() | nil) :: String.t() | nil + def full_name(nil), do: nil + def full_name(name), do: if(String.contains?(name, ":"), do: name, else: name <> ":latest") + + defp record(opts, result) do + entry = {pin(opts), result} + previous = :persistent_term.get(@check_key, nil) + + if previous != entry do + :persistent_term.put(@check_key, entry) + + case result do + :ok -> + Logger.info("memory: the embedding service matches its pin") + + {:off, reason} -> + Logger.warning("memory: the embedding service is OFF: #{inspect(reason)}") + end + end + + :ok + end + + @doc "Forgets the last check and a recorded truncation (a test starting clean; a restart does the same)." + @spec reset() :: :ok + def reset do + for key <- [@check_key, @truncated_key], :persistent_term.get(key, nil) != nil do + :persistent_term.erase(key) + end + + :ok + end + + # What a check result and a truncation record belong to: a change to any of these is a new + # pin, and nothing recorded for the old one applies. + defp pin(opts), + do: {opts[:base_url], full_name(opts[:model]), opts[:digest], opts[:runtime_version]} + + ## The tokenizer + + @doc """ + The tokenizer the configuration pins, loaded once per VM and its SHA-256 checked: + `{:off, :tokenizer_missing}` or `{:off, :tokenizer_digest_mismatch}` otherwise. + """ + @spec tokenizer(keyword()) :: {:ok, BPE.t()} | {:off, term()} + def tokenizer(opts) do + path = tokenizer_path(opts) + key = {__MODULE__, :tokenizer, path, opts[:tokenizer_sha256]} + + case :persistent_term.get(key, nil) do + nil -> + result = load_tokenizer(path, opts[:tokenizer_sha256]) + if match?({:ok, _}, result), do: :persistent_term.put(key, result) + result + + result -> + result + end + end + + @doc "Where the tokenizer file is: `tokenizer_path:`, else `/models/tier3/.bpe.json`." + @spec tokenizer_path(keyword()) :: Path.t() + def tokenizer_path(opts) do + opts[:tokenizer_path] || + Path.join([ + Trinity.Paths.data_dir(), + "models", + "tier3", + "#{opts[:tokenizer_sha256]}.bpe.json" + ]) + end + + # sobelow_skip reason: Traversal.FileModule: the path is the operator's configured tokenizer + # file, or a name under the data directory made of a SHA-256 the configuration validated as + # 64 hex characters; the bytes read are checked against that SHA-256 before use. + @sobelow_skip ["Traversal.FileModule"] + defp load_tokenizer(path, sha) do + with {:ok, bin} <- File.read(path), + ^sha <- :crypto.hash(:sha256, bin) |> Base.encode16(case: :lower), + {:ok, tok} <- BPE.from_file(bin) do + {:ok, tok} + else + {:error, :enoent} -> {:off, :tokenizer_missing} + {:error, reason} -> {:off, {:tokenizer_unreadable, reason}} + _digest -> {:off, :tokenizer_digest_mismatch} + end + end + + ## HTTP + + defp http(method, opts, path, extra) do + Req.request( + [ + method: method, + url: String.trim_trailing(opts[:base_url], "/") <> path, + retry: false, + receive_timeout: opts[:timeout_ms], + connect_options: [timeout: opts[:timeout_ms]] + ] ++ extra + ) + end + + defp opt(key) do + Application.get_env(:trinity, :memory, []) |> Keyword.get(:ollama, []) |> Keyword.get(key) + end +end diff --git a/lib/trinity/memory/embedders/ollama/watch.ex b/lib/trinity/memory/embedders/ollama/watch.ex new file mode 100644 index 0000000..b33be94 --- /dev/null +++ b/lib/trinity/memory/embedders/ollama/watch.ex @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Embedders.Ollama.Watch do + @moduledoc """ + Re-checks the Tier 3 service against its pin (slice 134, AC2): once as soon as it starts, so a + boot never serves an unchecked model, and then every `check_interval_ms` + (`Trinity.Memory.Embedders.Ollama.check/0`). The result is what the embedder's + `availability/0` answers, so a changed digest turns semantic memory OFF with + `:model_digest_changed` within one interval, full-text recall goes on, and no other space is + selected: nothing here touches the active pointer. + + A check that cannot reach the service records `:endpoint_unreachable`; the next one that can + replaces it. The interval is read before each wait, so a test (or an operator's configuration + reload) can shorten it. + """ + use GenServer + + alias Trinity.Memory.Embedders.Ollama + + @doc false + def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__) + + @doc "Runs a check now and answers its result (the timer is unchanged)." + @spec check_now() :: :ok | {:off, term()} + def check_now, do: GenServer.call(__MODULE__, :check, 60_000) + + @impl true + def init(_opts), do: {:ok, %{}, {:continue, :check}} + + @impl true + def handle_continue(:check, state), do: {:noreply, run(state)} + + @impl true + def handle_info(:check, state), do: {:noreply, run(state)} + + @impl true + def handle_call(:check, _from, state), do: {:reply, Ollama.check(), state} + + defp run(state) do + Ollama.check() + Process.send_after(self(), :check, interval()) + state + end + + defp interval do + case Ollama.config() do + {:ok, opts} -> opts[:check_interval_ms] + {:error, _} -> 300_000 + end + end +end diff --git a/lib/trinity/memory/gguf.ex b/lib/trinity/memory/gguf.ex new file mode 100644 index 0000000..b08e23c --- /dev/null +++ b/lib/trinity/memory/gguf.ex @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.GGUF do + @moduledoc """ + Reads a GGUF file's metadata (slice 134): the key/value header llama.cpp and Ollama read before + the tensors. Trinity needs it once, at import, to take the tokenizer out of verified weights + (`Trinity.Memory.Tier3.Import`); it never reads the tensors. + + The format (GGUF version 2 and 3, little-endian): the magic `GGUF`, a `u32` version, a `u64` + tensor count, a `u64` key/value count, then each pair: the key as a string (a `u64` length and + the bytes), a `u32` value type and the value. Types: 0 `u8`, 1 `i8`, 2 `u16`, 3 `i16`, 4 `u32`, + 5 `i32`, 6 `f32`, 7 `bool`, 8 string, 9 array (a `u32` element type, a `u64` count, the + elements), 10 `u64`, 11 `i64`, 12 `f64`. + + The header is read from the start of the file in growing pieces, so a 1 GB file costs only + the few megabytes its metadata takes. Anything the format does not allow (another magic, an + unknown type, a length past the file) is `{:error, reason}`, never a guess. + """ + + Module.register_attribute(__MODULE__, :sobelow_skip, persist: true) + + @first_read 4 * 1024 * 1024 + @max_header 512 * 1024 * 1024 + + @type value :: integer() | float() | boolean() | String.t() | [value()] + + @doc "The metadata of the GGUF file at `path`, as a map from key to value." + @spec metadata(Path.t()) :: {:ok, %{String.t() => value()}} | {:error, term()} + def metadata(path) do + with {:ok, io} <- open(path) do + try do + read_growing(io, @first_read) + after + File.close(io) + end + end + end + + # sobelow_skip reason: Traversal.FileModule: the path is the operator's own weights file, given + # to the import task on its command line, opened read-only after its signatures were verified. + @sobelow_skip ["Traversal.FileModule"] + defp open(path) do + case File.open(path, [:read, :binary, :raw]) do + {:ok, io} -> {:ok, io} + {:error, reason} -> {:error, {:unreadable, reason}} + end + end + + defp read_growing(io, size) do + case :file.pread(io, 0, size) do + {:ok, bin} -> + case parse(bin) do + {:error, :truncated} when byte_size(bin) == size and size < @max_header -> + read_growing(io, size * 4) + + other -> + other + end + + :eof -> + {:error, :empty} + + {:error, reason} -> + {:error, {:unreadable, reason}} + end + end + + @doc """ + Parses the metadata from the start of a GGUF file held in `bin` (which may stop anywhere after + the header). `{:error, :truncated}` when the header runs past the end of `bin`. + """ + @spec parse(binary()) :: {:ok, %{String.t() => value()}} | {:error, term()} + def parse(<<"GGUF", version::little-32, _tensors::little-64, count::little-64, rest::binary>>) + when version in [2, 3] do + pairs(rest, count, %{}) + catch + :truncated -> {:error, :truncated} + {:bad, reason} -> {:error, reason} + end + + def parse(<<"GGUF", version::little-32, _::binary>>), + do: {:error, {:unsupported_version, version}} + + def parse(bin) when byte_size(bin) < 24, do: {:error, :truncated} + def parse(_), do: {:error, :not_gguf} + + defp pairs(_rest, 0, acc), do: {:ok, acc} + + defp pairs(bin, n, acc) do + {key, bin} = string(bin) + {type, bin} = u32(bin) + {value, bin} = value(type, bin) + pairs(bin, n - 1, Map.put(acc, key, value)) + end + + defp u32(<>), do: {v, rest} + defp u32(_), do: throw(:truncated) + + defp string(<>) do + case rest do + <> -> {s, rest} + _ when len > @max_header -> throw({:bad, {:string_too_long, len}}) + _ -> throw(:truncated) + end + end + + defp string(_), do: throw(:truncated) + + defp value(0, <>), do: {v, r} + defp value(1, <>), do: {v, r} + defp value(2, <>), do: {v, r} + defp value(3, <>), do: {v, r} + defp value(4, <>), do: {v, r} + defp value(5, <>), do: {v, r} + defp value(6, <>), do: {v, r} + defp value(7, <>), do: {v != 0, r} + defp value(8, bin), do: string(bin) + defp value(10, <>), do: {v, r} + defp value(11, <>), do: {v, r} + defp value(12, <>), do: {v, r} + + defp value(9, <>) do + if type == 9, do: throw({:bad, :nested_array}) + if count > @max_header, do: throw({:bad, {:array_too_long, count}}) + array(type, count, rest, []) + end + + defp value(type, _bin) when type in 0..12, do: throw(:truncated) + defp value(type, _bin), do: throw({:bad, {:unknown_type, type}}) + + defp array(_type, 0, rest, acc), do: {Enum.reverse(acc), rest} + + defp array(type, n, bin, acc) do + {v, rest} = value(type, bin) + array(type, n - 1, rest, [v | acc]) + end +end diff --git a/lib/trinity/memory/semantic.ex b/lib/trinity/memory/semantic.ex index d28b82b..9f16130 100644 --- a/lib/trinity/memory/semantic.ex +++ b/lib/trinity/memory/semantic.ex @@ -175,6 +175,32 @@ defmodule Trinity.Memory.Semantic do def describe({:off, {:mixed_space, n}}), do: "semantic recall is unavailable: #{n} stored vector(s) do not fit the active space" + def describe({:off, :model_digest_changed}), + do: + "semantic recall is unavailable: the embedding service's model digest is not the pinned one " <> + "(the model behind the tag changed; an operator re-import and re-tier changes the space)" + + def describe({:off, :runtime_version_changed}), + do: "semantic recall is unavailable: the embedding service's version is not the pinned one" + + def describe({:off, :digest_unchecked}), + do: + "semantic recall is unavailable: the embedding service has not been checked against its pin yet" + + def describe({:off, :model_not_served}), + do: "semantic recall is unavailable: the embedding service does not serve the pinned model" + + def describe({:off, {:service_truncated, %{counted: c, served: s}}}), + do: + "semantic recall is unavailable: the embedding service embedded #{s} of #{c} tokens " <> + "despite truncate:false (restart after the service is fixed)" + + def describe({:off, :tokenizer_missing}), + do: "semantic recall is unavailable: the Tier 3 tokenizer file is not installed" + + def describe({:off, :tokenizer_digest_mismatch}), + do: "semantic recall is unavailable: the Tier 3 tokenizer file failed its digest check" + def describe({:off, {:config, reason}}), do: "semantic recall is unavailable: the embedder configuration is refused (#{inspect(reason)})" @@ -205,17 +231,26 @@ defmodule Trinity.Memory.Semantic do @doc """ Embeds texts for the store: through the serving embedder, tagged with the space the vectors - belong to. A failure is recorded as the tier's runtime fault (an endpoint error as - `:endpoint_unreachable`); a success clears it. + belong to. `kind` is `:document` (a memory) or `:query` (a search; slice 134: the embedder's + `embed_query/1` when it has one). A failure is recorded as the tier's runtime fault (an + endpoint error as `:endpoint_unreachable`); a success clears it. Slice 134: + `{:error, :input_too_long}` is returned and **not** recorded, because it is a property of that + one input, not of the tier (NOTES decision 4). """ - @spec embed([String.t()]) :: {:ok, [Embedder.vector()], Space.id()} | {:error, term()} - def embed(texts) do + @spec embed([String.t()], :document | :query) :: + {:ok, [Embedder.vector()], Space.id()} | {:error, term()} + def embed(texts, kind \\ :document) do with {:ok, %{module: m, space_id: id}} <- serving_on() do - case m.embed(texts) do + result = if kind == :query, do: Embedder.embed_query(m, texts), else: m.embed(texts) + + case result do {:ok, vectors} -> clear_fault(id, :embed) {:ok, vectors, id} + {:error, :input_too_long} -> + {:error, :input_too_long} + {:error, reason} -> {:error, put_fault(id, runtime_reason(reason))} end @@ -246,7 +281,7 @@ defmodule Trinity.Memory.Semantic do @spec search(String.t(), [String.t()], String.t(), pos_integer()) :: {:ok, [VectorStore.hit()]} | {:error, term()} def search(persona_id, scopes, query, k) do - with {:ok, [vector], space_id} <- embed([query]) do + with {:ok, [vector], space_id} <- embed([query], :query) do search_vector(persona_id, scopes, vector, space_id, k) end end diff --git a/lib/trinity/memory/supervisor.ex b/lib/trinity/memory/supervisor.ex index c328527..f54cdf7 100644 --- a/lib/trinity/memory/supervisor.ex +++ b/lib/trinity/memory/supervisor.ex @@ -3,7 +3,8 @@ defmodule Trinity.Memory.Supervisor do @moduledoc """ The memory side's processes (slice 032): the task supervisor the observer runs under, and - the embedding serving when the local embedder can serve. The serving is started at boot + the embedding serving when the local embedder can serve; at 134, whatever a configured + embedder's `children/0` names (the Tier 3 client's pin watcher). The serving is started at boot when the model is present and on demand after a download (`ensure_embedding/0`); when it cannot start, the tier is off and the reason is logged once, and the tree boots. """ @@ -17,7 +18,10 @@ defmodule Trinity.Memory.Supervisor do @impl true def init(_opts) do - children = [{Task.Supervisor, name: Trinity.Memory.TaskSupervisor}] ++ embedding_children() + children = + [{Task.Supervisor, name: Trinity.Memory.TaskSupervisor}] ++ + embedding_children() ++ embedder_children() + Supervisor.init(children, strategy: :one_for_one) end @@ -56,6 +60,17 @@ defmodule Trinity.Memory.Supervisor do end end + # Slice 134: what each configured embedder asks for (`children/0`), e.g. the Tier 3 client's + # pin watcher. An embedder named twice is started once. + defp embedder_children do + Embedder.modules() + |> Enum.uniq() + |> Enum.flat_map(fn m -> + Code.ensure_loaded(m) + if function_exported?(m, :children, 0), do: m.children(), else: [] + end) + end + defp embedding_child do with :ok <- Embedders.Bumblebee.availability(), {:ok, serving} <- Embedders.Bumblebee.serving() do diff --git a/lib/trinity/memory/tier3/import.ex b/lib/trinity/memory/tier3/import.ex new file mode 100644 index 0000000..dc6ba21 --- /dev/null +++ b/lib/trinity/memory/tier3/import.ex @@ -0,0 +1,586 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Tier3.Import do + @moduledoc """ + The Tier 3 weights import path (slice 134, D6): signed weights in, a pinned model on the + operator's Ollama out, or a refusal with the reason. Offline: the only peer is the operator's + own service. + + The input is an OCI image layout as `cosign save` writes it: one artifact whose layers carry + `org.opencontainers.image.title` (the ORAS file convention), one of them a `.gguf` and one the + OMS signature `model.sig`, with its cosign signature beside it. In order, any step refusing + ending the import: + + 1. **Stage.** The layout is copied into a private directory (mode 0700); a symbolic link or + anything but a regular file or directory in it is refused. Everything after reads the copy. + 2. **cosign**, offline, against the operator's cosign public key (`Tier3.Verifier`). + 3. **Unpack.** The index must list exactly one image manifest; its bytes, and each layer's, + must have the SHA-256 and size their descriptor names; each title must be a plain file name, + used once. The copy is what is hashed, and the copy is what is used. + 4. **OMS**, offline, against the operator's OMS public key, over the unpacked model files. + 5. **The statement.** The GGUF's SHA-256, computed here, must be the one the OMS statement + lists for it: the file Ollama receives is the file that was signed. + 6. **The tokenizer** is read from the GGUF (`Trinity.Memory.GGUF`, `Trinity.Memory.BPE`) and + written under its own SHA-256 into the output directory. + 7. **Ollama.** The GGUF is sent with `/api/blobs` (by its SHA-256, which Ollama checks) and the + model made with `/api/create`; `/api/show` must name that same blob; `/api/tags`' digest and + `/api/version` are read. + 8. **The record**: every digest above, written as JSON beside the tokenizer and returned. + + The import never configures Trinity: the record and `config_block/2` give the operator the + pin to set, because the active space moves only by the operator's action (D3). + """ + + alias Trinity.Memory.{BPE, GGUF} + alias Trinity.Memory.Embedders.Ollama + + require Logger + + Module.register_attribute(__MODULE__, :sobelow_skip, persist: true) + + @manifest_types [ + "application/vnd.oci.image.manifest.v1+json", + "application/vnd.docker.distribution.manifest.v2+json" + ] + @title "org.opencontainers.image.title" + @signature "model.sig" + @plain_name ~r/\A[A-Za-z0-9][A-Za-z0-9._-]{0,254}\z/ + @hex64 ~r/\A[0-9a-f]{64}\z/ + @pooling %{0 => "none", 1 => "mean", 2 => "cls", 3 => "last", 4 => "rank"} + + @type opts :: [ + layout: Path.t(), + cosign_key: Path.t(), + oms_key: Path.t(), + base_url: String.t(), + model: String.t(), + out: Path.t(), + verifier: module(), + verifier_opts: keyword(), + staging: Path.t(), + timeout_ms: pos_integer() + ] + + @doc """ + Runs the import. Options: `layout:`, `cosign_key:`, `oms_key:`, `base_url:`, `model:` (the name + to create on the service), `out:` (where the tokenizer and the record go; the data directory's + `models/tier3` by default), `verifier:` (`Tier3.Verifier.CLI`) with `verifier_opts:` + (`cosign:`, `model_signing:`), `staging:` (the parent of the private staging directory) and + `timeout_ms:`. `{:ok, record}` or `{:error, reason}`. + """ + @spec run(opts()) :: {:ok, map()} | {:error, term()} + # sobelow_skip reason: Traversal.FileModule: the staging directory removed is the one + # `stage_dir/1` made under the parent the operator named, with a random suffix; nothing else is + # removed. + @sobelow_skip ["Traversal.FileModule"] + def run(opts) do + with {:ok, opts} <- required(opts), + {:ok, stage} <- stage_dir(opts) do + try do + steps(opts, stage) + after + File.rm_rf(stage) + end + end + end + + defp required(opts) do + case Enum.find([:layout, :cosign_key, :oms_key, :base_url, :model], &(opts[&1] in [nil, ""])) do + nil -> {:ok, opts} + key -> {:error, {:option_missing, key}} + end + end + + defp steps(opts, stage) do + verifier = Keyword.get(opts, :verifier, Trinity.Memory.Tier3.Verifier.CLI) + vopts = Keyword.get(opts, :verifier_opts, []) + layout = Path.join(stage, "layout") + model_dir = Path.join(stage, "model") + sig = Path.join(stage, @signature) + + with :ok <- copy_tree(opts[:layout], layout), + :ok <- verifier.cosign(layout, opts[:cosign_key], vopts), + {:ok, unpacked} <- unpack(layout, model_dir, sig), + :ok <- verifier.oms(model_dir, sig, opts[:oms_key], vopts), + {:ok, oms} <- statement(sig, unpacked.gguf, unpacked.files), + {:ok, tok} <- tokenizer(Path.join(model_dir, unpacked.gguf), out_dir(opts)), + {:ok, served} <- ollama(opts, Path.join(model_dir, unpacked.gguf), unpacked) do + record(opts, unpacked, oms, tok, served) + end + end + + ## 1. Stage + + # sobelow_skip reason: Traversal.FileModule: the parent directory is the operator's (an option, + # default the system temporary directory) and the child's name is a fixed prefix and random hex; + # it is created mode 0700. + @sobelow_skip ["Traversal.FileModule"] + defp stage_dir(opts) do + parent = Keyword.get(opts, :staging, System.tmp_dir!()) + + dir = + Path.join( + parent, + "tier3-import-" <> Base.encode16(:crypto.strong_rand_bytes(8), case: :lower) + ) + + with :ok <- File.mkdir_p(parent), + :ok <- File.mkdir(dir), + :ok <- File.chmod(dir, 0o700) do + {:ok, dir} + else + {:error, reason} -> {:error, {:staging, reason}} + end + end + + # sobelow_skip reason: Traversal.FileModule: the source is the layout directory the operator + # named, read with lstat so a symbolic link is refused rather than followed; the destination is + # inside the private staging directory this module created. + @sobelow_skip ["Traversal.FileModule"] + defp copy_tree(src, dest) do + case File.lstat(src) do + {:ok, %File.Stat{type: :directory}} -> + src |> copy_dir(dest) |> layout_error(src) + + {:ok, %File.Stat{type: :regular}} -> + File.cp(src, dest) |> layout_error(src) + + {:ok, %File.Stat{type: type}} -> + {:error, {:layout_entry_refused, type, src}} + + {:error, reason} -> + {:error, {:layout_unreadable, reason, src}} + end + end + + # sobelow_skip reason: Traversal.FileModule: as `copy_tree/2`: the source is the operator's + # layout, the destination inside the private staging directory. + @sobelow_skip ["Traversal.FileModule"] + defp copy_dir(src, dest) do + with :ok <- File.mkdir(dest), {:ok, names} <- File.ls(src) do + names |> Enum.sort() |> Enum.reduce_while(:ok, ©_entry(&1, &2, src, dest)) + end + end + + defp copy_entry(name, :ok, src, dest) do + case copy_tree(Path.join(src, name), Path.join(dest, name)) do + :ok -> {:cont, :ok} + error -> {:halt, error} + end + end + + defp layout_error(:ok, _src), do: :ok + defp layout_error({:error, {_, _, _} = reason}, _src), do: {:error, reason} + defp layout_error({:error, reason}, src), do: {:error, {:layout_unreadable, reason, src}} + + ## 3. Unpack + + @doc false + @spec unpack(Path.t(), Path.t(), Path.t()) :: {:ok, map()} | {:error, term()} + # sobelow_skip reason: Traversal.FileModule: the layout and the model directory are inside the + # private staging directory. + @sobelow_skip ["Traversal.FileModule"] + def unpack(layout, model_dir, sig) do + with {:ok, index} <- read_json(Path.join(layout, "index.json"), :index_unreadable), + {:ok, desc} <- one_manifest(index), + {:ok, bytes} <- blob_bytes(layout, desc), + {:ok, manifest} <- decode(bytes, :manifest_unreadable), + {:ok, layers} <- layers(manifest), + :ok <- File.mkdir_p(model_dir), + {:ok, files} <- unpack_layers(layout, layers, model_dir, sig), + {:ok, gguf} <- one_gguf(files), + :ok <- has_signature(files) do + {:ok, %{manifest_digest: desc["digest"], gguf: gguf, files: Map.delete(files, @signature)}} + end + end + + defp one_manifest(%{"manifests" => manifests}) when is_list(manifests) do + case Enum.filter(manifests, &(&1["mediaType"] in @manifest_types)) do + [desc] -> {:ok, desc} + other -> {:error, {:index_manifests, length(other)}} + end + end + + defp one_manifest(_), do: {:error, {:index_manifests, 0}} + + defp layers(%{"layers" => layers}) when is_list(layers) and layers != [], do: {:ok, layers} + defp layers(_), do: {:error, :manifest_without_layers} + + defp unpack_layers(layout, layers, model_dir, sig) do + Enum.reduce_while(layers, {:ok, %{}}, fn layer, {:ok, acc} -> + case unpack_layer(layout, layer, model_dir, sig, acc) do + {:ok, name, sha} -> {:cont, {:ok, Map.put(acc, name, sha)}} + error -> {:halt, error} + end + end) + end + + defp unpack_layer(layout, layer, model_dir, sig, seen) do + name = get_in(layer, ["annotations", @title]) + + cond do + not is_binary(name) -> + {:error, :layer_without_title} + + not Regex.match?(@plain_name, name) -> + {:error, {:layer_title_refused, name}} + + Map.has_key?(seen, name) -> + {:error, {:layer_title_twice, name}} + + true -> + copy_layer( + layout, + layer, + name, + if(name == @signature, do: sig, else: Path.join(model_dir, name)) + ) + end + end + + # sobelow_skip reason: Traversal.FileModule: the source is a blob path made of a validated 64-hex + # digest under the staged layout; the destination's file name is a title matched against a plain- + # name pattern (no separator, no leading dot) inside the private staging directory. + @sobelow_skip ["Traversal.FileModule"] + defp copy_layer(layout, layer, name, dest) do + with {:ok, src} <- blob_path(layout, layer), + :ok <- File.cp(src, dest) |> fs_error(name), + {:ok, sha, size} <- file_sha256(dest), + :ok <- same_blob(layer, sha, size, name) do + {:ok, name, sha} + end + end + + # sobelow_skip reason: Traversal.FileModule: the blob path is made of a validated 64-hex digest + # under the staged layout. + @sobelow_skip ["Traversal.FileModule"] + defp blob_bytes(layout, desc) do + with {:ok, path} <- blob_path(layout, desc), + {:ok, bytes} <- File.read(path) |> fs_error(desc["digest"]), + :ok <- same_blob(desc, sha256(bytes), byte_size(bytes), "manifest") do + {:ok, bytes} + end + end + + defp blob_path(layout, %{"digest" => "sha256:" <> hex}) do + if Regex.match?(@hex64, hex), + do: {:ok, Path.join([layout, "blobs", "sha256", hex])}, + else: {:error, {:digest_refused, hex}} + end + + defp blob_path(_layout, desc), do: {:error, {:digest_refused, desc["digest"]}} + + defp same_blob(desc, sha, size, name) do + cond do + desc["digest"] != "sha256:" <> sha -> {:error, {:blob_digest_mismatch, name}} + desc["size"] != size -> {:error, {:blob_size_mismatch, name}} + true -> :ok + end + end + + defp one_gguf(files) do + case Enum.filter(Map.keys(files), &String.ends_with?(String.downcase(&1), ".gguf")) do + [gguf] -> {:ok, gguf} + other -> {:error, {:gguf_count, length(other)}} + end + end + + defp has_signature(files), + do: if(Map.has_key?(files, @signature), do: :ok, else: {:error, :oms_signature_missing}) + + ## 5. The OMS statement + + @doc false + @spec statement(Path.t(), String.t(), map()) :: {:ok, map()} | {:error, term()} + def statement(sig, gguf, files) do + with {:ok, bundle} <- read_json(sig, :oms_signature_unreadable), + {:ok, payload} <- payload(bundle), + {:ok, resources} <- resources(payload) do + listed = Map.new(resources, fn r -> {r["name"], {r["algorithm"], r["digest"]}} end) + + case Map.fetch(listed, gguf) do + {:ok, {"sha256", digest}} -> same_digest(digest, files[gguf], gguf, payload) + _ -> {:error, {:oms_statement_lacks, gguf}} + end + end + end + + defp same_digest(digest, digest, _gguf, payload), + do: {:ok, %{subject: subject(payload), predicate_type: payload["predicateType"]}} + + defp same_digest(_listed, _computed, gguf, _payload), + do: {:error, {:oms_statement_mismatch, gguf}} + + defp payload(%{"dsseEnvelope" => %{"payload" => b64}}) when is_binary(b64) do + with {:ok, json} <- Base.decode64(b64), {:ok, p} <- Jason.decode(json) do + {:ok, p} + else + _ -> {:error, :oms_payload_unreadable} + end + end + + defp payload(_), do: {:error, :oms_payload_unreadable} + + defp resources(%{ + "predicateType" => "https://model_signing/signature/v1" <> _, + "predicate" => p + }) do + case p["resources"] do + list when is_list(list) -> {:ok, list} + _ -> {:error, :oms_resources_missing} + end + end + + defp resources(p), do: {:error, {:oms_predicate_type, p["predicateType"]}} + + defp subject(%{"subject" => [%{"digest" => %{"sha256" => d}} | _]}), do: d + defp subject(_), do: nil + + ## 6. The tokenizer + + # sobelow_skip reason: Traversal.FileModule: the GGUF is the verified file in the staging + # directory; the output directory is the operator's (an option, default the data directory) and + # the file name a SHA-256. + @sobelow_skip ["Traversal.FileModule"] + defp tokenizer(gguf, out) do + with {:ok, meta} <- GGUF.metadata(gguf), + {:ok, bpe} <- BPE.from_gguf(meta) do + bin = BPE.to_file(bpe) + sha = sha256(bin) + path = Path.join(out, "#{sha}.bpe.json") + + with :ok <- File.mkdir_p(out) |> fs_error(out), + :ok <- File.write(path, bin) |> fs_error(path) do + {:ok, %{sha256: sha, path: path, info: model_info(meta)}} + end + end + end + + defp model_info(meta) do + arch = meta["general.architecture"] + + %{ + "architecture" => arch, + "name" => meta["general.name"], + "license" => meta["general.license"], + "file_type" => meta["general.file_type"], + "dim" => meta["#{arch}.embedding_length"], + "context_length" => meta["#{arch}.context_length"], + "pooling" => Map.get(@pooling, meta["#{arch}.pooling_type"], "unrecorded") + } + end + + ## 7. Ollama + + defp ollama(opts, gguf, unpacked) do + sha = unpacked.files[unpacked.gguf] + ref = "sha256:" <> sha + o = [base_url: opts[:base_url], model: opts[:model], timeout_ms: opts[:timeout_ms] || 600_000] + + with :ok <- upload(o, gguf, ref), + :ok <- create(o, unpacked.gguf, ref), + :ok <- served_blob(o, sha), + {:ok, digest} <- tags_digest(o), + {:ok, version} <- version(o) do + {:ok, %{model: Ollama.full_name(opts[:model]), digest: digest, version: version}} + end + end + + # sobelow_skip reason: Traversal.FileModule: the GGUF streamed is the verified file in the private + # staging directory. + @sobelow_skip ["Traversal.FileModule"] + defp upload(o, gguf, ref) do + case http(o, :head, "/api/blobs/" <> ref, []) do + {:ok, %{status: 200}} -> + :ok + + {:ok, %{status: 404}} -> + case http(o, :post, "/api/blobs/" <> ref, body: File.stream!(gguf, 1_048_576)) do + {:ok, %{status: s}} when s in [200, 201] -> :ok + other -> {:error, {:upload_refused, summary(other)}} + end + + other -> + {:error, {:service_unreachable, summary(other)}} + end + end + + defp create(o, name, ref) do + body = %{model: o[:model], files: %{name => ref}, stream: false} + + case http(o, :post, "/api/create", json: body) do + {:ok, %{status: 200, body: %{"status" => "success"}}} -> :ok + other -> {:error, {:create_refused, summary(other)}} + end + end + + defp served_blob(o, sha) do + case http(o, :post, "/api/show", json: %{model: o[:model]}) do + {:ok, %{status: 200, body: %{"modelfile" => mf}}} when is_binary(mf) -> + case Regex.run(~r/^FROM\s+\S*sha256[-:]([0-9a-f]{64})\s*$/m, mf) do + [_, ^sha] -> :ok + [_, other] -> {:error, {:served_blob_mismatch, other}} + nil -> {:error, :served_blob_unknown} + end + + other -> + {:error, {:show_refused, summary(other)}} + end + end + + # The digest names the record's file, and it comes from the service: anything but 64 hex + # characters is refused rather than used as a path. + defp tags_digest(o) do + case Ollama.served_digest(o) do + {:ok, d} -> if Regex.match?(@hex64, d), do: {:ok, d}, else: {:error, {:digest_refused, d}} + {:off, reason} -> {:error, reason} + end + end + + defp version(o) do + case http(o, :get, "/api/version", []) do + {:ok, %{status: 200, body: %{"version" => v}}} -> {:ok, v} + other -> {:error, {:version_unreadable, summary(other)}} + end + end + + defp http(o, method, path, extra) do + Req.request( + [ + method: method, + url: String.trim_trailing(o[:base_url], "/") <> path, + retry: false, + receive_timeout: o[:timeout_ms] + ] ++ extra + ) + end + + defp summary({:ok, %{status: s, body: b}}), do: {s, b} + defp summary({:error, e}), do: Exception.message(e) + + ## 8. The record + + # sobelow_skip reason: Traversal.FileModule: the record is written to the operator's output + # directory under a name made of the 64-hex digest the service returned (`served_digest/1` checks + # nothing about its shape, so the name is checked here). + @sobelow_skip ["Traversal.FileModule"] + defp record(opts, unpacked, oms, tok, served) do + record = %{ + "format" => "trinity-tier3-import-1", + "imported_at" => DateTime.utc_now() |> DateTime.truncate(:second) |> DateTime.to_iso8601(), + "layout" => Path.expand(opts[:layout]), + "manifest_digest" => unpacked.manifest_digest, + "cosign_key_sha256" => file_digest(opts[:cosign_key]), + "oms_key_sha256" => file_digest(opts[:oms_key]), + "oms_subject_sha256" => oms.subject, + "oms_predicate_type" => oms.predicate_type, + "files" => unpacked.files, + "gguf" => unpacked.gguf, + "weights_sha256" => unpacked.files[unpacked.gguf], + "tokenizer_sha256" => tok.sha256, + "tokenizer_path" => tok.path, + "model_info" => tok.info, + "ollama" => %{ + "base_url" => opts[:base_url], + "model" => served.model, + "digest" => served.digest, + "version" => served.version + } + } + + path = Path.join(out_dir(opts), "#{served.digest}.import.json") + + with :ok <- File.write(path, Jason.encode_to_iodata!(record, pretty: true)) |> fs_error(path) do + Logger.info("tier3 import: #{served.model} digest #{served.digest}") + {:ok, Map.put(record, "record_path", path)} + end + end + + defp file_digest(path) do + case file_sha256(path) do + {:ok, sha, _} -> sha + _ -> nil + end + end + + @doc """ + The configuration block for a record: what the operator sets to pin the embedder to the model + just imported. `opts`: `model_id:`, `num_ctx:` (8192), `max_input_tokens:` (`num_ctx - 1`), + `query_prompt:`, `document_prompt:`. + """ + @spec config_block(map(), keyword()) :: String.t() + def config_block(record, opts \\ []) do + num_ctx = Keyword.get(opts, :num_ctx, 8192) + o = record["ollama"] + info = record["model_info"] + + """ + config :trinity, :memory, + embedder: :ollama, + locality: :within_boundary, # or :external with external_opt_in: true; declare it + ollama: [ + base_url: #{inspect(o["base_url"])}, + model: #{inspect(o["model"])}, + model_id: #{inspect(Keyword.get(opts, :model_id, record["gguf"]))}, + digest: #{inspect(o["digest"])}, + weights_sha256: #{inspect(record["weights_sha256"])}, + tokenizer_sha256: #{inspect(record["tokenizer_sha256"])}, + tokenizer_path: #{inspect(record["tokenizer_path"])}, + runtime_version: #{inspect(o["version"])}, + num_ctx: #{num_ctx}, + max_input_tokens: #{Keyword.get(opts, :max_input_tokens, num_ctx - 1)}, + dim: #{inspect(info["dim"])}, + pooling: #{inspect(info["pooling"])}, + query_prompt: #{inspect(Keyword.get(opts, :query_prompt, ""))}, + document_prompt: #{inspect(Keyword.get(opts, :document_prompt, ""))} + ] + """ + end + + ## Files + + defp out_dir(opts), + do: Keyword.get(opts, :out) || Path.join([Trinity.Paths.data_dir(), "models", "tier3"]) + + # sobelow_skip reason: Traversal.FileModule: the paths read are files of the staged copy (inside + # the private staging directory) or the operator's key files named on the command line. + @sobelow_skip ["Traversal.FileModule"] + defp read_json(path, reason) do + case File.read(path) do + {:ok, bin} -> decode(bin, reason) + {:error, _} -> {:error, reason} + end + end + + defp decode(bin, reason) do + case Jason.decode(bin) do + {:ok, %{} = map} -> {:ok, map} + _ -> {:error, reason} + end + end + + # sobelow_skip reason: Traversal.FileModule: as `read_json/2`. + @sobelow_skip ["Traversal.FileModule"] + defp file_sha256(path) do + case File.stat(path) do + {:ok, %File.Stat{type: :regular, size: size}} -> + sha = + path + |> File.stream!(1_048_576) + |> Enum.reduce(:crypto.hash_init(:sha256), &:crypto.hash_update(&2, &1)) + |> :crypto.hash_final() + |> Base.encode16(case: :lower) + + {:ok, sha, size} + + _ -> + {:error, {:unreadable, path}} + end + end + + defp sha256(bin), do: :crypto.hash(:sha256, bin) |> Base.encode16(case: :lower) + + defp fs_error(:ok, _what), do: :ok + defp fs_error({:ok, v}, _what), do: {:ok, v} + defp fs_error({:error, reason}, what), do: {:error, {:file, reason, what}} +end diff --git a/lib/trinity/memory/tier3/verifier.ex b/lib/trinity/memory/tier3/verifier.ex new file mode 100644 index 0000000..354a436 --- /dev/null +++ b/lib/trinity/memory/tier3/verifier.ex @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Tier3.Verifier do + @moduledoc """ + The two signature checks of the Tier 3 import path (slice 134, D6), offline, against keys the + operator holds: **cosign** over the OCI artifact as delivered, **OMS** (OpenSSF Model Signing) + over the model files once unpacked. + + The implementation that ships is `Trinity.Memory.Tier3.Verifier.CLI`, which runs the reference + programs (`cosign`, `model_signing`). Trinity does not carry a signature verifier of its own: + one would be a second thing to trust and audit (slice 134 NOTES, decision 10). The behaviour + exists so the import path's own checks can be tested on a machine without the programs. + """ + + @doc "`:ok` when the OCI layout's signature verifies against the cosign public key, offline." + @callback cosign(layout :: Path.t(), public_key :: Path.t(), opts :: keyword()) :: + :ok | {:error, term()} + + @doc "`:ok` when the OMS signature over the model directory verifies against the public key." + @callback oms(model_dir :: Path.t(), signature :: Path.t(), public_key :: Path.t(), keyword()) :: + :ok | {:error, term()} +end diff --git a/lib/trinity/memory/tier3/verifier/cli.ex b/lib/trinity/memory/tier3/verifier/cli.ex new file mode 100644 index 0000000..2c39338 --- /dev/null +++ b/lib/trinity/memory/tier3/verifier/cli.ex @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Tier3.Verifier.CLI do + @moduledoc """ + The verifier as the reference programs (slice 134), from paths in the options (`cosign:`, + `model_signing:`) or found on `PATH`: + + cosign verify --key --offline=true --local-image --insecure-ignore-tlog=true + model_signing verify key --signature --public_key + + `--insecure-ignore-tlog=true` because an air-gapped import host has no transparency log to + consult: the operator's key is the trust (D6; the Sigstore Cosign 2.0 notes say a signature + made with `--tlog-upload=false` is verified this way). Exit status 0 is a pass; anything else, + or a program that cannot be found, is a refusal carrying the program's last lines. + """ + @behaviour Trinity.Memory.Tier3.Verifier + + Module.register_attribute(__MODULE__, :sobelow_skip, persist: true) + + @impl true + def cosign(layout, public_key, opts) do + args = [ + "verify", + "--key", + public_key, + "--offline=true", + "--local-image", + "--insecure-ignore-tlog=true", + layout + ] + + with {:ok, program} <- program(opts, :cosign, "cosign") do + run(program, args, :cosign_refused) + end + end + + @impl true + def oms(model_dir, signature, public_key, opts) do + args = ["verify", "key", "--signature", signature, "--public_key", public_key, model_dir] + + with {:ok, program} <- program(opts, :model_signing, "model_signing") do + run(program, args, :oms_refused) + end + end + + defp program(opts, key, name) do + case Keyword.get(opts, key) || System.find_executable(name) do + nil -> {:error, {:program_missing, name}} + path -> if File.regular?(path), do: {:ok, path}, else: {:error, {:program_missing, path}} + end + end + + # sobelow_skip reason: CI.System: the program is the operator's own cosign or model_signing + # (an option of the import task, or found on PATH), run with a fixed argument list whose only + # variables are paths the import created or the operator named; nothing comes from a model or a + # network peer. + @sobelow_skip ["CI.System"] + defp run(program, args, refusal) do + case System.cmd(program, args, stderr_to_stdout: true) do + {_out, 0} -> :ok + {out, status} -> {:error, {refusal, status, tail(out)}} + end + rescue + e in ErlangError -> {:error, {refusal, :not_run, Exception.message(e)}} + end + + defp tail(out), do: out |> String.split("\n", trim: true) |> Enum.take(-3) |> Enum.join("\n") +end diff --git a/scripts/tier3/bench.exs b/scripts/tier3/bench.exs new file mode 100644 index 0000000..a3dfca9 --- /dev/null +++ b/scripts/tier3/bench.exs @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Gate 5, throughput (slice 134, AC9): p50 and p95 of an embedding call through Trinity's client +# at batch 1 and batch 32, recorded with no pass mark. +# +# mix run --no-start scripts/tier3/bench.exs --record --fixtures \ +# [--num-ctx 2048] [--n1 200] [--n32 40] +# +# The texts are the parity fixtures' natural sentences (documents, no template), cycled; each +# call is timed end to end from the client (tokenizing, the request, the answer's checks). +Code.require_file("gate_lib.exs", __DIR__) +{:ok, _} = Application.ensure_all_started(:req) + +alias Trinity.Memory.Embedders.Ollama + +opts = + Tier3Gate.args(System.argv(), + record: :string, + fixtures: :string, + num_ctx: :integer, + n1: :integer, + n32: :integer, + base_url: :string + ) + +record = Tier3Gate.record!(opts[:record]) +num_ctx = Keyword.get(opts, :num_ctx, 2048) +Tier3Gate.configure!(record, num_ctx: num_ctx, base_url: Keyword.get(opts, :base_url, record["ollama"]["base_url"])) +:ok = Ollama.check() + +texts = + Tier3Gate.fixtures(opts[:fixtures], "parity.jsonl") + |> Enum.filter(&(&1["role"] == "document" and &1["count"] <= 200)) + |> Enum.map(& &1["text"]) + +tokens = Enum.map(texts, &Trinity.Memory.BPE.count(elem(Ollama.tokenizer(elem(Ollama.config(), 1)), 1), &1)) + +run = fn batch, n, warm -> + texts + |> Stream.cycle() + |> Stream.chunk_every(batch) + |> Enum.take(n + warm) + |> Enum.with_index() + |> Enum.flat_map(fn {chunk, i} -> + {us, {:ok, vs}} = :timer.tc(fn -> Ollama.embed(chunk) end) + true = length(vs) == batch + if i < warm, do: [], else: [us / 1000] + end) +end + +IO.puts( + "GATE5 #{record["gguf"]} as #{record["ollama"]["model"]}, Ollama #{record["ollama"]["version"]}, num_ctx #{num_ctx}; " <> + "#{length(texts)} sentences, #{Enum.min(tokens)} to #{Enum.max(tokens)} tokens (median #{Tier3Gate.percentile(tokens, 50)})" +) + +for {batch, n, warm} <- [{1, Keyword.get(opts, :n1, 200), 10}, {32, Keyword.get(opts, :n32, 40), 3}] do + ms = run.(batch, n, warm) + + IO.puts( + " batch #{batch}: #{length(ms)} calls after #{warm} warm-ups, p50 #{Float.round(Tier3Gate.percentile(ms, 50), 1)} ms, " <> + "p95 #{Float.round(Tier3Gate.percentile(ms, 95), 1)} ms, max #{Float.round(Enum.max(ms), 1)} ms; " <> + "#{Float.round(batch * 1000 / Tier3Gate.percentile(ms, 50), 1)} texts/s at p50" + ) +end + +IO.puts("GATE5 MEASURED (no pass mark)") +IO.puts(Tier3Gate.env_line()) diff --git a/scripts/tier3/digest_pin.exs b/scripts/tier3/digest_pin.exs new file mode 100644 index 0000000..866856f --- /dev/null +++ b/scripts/tier3/digest_pin.exs @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Gate 3, the digest pin (slice 134, AC7): replacing the blob behind the tag changes the +# /api/tags digest, and a running Trinity goes OFF with :model_digest_changed. +# +# XDG_DATA_HOME= MIX_ENV=dev mix run --no-start scripts/tier3/digest_pin.exs \ +# --record --swap-gguf [--interval-ms 2000] [--num-ctx 2048] +# +# A real boot: its own databases under XDG_DATA_HOME, migrated, the application started with the +# Tier 3 embedder configured from the import record, so `Trinity.Memory.Supervisor` starts the +# watcher the way a deployment does. The store is pinned to the record's space, a memory written +# and a message stored. Then the script itself replaces the blob behind the tag, as an operator or +# an attacker with access to the service would (/api/blobs and /api/create with the other GGUF, +# same name), and polls the tier's status until it is OFF, printing the time. +Code.require_file("gate_lib.exs", __DIR__) + +alias Trinity.Memory.{AlwaysOn, Retriever, Semantic, Space, Spaces} +alias Trinity.Memory.Embedders.Ollama + +opts = + Tier3Gate.args(System.argv(), + record: :string, + swap_gguf: :string, + interval_ms: :integer, + num_ctx: :integer, + base_url: :string + ) + +record = Tier3Gate.record!(opts[:record]) +interval = Keyword.get(opts, :interval_ms, 2000) +data = System.fetch_env!("XDG_DATA_HOME") +File.mkdir_p!(data) + +# The node's own databases and no HTTP listener. +Application.put_env(:trinity, Trinity.Repo, Keyword.put(Application.get_env(:trinity, Trinity.Repo), :database, Path.join(data, "gate3.db"))) + +Application.put_env( + :trinity, + Trinity.Repo.Receipts, + Keyword.put(Application.get_env(:trinity, Trinity.Repo.Receipts), :database, Path.join(data, "gate3_receipts.db")) +) + +Application.put_env(:trinity, TrinityWeb.Endpoint, Keyword.put(Application.get_env(:trinity, TrinityWeb.Endpoint, []), :server, false)) +Trinity.Release.migrate() +{:ok, _} = Application.ensure_all_started(:req) +o = Tier3Gate.configure!(record, num_ctx: Keyword.get(opts, :num_ctx, 2048), check_interval_ms: interval, base_url: Keyword.get(opts, :base_url, record["ollama"]["base_url"])) +{:ok, _} = Application.ensure_all_started(:trinity) + +t0 = System.monotonic_time(:millisecond) +now = fn -> System.monotonic_time(:millisecond) - t0 end +say = fn line -> IO.puts("GATE3 t=#{now.()}ms #{line}") end + +wait = fn want, limit_ms -> + Enum.find_value(0..div(limit_ms, 20), fn _ -> + if Semantic.status() == want, do: now.(), else: Process.sleep(20) && nil + end) +end + +say.("booted; watcher #{inspect(Process.whereis(Ollama.Watch) |> is_pid())}; status #{inspect(Semantic.status())}") +row = Trinity.Memory.Spaces.pin_if_empty(Ollama.space()) |> elem(1) +say.("pinned the empty store to space #{Space.short(row.id)} (revision = the /api/tags digest #{o[:digest]})") +true = is_integer(wait.(:on, 30_000)) +say.("status :on") + +{:ok, persona} = Trinity.Sessions.create_persona(%{name: "gate3", soul: "gate", model: nil}) +scope = AlwaysOn.persona_scope(persona.id) +{:ok, _} = Semantic.add(%{persona_id: persona.id, scope: scope, key: "dog", body: "The person's dog is called Rex."}, by: "gate3") +{:ok, session} = Trinity.Sessions.create_session(%{persona_id: persona.id}) +{:ok, _} = Trinity.Sessions.append_message(session.id, %{role: "user", content: "my sister lives in Boston"}) +{:ok, [before]} = Ollama.embed_query(["what is my dog called?"]) +say.("memory written; vectors in the space: #{Spaces.count(row.id)}; recall #{inspect(Retriever.relevant(persona.id, nil, "what is my dog called?", touch: false) |> Enum.map(&{&1.kind, &1.found_by}))}") + +# Replace the blob behind the tag. +gguf = opts[:swap_gguf] +sha = :crypto.hash(:sha256, File.read!(gguf)) |> Base.encode16(case: :lower) +base = o[:base_url] +# As the import path does: send the blob only when the service does not hold it already. +s1 = + case Req.head!(base <> "/api/blobs/sha256:" <> sha, retry: false) do + %{status: 200} -> "already held" + %{status: 404} -> Req.post!(base <> "/api/blobs/sha256:" <> sha, body: File.stream!(gguf, 1_048_576), retry: false, receive_timeout: 600_000).status + end + +%{status: 200} = Req.post!(base <> "/api/create", json: %{model: o[:model], files: %{Path.basename(gguf) => "sha256:" <> sha}, stream: false}, retry: false, receive_timeout: 600_000) +{:ok, new_digest} = Ollama.served_digest(o) +swapped = now.() +say.("replaced the blob behind #{o[:model]} with #{Path.basename(gguf)} (blob: #{s1}); /api/tags digest now #{new_digest}") + +# What the service now answers for the same query, read directly (Trinity is not asked). +{200, %{"embeddings" => [after_v]}} = + Tier3Gate.embed_raw(base, o[:model], [o[:query_prompt] <> "what is my dog called?"], o[:num_ctx], false) + +say.("the service's vector for the same query, before and after the swap: cosine #{Float.round(Tier3Gate.cosine(before, after_v), 6)}") + +off = wait.({:off, :model_digest_changed}, 3 * interval + 1000) + +if off do + say.("status #{inspect(Semantic.status())} #{off - swapped} ms after the swap (interval #{interval} ms)") +else + say.("still #{inspect(Semantic.status())} #{now.() - swapped} ms after the swap (interval #{interval} ms)") +end + +say.("search: #{inspect(Semantic.search(persona.id, [scope], "what is my dog called?", 3) |> then(fn {:error, r} -> {:error, r}; {:ok, h} -> {:ok, length(h)} end))}") +say.("recall: #{inspect(Retriever.relevant(persona.id, nil, "sister Boston", touch: false) |> Enum.map(&{&1.kind, &1.found_by}))}") +say.("active space #{Space.short(Spaces.active().id)} (unchanged: #{Spaces.active().id == row.id}); spaces #{length(Spaces.list())}; jobs #{Trinity.Repo.aggregate(Oban.Job, :count)}") +IO.puts("GATE3 VERDICT #{if off && off - swapped <= interval + 1000, do: "PASS", else: "FAIL"}") +System.halt(0) diff --git a/scripts/tier3/gate_lib.exs b/scripts/tier3/gate_lib.exs new file mode 100644 index 0000000..f8327d3 --- /dev/null +++ b/scripts/tier3/gate_lib.exs @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Shared by the Tier 3 gate scripts (slice 134): the import record, the client's configuration +# from it, a raw `/api/embed` call, and the statistics. Loaded with `Code.require_file/1`. +defmodule Tier3Gate do + @moduledoc false + + alias Trinity.Memory.Embedders.Ollama + + def args(argv, switches) do + {opts, _, invalid} = OptionParser.parse(argv, strict: switches) + if invalid != [], do: raise("unknown options #{inspect(invalid)}") + opts + end + + def record!(path), do: path |> File.read!() |> Jason.decode!() + + @doc "Points the Tier 3 client at the record's model, pinned as the record says." + def configure!(record, opts) do + num_ctx = Keyword.fetch!(opts, :num_ctx) + + ollama = [ + base_url: Keyword.get(opts, :base_url, record["ollama"]["base_url"]), + model: record["ollama"]["model"], + model_id: record["gguf"], + digest: record["ollama"]["digest"], + weights_sha256: record["weights_sha256"], + tokenizer_sha256: record["tokenizer_sha256"], + tokenizer_path: record["tokenizer_path"], + runtime_version: record["ollama"]["version"], + num_ctx: num_ctx, + max_input_tokens: Keyword.get(opts, :max_input_tokens, num_ctx - 1), + dim: record["model_info"]["dim"], + pooling: record["model_info"]["pooling"], + query_prompt: Keyword.get(opts, :query_prompt, ""), + document_prompt: "", + check_interval_ms: Keyword.get(opts, :check_interval_ms, 300_000), + timeout_ms: 600_000 + ] + + memory = Application.get_env(:trinity, :memory, []) + + Application.put_env( + :trinity, + :memory, + Keyword.merge(memory, embedder: :ollama, locality: :within_boundary, ollama: ollama) + ) + + Ollama.reset() + ollama + end + + @doc "One raw `/api/embed` call: `{status, body}`." + def embed_raw(base_url, model, inputs, num_ctx, truncate) do + body = + %{model: model, input: inputs, options: %{num_ctx: num_ctx}} + |> then(fn b -> if truncate == :omit, do: b, else: Map.put(b, :truncate, truncate) end) + + %{status: s, body: b} = + Req.post!(base_url <> "/api/embed", json: body, retry: false, receive_timeout: 600_000) + + {s, b} + end + + def cosine(a, b) do + {d, na, nb} = + Enum.zip_reduce(a, b, {0.0, 0.0, 0.0}, fn x, y, {d, p, q} -> {d + x * y, p + x * x, q + y * y} end) + + d / (:math.sqrt(na) * :math.sqrt(nb)) + end + + def percentile(values, p) do + sorted = Enum.sort(values) + Enum.at(sorted, min(length(sorted) - 1, round(p / 100 * (length(sorted) - 1)))) + end + + def fixtures(dir, name), do: Path.join(dir, name) |> File.stream!() |> Enum.map(&Jason.decode!/1) + + def manifest(dir), do: Path.join(dir, "fixtures-manifest.json") |> File.read!() |> Jason.decode!() + + def env_line do + "otp #{System.otp_release()} elixir #{System.version()} schedulers #{System.schedulers_online()}" + end +end diff --git a/scripts/tier3/gates.sh b/scripts/tier3/gates.sh new file mode 100755 index 0000000..d9bc796 --- /dev/null +++ b/scripts/tier3/gates.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# The Tier 3 gates' container steps (slice 134), one subcommand at a time, so the same commands +# rerun on another image (a hardened build of the same version: TIER3_IMAGE=/:0.40.0). +# +# scripts/tier3/gates.sh package sign a GGUF as a producer would (OMS, then an ORAS +# artifact cosign-signed in a loopback registry) and +# save it as an OCI layout: the delivery to import +# scripts/tier3/gates.sh serve start the service (CPU, --memory=16g, loopback port) +# scripts/tier3/gates.sh import run mix trinity.tier3.import on a delivery +# scripts/tier3/gates.sh offline Gate 4: a second container with --network none over +# a copy of the verified models directory +# scripts/tier3/gates.sh down remove the containers +# +# Gates 1, 2, 3 and 5 are mix scripts in this directory (parity.exs, truncation.exs, +# digest_pin.exs, bench.exs), run against the served model with the import's record. +# +# Environment: TIER3_IMAGE (ollama/ollama:0.40.0), TIER3_WORK (where keys, deliveries, records and +# the models directory go; outside the tree), TIER3_PORT (11534), COSIGN, ORAS, MODEL_SIGNING +# (the programs). Private keys are made here, in TIER3_WORK/keys, and never leave it. +set -euo pipefail + +image="${TIER3_IMAGE:-ollama/ollama:0.40.0}" +work="${TIER3_WORK:?set TIER3_WORK to a directory outside the tree}" +port="${TIER3_PORT:-11534}" +cosign="${COSIGN:?set COSIGN to the cosign program}" +oras="${ORAS:?set ORAS to the oras program}" +model_signing="${MODEL_SIGNING:?set MODEL_SIGNING to the model_signing program}" +serve_name=tier3-ollama +offline_name=tier3-offline +here="$(cd "$(dirname "$0")" && pwd)" +tree="$(cd "$here/../.." && pwd)" +mkdir -p "$work" + +keys() { + mkdir -p "$work/keys" + if [ ! -f "$work/keys/cosign.pub" ]; then + (cd "$work/keys" && COSIGN_PASSWORD= "$cosign" generate-key-pair >/dev/null) + fi + if [ ! -f "$work/keys/oms.pub" ]; then + openssl ecparam -name prime256v1 -genkey -noout -out "$work/keys/oms.key" + openssl ec -in "$work/keys/oms.key" -pubout -out "$work/keys/oms.pub" 2>/dev/null + fi + chmod 600 "$work/keys/"*.key +} + +package() { + local gguf="$1" name="$2" d="$work/delivery/$2" + keys + rm -rf "$d"; mkdir -p "$d/model" + cp "$gguf" "$d/model/" + echo "== OMS: model_signing sign key over $d/model" + "$model_signing" sign key --private_key "$work/keys/oms.key" --signature "$d/model.sig" "$d/model" + echo "== a loopback registry, oras push, cosign sign (no transparency log), cosign save" + docker run -d --rm --name tier3-registry -p 127.0.0.1:5134:5000 registry:2 >/dev/null + trap 'docker stop tier3-registry >/dev/null 2>&1 || true' RETURN + sleep 2 + local ref="127.0.0.1:5134/tier3/$name:v1" + # The layers' titles are the files' base names: push from a directory holding both. + mkdir -p "$d/push"; ln "$d/model/$(basename "$gguf")" "$d/model.sig" "$d/push/" + (cd "$d/push" && "$oras" push --plain-http "$ref" --artifact-type application/vnd.trinity.tier3.model.v1 \ + "$(basename "$gguf"):application/octet-stream" "model.sig:application/vnd.dev.sigstore.bundle.v0.3+json" >/dev/null) + rm -rf "$d/push" + local digest; digest="$("$oras" resolve --plain-http "$ref")" + echo "artifact $ref@$digest" + COSIGN_PASSWORD= "$cosign" sign --key "$work/keys/cosign.key" --use-signing-config=false --tlog-upload=false -y \ + "127.0.0.1:5134/tier3/$name@$digest" 2>&1 | grep -v -i deprecated + "$cosign" save --dir "$d/layout" "127.0.0.1:5134/tier3/$name@$digest" + echo "delivery $d/layout ($(du -sh "$d/layout" | cut -f1))" +} + +serve() { + mkdir -p "$work/ollama-models" + docker rm -f "$serve_name" >/dev/null 2>&1 || true + docker run -d --name "$serve_name" --memory=16g --user "$(id -u):$(id -g)" -e HOME=/tmp \ + -e OLLAMA_MODELS=/models -v "$work/ollama-models:/models" -p "127.0.0.1:$port:11434" "$image" >/dev/null + for _ in $(seq 1 60); do + curl -fsS "http://127.0.0.1:$port/api/version" 2>/dev/null && echo && break + sleep 1 + done + docker image inspect "$image" --format 'image {{.Id}} {{json .RepoDigests}}' +} + +import_() { + local name="$1" model="$2" d="$work/delivery/$1" + mkdir -p "$work/records" + (cd "$tree" && mix trinity.tier3.import --layout "$d/layout" --cosign-key "$work/keys/cosign.pub" \ + --oms-key "$work/keys/oms.pub" --base-url "http://127.0.0.1:$port" --model "$model" \ + --cosign "$cosign" --model-signing "$model_signing" --out "$work/records") +} + +offline() { + local name="$1" + rm -rf "$work/offline-models"; cp -a "$work/ollama-models" "$work/offline-models" + echo "== the directory, verified before the start: every blob's SHA-256 is its name" + local bad=0 + for b in "$work/offline-models/blobs/"sha256-*; do + [ "sha256-$(sha256sum "$b" | cut -c1-64)" = "$(basename "$b")" ] || { echo "MISMATCH $b"; bad=1; } + done + echo "blobs $(ls "$work/offline-models/blobs" | wc -l), mismatches $bad" + local m="$work/offline-models/manifests-v2/ollama.com/library/$name/latest" + echo "manifest $(sha256sum "$m" | cut -c1-64) (the /api/tags digest), layers: $(python3 -c 'import json,sys; print([l["digest"] for l in json.load(open(sys.argv[1]))["layers"]])' "$m")" + [ "$bad" = 0 ] || return 1 + docker rm -f "$offline_name" >/dev/null 2>&1 || true + # Read-write: Ollama 0.40.0 removes the legacy manifests/ directory at start (a read-only mount + # stops it with "remove /models/manifests: read-only file system"). The copy is disposable. + docker run -d --name "$offline_name" --network none --memory=16g --user "$(id -u):$(id -g)" -e HOME=/tmp \ + -e OLLAMA_MODELS=/models -v "$work/offline-models:/models" "$image" >/dev/null + sleep 3 + echo "== the container's network: mode $(docker inspect -f '{{.HostConfig.NetworkMode}}' "$offline_name"); interfaces:" + docker run --rm --network "container:$offline_name" alpine:3.19 sh -c 'ls /sys/class/net; ip route 2>/dev/null || true' + local probe=(docker run --rm --network "container:$offline_name" alpine:3.19) + echo "== /api/version and /api/tags, from inside its network namespace" + "${probe[@]}" wget -qO- http://127.0.0.1:11434/api/version; echo + "${probe[@]}" wget -qO- http://127.0.0.1:11434/api/tags; echo + echo "== /api/embed, truncate:false" + "${probe[@]}" wget -qO- --header 'Content-Type: application/json' \ + --post-data "{\"model\":\"$name\",\"input\":[\"The person's dog is called Rex.\"],\"truncate\":false,\"options\":{\"num_ctx\":2048}}" \ + http://127.0.0.1:11434/api/embed | python3 -c 'import json,sys; d=json.load(sys.stdin); v=d["embeddings"][0]; print("embeddings", len(d["embeddings"]), "dim", len(v), "norm2 %.6f" % sum(x*x for x in v), "prompt_eval_count", d["prompt_eval_count"])' + echo "== a pull, which needs the network" + docker exec "$offline_name" ollama pull all-minilm 2>&1 | tail -1 || true + echo "== the server's log" + docker logs "$offline_name" 2>&1 | grep -v GIN | tail -5 +} + +down() { + docker rm -f "$serve_name" "$offline_name" tier3-registry >/dev/null 2>&1 || true +} + +case "${1:-}" in + package) package "$2" "$3" ;; + serve) serve ;; + import) import_ "$2" "$3" ;; + offline) offline "$2" ;; + down) down ;; + *) sed -n 5,22p "$0"; exit 2 ;; +esac diff --git a/scripts/tier3/parity.exs b/scripts/tier3/parity.exs new file mode 100644 index 0000000..bf155f3 --- /dev/null +++ b/scripts/tier3/parity.exs @@ -0,0 +1,129 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Gate 1, parity (slice 134, AC5): every one of the 500 fixtures embedded by the service has +# cosine >= 0.999 with the sentence-transformers reference, at a pinned num_ctx. +# +# mix run --no-start scripts/tier3/parity.exs --record --fixtures \ +# [--num-ctx 2048] [--mode client|raw] [--truncate true|false] [--out ] +# +# `client` (the default) embeds through Trinity's own client, configured from the import record: +# its own token count, truncate:false, the query template on queries, the pin checked first. +# `raw` posts to /api/embed one item at a time with the given num_ctx and truncate, and also +# compares each item's prompt_eval_count with the reference tokenizer's count; it is how the +# red is run (a num_ctx that does not fit the long items, with Ollama's default truncation). +Code.require_file("gate_lib.exs", __DIR__) +{:ok, _} = Application.ensure_all_started(:req) + +alias Trinity.Memory.Embedders.Ollama + +opts = + Tier3Gate.args(System.argv(), + record: :string, + fixtures: :string, + num_ctx: :integer, + mode: :string, + truncate: :string, + out: :string, + base_url: :string, + batch: :integer + ) + +record = Tier3Gate.record!(opts[:record]) +dir = opts[:fixtures] +manifest = Tier3Gate.manifest(dir) +rows = Tier3Gate.fixtures(dir, "parity.jsonl") +num_ctx = Keyword.get(opts, :num_ctx, 2048) +mode = Keyword.get(opts, :mode, "client") +prompt = manifest["query_prompt"] +min_cosine = 0.999 +base_url = Keyword.get(opts, :base_url, record["ollama"]["base_url"]) + +text = fn r -> if r["role"] == "query", do: prompt <> r["text"], else: r["text"] end + +results = + case mode do + "client" -> + Tier3Gate.configure!(record, num_ctx: num_ctx, query_prompt: prompt, base_url: base_url) + :ok = Ollama.check() + batch = Keyword.get(opts, :batch, 8) + + rows + |> Enum.chunk_by(& &1["role"]) + |> Enum.flat_map(fn group -> + group + |> Enum.chunk_every(batch) + |> Enum.flat_map(fn chunk -> + texts = Enum.map(chunk, & &1["text"]) + role = hd(chunk)["role"] + {:ok, vs} = if role == "query", do: Ollama.embed_query(texts), else: Ollama.embed(texts) + Enum.zip_with(chunk, vs, fn r, v -> %{row: r, vector: v, served: nil, status: 200} end) + end) + end) + + "raw" -> + truncate = + case Keyword.get(opts, :truncate, "false") do + "true" -> true + "false" -> false + "omit" -> :omit + end + + Enum.map(rows, fn r -> + case Tier3Gate.embed_raw(base_url, record["ollama"]["model"], [text.(r)], num_ctx, truncate) do + {200, %{"embeddings" => [v], "prompt_eval_count" => n}} -> + %{row: r, vector: v, served: n, status: 200} + + {status, body} -> + %{row: r, vector: nil, served: nil, status: status, error: body["error"]} + end + end) + end + +scored = + Enum.map(results, fn res -> + cos = if res.vector, do: Tier3Gate.cosine(res.vector, res.row["embedding"]), else: nil + Map.put(res, :cosine, cos) + end) + +cosines = for %{cosine: c} when is_number(c) <- scored, do: c +failed = Enum.filter(scored, &(&1.cosine == nil or &1.cosine < min_cosine)) +counted = for %{served: s, row: r} when is_integer(s) <- scored, do: {s, r["count"]} +count_mismatch = Enum.count(counted, fn {s, c} -> s != c end) + +if out = opts[:out] do + File.write!( + out, + Enum.map_join(scored, "\n", fn s -> + Jason.encode!(%{ + id: s.row["id"], + role: s.row["role"], + reference_count: s.row["count"], + served: s.served, + status: s.status, + cosine: s.cosine + }) + end) <> "\n" + ) +end + +by_role = + scored + |> Enum.group_by(&{&1.row["role"], &1.row["count"] > 200}) + |> Enum.map(fn {{role, long}, xs} -> + cs = for %{cosine: c} when is_number(c) <- xs, do: c + label = if long, do: "#{role} (over 200 tokens)", else: role + " #{label}: #{length(xs)} items, min #{if cs == [], do: "none", else: Float.round(Enum.min(cs), 6)}" + end) + |> Enum.sort() + +IO.puts(""" +GATE1 #{record["gguf"]} as #{record["ollama"]["model"]} (digest #{record["ollama"]["digest"]}), Ollama #{record["ollama"]["version"]} + mode #{mode}, num_ctx #{num_ctx}#{if mode == "raw", do: ", truncate #{Keyword.get(opts, :truncate, "false")}", else: ", truncate false (the client)"} + items #{length(scored)}, vectors #{length(cosines)}, min cosine #{if cosines == [], do: "none", else: Float.round(Enum.min(cosines), 8)}, p01 #{if cosines == [], do: "none", else: Float.round(Tier3Gate.percentile(cosines, 1), 8)}, median #{if cosines == [], do: "none", else: Float.round(Tier3Gate.percentile(cosines, 50), 8)} +#{Enum.join(by_role, "\n")} + below #{min_cosine} or without a vector: #{length(failed)}#{if mode == "raw", do: "\n prompt_eval_count differing from the reference count: #{count_mismatch} of #{length(counted)}", else: ""} + worst: #{scored |> Enum.filter(& &1.cosine) |> Enum.sort_by(& &1.cosine) |> Enum.take(3) |> Enum.map_join("; ", &"id #{&1.row["id"]} #{&1.row["role"]} #{&1.row["count"]} tokens #{Float.round(&1.cosine, 6)}")} +GATE1 VERDICT #{if failed == [], do: "PASS", else: "FAIL"} (#{length(scored) - length(failed)} of #{length(scored)} at or over #{min_cosine}) +#{Tier3Gate.env_line()}\ +""") diff --git a/scripts/tier3/reference_fixtures.py b/scripts/tier3/reference_fixtures.py new file mode 100644 index 0000000..3410f62 --- /dev/null +++ b/scripts/tier3/reference_fixtures.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +"""Writes the Tier 3 reference fixtures into a model cache directory (slice 134). + +Derived from the pinned sentence-transformers reference of the model under test, and kept with the +weights, outside the tree: + + tokenizer-natural.jsonl the natural sentences and the reference tokenizer's ids (the client's + own count is held to these) + tokenizer-adversarial.jsonl generated hostile inputs and their ids + parity.jsonl 500 items (300 sentences as documents, 100 as queries with the query + prompt, 100 long passages as documents) and the reference's + L2-normalised embeddings: Gate 1's other side + fixtures-manifest.json the model revision, the inputs' digests, the seed, the library + versions and each output's SHA-256 + + HF_HUB_OFFLINE=1 python -I scripts/tier3/reference_fixtures.py \ + --model \ + --sentences \ + --adversarial --out /fixtures + +The texts are 133's (WikiText-103, test split, and its 1,000 adversarial items), so the two +embedders' fixtures share inputs; the extra adversarial items here are the ones a byte-level BPE +with added tokens can get wrong (added-token literals, newline and space runs, digit runs, mixed-case +contractions). +""" +import argparse +import hashlib +import json +import os +import random +import re +import sys + +import numpy as np +import pyarrow.parquet as pq +import sentence_transformers +import tokenizers +import torch +import transformers +from sentence_transformers import SentenceTransformer +from tokenizers import Tokenizer + +SEED = 20261008 + + +def sha256(path): + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def clean(line): + line = line.replace(" @-@ ", "-").replace(" @,@ ", ",").replace(" @.@ ", ".") + line = re.sub(r" ([,.;:!?)\]'])", r"\1", line).replace("( ", "(") + return line.strip() + + +def wikitext(parquet): + lines = [clean(l) for l in pq.read_table(parquet).column("text").to_pylist()] + return [l for l in lines if l and not l.startswith("=")] + + +def sentences(lines, rng, n): + out = set() + for line in lines: + for s in re.split(r"(?<=[.!?])\s+(?=[A-Z\"'(])", line): + if 20 <= len(s) <= 400: + out.add(s) + out = sorted(out) + rng.shuffle(out) + if len(out) < n: + sys.exit(f"only {len(out)} sentences, {n} wanted") + return out[:n] + + +def passages(lines, rng, tok, n, lo, hi): + """n passages of lo to hi tokens: consecutive paragraphs joined until the length is reached.""" + starts = list(range(len(lines))) + rng.shuffle(starts) + out = [] + for start in starts: + text, i = "", start + while i < len(lines) and len(tok.encode(text).ids) < lo: + text = (text + "\n\n" + lines[i]) if text else lines[i] + i += 1 + count = len(tok.encode(text).ids) + if lo <= count <= hi: + out.append(text) + if len(out) == n: + return out + sys.exit(f"only {len(out)} passages of {lo} to {hi} tokens") + + +def extra_adversarial(tok, rng): + added = [t.content for t in tok.get_added_tokens_decoder().values()] + items = [] + for t in added: + items += [t, f"before{t}after", f" {t} ", f"{t}{t}", t[:-1]] + items += ["\n" * k for k in (1, 2, 3, 7)] + [" " * k for k in (1, 2, 3, 9)] + items += ["a\n\n\nb", "x \n y", "\t\ttab", "\r\n\r\nwin", " \u00a0 nbsp", "\u3000ideographic space"] + items += ["1234567890", "3.14159", "1,000,000", "٣٤٥ arabic-indic", "Ⅻ roman", "²³ superscripts"] + items += ["I'M", "you'RE", "They'Ll", "it's", "IT'S", "we'd've", "rock'n'roll"] + items += ["e\u0301 decomposed", "é composed", "fi ligature", "Ａｂｃ fullwidth"] + items += ["👩\u200d👩\u200d👧\u200d👦 family", "🏳\ufe0f\u200d🌈", "1\ufe0f⃣", "日本語のテキスト", "한국어 텍스트", "中文文本，标点。"] + items += ["\u200b\u200c\u200d zero widths", "\x00\x01\x1f controls", "\ufeffbom", "퟿"] + items += ["".join(rng.choice("ab \n\t.,!?'\"-") for _ in range(rng.randint(5, 60))) for _ in range(50)] + return items + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--model", required=True) + ap.add_argument("--sentences", required=True) + ap.add_argument("--adversarial", required=True) + ap.add_argument("--out", required=True) + args = ap.parse_args() + os.makedirs(args.out, exist_ok=True) + rng = random.Random(SEED) + torch.manual_seed(SEED) + + tok = Tokenizer.from_file(os.path.join(args.model, "tokenizer.json")) + lines = wikitext(args.sentences) + natural = sentences(lines, rng, 5000) + with open(args.adversarial) as f: + adversarial = [json.loads(l)["text"] for l in f] + adversarial += extra_adversarial(tok, rng) + + def ids(text): + return tok.encode(text, add_special_tokens=False).ids + + def count(text): + return len(tok.encode(text, add_special_tokens=True).ids) + + files = {} + for name, texts in (("tokenizer-natural.jsonl", natural), ("tokenizer-adversarial.jsonl", adversarial)): + path = os.path.join(args.out, name) + with open(path, "w") as f: + for t in texts: + f.write(json.dumps({"text": t, "ids": ids(t), "count": count(t)}, ensure_ascii=False) + "\n") + files[name] = path + + model = SentenceTransformer(args.model, device="cpu") + query_prompt = model.prompts["query"] + docs = natural[:300] + queries = natural[300:400] + long = passages(lines, rng, tok, 100, 200, 1500) + + def encode(texts, prompt_name=None): + return model.encode(texts, prompt_name=prompt_name, batch_size=8, normalize_embeddings=True, + convert_to_numpy=True).astype(np.float64) + + items = [("document", t, v) for t, v in zip(docs, encode(docs))] + items += [("query", t, v) for t, v in zip(queries, encode(queries, "query"))] + items += [("document", t, v) for t, v in zip(long, encode(long))] + path = os.path.join(args.out, "parity.jsonl") + with open(path, "w") as f: + for i, (role, text, vec) in enumerate(items): + prompt = query_prompt if role == "query" else "" + f.write(json.dumps({"id": i, "role": role, "text": text, "count": count(prompt + text), + "embedding": [float(x) for x in vec]}, ensure_ascii=False) + "\n") + files["parity.jsonl"] = path + + manifest = { + "model": args.model, + "model_files": {n: sha256(os.path.join(args.model, n)) for n in ("model.safetensors", "tokenizer.json")}, + "sentences": args.sentences, + "sentences_sha256": sha256(args.sentences), + "adversarial_source_sha256": sha256(args.adversarial), + "seed": SEED, + "query_prompt": query_prompt, + "max_seq_length": model.max_seq_length, + "versions": {"sentence_transformers": sentence_transformers.__version__, + "transformers": transformers.__version__, "tokenizers": tokenizers.__version__, + "torch": torch.__version__, "numpy": np.__version__}, + "script_sha256": sha256(os.path.abspath(__file__)), + "files": {n: sha256(p) for n, p in files.items()}, + } + with open(os.path.join(args.out, "fixtures-manifest.json"), "w") as f: + json.dump(manifest, f, indent=2, ensure_ascii=False) + print(json.dumps(manifest["files"], indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/tier3/truncation.exs b/scripts/tier3/truncation.exs new file mode 100644 index 0000000..9a96a5b --- /dev/null +++ b/scripts/tier3/truncation.exs @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Gate 2, truncation (slice 134, AC6): against the real service, an input of max tokens + 1 with +# truncate:false returns an error, never a vector. +# +# mix run --no-start scripts/tier3/truncation.exs --record [--num-ctx 512] +# +# "Max tokens" is the most the service accepts at that num_ctx, which on Ollama 0.40.0 is +# num_ctx - 1 (slice 134 NOTES); the script finds the boundary rather than assuming it. Two +# inputs are built to an exact count with Trinity's tokenizer (a repeated word, and natural +# prose): max (expected: a vector, prompt_eval_count = max) and max + 1 (expected: an error and +# no vector with truncate:false). The red is the same max + 1 input with truncate:true and with +# truncate omitted (Ollama's default), which the gate exists because of: a vector comes back. +# Then the same through Trinity's client, whose own limit refuses max + 1 before sending. +Code.require_file("gate_lib.exs", __DIR__) +{:ok, _} = Application.ensure_all_started(:req) + +alias Trinity.Memory.BPE +alias Trinity.Memory.Embedders.Ollama + +opts = Tier3Gate.args(System.argv(), record: :string, num_ctx: :integer, base_url: :string, prose: :string) +record = Tier3Gate.record!(opts[:record]) +num_ctx = Keyword.get(opts, :num_ctx, 512) +base = Keyword.get(opts, :base_url, record["ollama"]["base_url"]) +model = record["ollama"]["model"] +{:ok, tok} = record["tokenizer_path"] |> File.read!() |> BPE.from_file() + +# Exact-count inputs: n tokens with the end token, from a repeated word and from prose. +repeat = fn n -> String.duplicate(" a", n - 1) end +prose_words = String.split(opts[:prose] || File.read!(Path.join(__DIR__, "../../README.md"))) + +prose = fn n -> + Enum.reduce_while(prose_words |> Stream.cycle() |> Enum.take(n * 3), "", fn w, acc -> + next = if acc == "", do: w, else: acc <> " " <> w + c = BPE.count(tok, next) + + cond do + c == n -> {:halt, next} + c > n -> {:halt, acc <> String.duplicate(" a", n - BPE.count(tok, acc))} + true -> {:cont, next} + end + end) +end + +show = fn {status, body} -> + cond do + is_map(body) and Map.has_key?(body, "embeddings") -> + v = hd(body["embeddings"]) + "HTTP #{status}, a vector of #{length(v)}, prompt_eval_count #{body["prompt_eval_count"]}" + + true -> + "HTTP #{status}, no vector, error #{inspect(body["error"])}" + end +end + +IO.puts("GATE2 #{model} (digest #{record["ollama"]["digest"]}), Ollama #{record["ollama"]["version"]}, num_ctx #{num_ctx}") + +# The boundary, found: the largest count answered with truncate:false. +max = + Enum.find((num_ctx + 1)..(num_ctx - 3)//-1, fn n -> + match?({200, _}, Tier3Gate.embed_raw(base, model, [repeat.(n)], num_ctx, false)) + end) + +IO.puts(" the largest input the service accepts at num_ctx #{num_ctx}: #{max} tokens") + +verdicts = + for {name, build} <- [repeat: repeat, prose: prose] do + at_max = build.(max) + over = build.(max + 1) + IO.puts(" #{name}: Trinity counts #{BPE.count(tok, at_max)} and #{BPE.count(tok, over)}") + a = Tier3Gate.embed_raw(base, model, [at_max], num_ctx, false) + b = Tier3Gate.embed_raw(base, model, [over], num_ctx, false) + red_true = Tier3Gate.embed_raw(base, model, [over], num_ctx, true) + red_omit = Tier3Gate.embed_raw(base, model, [over], num_ctx, :omit) + IO.puts(" max (#{max}), truncate:false -> #{show.(a)}") + IO.puts(" max + 1 (#{max + 1}), truncate:false -> #{show.(b)}") + IO.puts(" red: max + 1, truncate:true -> #{show.(red_true)}") + IO.puts(" red: max + 1, truncate omitted -> #{show.(red_omit)}") + {at_status, at_body} = a + {over_status, over_body} = b + + at_status == 200 and at_body["prompt_eval_count"] == max and over_status != 200 and + not Map.has_key?(over_body, "embeddings") + end + +# Through Trinity's client, limit max (= num_ctx - 1): max + 1 refused before sending. +Tier3Gate.configure!(record, num_ctx: num_ctx, max_input_tokens: max, base_url: base) +:ok = Ollama.check() +IO.puts(" client, max_input_tokens #{max}: max -> #{inspect(Ollama.embed([repeat.(max)]) |> elem(0))}, max + 1 -> #{inspect(Ollama.embed([repeat.(max + 1)]))}") + +IO.puts("GATE2 VERDICT #{if Enum.all?(verdicts), do: "PASS", else: "FAIL"}") +IO.puts(Tier3Gate.env_line()) diff --git a/test/support/excluded_counter.ex b/test/support/excluded_counter.ex index a3f812e..2c9e7a8 100644 --- a/test/support/excluded_counter.ex +++ b/test/support/excluded_counter.ex @@ -19,6 +19,8 @@ defmodule Trinity.ExcludedCounter do @reasons %{ static_weights: "TRINITY_STATIC_MODEL_DIR unset", + tier3_model: "TRINITY_TIER3_MODEL_DIR unset", + signing_tools: "TRINITY_COSIGN or TRINITY_MODEL_SIGNING unset", local_model: "TRINITY_LOCAL_MODEL_CACHE unset", postgres: "not the Postgres adapter", fips: "TRINITY_FIPS_LEG unset" diff --git a/test/support/fake_ollama.ex b/test/support/fake_ollama.ex new file mode 100644 index 0000000..bc93793 --- /dev/null +++ b/test/support/fake_ollama.ex @@ -0,0 +1,231 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.FakeOllama do + @moduledoc """ + Test-only: the parts of Ollama's HTTP API the Tier 3 client and its import path use (slice + 134), served on the loopback by Bandit, so the whole client path (Req, the JSON, the status + codes) runs with nothing leaving the machine. + + It behaves as Ollama 0.40.0 was observed to (slice 134 NOTES, "Facts gathered"): `/api/embed` + counts each input's tokens with the tokenizer it is given, accepts at most `num_ctx - 1`, and + with `truncate: false` answers an over-length input `400 {"error": "the input length exceeds + the context length"}`; `prompt_eval_count` is the batch's sum; `/api/tags` lists each model + with a 64-hex digest and the `:latest` tag added. A test changes its behaviour through the + Agent: + + * `mode: {:truncate_silently, n}`: every input is cut to `n` tokens and answered 200, + whatever `truncate` says (the service AC1's red is about); + * `mode: {:ctx, n}`: the service enforces its own context of `n`, not the request's; + * `mode: :down`: every request is answered 503; + * `set_digest/3`, `set_version/2`, `remove_model/2`: what `/api/tags` and `/api/version` say; + * `dim:` and `norm:`: the vectors' width and length. + + Every request is recorded (`requests/1`). + """ + import Plug.Conn + + alias Trinity.Memory.BPE + + @doc "Starts the Agent and the server under the test's supervisor; `%{url: _, agent: _}`." + @spec start!(BPE.t(), keyword()) :: %{url: String.t(), agent: pid()} + def start!(tokenizer, opts \\ []) do + state = %{ + tokenizer: tokenizer, + models: %{}, + version: Keyword.get(opts, :version, "0.40.0"), + mode: :honest, + dim: Keyword.get(opts, :dim, 8), + norm: 1.0, + requests: [], + blobs: %{}, + show_blob: nil, + create_digest: nil + } + + agent = ExUnit.Callbacks.start_supervised!({Agent, fn -> state end}, id: make_ref()) + + server = + ExUnit.Callbacks.start_supervised!( + {Bandit, plug: {__MODULE__, agent}, ip: :loopback, port: 0, startup_log: false}, + id: make_ref() + ) + + {:ok, {_ip, port}} = ThousandIsland.listener_info(server) + %{url: "http://127.0.0.1:#{port}", agent: agent} + end + + @doc "Registers a model as `/api/create` would, with a given digest." + @spec put_model(pid(), String.t(), String.t(), String.t()) :: :ok + def put_model(agent, name, digest, blob \\ String.duplicate("b", 64)) do + Agent.update(agent, &put_in(&1, [:models, full(name)], %{digest: digest, blob: blob})) + end + + @doc "Changes the digest `/api/tags` lists for a model (a new blob behind the tag)." + @spec set_digest(pid(), String.t(), String.t()) :: :ok + def set_digest(agent, name, digest), + do: Agent.update(agent, &put_in(&1, [:models, full(name), :digest], digest)) + + @doc "Removes a model from `/api/tags`." + @spec remove_model(pid(), String.t()) :: :ok + def remove_model(agent, name), + do: Agent.update(agent, &update_in(&1, [:models], fn m -> Map.delete(m, full(name)) end)) + + @doc "Sets what `/api/version` answers." + @spec set_version(pid(), String.t()) :: :ok + def set_version(agent, v), do: Agent.update(agent, &Map.put(&1, :version, v)) + + @doc "Sets a field of the state (`mode`, `dim`, `norm`, `show_blob`, `create_digest`)." + @spec set(pid(), atom(), term()) :: :ok + def set(agent, key, value), do: Agent.update(agent, &Map.put(&1, key, value)) + + @doc "The recorded requests, oldest first, as `{method, path, decoded body}`." + @spec requests(pid(), String.t() | nil) :: [{String.t(), String.t(), term()}] + def requests(agent, path \\ nil) do + agent + |> Agent.get(& &1.requests) + |> Enum.reverse() + |> Enum.filter(fn {_, p, _} -> path == nil or p == path end) + end + + @doc "The blobs received, by SHA-256." + @spec blobs(pid()) :: %{String.t() => binary()} + def blobs(agent), do: Agent.get(agent, & &1.blobs) + + defp full(name), do: if(String.contains?(name, ":"), do: name, else: name <> ":latest") + + ## The plug + + def init(agent), do: agent + + def call(conn, agent) do + {:ok, raw, conn} = read_all(conn, "") + body = if raw != "" and json?(conn), do: Jason.decode!(raw), else: raw + path = "/" <> Enum.join(conn.path_info, "/") + Agent.update(agent, &Map.update!(&1, :requests, fn r -> [{conn.method, path, body} | r] end)) + state = Agent.get(agent, & &1) + + if state.mode == :down, + do: reply(conn, 503, %{"error" => "down"}), + else: route(conn, conn.method, path, body, state, agent) + end + + defp read_all(conn, acc) do + case read_body(conn, length: 64_000_000) do + {:ok, data, conn} -> {:ok, acc <> data, conn} + {:more, data, conn} -> read_all(conn, acc <> data) + end + end + + defp json?(conn), do: not String.starts_with?(conn.request_path, "/api/blobs") + + defp route(conn, "GET", "/api/version", _body, state, _agent), + do: reply(conn, 200, %{"version" => state.version}) + + defp route(conn, "GET", "/api/tags", _body, state, _agent) do + models = + for {name, m} <- Enum.sort(state.models), + do: %{"name" => name, "model" => name, "digest" => m.digest, "size" => 1} + + reply(conn, 200, %{"models" => models}) + end + + defp route(conn, "POST", "/api/embed", body, state, _agent), do: embed(conn, body, state) + + defp route(conn, "HEAD", "/api/blobs/sha256:" <> sha, _body, state, _agent), + do: + if(Map.has_key?(state.blobs, sha), + do: send_resp(conn, 200, ""), + else: send_resp(conn, 404, "") + ) + + defp route(conn, "POST", "/api/blobs/sha256:" <> sha, raw, _state, agent) do + if Base.encode16(:crypto.hash(:sha256, raw), case: :lower) == sha do + Agent.update(agent, &put_in(&1, [:blobs, sha], raw)) + send_resp(conn, 201, "") + else + reply(conn, 400, %{"error" => "digest mismatch"}) + end + end + + defp route(conn, "POST", "/api/create", %{"model" => name, "files" => files}, state, agent) do + [{_file, "sha256:" <> sha}] = Map.to_list(files) + + if Map.has_key?(state.blobs, sha) do + digest = + state.create_digest || + Base.encode16(:crypto.hash(:sha256, full(name) <> sha), case: :lower) + + put_model(agent, name, digest, sha) + reply(conn, 200, %{"status" => "success"}) + else + reply(conn, 400, %{"error" => "blob not found"}) + end + end + + defp route(conn, "POST", "/api/show", %{"model" => name}, state, _agent) do + case state.models[full(name)] do + nil -> + reply(conn, 404, %{"error" => "model not found"}) + + m -> + blob = state.show_blob || m.blob + + reply(conn, 200, %{ + "modelfile" => "FROM /models/blobs/sha256-#{blob}\nTEMPLATE {{ .Prompt }}\n" + }) + end + end + + defp route(conn, _method, _path, _body, _state, _agent), + do: reply(conn, 404, %{"error" => "not found"}) + + defp embed(conn, %{"model" => name, "input" => inputs} = body, state) do + inputs = List.wrap(inputs) + ctx = context(state.mode, get_in(body, ["options", "num_ctx"]) || 4096) + counts = Enum.map(inputs, &BPE.count(state.tokenizer, &1)) + + cond do + not Map.has_key?(state.models, full(name)) -> + reply(conn, 404, %{"error" => "model \"#{name}\" not found"}) + + match?({:truncate_silently, _}, state.mode) -> + {:truncate_silently, n} = state.mode + answer(conn, name, inputs, Enum.map(counts, &min(&1, n)), state) + + body["truncate"] == false and Enum.any?(counts, &(&1 > ctx - 1)) -> + reply(conn, 400, %{"error" => "the input length exceeds the context length"}) + + true -> + answer(conn, name, inputs, Enum.map(counts, &min(&1, ctx - 1)), state) + end + end + + defp context({:ctx, n}, _requested), do: n + defp context(_mode, requested), do: requested + + defp answer(conn, name, inputs, served, state) do + reply(conn, 200, %{ + "model" => name, + "embeddings" => Enum.map(inputs, &vector(&1, state.dim, state.norm)), + "prompt_eval_count" => Enum.sum(served) + }) + end + + @doc "The fake's deterministic vector for a text: `dim` components from its SHA-512, of length `norm`." + @spec vector(String.t(), pos_integer(), float()) :: [float()] + def vector(text, dim, norm \\ 1.0) do + bytes = + Stream.iterate(0, &(&1 + 1)) + |> Stream.map(&:crypto.hash(:sha512, [text, <<&1::32>>])) + |> Enum.take(div(dim, 64) + 1) + |> IO.iodata_to_binary() + + raw = for <>, do: b / 1.0 + len = :math.sqrt(Enum.reduce(raw, 0.0, &(&1 * &1 + &2))) + Enum.map(raw, &(&1 / len * norm)) + end + + defp reply(conn, status, map) do + conn |> put_resp_content_type("application/json") |> send_resp(status, Jason.encode!(map)) + end +end diff --git a/test/support/oci_layout.ex b/test/support/oci_layout.ex new file mode 100644 index 0000000..05ea1d2 --- /dev/null +++ b/test/support/oci_layout.ex @@ -0,0 +1,179 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.OCILayout do + @moduledoc """ + Test-only (slice 134): builds the OCI image layout the Tier 3 import path takes, signed in the + shape cosign v3.1.3 writes with `cosign sign --key ... --use-signing-config=false + --tlog-upload=false` and `cosign save` (observed 2026-10-08, slice 134 NOTES): `index.json` + lists the image manifest only; the signature is a referrer manifest whose `subject` is the image + manifest and whose one layer is a Sigstore bundle v0.3 holding a DSSE envelope over an in-toto + statement naming the image manifest's digest. + + The keys are made by the test (`key!/0`, P-256) and written only under the test's temporary + directory. The real `cosign` is what verifies the result (the `:signing_tools` cases), so a + layout it would not accept fails the valid case rather than passing a test of mine. + """ + + @empty_config ~s({}) + @oci_manifest "application/vnd.oci.image.manifest.v1+json" + @bundle_type "application/vnd.dev.sigstore.bundle.v0.3+json" + @secp256r1 {1, 2, 840, 10_045, 3, 1, 7} + + @doc "A fresh P-256 key pair: `%{private: record, public_pem: binary, private_pem: binary, der: binary}`." + @spec key!() :: map() + def key! do + priv = :public_key.generate_key({:namedCurve, @secp256r1}) + {:ECPrivateKey, _v, _d, _params, point, _attrs} = priv + spki = {{:ECPoint, point}, {:namedCurve, @secp256r1}} + {_, der, _} = entry = :public_key.pem_entry_encode(:SubjectPublicKeyInfo, spki) + + %{ + private: priv, + der: der, + public_pem: :public_key.pem_encode([entry]), + private_pem: :public_key.pem_encode([:public_key.pem_entry_encode(:ECPrivateKey, priv)]) + } + end + + @doc "Writes a key pair's halves under `dir`: `{public_path, private_path}`." + @spec write_key!(map(), Path.t(), String.t()) :: {Path.t(), Path.t()} + def write_key!(key, dir, name) do + pub = Path.join(dir, name <> ".pub") + priv = Path.join(dir, name <> ".key") + File.write!(pub, key.public_pem) + File.write!(priv, key.private_pem) + File.chmod!(priv, 0o600) + {pub, priv} + end + + @doc """ + Writes a layout into `dir` holding `files` (`[{title, bytes}]`) as layers, signed with + `cosign_key` (a `key!/0` map, or nil for none). Options: `sign_digest:` signs that manifest + digest instead of this one's (a valid signature over another artifact); `flip_signature:` flips + one bit of the signature. Returns the image manifest's digest. + """ + @spec build!(Path.t(), [{String.t(), binary()}], map() | nil, keyword()) :: String.t() + def build!(dir, files, cosign_key, opts \\ []) do + File.mkdir_p!(Path.join([dir, "blobs", "sha256"])) + File.write!(Path.join(dir, "oci-layout"), ~s({"imageLayoutVersion":"1.0.0"})) + config = blob!(dir, @empty_config) + + layers = + for {title, bytes} <- files do + bytes + |> then(&blob!(dir, &1)) + |> Map.merge(%{ + "mediaType" => "application/octet-stream", + "annotations" => %{"org.opencontainers.image.title" => title} + }) + end + + manifest = + Jason.encode!(%{ + "schemaVersion" => 2, + "mediaType" => @oci_manifest, + "artifactType" => "application/vnd.trinity.tier3.model.v1", + "config" => Map.put(config, "mediaType", "application/vnd.oci.empty.v1+json"), + "layers" => layers + }) + + image = blob!(dir, manifest) |> Map.put("mediaType", @oci_manifest) + + index = %{ + "schemaVersion" => 2, + "mediaType" => "application/vnd.oci.image.index.v1+json", + "manifests" => [ + Map.put(image, "annotations", %{"kind" => "dev.cosignproject.cosign/image"}) + ] + } + + File.write!(Path.join(dir, "index.json"), Jason.encode!(index)) + if cosign_key, do: sign!(dir, image, cosign_key, config, opts) + image["digest"] + end + + defp sign!(dir, image, key, config, opts) do + "sha256:" <> hex = Keyword.get(opts, :sign_digest, image["digest"]) + + statement = + Jason.encode!(%{ + "_type" => "https://in-toto.io/Statement/v1", + "subject" => [%{"digest" => %{"sha256" => hex}, "annotations" => %{}}], + "predicateType" => "https://sigstore.dev/cosign/sign/v1", + "predicate" => %{} + }) + + type = "application/vnd.in-toto+json" + pae = "DSSEv1 #{byte_size(type)} #{type} #{byte_size(statement)} #{statement}" + sig = :public_key.sign(pae, :sha256, key.private) + sig = if Keyword.get(opts, :flip_signature), do: flip(sig, 10), else: sig + + bundle = + Jason.encode!(%{ + "mediaType" => @bundle_type, + "verificationMaterial" => %{ + "publicKey" => %{"hint" => Base.encode64(:crypto.hash(:sha256, key.der))} + }, + "dsseEnvelope" => %{ + "payload" => Base.encode64(statement), + "payloadType" => type, + "signatures" => [%{"sig" => Base.encode64(sig)}] + } + }) + + bundle_desc = blob!(dir, bundle) |> Map.put("mediaType", @bundle_type) + + referrer = + Jason.encode!(%{ + "schemaVersion" => 2, + "mediaType" => @oci_manifest, + "artifactType" => @bundle_type, + "config" => + Map.merge(config, %{ + "mediaType" => "application/vnd.oci.empty.v1+json", + "artifactType" => @bundle_type + }), + "layers" => [bundle_desc], + "annotations" => %{ + "dev.sigstore.bundle.content" => "dsse-envelope", + "dev.sigstore.bundle.predicateType" => "https://sigstore.dev/cosign/sign/v1" + }, + "subject" => image + }) + + blob!(dir, referrer) + end + + @doc ~s(Writes bytes as a blob: the descriptor `%{"digest", "size"}`.) + @spec blob!(Path.t(), binary()) :: map() + def blob!(dir, bytes) do + hex = :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower) + File.write!(Path.join([dir, "blobs", "sha256", hex]), bytes) + %{"digest" => "sha256:" <> hex, "size" => byte_size(bytes)} + end + + @doc "One bit of `bin` flipped at byte `at`." + @spec flip(binary(), non_neg_integer()) :: binary() + def flip(bin, at) do + <> = bin + <> + end + + @doc "Flips one bit of the blob holding layer `title` in place (a byte flipped in transit)." + @spec flip_layer!(Path.t(), String.t(), non_neg_integer()) :: :ok + def flip_layer!(dir, title, at \\ 100) do + [desc] = + layers(dir) |> Enum.filter(&(&1["annotations"]["org.opencontainers.image.title"] == title)) + + "sha256:" <> hex = desc["digest"] + path = Path.join([dir, "blobs", "sha256", hex]) + File.write!(path, flip(File.read!(path), at)) + end + + defp layers(dir) do + %{"manifests" => [%{"digest" => "sha256:" <> hex}]} = + Jason.decode!(File.read!(Path.join(dir, "index.json"))) + + Jason.decode!(File.read!(Path.join([dir, "blobs", "sha256", hex])))["layers"] + end +end diff --git a/test/support/tier3_helper.ex b/test/support/tier3_helper.ex new file mode 100644 index 0000000..89243f5 --- /dev/null +++ b/test/support/tier3_helper.ex @@ -0,0 +1,131 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Tier3Helper do + @moduledoc """ + Test-only (slice 134): a small byte-level BPE vocabulary written by hand, a GGUF file carrying + it (`gguf/1`, through a writer of the format's header), and the Ollama embedder's + configuration pointed at a `Trinity.FakeOllama`. + + The vocabulary is every byte's GPT-2 stand-in (ids 0 to 255), five merges, and + `<|endoftext|>` appended as the end token, so `BPE.count/2` of `"hello"` is + `h e ll o` merged by the rules below plus one: small enough to reason about in a test. + """ + alias Trinity.Memory.BPE + + @merges ["Ġ a", "h e", "l l", "he ll", "Ġ w"] + @eos "<|endoftext|>" + + @doc "The tokens, in id order." + @spec tokens() :: [String.t()] + def tokens do + bytes = for b <- 0..255, do: BPE.byte_char(b) + merged = Enum.map(@merges, &String.replace(&1, " ", "")) + bytes ++ merged ++ [@eos] + end + + @doc "The merges, in rank order." + @spec merges() :: [String.t()] + def merges, do: @merges + + @doc "The end token's id." + @spec eos() :: non_neg_integer() + def eos, do: length(tokens()) - 1 + + @doc "The tokenizer." + @spec tokenizer() :: BPE.t() + def tokenizer, do: BPE.new(tokens(), merges(), added: [eos()], eos: eos()) + + @doc "Writes the tokenizer file into `dir`: `{path, sha256}`." + @spec write_tokenizer!(Path.t()) :: {Path.t(), String.t()} + def write_tokenizer!(dir) do + bin = BPE.to_file(tokenizer()) + sha = sha256(bin) + path = Path.join(dir, "#{sha}.bpe.json") + File.mkdir_p!(dir) + File.write!(path, bin) + {path, sha} + end + + @doc "A GGUF file's bytes with this tokenizer and a Qwen3-like header (`extra:` more pairs)." + @spec gguf(keyword()) :: binary() + def gguf(opts \\ []) do + types = Enum.map(tokens(), fn t -> if t == @eos, do: 3, else: 1 end) + + pairs = + [ + {"general.architecture", {:string, "qwen3"}}, + {"general.name", {:string, "test model"}}, + {"general.license", {:string, "apache-2.0"}}, + {"general.file_type", {:u32, 1}}, + {"qwen3.embedding_length", {:u32, Keyword.get(opts, :dim, 8)}}, + {"qwen3.context_length", {:u32, 512}}, + {"qwen3.pooling_type", {:u32, 3}}, + {"tokenizer.ggml.model", {:string, "gpt2"}}, + {"tokenizer.ggml.pre", {:string, "qwen2"}}, + {"tokenizer.ggml.tokens", {:array, :string, tokens()}}, + {"tokenizer.ggml.merges", {:array, :string, merges()}}, + {"tokenizer.ggml.token_type", {:array, :i32, types}}, + {"tokenizer.ggml.add_bos_token", {:bool, false}}, + {"tokenizer.ggml.add_eos_token", {:bool, true}}, + {"tokenizer.ggml.eos_token_id", {:u32, eos()}} + ] ++ Keyword.get(opts, :extra, []) + + header = <<"GGUF", 3::little-32, 0::little-64, length(pairs)::little-64>> + body = for {k, v} <- pairs, into: <<>>, do: str(k) <> value(v) + # Stand-in tensor bytes: the reader never looks past the header. + header <> body <> :crypto.strong_rand_bytes(Keyword.get(opts, :tail, 4096)) + end + + defp str(s), do: <> + + defp value({:string, s}), do: <<8::little-32>> <> str(s) + defp value({:u32, n}), do: <<4::little-32, n::little-32>> + defp value({:bool, b}), do: <<7::little-32, if(b, do: 1, else: 0)::8>> + defp value({:f32, f}), do: <<6::little-32, f::little-float-32>> + + defp value({:array, type, items}) do + code = %{string: 8, i32: 5, u32: 4}[type] + + <<9::little-32, code::little-32, length(items)::little-64>> <> + for(i <- items, into: <<>>, do: elem_value(type, i)) + end + + defp elem_value(:string, s), do: str(s) + defp elem_value(:i32, n), do: <> + defp elem_value(:u32, n), do: <> + + @doc """ + The memory configuration for an Ollama embedder at `url` (a `FakeOllama`), with the tokenizer + file at `tokenizer`, overridden by `overrides` (keys of the `ollama:` list). + """ + @spec memory(String.t(), {Path.t(), String.t()}, keyword()) :: keyword() + def memory(url, {tok_path, tok_sha}, overrides \\ []) do + ollama = + Keyword.merge( + [ + base_url: url, + model: "embed-test", + model_id: "test/embed-test", + digest: String.duplicate("d", 64), + weights_sha256: String.duplicate("a", 64), + tokenizer_sha256: tok_sha, + tokenizer_path: tok_path, + runtime_version: "0.40.0", + num_ctx: 64, + max_input_tokens: 63, + dim: 8, + query_prompt: "Q: ", + document_prompt: "", + check_interval_ms: 300_000, + timeout_ms: 5_000 + ], + overrides + ) + + [embedder: :ollama, locality: :within_boundary, ollama: ollama] + end + + @doc "SHA-256, lowercase hex." + @spec sha256(iodata()) :: String.t() + def sha256(data), do: :crypto.hash(:sha256, data) |> Base.encode16(case: :lower) +end diff --git a/test/test_helper.exs b/test/test_helper.exs index f7389cf..af336c4 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -14,11 +14,22 @@ # where TRINITY_STATIC_MODEL_DIR names the directory holding them; excluded by tag elsewhere, # and where the variable is set a missing file is a failure, never a skip (the owner's rule, # 2026-10-07). No weights are in the tree before legal review (slice 133 NOTES, D7). +# Slice 134: :tier3_model (the Tier 3 tokenizer's parity with the reference, over the GGUF and +# the fixtures in the model cache) runs where TRINITY_TIER3_MODEL_DIR names that directory, and +# :signing_tools (the import path's signature cases, through the real cosign and model_signing) +# where TRINITY_COSIGN and TRINITY_MODEL_SIGNING name the two programs; excluded by tag +# elsewhere, and where set, a missing file is a failure, never a skip. # :postgres (EXPLAIN plans and the per-space HNSW indexes) runs only on the Postgres adapter. # `Trinity.ExcludedCounter` prints, after the summary, how many each of these tags excluded. fips_leg? = System.get_env("TRINITY_FIPS_LEG") == "1" local_model? = System.get_env("TRINITY_LOCAL_MODEL_CACHE") not in [nil, ""] static_weights? = System.get_env("TRINITY_STATIC_MODEL_DIR") not in [nil, ""] +tier3_model? = System.get_env("TRINITY_TIER3_MODEL_DIR") not in [nil, ""] + +signing_tools? = + System.get_env("TRINITY_COSIGN") not in [nil, ""] and + System.get_env("TRINITY_MODEL_SIGNING") not in [nil, ""] + postgres? = Application.get_env(:trinity, :db_adapter) == Ecto.Adapters.Postgres ExUnit.start( @@ -27,6 +38,8 @@ ExUnit.start( if(fips_leg?, do: [], else: [:fips]) ++ if(local_model?, do: [], else: [:local_model]) ++ if(static_weights?, do: [], else: [:static_weights]) ++ + if(tier3_model?, do: [], else: [:tier3_model]) ++ + if(signing_tools?, do: [], else: [:signing_tools]) ++ if(postgres?, do: [], else: [:postgres]), formatters: [ExUnit.CLIFormatter, Trinity.ExcludedCounter] ) diff --git a/test/trinity/embedder_boot_node_test.exs b/test/trinity/embedder_boot_node_test.exs index 3335441..f46c847 100644 --- a/test/trinity/embedder_boot_node_test.exs +++ b/test/trinity/embedder_boot_node_test.exs @@ -278,4 +278,57 @@ defmodule Trinity.EmbedderBootNodeTest do assert line(out, "SEMANTIC") == "SEMANTIC {:off, :weights_digest_mismatch}" end end + + describe "slice 134: the Tier 3 client's own configuration faults refuse a regulated boot" do + # The memory configuration of a Tier 3 client on an allow-listed endpoint; `overrides` go into + # its `ollama:` list. Nothing listens there: a correct configuration boots, and the watcher's + # first check records the service as unreachable, which is a runtime fault, not a refusal. + defp ollama_memory(overrides) do + ollama = + Keyword.merge( + [ + base_url: "https://models.internal:11434", + model: "embed", + digest: String.duplicate("d", 64), + weights_sha256: String.duplicate("a", 64), + tokenizer_sha256: String.duplicate("c", 64), + runtime_version: "0.40.0", + num_ctx: 2048, + max_input_tokens: 2047, + dim: 1024, + timeout_ms: 200 + ], + overrides + ) + + [embedder: :ollama, locality: :within_boundary, ollama: ollama] + end + + # The chat model on the allow-list (as the cases above), then the Tier 3 memory configuration. + defp ollama_script(memory), + do: @fixture <> models_and_memory("https://models.internal/v1", memory) <> @report + + test "max_input_tokens not below num_ctx: refused with :ollama_config" do + script = ollama_script(ollama_memory(max_input_tokens: 2048)) + {out, status} = boot("ollama_bad_limit", regulated(), script) + refusal = refused!(out, status, "ollama_config") + assert refusal =~ "max_input_not_below_num_ctx" + end + + test "a correct configuration boots, and the unreachable service is a runtime fault" do + {out, status} = boot("ollama_good", regulated(), ollama_script(ollama_memory([]))) + + assert status == 0, """ + a correctly configured regulated node refused: if this fails while the refusal passes, the + check refuses everything and proves nothing. + #{String.slice(out, -3000, 3000)} + """ + + assert line(out, "BOOT_OK") == "BOOT_OK" + IO.puts("\n" <> line(out, "SEMANTIC")) + + assert line(out, "SEMANTIC") =~ + ~r/^SEMANTIC \{:off, :(digest_unchecked|tokenizer_missing|endpoint_unreachable)\}$/ + end + end end diff --git a/test/trinity/memory/bpe_parity_test.exs b/test/trinity/memory/bpe_parity_test.exs new file mode 100644 index 0000000..461b3a1 --- /dev/null +++ b/test/trinity/memory/bpe_parity_test.exs @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.BPEParityTest do + @moduledoc """ + Slice 134: the Tier 3 client counts tokens exactly as the reference does, over the model's own + vocabulary, taken from its GGUF as the import path takes it. + + `:tier3_model`: runs where `TRINITY_TIER3_MODEL_DIR` names the directory holding + `gguf/Qwen3-Embedding-0.6B-f16.gguf` and the fixtures `scripts/tier3/reference_fixtures.py` + wrote into `fixtures/`; a missing file there is a failure, never a skip. Nothing here is in the + tree: the vocabulary is the model's. + """ + use ExUnit.Case, async: true + + alias Trinity.Memory.{BPE, GGUF} + + @moduletag :tier3_model + @moduletag timeout: 300_000 + + setup_all do + dir = System.fetch_env!("TRINITY_TIER3_MODEL_DIR") + gguf = Path.join([dir, "gguf", "Qwen3-Embedding-0.6B-f16.gguf"]) + assert File.regular?(gguf), "TRINITY_TIER3_MODEL_DIR is set and #{gguf} is absent" + {:ok, meta} = GGUF.metadata(gguf) + {:ok, tok} = BPE.from_gguf(meta) + {:ok, tok: tok, dir: dir} + end + + defp rows(dir, name) do + path = Path.join([dir, "fixtures", name]) + assert File.regular?(path), "#{path} is absent" + path |> File.stream!() |> Enum.map(&Jason.decode!/1) + end + + defp differing(tok, rows) do + Enum.reject(rows, fn r -> + BPE.encode(tok, r["text"]) == r["ids"] and BPE.count(tok, r["text"]) == r["count"] + end) + end + + test "5,000 natural sentences: every id and every count identical", %{tok: tok, dir: dir} do + rows = rows(dir, "tokenizer-natural.jsonl") + assert length(rows) == 5000 + bad = differing(tok, rows) + IO.puts("\nBPE parity: #{length(bad)} of #{length(rows)} natural sentences differ") + assert bad == [] + end + + test "the adversarial set: every id and every count identical", %{tok: tok, dir: dir} do + rows = rows(dir, "tokenizer-adversarial.jsonl") + texts = Enum.map(rows, & &1["text"]) + assert Enum.any?(texts, &String.contains?(&1, "<|endoftext|>")) + assert Enum.any?(texts, &String.contains?(&1, "\u180e")) + bad = differing(tok, rows) + IO.puts("\nBPE parity: #{length(bad)} of #{length(rows)} adversarial items differ") + assert bad == [], inspect(Enum.take(bad, 3), printable_limit: 200) + end + + test "Gate 1's 500 items: the count the client would send equals the reference's", %{ + tok: tok, + dir: dir + } do + rows = rows(dir, "parity.jsonl") + manifest = Jason.decode!(File.read!(Path.join([dir, "fixtures", "fixtures-manifest.json"]))) + prompt = manifest["query_prompt"] + + bad = + Enum.reject(rows, fn r -> + text = if r["role"] == "query", do: prompt <> r["text"], else: r["text"] + BPE.count(tok, text) == r["count"] + end) + + assert length(rows) == 500 + assert bad == [] + end +end diff --git a/test/trinity/memory/bpe_test.exs b/test/trinity/memory/bpe_test.exs new file mode 100644 index 0000000..e2ad1ab --- /dev/null +++ b/test/trinity/memory/bpe_test.exs @@ -0,0 +1,167 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.BPETest do + @moduledoc """ + Slice 134: the byte-level BPE's rules on a hand-written vocabulary (`Trinity.Tier3Helper`), + on every leg, and the GGUF header reader on a file the test writes. Parity with the reference + over the real vocabulary is `bpe_parity_test.exs` (`:tier3_model`). + """ + use ExUnit.Case, async: true + + alias Trinity.Memory.{BPE, GGUF} + alias Trinity.Tier3Helper + + defp tok, do: Tier3Helper.tokenizer() + defp id(token), do: Enum.find_index(Tier3Helper.tokens(), &(&1 == token)) + defp b(char), do: id(BPE.byte_char(char)) + + test "merges apply by rank: hello is hell + o" do + assert BPE.encode(tok(), "hello") == [id("hell"), b(?o)] + assert BPE.count(tok(), "hello") == 3 + end + + test "a leading space is the byte map's Ġ, and a space-a merges" do + assert BPE.encode(tok(), " a") == [id("Ġa")] + assert BPE.encode(tok(), "a a") == [b(?a), id("Ġa")] + end + + test "every byte of any input has a symbol: nothing is unknown" do + text = "naïve 日本 👩\u200d👩\u200d👧 \u0000" + ids = BPE.encode(tok(), text) + assert length(ids) == byte_size(:unicode.characters_to_nfc_binary(text)) + end + + test "an added token in the raw text is one id; the text around it is tokenized alone" do + eos = Tier3Helper.eos() + assert BPE.encode(tok(), "he<|endoftext|>he") == [id("he"), eos, id("he")] + assert BPE.encode(tok(), "<|endoftext|>") == [eos] + end + + test "the end token is counted once per text" do + assert BPE.count(tok(), "") == 1 + assert BPE.count(tok(), "hello") == length(BPE.encode(tok(), "hello")) + 1 + end + + test "the pre-tokenizer is Qwen2's: contractions, letters, single digits, punctuation, spaces" do + assert BPE.pre_tokenize("I'M here, 2026!\n\n ok") == + ["I", "'M", " here", ",", " ", "2", "0", "2", "6", "!\n\n", " ", " ok"] + end + + test "whitespace is Unicode's White_Space: U+180E is not a space, U+2009 is" do + # PCRE's \s matches U+180E; the reference's engine does not (slice 134 NOTES, decision 14). + assert BPE.pre_tokenize("U\u180e.S") == ["U", "\u180e.", "S"] + assert BPE.pre_tokenize("a\u2009b") == ["a", "\u2009b"] + end + + test "NFC before splitting: a decomposed é is the composed one's bytes" do + assert BPE.encode(tok(), "e\u0301") == BPE.encode(tok(), "\u00e9") + end + + test "the file is deterministic and reads back to the same tokenizer" do + bin = BPE.to_file(tok()) + assert bin == BPE.to_file(tok()) + assert {:ok, back} = BPE.from_file(bin) + assert back == tok() + assert BPE.from_file("{}") == {:error, :not_a_tokenizer_file} + assert BPE.from_file("not json") == {:error, :not_a_tokenizer_file} + end + + describe "GGUF" do + test "the header's metadata, and the tokenizer it describes" do + bin = Tier3Helper.gguf(dim: 16) + assert {:ok, meta} = GGUF.parse(bin) + assert meta["general.architecture"] == "qwen3" + assert meta["qwen3.embedding_length"] == 16 + assert meta["tokenizer.ggml.add_eos_token"] == true + assert meta["tokenizer.ggml.tokens"] == Tier3Helper.tokens() + assert {:ok, from_gguf} = BPE.from_gguf(meta) + assert BPE.to_file(from_gguf) == BPE.to_file(tok()) + end + + @tag :tmp_dir + test "a file is read in growing pieces, only as far as the header", %{tmp_dir: dir} do + path = Path.join(dir, "m.gguf") + File.write!(path, Tier3Helper.gguf(tail: 3_000_000)) + assert {:ok, %{"general.name" => "test model"}} = GGUF.metadata(path) + end + + test "every scalar type and an array of each" do + pairs = [ + {"u8", <<0::little-32, 200>>}, + {"i8", <<1::little-32, -3::signed-8>>}, + {"u16", <<2::little-32, 60_000::little-16>>}, + {"i16", <<3::little-32, -300::little-signed-16>>}, + {"u32", <<4::little-32, 4_000_000_000::little-32>>}, + {"i32", <<5::little-32, -70_000::little-signed-32>>}, + {"f32", <<6::little-32, 1.5::little-float-32>>}, + {"bool", <<7::little-32, 0>>}, + {"u64", <<10::little-32, 2 ** 40::little-64>>}, + {"i64", <<11::little-32, -(2 ** 40)::little-signed-64>>}, + {"f64", <<12::little-32, 0.25::little-float-64>>}, + {"ints", + <<9::little-32, 5::little-32, 2::little-64, 7::little-signed-32, -7::little-signed-32>>} + ] + + body = + for {k, v} <- pairs, into: <<>>, do: <> + + bin = <<"GGUF", 2::little-32, 0::little-64, length(pairs)::little-64, body::binary>> + + assert {:ok, meta} = GGUF.parse(bin) + + assert meta == %{ + "u8" => 200, + "i8" => -3, + "u16" => 60_000, + "i16" => -300, + "u32" => 4_000_000_000, + "i32" => -70_000, + "f32" => 1.5, + "bool" => false, + "u64" => 2 ** 40, + "i64" => -(2 ** 40), + "f64" => 0.25, + "ints" => [7, -7] + } + + nested = + <<"GGUF", 3::little-32, 0::64, 1::little-64, 1::little-64, "n", 9::little-32, + 9::little-32, 1::little-64>> + + assert GGUF.parse(nested) == {:error, :nested_array} + assert GGUF.parse(binary_part(bin, 0, byte_size(bin) - 3)) == {:error, :truncated} + end + + @tag :tmp_dir + test "a missing or empty file is refused", %{tmp_dir: dir} do + assert GGUF.metadata(Path.join(dir, "absent.gguf")) == {:error, {:unreadable, :enoent}} + File.write!(Path.join(dir, "empty.gguf"), "") + assert GGUF.metadata(Path.join(dir, "empty.gguf")) == {:error, :empty} + end + + test "refusals: not a GGUF, cut short, an unknown type, another version" do + bin = Tier3Helper.gguf() + assert GGUF.parse("GGML" <> binary_part(bin, 4, 100)) == {:error, :not_gguf} + assert GGUF.parse(binary_part(bin, 0, 200)) == {:error, :truncated} + + assert GGUF.parse(<<"GGUF", 1::little-32, 0::64, 0::64>>) == + {:error, {:unsupported_version, 1}} + + bad = <<"GGUF", 3::little-32, 0::little-64, 1::little-64, 1::little-64, "k", 99::little-32>> + assert GGUF.parse(bad) == {:error, {:unknown_type, 99}} + end + + test "a tokenizer this module does not reproduce is refused" do + {:ok, meta} = GGUF.parse(Tier3Helper.gguf()) + + assert BPE.from_gguf(%{meta | "tokenizer.ggml.pre" => "llama3"}) == + {:error, {:unsupported_pre_tokenizer, "llama3"}} + + assert BPE.from_gguf(%{meta | "tokenizer.ggml.model" => "llama"}) == + {:error, {:unsupported_tokenizer, "llama"}} + + assert BPE.from_gguf(Map.delete(meta, "tokenizer.ggml.merges")) == + {:error, :tokenizer_metadata_missing} + end + end +end diff --git a/test/trinity/memory/ollama_digest_test.exs b/test/trinity/memory/ollama_digest_test.exs new file mode 100644 index 0000000..7d09eee --- /dev/null +++ b/test/trinity/memory/ollama_digest_test.exs @@ -0,0 +1,131 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.OllamaDigestTest do + @moduledoc """ + Slice 134, AC2: changing the fake `/api/tags` digest leaves the store OFF with + `:model_digest_changed` within one check interval, with FTS5 continuing and no other space + selected. + + The watcher (`Embedders.Ollama.Watch`) runs as the application would run it, under the test's + supervisor, with a short interval; the service is `Trinity.FakeOllama` on the loopback. + """ + use Trinity.DataCase + + alias Trinity.{Factory, FakeOllama, Tier3Helper} + alias Trinity.Memory.{AlwaysOn, Retriever, Semantic, SpaceRow, Spaces} + alias Trinity.Memory.Embedders.Ollama + + @moduletag :tmp_dir + @interval 200 + + setup %{tmp_dir: dir} do + memory = Application.get_env(:trinity, :memory, []) + Ollama.reset() + + on_exit(fn -> + Application.put_env(:trinity, :memory, memory) + Ollama.reset() + end) + + tok = Tier3Helper.write_tokenizer!(dir) + fake = FakeOllama.start!(Tier3Helper.tokenizer()) + config = Tier3Helper.memory(fake.url, tok, check_interval_ms: @interval) + Application.put_env(:trinity, :memory, Keyword.merge(memory, config)) + FakeOllama.put_model(fake.agent, "embed-test", config[:ollama][:digest]) + {:ok, fake: fake, pinned: config[:ollama][:digest]} + end + + # Polls every 5 ms until `status/0` answers `want`, up to `limit` ms: the elapsed ms, or nil. + defp until_status(want, limit) do + t0 = System.monotonic_time(:millisecond) + + Enum.find_value(0..div(limit, 5), fn _ -> + if Semantic.status() == want, + do: System.monotonic_time(:millisecond) - t0, + else: Process.sleep(5) && nil + end) + end + + test "AC2: a changed /api/tags digest is OFF with :model_digest_changed within one interval; FTS goes on; no other space", + %{fake: fake, pinned: pinned} do + persona = Factory.persona!() + scope = AlwaysOn.persona_scope(persona.id) + row = Trinity.SpacesHelper.pin!(Ollama.space()) + + # Before the first check has answered, nothing is served. + assert Semantic.status() == {:off, :digest_unchecked} + start_supervised!(Ollama.Watch) + assert until_status(:on, 2_000), "the tier never came on after the first check" + + {:ok, _} = + Semantic.add(%{persona_id: persona.id, scope: scope, key: "k", body: "the dog is Rex"}, + by: "t" + ) + + past = Factory.session!(%{persona_id: persona.id, title: "Earlier"}) + Factory.message!(past.id, %{role: "user", content: "my sister lives in Boston"}) + spaces_before = Repo.aggregate(SpaceRow, :count) + + # A new blob behind the tag. + FakeOllama.set_digest(fake.agent, "embed-test", String.duplicate("e", 64)) + elapsed = until_status({:off, :model_digest_changed}, 5 * @interval) + assert elapsed, "still #{inspect(Semantic.status())} after #{5 * @interval} ms" + assert elapsed <= @interval + 100, "OFF after #{elapsed} ms, interval #{@interval} ms" + + # Full-text recall goes on; the vector leg stands down. + hits = Retriever.relevant(persona.id, nil, "sister Boston", touch: false) + assert [%{kind: :message, found_by: [:fts]}] = hits + + # Nothing moved: the same active space, no space registered, no job enqueued, the vector kept. + assert Spaces.active().id == row.id + assert Repo.aggregate(SpaceRow, :count) == spaces_before + assert Repo.aggregate(Oban.Job, :count) == 0 + assert Spaces.count(row.id) == 1 + assert Ollama.space() |> Trinity.Memory.Space.id() == row.id + + IO.puts( + "\nAC2: OFF #{elapsed} ms after the digest changed (interval #{@interval} ms); status #{inspect(Semantic.status())}; " <> + "hits #{inspect(Enum.map(hits, &{&1.kind, &1.found_by}))}; active #{Trinity.Memory.Space.short(Spaces.active().id)} unchanged" + ) + + # The pinned digest again (the same weights) is the same space: on again at the next check. + FakeOllama.set_digest(fake.agent, "embed-test", pinned) + assert until_status(:on, 5 * @interval) + end + + test "another runtime version is OFF with :runtime_version_changed", %{fake: fake} do + Trinity.SpacesHelper.pin!(Ollama.space()) + start_supervised!(Ollama.Watch) + assert until_status(:on, 2_000) + FakeOllama.set_version(fake.agent, "0.40.1") + assert until_status({:off, :runtime_version_changed}, 5 * @interval) + end + + test "a model the service no longer lists is OFF with :model_not_served", %{fake: fake} do + Trinity.SpacesHelper.pin!(Ollama.space()) + start_supervised!(Ollama.Watch) + assert until_status(:on, 2_000) + FakeOllama.remove_model(fake.agent, "embed-test") + assert until_status({:off, :model_not_served}, 5 * @interval) + end + + test "a service that stops answering is OFF with :endpoint_unreachable", %{fake: fake} do + Trinity.SpacesHelper.pin!(Ollama.space()) + start_supervised!(Ollama.Watch) + assert until_status(:on, 2_000) + FakeOllama.set(fake.agent, :mode, :down) + assert until_status({:off, :endpoint_unreachable}, 5 * @interval) + end + + test "check_now/0 answers the current comparison", %{fake: fake} do + start_supervised!(Ollama.Watch) + assert Ollama.Watch.check_now() == :ok + FakeOllama.set_digest(fake.agent, "embed-test", String.duplicate("e", 64)) + assert Ollama.Watch.check_now() == {:off, :model_digest_changed} + end + + test "the describe line names the reason" do + assert Semantic.describe({:off, :model_digest_changed}) =~ + "model digest is not the pinned one" + end +end diff --git a/test/trinity/memory/ollama_embedder_test.exs b/test/trinity/memory/ollama_embedder_test.exs new file mode 100644 index 0000000..e5314e2 --- /dev/null +++ b/test/trinity/memory/ollama_embedder_test.exs @@ -0,0 +1,244 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.OllamaEmbedderTest do + @moduledoc """ + Slice 134, AC1: the Tier 3 client always sends `truncate:false`, refuses an over-length input + itself with `{:error, :input_too_long}` and no vector, and catches a service that truncates + silently through `prompt_eval_count` below its own count. + + The service is `Trinity.FakeOllama` on the loopback, behaving as Ollama 0.40.0 was observed + to; the client runs its whole path (its tokenizer, Req, the JSON, the status codes). + """ + use Trinity.DataCase + + alias Trinity.{Factory, FakeOllama, Tier3Helper} + alias Trinity.Memory.{AlwaysOn, BPE, EmbedderConfig, Retriever, Semantic, Spaces} + alias Trinity.Memory.Embedders.Ollama + + @moduletag :tmp_dir + + setup %{tmp_dir: dir} do + memory = Application.get_env(:trinity, :memory, []) + Ollama.reset() + + on_exit(fn -> + Application.put_env(:trinity, :memory, memory) + Ollama.reset() + end) + + tok = Tier3Helper.write_tokenizer!(dir) + fake = FakeOllama.start!(Tier3Helper.tokenizer()) + config = Tier3Helper.memory(fake.url, tok) + Application.put_env(:trinity, :memory, Keyword.merge(memory, config)) + FakeOllama.put_model(fake.agent, "embed-test", config[:ollama][:digest]) + :ok = Ollama.check() + {:ok, fake: fake, config: config} + end + + defp count(text), do: BPE.count(Tier3Helper.tokenizer(), text) + + test "every request carries truncate:false and the pinned num_ctx; queries carry the query template", + %{fake: fake} do + assert {:ok, [d1, d2]} = Ollama.embed(["hello world", "a second fact"]) + assert {:ok, [q]} = Ollama.embed_query(["what did I say?"]) + assert length(d1) == 8 and length(d2) == 8 and length(q) == 8 + + requests = FakeOllama.requests(fake.agent, "/api/embed") + assert length(requests) == 2 + + for {"POST", _, body} <- requests do + assert body["truncate"] == false + assert body["options"] == %{"num_ctx" => 64} + assert body["model"] == "embed-test" + end + + assert [ + {_, _, %{"input" => ["hello world", "a second fact"]}}, + {_, _, %{"input" => ["Q: what did I say?"]}} + ] = + requests + end + + test "AC1: an input over Trinity's own limit is {:error, :input_too_long}, and nothing is sent", + %{fake: fake} do + long = String.duplicate("x", 100) + assert count(long) > 63 + + assert Ollama.embed([long]) == {:error, :input_too_long} + assert Ollama.embed(["short", long]) == {:error, :input_too_long} + assert FakeOllama.requests(fake.agent, "/api/embed") == [] + + # At the limit exactly, it is sent and answered. + at_limit = String.duplicate("x", 62) + assert count(at_limit) == 63 + assert {:ok, [_]} = Ollama.embed([at_limit]) + + # The refusal is about that input: the embedder stays available. + assert Ollama.availability() == :ok + end + + test "AC1 red: a service that truncates silently is caught through prompt_eval_count", + %{fake: fake} do + FakeOllama.set(fake.agent, :mode, {:truncate_silently, 4}) + text = "my sister lives in Boston" + counted = count(text) + assert counted > 4 and counted <= 63 + + # The service answers 200 with a vector; the client returns no vector. + assert Ollama.embed([text]) == {:error, :input_too_long} + assert [{_, _, %{"truncate" => false}}] = FakeOllama.requests(fake.agent, "/api/embed") + + # And the embedder is OFF with the counts, until a restart: a short input that fits the + # service's cut does not bring it back. + assert Ollama.availability() == {:off, {:service_truncated, %{counted: counted, served: 4}}} + assert Ollama.embed(["hi"]) == {:error, {:service_truncated, %{counted: counted, served: 4}}} + + IO.puts( + "\nAC1: counted #{counted}, the service reported #{4}; availability #{inspect(Ollama.availability())}" + ) + end + + test "AC1 through the store: the truncating service stores no vector, and full text still answers", + %{fake: fake} do + persona = Factory.persona!() + scope = AlwaysOn.persona_scope(persona.id) + Trinity.SpacesHelper.pin!(Ollama.space()) + assert Semantic.status() == :on + + past = Factory.session!(%{persona_id: persona.id, title: "Earlier"}) + Factory.message!(past.id, %{role: "user", content: "my sister lives in Boston"}) + + FakeOllama.set(fake.agent, :mode, {:truncate_silently, 4}) + + assert {:error, :input_too_long} = + Semantic.add( + %{ + persona_id: persona.id, + scope: scope, + key: "k", + body: "my sister lives in Boston" + }, + by: "test" + ) + + assert Semantic.entries(persona.id, [scope]) == [] + assert Spaces.count(Spaces.active().id) == 0 + assert {:off, {:service_truncated, _}} = Semantic.status() + + hits = Retriever.relevant(persona.id, nil, "sister Boston", touch: false) + assert [%{kind: :message, found_by: [:fts]}] = hits + end + + test "an over-length input refused by Trinity is not a fault of the tier", %{fake: _fake} do + Trinity.SpacesHelper.pin!(Ollama.space()) + assert Semantic.embed([String.duplicate("x", 100)]) == {:error, :input_too_long} + assert Semantic.status() == :on + end + + test "the service's own refusal on context length is :input_too_long", %{fake: fake} do + FakeOllama.set(fake.agent, :mode, {:ctx, 8}) + text = "my sister lives in Boston" + assert count(text) > 7 + assert Ollama.embed([text]) == {:error, :input_too_long} + assert Ollama.availability() == :ok + end + + test "a vector of the wrong width or length is refused", %{fake: fake} do + FakeOllama.set(fake.agent, :dim, 7) + assert {:error, {:vector_shape, 7, 8}} = Ollama.embed(["hello"]) + FakeOllama.set(fake.agent, :dim, 8) + FakeOllama.set(fake.agent, :norm, 0.9) + assert {:error, {:vector_shape, 8, 8}} = Ollama.embed(["hello"]) + end + + test "an endpoint answering 503 is :endpoint_unreachable, and the store records it", %{ + fake: fake + } do + Trinity.SpacesHelper.pin!(Ollama.space()) + FakeOllama.set(fake.agent, :mode, :down) + assert Ollama.embed(["hello"]) == {:error, :endpoint_unreachable} + assert {:error, :endpoint_unreachable} = Semantic.embed(["hello"]) + assert Semantic.status() == {:off, :endpoint_unreachable} + end + + test "a search embeds its query with the query template; a memory with the document template", + %{fake: fake} do + persona = Factory.persona!() + scope = AlwaysOn.persona_scope(persona.id) + Trinity.SpacesHelper.pin!(Ollama.space()) + + attrs = %{persona_id: persona.id, scope: scope, key: "k", body: "the dog is Rex"} + assert {:ok, _} = Semantic.add(attrs, by: "t") + + assert {:ok, _} = Semantic.search(persona.id, [scope], "dog name", 5) + + inputs = for {_, _, %{"input" => i}} <- FakeOllama.requests(fake.agent, "/api/embed"), do: i + assert ["the dog is Rex"] in inputs + assert ["Q: dog name"] in inputs + end + + describe "configuration faults (refuse a regulated boot, OFF under :default)" do + test "max_input_tokens not below num_ctx", %{config: config} do + bad = put_in(config, [:ollama, :max_input_tokens], 64) + + assert EmbedderConfig.check(:default, bad, [], nil) == + {:error, {:ollama_config, {:max_input_not_below_num_ctx, 64, 64}}} + end + + test "a pin that is not a SHA-256, and a missing pin", %{config: config} do + assert {:error, {:ollama_config, {:not_a_sha256, :digest}}} = + EmbedderConfig.check( + :default, + put_in(config, [:ollama, :digest], "latest"), + [], + nil + ) + + missing = Keyword.update!(config, :ollama, &Keyword.delete(&1, :digest)) + + assert {:error, {:ollama_config, {:option, :digest, _}}} = + EmbedderConfig.check(:default, missing, [], nil) + end + + test "the endpoint's locality must be declared, and allow-listed under :regulated", %{ + config: config + } do + undeclared = Keyword.delete(config, :locality) + + assert {:error, {:embedder_locality_undeclared, :ollama, _}} = + EmbedderConfig.check(:default, undeclared, [], nil) + + assert {:error, {:not_allow_listed, :ollama, "http://127.0.0.1"}} = + EmbedderConfig.check(:regulated, config, [], "https://elsewhere.internal") + + assert :ok = EmbedderConfig.check(:regulated, config, [], "http://127.0.0.1") + end + + test "under :default a fault is OFF with {:config, reason}", %{config: config} do + memory = Application.get_env(:trinity, :memory) + Application.put_env(:trinity, :memory, put_in(memory, [:ollama, :max_input_tokens], 99)) + + assert {:off, {:config, {:ollama_config, {:max_input_not_below_num_ctx, 99, 64}}}} = + Semantic.status() + + _ = config + end + end + + test "a tokenizer file that is not the pinned one is OFF with :tokenizer_digest_mismatch", + %{config: config, tmp_dir: dir} do + path = Path.join(dir, "other.bpe.json") + File.write!(path, File.read!(config[:ollama][:tokenizer_path]) <> " ") + memory = Application.get_env(:trinity, :memory) + Application.put_env(:trinity, :memory, put_in(memory, [:ollama, :tokenizer_path], path)) + assert Ollama.availability() == {:off, :tokenizer_digest_mismatch} + + Application.put_env( + :trinity, + :memory, + put_in(memory, [:ollama, :tokenizer_path], path <> ".absent") + ) + + assert Ollama.availability() == {:off, :tokenizer_missing} + end +end diff --git a/test/trinity/memory/ollama_space_test.exs b/test/trinity/memory/ollama_space_test.exs new file mode 100644 index 0000000..ec01d9e --- /dev/null +++ b/test/trinity/memory/ollama_space_test.exs @@ -0,0 +1,126 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.OllamaSpaceTest do + @moduledoc """ + Slice 134, AC3: `num_ctx`, the query and document prompt templates and the runtime version + are in the Tier 3 embedder's space identity, so changing any of them is a new space (133's + property, `test/property/space_identity_test.exs`, extended to this embedder). + + The population is the embedder's configuration keys that `space/0` reads, enumerated from + `Trinity.Memory.Embedders.Ollama`'s schema minus the ones that do not describe the vectors + (where the service is, how long to wait, how often to check, where the tokenizer file sits, + the thresholds), so a key added later is in the test without editing it. + """ + use ExUnit.Case, async: false + + alias Trinity.Memory.Embedders.Ollama + alias Trinity.Memory.Space + + # The keys that say where and how the service is reached, not what a vector means. + @not_identity [:base_url, :model, :tokenizer_path, :check_interval_ms, :timeout_ms, :thresholds] + + @base [ + base_url: "http://embed.internal:11434", + model: "embed-test", + model_id: "Qwen/Qwen3-Embedding-0.6B-GGUF", + digest: String.duplicate("d", 64), + weights_sha256: String.duplicate("a", 64), + tokenizer_sha256: String.duplicate("c", 64), + runtime_version: "0.40.0", + num_ctx: 8192, + max_input_tokens: 8191, + dim: 1024, + pooling: "last", + normalisation: "l2", + query_prompt: + "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:", + document_prompt: "" + ] + + @changed %{ + model_id: "Qwen/Qwen3-Embedding-4B-GGUF", + digest: String.duplicate("e", 64), + weights_sha256: String.duplicate("b", 64), + tokenizer_sha256: String.duplicate("f", 64), + runtime_version: "0.40.1", + num_ctx: 8193, + max_input_tokens: 8190, + dim: 768, + pooling: "mean", + normalisation: "none", + query_prompt: "Query: ", + document_prompt: "Document: " + } + + setup do + memory = Application.get_env(:trinity, :memory, []) + on_exit(fn -> Application.put_env(:trinity, :memory, memory) end) + :ok + end + + defp space_with(ollama, locality \\ :within_boundary) do + Application.put_env(:trinity, :memory, embedder: :ollama, locality: locality, ollama: ollama) + Ollama.space() + end + + defp identity_keys do + Ollama.options_doc() + |> then(&Regex.scan(~r/^\s*\* `:([a-z0-9_]+)`/m, &1)) + |> Enum.map(fn [_, k] -> String.to_existing_atom(k) end) + |> Kernel.--(@not_identity) + end + + test "the population: every identity key has a changed value in this test" do + keys = identity_keys() + + assert :num_ctx in keys and :query_prompt in keys and :document_prompt in keys and + :runtime_version in keys + + assert Enum.sort(keys) == Enum.sort(Map.keys(@changed)) + end + + test "AC3: changing any one identity key changes the space ID" do + base = Space.id(space_with(@base)) + + unchanged = + for key <- identity_keys(), + Space.id(space_with(Keyword.put(@base, key, Map.fetch!(@changed, key)))) == base, + do: key + + assert unchanged == [], "the space ID did not change when #{inspect(unchanged)} changed" + + IO.puts( + "\nAC3: #{length(identity_keys())} identity keys, each a new space: #{inspect(identity_keys())}" + ) + end + + test "AC3: num_ctx, both prompt templates and the runtime version each have their own field" do + s = space_with(@base) + assert s.num_ctx == 8192 + assert s.query_prompt == @base[:query_prompt] + assert s.document_prompt == "" + assert s.runtime == "ollama" and s.runtime_version == "0.40.0" + assert s.revision == @base[:digest] + assert s.weights_digest == @base[:weights_sha256] + assert s.tokenizer_digest == @base[:tokenizer_sha256] + assert s.max_input_tokens == 8191 and s.truncation == "refuse" + end + + test "the declared locality is part of the space" do + refute Space.id(space_with(@base, :within_boundary)) == Space.id(space_with(@base, :external)) + end + + test "the keys that do not describe a vector leave the space alone" do + base = Space.id(space_with(@base)) + + for {key, value} <- [ + base_url: "http://other:11434", + model: "renamed", + check_interval_ms: 1, + timeout_ms: 1 + ] do + assert Space.id(space_with(Keyword.put(@base, key, value))) == base, + "#{key} moved the space" + end + end +end diff --git a/test/trinity/memory/tier3_import_test.exs b/test/trinity/memory/tier3_import_test.exs new file mode 100644 index 0000000..fec106d --- /dev/null +++ b/test/trinity/memory/tier3_import_test.exs @@ -0,0 +1,468 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Tier3ImportTest do + @moduledoc """ + Slice 134, AC4: the import path refuses weights whose OMS or cosign signature does not verify + against the operator's key, offline, and records the resulting digest when they do. Red: one + flipped byte. + + Two groups: + + * **Everywhere**, through a stand-in verifier that passes: the import path's own checks (a + flipped byte in a blob against its descriptor, a title that is a path, a symbolic link in + the layout, the OMS statement against the GGUF, the blob the service names) and the record. + * **`:signing_tools`**, through the real `cosign` and `model_signing` (`TRINITY_COSIGN`, + `TRINITY_MODEL_SIGNING`; a failure, never a skip, when set and absent): keys made by the + test, the OMS signature made by `model_signing` itself, the layout signed in cosign's + shape (`Trinity.OCILayout`), and each signature broken by one flipped bit. + + The service is `Trinity.FakeOllama` on the loopback. Nothing reaches a network. + """ + use ExUnit.Case, async: false + + alias Trinity.{FakeOllama, OCILayout, Tier3Helper} + alias Trinity.Memory.Tier3.Import + + @moduletag :tmp_dir + + defmodule Pass do + @moduledoc false + @behaviour Trinity.Memory.Tier3.Verifier + @impl true + def cosign(_layout, _key, _opts), do: :ok + @impl true + def oms(_dir, _sig, _key, _opts), do: :ok + end + + setup %{tmp_dir: dir} do + fake = FakeOllama.start!(Tier3Helper.tokenizer()) + gguf = Tier3Helper.gguf() + {:ok, fake: fake, gguf: gguf, dir: dir} + end + + # An OMS bundle whose statement lists `files` (`[{name, bytes}]`), unsigned: for the stand-in + # verifier, which never looks at the signature. + defp unsigned_oms(files) do + resources = + for {n, b} <- files, + do: %{"name" => n, "algorithm" => "sha256", "digest" => Tier3Helper.sha256(b)} + + payload = + Jason.encode!(%{ + "_type" => "https://in-toto.io/Statement/v1", + "subject" => [%{"name" => "model", "digest" => %{"sha256" => String.duplicate("0", 64)}}], + "predicateType" => "https://model_signing/signature/v1.0", + "predicate" => %{"resources" => resources} + }) + + Jason.encode!(%{"dsseEnvelope" => %{"payload" => Base.encode64(payload), "signatures" => []}}) + end + + defp import(dir, layout, fake, extra \\ []) do + Import.run( + Keyword.merge( + [ + layout: layout, + cosign_key: Path.join(dir, "absent-cosign.pub"), + oms_key: Path.join(dir, "absent-oms.pub"), + base_url: fake.url, + model: "embed-test", + out: Path.join(dir, "out"), + staging: Path.join(dir, "staging"), + verifier: Pass + ], + extra + ) + ) + end + + defp layout!(dir, files, key \\ nil, opts \\ []) do + path = Path.join(dir, "layout-#{System.unique_integer([:positive])}") + OCILayout.build!(path, files, key, opts) + path + end + + describe "the import path's own checks (stand-in verifier)" do + test "a valid layout is imported and its digests recorded", %{ + dir: dir, + fake: fake, + gguf: gguf + } do + files = [{"m.gguf", gguf}, {"LICENSE", "Apache-2.0\n"}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + + assert {:ok, record} = import(dir, layout, fake) + + sha = Tier3Helper.sha256(gguf) + assert record["weights_sha256"] == sha + assert FakeOllama.blobs(fake.agent) == %{sha => gguf} + + assert [{"POST", "/api/create", %{"files" => %{"m.gguf" => "sha256:" <> ^sha}}}] = + FakeOllama.requests(fake.agent, "/api/create") + + assert [%{"digest" => digest}] = Req.get!(fake.url <> "/api/tags").body["models"] + + assert record["ollama"] == %{ + "base_url" => fake.url, + "model" => "embed-test:latest", + "digest" => digest, + "version" => "0.40.0" + } + + assert record["files"] == %{ + "m.gguf" => sha, + "LICENSE" => Tier3Helper.sha256("Apache-2.0\n") + } + + assert record["model_info"]["pooling"] == "last" and record["model_info"]["dim"] == 8 + + # The tokenizer is the one in the GGUF, under its own digest; the record is on disk. + assert File.read!(record["tokenizer_path"]) |> Tier3Helper.sha256() == + record["tokenizer_sha256"] + + assert File.read!(record["tokenizer_path"]) == + Trinity.Memory.BPE.to_file(Tier3Helper.tokenizer()) + + assert Jason.decode!(File.read!(record["record_path"]))["ollama"]["digest"] == digest + + # The staging directory is gone, and the configuration block names the pin. + assert File.ls!(Path.join(dir, "staging")) == [] + block = Import.config_block(record, num_ctx: 512) + assert block =~ ~s(digest: "#{digest}") + assert block =~ "max_input_tokens: 511" + end + + test "AC4 red: one flipped byte in the GGUF's blob is refused before anything reaches the service", + %{dir: dir, fake: fake, gguf: gguf} do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + OCILayout.flip_layer!(layout, "m.gguf") + + assert import(dir, layout, fake) == {:error, {:blob_digest_mismatch, "m.gguf"}} + assert FakeOllama.requests(fake.agent) == [] + end + + test "the OMS statement must list the GGUF that is imported", %{ + dir: dir, + fake: fake, + gguf: gguf + } do + other = Tier3Helper.gguf() + layout = layout!(dir, [{"m.gguf", gguf}, {"model.sig", unsigned_oms([{"m.gguf", other}])}]) + assert import(dir, layout, fake) == {:error, {:oms_statement_mismatch, "m.gguf"}} + + layout = layout!(dir, [{"m.gguf", gguf}, {"model.sig", unsigned_oms([{"x.gguf", gguf}])}]) + assert import(dir, layout, fake) == {:error, {:oms_statement_lacks, "m.gguf"}} + end + + test "titles must be plain file names, used once; one GGUF; the signature present", + %{dir: dir, fake: fake, gguf: gguf} do + sig = unsigned_oms([{"m.gguf", gguf}]) + + assert import(dir, layout!(dir, [{"../m.gguf", gguf}, {"model.sig", sig}]), fake) == + {:error, {:layer_title_refused, "../m.gguf"}} + + assert import(dir, layout!(dir, [{"a/m.gguf", gguf}, {"model.sig", sig}]), fake) == + {:error, {:layer_title_refused, "a/m.gguf"}} + + assert import( + dir, + layout!(dir, [{"m.gguf", gguf}, {"m.gguf", gguf}, {"model.sig", sig}]), + fake + ) == {:error, {:layer_title_twice, "m.gguf"}} + + assert import( + dir, + layout!(dir, [{"a.gguf", gguf}, {"b.gguf", gguf}, {"model.sig", sig}]), + fake + ) == {:error, {:gguf_count, 2}} + + assert import(dir, layout!(dir, [{"m.gguf", gguf}]), fake) == + {:error, :oms_signature_missing} + + assert FakeOllama.requests(fake.agent) == [] + end + + test "a symbolic link in the layout is refused, not followed", %{ + dir: dir, + fake: fake, + gguf: gguf + } do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + File.ln_s!("/etc/hostname", Path.join(layout, "extra")) + assert {:error, {:layout_entry_refused, :symlink, _}} = import(dir, layout, fake) + end + + test "the service must name the blob that was sent", %{dir: dir, fake: fake, gguf: gguf} do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + FakeOllama.set(fake.agent, :show_blob, String.duplicate("9", 64)) + + assert import(dir, layout, fake) == + {:error, {:served_blob_mismatch, String.duplicate("9", 64)}} + end + + test "a digest from the service that is not 64 hex characters is refused, not used as a file name", + %{dir: dir, fake: fake, gguf: gguf} do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + FakeOllama.set(fake.agent, :create_digest, "../../escape") + assert import(dir, layout, fake) == {:error, {:digest_refused, "../../escape"}} + assert Path.wildcard(Path.join(dir, "**/*.import.json")) == [] + end + + test "a verifier's refusal ends the import", %{dir: dir, fake: fake, gguf: gguf} do + defmodule RefuseCosign do + @moduledoc false + @behaviour Trinity.Memory.Tier3.Verifier + @impl true + def cosign(_l, _k, _o), do: {:error, {:cosign_refused, 1, "no"}} + @impl true + def oms(_d, _s, _k, _o), do: :ok + end + + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + + assert import(dir, layout, fake, verifier: RefuseCosign) == + {:error, {:cosign_refused, 1, "no"}} + + assert FakeOllama.requests(fake.agent) == [] + end + + test "the programs missing is a refusal, never a pass", %{dir: dir, fake: fake, gguf: gguf} do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + + assert import(dir, layout, fake, + verifier: Trinity.Memory.Tier3.Verifier.CLI, + verifier_opts: [cosign: Path.join(dir, "no-cosign")] + ) == {:error, {:program_missing, Path.join(dir, "no-cosign")}} + end + end + + describe "the CLI verifier and the mix task, with stand-in programs" do + # Programs that only exit: the wrapper's contract is the exit status, and these hold it to + # that on every leg. The real programs are the :signing_tools cases below. + defp program!(dir, name, status, out) do + path = Path.join(dir, name) + File.write!(path, "#!/bin/sh\necho '#{out}'\nexit #{status}\n") + File.chmod!(path, 0o755) + path + end + + test "exit 0 from both is a pass; the task prints the record and the configuration block", + %{dir: dir, fake: fake, gguf: gguf} do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + + argv = [ + "--layout", + layout, + "--cosign-key", + "k.pub", + "--oms-key", + "o.pub", + "--base-url", + fake.url, + "--model", + "embed-test", + "--out", + Path.join(dir, "out"), + "--cosign", + program!(dir, "cosign-ok", 0, "verified"), + "--model-signing", + program!(dir, "oms-ok", 0, "Verification succeeded"), + "--num-ctx", + "512", + "--query-prompt", + "Q: " + ] + + Mix.shell(Mix.Shell.Process) + on_exit(fn -> Mix.shell(Mix.Shell.IO) end) + Mix.Tasks.Trinity.Tier3.Import.run(argv) + assert_received {:mix_shell, :info, [out]} + assert out =~ "imported embed-test:latest: digest " + assert out =~ "max_input_tokens: 511" + assert out =~ ~s(query_prompt: "Q: ") + end + + test "a non-zero exit is a refusal carrying the program's last lines", %{ + dir: dir, + fake: fake, + gguf: gguf + } do + files = [{"m.gguf", gguf}] + layout = layout!(dir, files ++ [{"model.sig", unsigned_oms(files)}]) + bad = program!(dir, "cosign-bad", 1, "Error: no matching attestations") + + assert import(dir, layout, fake, + verifier: Trinity.Memory.Tier3.Verifier.CLI, + verifier_opts: [cosign: bad, model_signing: program!(dir, "oms-ok", 0, "ok")] + ) == {:error, {:cosign_refused, 1, "Error: no matching attestations"}} + + assert import(dir, layout, fake, + verifier: Trinity.Memory.Tier3.Verifier.CLI, + verifier_opts: [ + cosign: program!(dir, "cosign-ok", 0, "ok"), + model_signing: program!(dir, "oms-bad", 1, "Verification failed") + ] + ) == {:error, {:oms_refused, 1, "Verification failed"}} + + assert FakeOllama.requests(fake.agent) == [] + end + + test "the task refuses unknown options, a missing option, and a refusal, by raising", %{ + dir: dir + } do + assert_raise Mix.Error, ~r/unknown options/, fn -> + Mix.Tasks.Trinity.Tier3.Import.run(["--nope", "x"]) + end + + assert_raise Mix.Error, ~r/refused: \{:option_missing, :layout\}/, fn -> + Mix.Tasks.Trinity.Tier3.Import.run(["--out", dir]) + end + end + end + + describe "the real verifiers (:signing_tools)" do + @describetag :signing_tools + + setup %{dir: dir} do + tools = [ + cosign: System.fetch_env!("TRINITY_COSIGN"), + model_signing: System.fetch_env!("TRINITY_MODEL_SIGNING") + ] + + for {_, p} <- tools, do: assert(File.regular?(p), "#{p} is named and absent") + + cosign = OCILayout.key!() + oms = OCILayout.key!() + {cosign_pub, _} = OCILayout.write_key!(cosign, dir, "cosign") + {oms_pub, oms_priv} = OCILayout.write_key!(oms, dir, "oms") + + {:ok, + tools: tools, cosign: cosign, cosign_pub: cosign_pub, oms_pub: oms_pub, oms_priv: oms_priv} + end + + # The model directory signed by model_signing itself: `[{name, bytes}]` plus model.sig. + defp oms_signed!(ctx, files) do + model = Path.join(ctx.dir, "model-#{System.unique_integer([:positive])}") + File.mkdir_p!(model) + for {n, b} <- files, do: File.write!(Path.join(model, n), b) + sig = model <> ".sig" + + {out, status} = + System.cmd( + ctx.tools[:model_signing], + ["sign", "key", "--private_key", ctx.oms_priv, "--signature", sig, model], + stderr_to_stdout: true + ) + + assert status == 0, out + files ++ [{"model.sig", File.read!(sig)}] + end + + defp real(ctx, layout, extra \\ []) do + import( + ctx.dir, + layout, + ctx.fake, + Keyword.merge( + [ + cosign_key: ctx.cosign_pub, + oms_key: ctx.oms_pub, + verifier: Trinity.Memory.Tier3.Verifier.CLI, + verifier_opts: ctx.tools + ], + extra + ) + ) + end + + test "AC4: signed by both keys, verified offline, imported, and the digest recorded", ctx do + layout = layout!(ctx.dir, oms_signed!(ctx, [{"m.gguf", ctx.gguf}]), ctx.cosign) + assert {:ok, record} = real(ctx, layout) + assert record["weights_sha256"] == Tier3Helper.sha256(ctx.gguf) + assert record["cosign_key_sha256"] == Tier3Helper.sha256(File.read!(ctx.cosign_pub)) + assert record["oms_predicate_type"] == "https://model_signing/signature/v1.0" + assert record["ollama"]["digest"] =~ ~r/\A[0-9a-f]{64}\z/ + + IO.puts( + "\nAC4: imported #{record["ollama"]["model"]}, digest #{record["ollama"]["digest"]}" + ) + end + + test "AC4 red: one flipped bit in the cosign signature is refused by cosign", ctx do + layout = + layout!(ctx.dir, oms_signed!(ctx, [{"m.gguf", ctx.gguf}]), ctx.cosign, + flip_signature: true + ) + + assert {:error, {:cosign_refused, 1, out}} = real(ctx, layout) + assert out =~ "accepted signatures do not match threshold" + assert FakeOllama.requests(ctx.fake.agent) == [] + IO.puts("\nAC4 red (cosign signature): #{out |> String.split("\n") |> List.last()}") + end + + test "AC4 red: one flipped byte in the weights, repackaged and cosign-signed again, is refused by OMS", + ctx do + [{"m.gguf", _}, {"model.sig", sig}] = oms_signed!(ctx, [{"m.gguf", ctx.gguf}]) + tampered = OCILayout.flip(ctx.gguf, byte_size(ctx.gguf) - 10) + layout = layout!(ctx.dir, [{"m.gguf", tampered}, {"model.sig", sig}], ctx.cosign) + assert {:error, {:oms_refused, 1, out}} = real(ctx, layout) + assert out =~ "m.gguf" + assert FakeOllama.requests(ctx.fake.agent) == [] + + IO.puts( + "\nAC4 red (weights, cosign valid): #{out |> String.split("\n") |> List.last() |> String.slice(0, 120)}" + ) + end + + test "AC4 red: one flipped byte in the OMS signature, repackaged and cosign-signed again, is refused by OMS", + ctx do + [{"m.gguf", g}, {"model.sig", sig}] = oms_signed!(ctx, [{"m.gguf", ctx.gguf}]) + bundle = Jason.decode!(sig) + raw = bundle["dsseEnvelope"]["signatures"] |> hd() |> Map.fetch!("sig") |> Base.decode64!() + + flipped = + put_in(bundle, ["dsseEnvelope", "signatures"], [ + %{"sig" => Base.encode64(OCILayout.flip(raw, 10))} + ]) + + layout = + layout!(ctx.dir, [{"m.gguf", g}, {"model.sig", Jason.encode!(flipped)}], ctx.cosign) + + assert {:error, {:oms_refused, 1, _}} = real(ctx, layout) + assert FakeOllama.requests(ctx.fake.agent) == [] + end + + test "another operator's keys are refused, each by its own verifier", ctx do + layout = layout!(ctx.dir, oms_signed!(ctx, [{"m.gguf", ctx.gguf}]), ctx.cosign) + {other_pub, _} = OCILayout.write_key!(OCILayout.key!(), ctx.dir, "other") + assert {:error, {:cosign_refused, 1, _}} = real(ctx, layout, cosign_key: other_pub) + assert {:error, {:oms_refused, 1, _}} = real(ctx, layout, oms_key: other_pub) + end + + test "a valid cosign signature over another artifact does not vouch for this one", ctx do + files = oms_signed!(ctx, [{"m.gguf", ctx.gguf}]) + other = layout!(ctx.dir, [{"x.gguf", "other"}], nil) + + other_digest = + Jason.decode!(File.read!(Path.join(other, "index.json")))["manifests"] + |> hd() + |> Map.fetch!("digest") + + layout = layout!(ctx.dir, files, ctx.cosign, sign_digest: other_digest) + assert {:error, {:cosign_refused, 1, _}} = real(ctx, layout) + end + + test "an unsigned layout is refused by cosign", ctx do + layout = layout!(ctx.dir, oms_signed!(ctx, [{"m.gguf", ctx.gguf}]), nil) + assert {:error, {:cosign_refused, 1, out}} = real(ctx, layout) + assert out =~ "no signatures" + end + end +end