diff --git a/.sobelow-skips b/.sobelow-skips index 1e54748..38ecd26 100644 --- a/.sobelow-skips +++ b/.sobelow-skips @@ -1,5 +1,6 @@ Config.CSP: Missing Content-Security-Policy,lib/trinity_web/router.ex:14,1DC4768 Traversal.FileModule: Directory Traversal in `File.mkdir_p`,lib/trinity/data_dir/lock.ex:60,4ECF02C Traversal.FileModule: Directory Traversal in `File.read`,lib/trinity/data_dir/lock.ex:146,5EFB666 +Traversal.FileModule: Directory Traversal in `File.read`,lib/trinity/sandbox/skill.ex:75,30481DE Traversal.FileModule: Directory Traversal in `File.rm`,lib/trinity/data_dir/lock.ex:85,7D2B826 Traversal.FileModule: Directory Traversal in `File.write`,lib/trinity/data_dir/lock.ex:135,42D2817 diff --git a/.sobelow-skips.reasons b/.sobelow-skips.reasons index 3e18106..972e0d2 100644 --- a/.sobelow-skips.reasons +++ b/.sobelow-skips.reasons @@ -14,3 +14,4 @@ 5EFB666 Rekeyed at slice 013 (the moduledoc grew three lines for the macOS and Windows liveness check). Slice 010, Trinity.DataDir.Lock read/1: reads /LOCK, a path this module composes from the data directory and a constant file name; no caller supplies it. 7D2B826 Rekeyed at slice 013 (the moduledoc grew three lines for the macOS and Windows liveness check). Slice 010, Trinity.DataDir.Lock.release/2: removes /LOCK only when the file carries this process's own token; same composed path as the read. 42D2817 Rekeyed at slice 013 (the moduledoc grew three lines for the macOS and Windows liveness check). Slice 010, Trinity.DataDir.Lock take_over/2: rewrites /LOCK after the recorded holder's pid is found dead; same composed path, same constant name. +30481DE Slice 110, the skill script runner: the path sobelow cannot trace is the return of `Trinity.Sandbox.Skill.script_path/2`, which resolves it with `Path.expand/2` and refuses anything landing outside the skill's own `scripts/` directory. sobelow sees a function parameter and stops there; it cannot follow the containment into the previous call. The check is public precisely so it can be tested rather than trusted, and `test/trinity/sandbox/skill_test.exs` asserts the traversal case, the absolute-path case and the pathless-skill case against the real function rather than a copy of it. This is the only finding sobelow reports in the tree. diff --git a/CHANGELOG.md b/CHANGELOG.md index e34da9b..4e27afb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,41 @@ evidence for each increment is retained by the maintainers and summarised here. ## 2026-09-24 +### `slice/110` — Executable skills in an in-process sandbox +Skills that can execute need a real programming surface. Trinity now has one that costs no trust: a +`run_lua` tool and skill-owned scripts run Lua inside this process, with no operating system +process, no filesystem handle and no socket. A model asked to total a column produces a number that +looks right; a script produces one that is right and leaves the script in the transcript. + +The limits are the virtual machine's rather than a promise. The runner sets its own heap ceiling and +the machine kills it at the boundary; a wall clock bounds the run; the pool refuses an over-limit run +by name instead of queueing it invisibly, because a run that waits an unbounded time to start has no +wall-clock bound at all. + +**One limit is reported rather than claimed, and the measurement is published.** The library offers a +reduction cap enforced by polling every hundred milliseconds, so a cap of a thousand and a cap of a +million both stop a tight loop at twenty to thirty-four million reductions. Reductions are therefore +a statistic here and the bound that stops a runaway script is time. `docs/sandbox.md` carries the +numbers. + +What a script wants *done* it asks for through the same executor a model's own tool call uses, so it +is decided by the same permission gate and leaves the same receipts. There is no second door, and a +test asserts a denied call gives the script nothing. A call needing a person fails immediately rather +than holding the sandbox open, because a run is bounded in seconds and a person is not: letting a +script block would turn every approval into a timeout. + +A skill may ship a script of its own. Where that script may read is checked by resolving the path +rather than by trusting it, so a traversal is caught by where it lands rather than by what it looks +like. The check is public so the test exercises the real function rather than a copy of it. + +The skill scanner gained a rule for scripts reaching at what the sandbox removes. It is a reviewer's +signal and the changelog says so: the control is that those globals are absent at run time, and a +regular expression over source text would be a poor last line. + +`docs/sandbox.md` states what this does not protect against as plainly as what it does, including +that the sandbox constrains the script and the permission gate constrains the effect, and that +reading the first as a guarantee about the second is the mistake the page exists to prevent. + ### `slice/014` — The prompt window is the recent conversation A session that held more than five hundred messages was building every subsequent request from its **first** five hundred. The query that fetched the window ordered oldest-first and then took a diff --git a/README.md b/README.md index 75c342d..a991c85 100644 --- a/README.md +++ b/README.md @@ -37,12 +37,13 @@ desktop application. Apache-2.0, developed in the open from the first commit. ## Status -Pre-alpha, and usable from source. Milestones M0 to M4 are approved, **M5a Automates** with them -(050, the scheduler; 059, the MCP measurement; 060, the MCP client; 061, the MCP server; 062, MCP -authorization), **M5b Reaches** in part (070, the gateway core; 080, subagents) and the first slice -of **M6 Ships** (090, observability and the cost ledger), with 002 (the supply chain slice) beside -them: 29 slices, each merged with a merge commit and tagged `slice/NNN` -(`git tag -l 'slice/*' | wc -l` → 30, on 2026-09-24: one more tag than slices, because +Pre-alpha, and usable from source. Milestones **M0** to **M4** are approved, **M5a Automates** with +them (050, the scheduler; 059, the MCP measurement; 060, the MCP client; 061, the MCP server; 062, +MCP authorization), **M5b Reaches** in part (070, the gateway core; 080, subagents), the first slice +of **M6 Ships** (090, observability and the cost ledger), **M7 Sandboxed** (110, executable skills in +an in-process sandbox), and the first slice of **M9 Donatable** (120, the open-source hygiene audit), +with 002 (the supply chain slice) beside them: 36 slices, each merged with a merge commit and tagged +`slice/NNN` (`git tag -l 'slice/*' | wc -l` → 37, on 2026-09-24: one more tag than slices, because `slice/090` was pushed at the wrong commit before its merge landed and this repository's ruleset forbids moving or deleting a tag; the correct one is `slice/090.1`, and `git tag -n20 slice/090.1` says so in its own annotation rather than leaving a reader to work it out). What that means in @@ -134,6 +135,16 @@ enforces it. by model, by session, by persona, against budgets that warn and, when you ask them to, refuse. `/activity` shows the live stream with the spend beside it; a LiveDashboard page lists the running sessions, asking each process its state rather than trusting a row. +- **Computes, rather than estimates.** A `run_lua` tool runs a Lua script inside this process: no + operating system process, no filesystem handle, no socket. A model asked to total a column + produces a number that looks right; a script produces one that is right and leaves the script in + the transcript for anyone to check. The limits are the virtual machine's rather than a promise: + the runner sets its own heap ceiling and is killed at it, a wall clock bounds the run, and the + pool refuses an over-limit run by name instead of queueing it invisibly. What a script wants + *done* it asks for through `trinity.tool`, which is decided by the same permission gate and leaves + the same receipts as any other call, so there is no second door. A skill can ship a script of its + own, and where that script may read is checked by resolving the path rather than by trusting it. + `docs/sandbox.md` states what this does **not** protect against as plainly as what it does. - **Runs on a schedule.** Tasks on the `/tasks` page: a prompt, a persona, the skills to hint, and when (a cron expression, a one-shot time, or a phrase like "every weekday at 9am" the model turns into cron). Each run is a fresh conversation you can open, its result waits on @@ -144,8 +155,9 @@ enforces it. Not there yet: the messaging platforms themselves (071 Telegram and 072 Discord, which need a bot token the project does not yet hold), the native desktop shell and signed releases (the rest of M6), -executable skills in a sandbox (M7). The [Milestones](#milestones) section below sets out the order -the remaining work is being built in. +store-and-forward receipts for disconnected operation (026, waiting on an answer from the external +authority plane). The [Milestones](#milestones) section below sets out the order the remaining work +is being built in. The interface is a local web page; the desktop shell exists as a packaging spike, not a product. If you want to follow along, watch the roadmap and the tags. @@ -244,7 +256,8 @@ rather than described by it. | Auditability | Decisions and effects are recorded in an Ed25519-signed hash chain with checkpoints and a verifier, so an operator can reconstruct what was done, by whom and under what decision. | | Untrusted content | Anything arriving from outside the machine is marked untrusted at the boundary and is never treated as instruction. | | Approved cryptography | A dedicated CI leg builds from source and runs the cryptographic properties inside a FIPS-mode container, so statements about approved algorithms are measured on that leg rather than asserted. | -| Per-file licensing | Every one of the 609 files in this repository carries copyright and licence information, following the REUSE Specification 3.3. `reuse lint` exits 0 and runs in CI. What is not this project's says so: the desktop shell's icons are Tauri's template icons and are attributed to Tauri, not to us. | +| Exercised by generated input | The canonicalisers, the argument validator and the receipt chain are tested with property-based testing as well as by example, because code that answers "is this the same thing" fails on the spellings nobody thought of. The suite carries a planted defect it catches and the example tests miss, so a property that has never failed is not mistaken for one that cannot. | +| Per-file licensing | Every one of the 630 files in this repository carries copyright and licence information, following the REUSE Specification 3.3. `reuse lint` exits 0 and runs in CI. What is not this project's says so: the desktop shell's icons are Tauri's template icons and are attributed to Tauri, not to us. | | Supply chain | Dependency and licence audits run on every commit; dependency versions are pinned in `VERSIONS.md` and verified against the lock file by the gate. A CycloneDX bill of materials is generated by the gate on every commit and ships beside every packaged binary, so whoever holds the artifact holds the list of what is in it; the bill states its own coverage inside the document, including what it does not cover. | | Provenance | Every commit carries a Developer Certificate of Origin sign-off, enforced by a hook and independently by CI. Every packaged binary carries build provenance attested through the workflow's own identity and recorded in a public transparency log, verified in the same run that produced it, so a holder can check where the bytes were built without trusting this page. | | Independent self-certification | The project holds the [OpenSSF Best Practices **silver** badge](https://www.bestpractices.dev/projects/14772) (awarded 2026-09-23), assessed against the Open Source Security Foundation's published criteria at two tiers. Every required criterion at both is met. What is not met is stated rather than stretched: at passing, three suggested criteria (semantic versioning, which begins at the first supported release, and two concerning dynamic analysis tooling the project does not run); at silver, a bus factor of two and an accessibility assessment, both suggested-tier, and signed version tags. | @@ -266,7 +279,7 @@ rather than described by it. | M7 Sandboxed | Executable skills in an in-VM sandbox | 110 approved | | M9 Donatable | Open-source hygiene audited, supply chain signed, shared libraries extracted | 002 and 120 to 123 approved | -M0 to M5a are approved as of 2026-09-22. Of M5b, 070 and 080 are approved as of 2026-09-23; 071 and 072 wait on a messaging bot token the project does not yet hold. Of M6, 090 is approved as of 2026-09-24. Slice numbers have gaps on purpose (000, 001, 010, 011 and so on) so that a slice can be inserted +M0 to M5a are approved as of 2026-09-22. Of M5b, 070 and 080 are approved as of 2026-09-23; 071 and 072 wait on a messaging bot token the project does not yet hold. Of M6, 090 is approved as of 2026-09-24. **M7** is approved as of 2026-09-24 (110). Of M9, 120 is approved as of 2026-09-24. Four slices merged on 2026-09-24 were not on the plan and are recorded as what they were: 004 (property-based testing, taken because it closes two OpenSSF gold criteria), 014 (a defect: a session past 500 messages built its prompt from the *first* 500), and 027, 028 and 029 from the connectome research. Slice numbers have gaps on purpose (000, 001, 010, 011 and so on) so that a slice can be inserted later without renumbering anything. **[`ROADMAP.md`](ROADMAP.md)** sets out what the project intends to do and what it intends not to @@ -283,6 +296,7 @@ do over the next year, and why the order is what it is. | `THIRD_PARTY_LICENSES.md` | Every dependency and its licence, generated from the CycloneDX bill. A dependency whose licence neither the bill nor a declared entry supplies fails the gate | | `LICENSES/` | The full text of every licence the tree uses, as the REUSE Specification requires | | `docs/effects-catalog.md` | The published effect catalogue: every tool that can cause an effect, its risk tier and effect class, with a version that is a hash of its own rows. Generated, and the gate fails when it and the tree disagree | +| `docs/sandbox.md` | What the Lua sandbox protects against, what it does not, and which of its limits are enforced by the virtual machine rather than sampled | | `docs/telemetry.md` | The telemetry event catalogue: every event this tree emits, its measurements and metadata, and the rule about what never appears in one | | `docs/adr/` | Architecture decision records. One is added whenever a decision changes | | `docs/10-assurance-case.md` | The structured argument that the security claims hold, with the evidence for each and the assumptions and limits named | diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md index b34aedc..ee4fb7c 100644 --- a/THIRD_PARTY_LICENSES.md +++ b/THIRD_PARTY_LICENSES.md @@ -4,7 +4,7 @@ hand: the gate regenerates it and fails when this file and the bill disagree. --> # Third-party licences -133 components, derived from `sbom.cdx.json`, the CycloneDX bill this project's +134 components, derived from `sbom.cdx.json`, the CycloneDX bill this project's quality gate generates on every commit. This file and that bill are the same data rendered twice, so they cannot disagree. @@ -75,6 +75,7 @@ it, and a reader should not infer from an entry here that the component is in a | `lazy_html` | `0.1.12` | Apache-2.0 | bom | | `llm_db` | `2026.9.4` | Apache-2.0 | bom | | `logger` | `1.20.4` | Apache-2.0 | bom | +| `luerl` | `1.5.1` | Apache-2.0 | bom | | `makeup` | `1.2.2` | BSD-2-Clause | bom | | `makeup_elixir` | `1.0.1` | BSD-2-Clause | bom | | `makeup_erlang` | `1.1.0` | BSD-2-Clause | bom | diff --git a/VERSIONS.md b/VERSIONS.md index 5af22d2..4cf769e 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -90,6 +90,7 @@ never pin a version hex marks as retired or vulnerable. | `pgvector` | ~> 0.4.1 | ✅ in `mix.lock` | Vectors on the Postgres path: `memories.embedding_vector vector(384)` under an HNSW cosine index, searched by `Trinity.Memory.VectorStores.Pgvector` (Slice 032). The postgres job runs on the `pgvector/pgvector:pg17` image. Split from the postgrex row at Slice 010. | | `oban` | ~> 2.24 | ✅ in `mix.lock` | Uses `Oban.Engines.Lite` on SQLite and `Oban.Engines.Basic` on Postgres (the adapter is compile time, ADR-0002, so the engine is too). ⚠️ Oban Pro Workflows/Smart engine are Postgres-only. Added at Slice 050 (2.24.1 measured on hex.pm 2026-09-22, Apache-2.0). | | `sbom` | ~> 0.11 | ✅ in `mix.lock` | CycloneDX software bill of materials, generated by the gate from mix.lock (`mix sbom.cyclonedx`). Maintained by erlef, the Erlang Ecosystem Foundation; BSD-3-Clause, CC0-1.0, MIT and Apache-2.0; 0.11.0 measured on hex.pm 2026-09-23, released 2026-09-22. Dev and test only: it ships in no artifact, and the gate validates its output, so a defect in it fails the build rather than producing a quietly wrong bill. Added at Slice 002. | +| `luerl` | ~> 1.5 | ✅ in `mix.lock` | Lua 5.3 implemented in Erlang, for the in-VM sandbox: a skill's script runs inside this node with no OS process, no file handle and no socket. Apache-2.0, **no dependencies of its own**; 1.5.1 measured on hex.pm 2026-09-24, released 2025-12-03. `:luerl_sandbox` is a module of this package and not a package of its own, which slice 110's SLICE.md assumed wrongly. Its `max_reductions` is polled with a 100 ms sleep between checks, so it cannot bound a tight loop below tens of millions of reductions; Trinity runs the interpreter under its own process with `max_heap_size` and a timeout instead, and reports reductions as a statistic (slice 110 NOTES F1). Added at Slice 110. | | `stream_data` | ~> 1.4 | ✅ in `mix.lock` | Property-based testing: generated input for `test/property`. Apache-2.0; 1.4.0 measured on hex.pm 2026-09-24, released 2026-07-14. Test only, so it ships in no artifact. Elixir has no focused fuzzing library and property-based testing is what the OpenSSF gold criterion `dynamic_analysis` accepts in its place, which is why this is a dependency rather than a habit. Added at Slice 004. | | `jose` | ~> 1.11 | ✅ in `mix.lock` | JOSE (JWK, JWS, JWKS) over OTP's crypto, for the MCP resource server's token validation and the personal profile's embedded authorization server. MIT; 1.11.12 measured on hex.pm 2026-09-22. Not joken, a layer over it this tree does not need. Added at Slice 062. | | `oban_web` | ~> 2.13 | ✅ in `mix.lock` | The Oban dashboard, a LiveView mounted at /oban (dev, and behind `config :trinity, :oban_web, true` elsewhere). Apache-2.0 on hex.pm since the 2.12 line (2.13.0 measured 2026-09-22); the plan of 2026-09-05 wrote it in when it was commercial. Added at Slice 050. | diff --git a/config/config.exs b/config/config.exs index afd97d4..f35b53e 100644 --- a/config/config.exs +++ b/config/config.exs @@ -86,7 +86,9 @@ config :trinity, :tools, Trinity.Tools.Memory, Trinity.Tools.Shell.Run, # Slice 080: delegation to a child session. - Trinity.Tools.Delegate + Trinity.Tools.Delegate, + # Slice 110: a Lua script in the in-VM sandbox. + Trinity.Sandbox.Tool ], toolsets: %{ fs: ["fs_read", "fs_write", "fs_edit", "fs_list", "fs_glob", "fs_grep"], diff --git a/config/test.exs b/config/test.exs index 30af5ae..bd05d9f 100644 --- a/config/test.exs +++ b/config/test.exs @@ -40,7 +40,9 @@ config :trinity, :tools, Trinity.Tools.Memory, Trinity.Tools.Shell.Run, # Slice 080: delegation to a child session. - Trinity.Tools.Delegate + Trinity.Tools.Delegate, + # Slice 110: a Lua script in the in-VM sandbox. + Trinity.Sandbox.Tool ], toolsets: %{ core: ["echo", "sleep", "crash", "big", "write_note"], @@ -79,9 +81,25 @@ config :trinity, :llm, model: "chat", caps: [:stream, :tools, :json], price: %{input: 1.0, output: 2.0}, - # Slice 023: a small window so a long fake conversation crosses the thresholds (the fourteen - # tools' schemas alone are about 1,140 estimated tokens of every request). - context_tokens: 6_000 + # A small window so a long fake conversation crosses the thresholds. The number is a balance + # and both sides of it are measured, because the first version of this comment was written + # when it was not. + # + # **Below** it: the tool definitions are in every request and cannot be compacted away. That + # comment said "the fourteen tools' schemas alone are about 1,140 estimated tokens"; there are + # now 24 tools at 3,516 tokens, and at a 6,000 window (soft 4,200) the baseline request had + # reached 4,228. Every session was therefore over the soft threshold before anyone said + # anything, so an ordinary two-turn test spent a scripted model response on a compaction and + # failed somewhere unrelated. Registering one tool at slice 110 is what crossed it. + # + # **Above** it: `Trinity.Memory.CompactionTest` needs a 200-turn conversation to exceed the + # soft threshold, and its naive prompt measures 9,298 tokens, so the soft threshold has to + # stay below that. + # + # 10,000 puts soft at 7,000: the tool surface is half of it rather than 84%, there is room for + # a conversation, and 9,298 still crosses it. Re-measure both numbers before changing this, + # rather than adjusting it until the suite goes green. + context_tokens: 10_000 }, %{ id: "fake:embed", diff --git a/coverage.tsv b/coverage.tsv index 75ae532..c228fc7 100644 --- a/coverage.tsv +++ b/coverage.tsv @@ -32,3 +32,4 @@ slice_id percent sha date 120 80.31 78ee950 2026-09-24 004 80.31 91de943 2026-09-24 014 80.31 1a5d480 2026-09-24 +110 80.31 90c3d0a 2026-09-24 diff --git a/docs/06-risk-register.md b/docs/06-risk-register.md index b0d03f9..f930d10 100644 --- a/docs/06-risk-register.md +++ b/docs/06-risk-register.md @@ -30,4 +30,5 @@ | R25 | `glib` 0.18.5 carries GHSA-wrw7-89jp-8q8g (medium, unsoundness in `Iterator` and `DoubleEndedIterator` impls); first patched in 0.20.0 | Low | Med | The advisory is open against `src-tauri/Cargo.lock`; measured 2026-09-23 | It is transitive through the GTK 0.18 stack (`gtk` 0.18.2, `webkit2gtk` 2.0.2) that Tauri 2.11.5 requires on Linux, so it is fixed by a Tauri release built on gtk-rs 0.20 and not by a direct bump. **Scope:** the desktop shell only. The `headless` release is `steps: [:assemble]` with no Burrito and no `src-tauri`, so a server deployment does not ship this dependency. Owner: the maintainer. Lift condition: a Tauri release whose Linux stack resolves `glib` >= 0.20, adopted by the desktop shell work | 100 | | R26 | An intermittent `Exqlite.Error: Database busy` fails the test suite on loaded CI runners, in a different test each time, always on a write to the main repository | Med | Low (test-only; no production path) | Failed the gate on pull requests 67, 70 and again on the commit that attempted to fix it; the same commit passed on its other run, so it is seed- and load-dependent. Not once reproduced locally across repeated runs | Ruled out: `busy_timeout`, raised from the shipped 5 s to 30 s in the suite, did not stop it. SQLite returns BUSY immediately, without the busy handler, when a read transaction upgrades to a write while another connection holds the write lock; no timeout affects that path. The suite runs two connections (one for the sandbox owner, one for Oban's boot check) and deliberately queues fifty concurrent writers at slice 070 AC7, which is the load that provokes it. Candidate fixes, not yet taken: drain the stress test's sessions before the suite continues; give that test its own tag and CI step; or remove the second connection's need. Owner: the maintainer. Lift condition: ten consecutive green CI runs of the full suite with the stress test in the default run | 070 | | R26 | Occurrence recorded 2026-09-23: the same class of failure appeared on the **FIPS leg** rather than the ordinary gate, which the row above does not cover. `Trinity.Gateways.ApprovalsTest` AC4 failed in its setup, with the crash in `Trinity.Receipts.ChainWriter.terminate/2` calling `insert_checkpoint/2` after the sandbox owner had gone: an ownership error at termination rather than a `Database busy` on a write. **Re-running the identical commit passed all four checks**, which is the evidence that matters: the code did not change between the red run and the green one, so the failure is in the environment or the ordering and not in the change under test. What this adds to the row above: the mechanism is not only write contention, it also reaches process termination racing sandbox teardown, and it is not confined to the ordinary gate's runners. A candidate fix specific to this shape: `ChainWriter.terminate/2` writes a checkpoint on the way out, and a terminate callback that touches the database after the test that owns the connection has finished is a race by construction; it should skip the write when the repo is in sandbox mode and has no owner, rather than attempting it and crashing. **Second occurrence the same day, and it changes the severity.** The very next pull request, which changed *only this file*, failed the FIPS leg again: `Trinity.Gateways.RouterTest` AC6, `Exqlite.Error: Database busy` on an `INSERT INTO gateway_identities` - the row's original signature, on the leg the row did not cover. A change that cannot affect behaviour failing a required check twice in a row means this is no longer an intermittent annoyance: **it blocks work**, and the honest reading is that the FIPS leg provokes it more reliably than the ordinary gate does, most likely because that leg is slower (163 s against about 110 s locally) and every contention window is correspondingly wider. The row above stands as written | **High** (was Med: it now blocks merges) | Low (test-only; no production path) | The lift condition in the row above, still unmet; two failures on the FIPS leg on 2026-09-23, one of them on a documentation-only change | docs/06-risk-register.md R26 | 070 | +| R26 | Occurrence recorded 2026-09-24, and this one **is** R26 rather than something wearing its signature. `Trinity.Telemetry.CostsTest`, "an unconfigured budget is not a budget of zero", failed with `** (Exqlite.Error) Database busy` on the ordinary `gate` leg of pull request 96, while the sibling run of the identical commit passed. The test is untouched by that change, the error is the row's original wording, and the failure is on a write: three of the three things the row names. Recorded separately from the counter-example above so that the register shows both kinds, because the counter-example's whole point is that one cannot be assumed from the signature and this one was checked before being filed. **What it adds:** the row above says it blocks work, and it blocked a merge again here, costing a full CI cycle on a branch whose own tests were green. The three candidate fixes in the original row remain untaken, which is now a decision rather than an oversight: it has cost four cycles across 2026-09-23 and 2026-09-24. Recommend taking it as a slice of its own rather than continuing to re-run. The rows above stand as written | **High**, unchanged | Low (test-only; no production path) | The lift condition in the original row, still unmet; a fourth blocking occurrence | docs/06-risk-register.md R26 | 070 | | R26 | Counter-example recorded 2026-09-24, in slice 090, and it corrects a heuristic the rows above state. At commit `b1bca1f` the `pull_request` run passed all four checks and the `push` run of the **same commit** failed `postgres` (runs 36001064067 and 36001057879). By the reading the 2026-09-23 entry offers - the code did not change between the green run and the red one, so the fault is the environment - this was R26 again. **It was not.** It was a real defect in slice 080's code, on a genuinely racy path: every call into a child session is a `gen_statem.call`, and a call into a process that dies exits the caller, so killing a child killed whoever called `delegate/3`. Whether the kill landed before or after the call returned decided the outcome, which is why the same commit could go either way. Fixed at `1f33f12`, after which both runs of the same commit passed. **What this changes:** "the same commit passed on its other run" is evidence that the failure is *timing-dependent*, which is not the same as evidence that it is environmental. A real race in library code produces exactly the signature R26 is recognised by, so the signature alone cannot be the diagnosis - the stack trace has to be read before a failure is filed here. This slice nearly filed one. The rows above stand as written | none (not an occurrence) | none | none | none | 080, 090 | diff --git a/docs/effects-catalog.md b/docs/effects-catalog.md index 5ae0049..03d63c0 100644 --- a/docs/effects-catalog.md +++ b/docs/effects-catalog.md @@ -4,7 +4,7 @@ regenerates it and fails when this file and the tree disagree. --> # The effect catalogue -Version `6de39824f78b4380`, which is a SHA-256 over the rows below rather than a number anyone +Version `372070cf9a41c640`, which is a SHA-256 over the rows below rather than a number anyone typed. Two trees with the same effect surface carry the same version. Built from `MIX_ENV=prod`, which `--write` and `--check` require to be `prod`, because this is a statement about what ships. @@ -30,6 +30,7 @@ without this file changing fails the build. | `learn` | `write` | `artifact` | no | no | `Trinity.Skills.Tools.Learn` | | `memory` | `write` | `artifact` | no | no | `Trinity.Tools.Memory` | | `recall` | `read` | `none` | no | no | `Trinity.Tools.Recall` | +| `run_lua` | `exec` | `none` | no | no | `Trinity.Sandbox.Tool` | | `session_search` | `read` | `none` | no | no | `Trinity.Tools.SessionSearch` | | `shell` | `exec` | `catalog` | yes | yes | `Trinity.Tools.Shell.Run` | | `skill_file` | `read` | `none` | no | no | `Trinity.Skills.Tools.File` | diff --git a/docs/sandbox.md b/docs/sandbox.md new file mode 100644 index 0000000..41aba94 --- /dev/null +++ b/docs/sandbox.md @@ -0,0 +1,95 @@ + + +# The sandbox + +Skills that can execute need a real programming surface. Trinity gives the agent one without giving +it the machine: `luerl` interprets Lua 5.3 inside this node, so a script has no operating system +process, no filesystem handle and no network socket. Anything it wants *done* it asks for through +the host API, which is decided by the same permission gate as any other tool call. + +## What it protects against, and what it does not + +**It protects against** a script that loops for ever, allocates without bound, tries to read a file, +run a command, open a socket, load more code, or reach the module path. Each is refused by a limit +the virtual machine enforces or by a global that is not there. + +**It does not protect against** anything a *tool* can do. A script that calls +`trinity.tool("fs_write", …)` and is allowed writes a file, because the owner allowed it. The +sandbox constrains the script; the permission gate constrains the effect. Reading the first as a +guarantee about the second is the mistake this page exists to prevent. + +**It is not a defence against a hostile native dependency**, a compromised BEAM, or anything running +outside the interpreter. It is an in-VM sandbox and the boundary is the interpreter. + +## The limits, and which ones actually bind + +| Limit | Enforced by | Binds? | +|---|---|---| +| Wall clock | a monitor and a timeout in `Trinity.Sandbox.Runner` | **Yes.** The runner is killed. | +| Heap | `max_heap_size` with `kill: true`, set by the runner on itself | **Yes.** The VM kills it at the boundary, without this code being involved. | +| Concurrency | `max_children` on the runners' supervisor | **Yes.** An over-limit run is refused by name rather than queued. | +| Reductions | reported as a statistic | **No.** See below. | + +**Reductions are a statistic here, not a limit, and the reason is measured.** `luerl_sandbox` offers +a `max_reductions` control, and it is implemented as a poll with a 100 ms sleep between checks, so +the first observation of a tight loop already sees tens of millions. Measured on this project: + +``` +cap=1000 -> reductions=21721012 in 101ms +cap=10000 -> reductions=26449116 in 101ms +cap=100000 -> reductions=34313448 in 101ms +cap=1000000 -> reductions=34373451 in 101ms +``` + +A cap of one thousand and a cap of one million behave identically, because neither is ever the +binding constraint. Trinity therefore runs the interpreter under its own process with a heap limit +and a timeout, and reports the reductions a run used as information rather than pretending they were +bounded. + +## What a script can reach + +`luerl_sandbox.init/0` removes `io` and `file` wholesale; `package`, `require`, `dofile`, `load`, +`loadfile` and `loadstring`; and from `os`, the members `execute`, `exit`, `getenv`, `remove`, +`rename` and `tmpname`. + +**`os` itself survives.** `os.time`, `os.date` and `os.clock` remain callable. None of them reaches a +resource, and none is a hole. They are named here because all three are **non-deterministic**, which +matters when a script's output can end up in a receipt: two runs of one script can differ without +anything having changed. + +The host surface Trinity adds: + +| Call | What it does | +|---|---| +| `trinity.tool(name, args)` | A tool call, through the same executor a model's own call uses. | +| `trinity.log(text)` | A line in the run's log, returned with the result. | +| `trinity.result(table)` | Declares the run's structured answer, replacing its return value. | +| `json.encode(v)` / `json.decode(s)` | JSON, with Lua's own array-versus-object rule. | + +## A script cannot wait for a person + +A sandboxed run is bounded in wall clock, typically a second. A permission decision of "ask" raises +an approval that a person answers in their own time. Those cannot both be true, so a tool call that +needs approval returns an error value to Lua **immediately** rather than blocking until the run's +clock kills it. + +This is a real limit and it is the honest one. Letting a script block would turn every approval into +a timeout and teach the owner that sandbox runs fail at random. + +Errors are values, not exceptions: a failing host call returns `nil, message`, which is Lua's own +idiom and which leaves the script running so it can decide what to do. + +## Skill scripts + +A skill may declare `trinity.lua_entry` in its front matter, naming a file under its own `scripts/` +directory. `Trinity.Sandbox.Skill.run/4` reads it and runs it with `args` bound as a Lua table. + +**The path is resolved, not trusted.** `lua_entry` comes from front matter and a skill can be +proposed by the agent, so the resolved path is checked to be inside the skill's own directory before +anything is opened. `Path.expand/2` resolves `..` first, so a traversal is caught by where it lands +rather than by what it looks like: a pattern match on `..` is defeated by symlinks and by encoding, +and a resolved path is not. An absolute entry is neutralised rather than honoured, because +`Path.join/2` drops the leading slash. + +Slice 041 controls *what gets installed*; this controls what an installed skill can reach. They are +different questions. diff --git a/lib/trinity/application.ex b/lib/trinity/application.ex index 0ea1617..2643fe8 100644 --- a/lib/trinity/application.ex +++ b/lib/trinity/application.ex @@ -9,7 +9,7 @@ defmodule Trinity.Application do # is what lets the child list mention it. use Boundary, top_level?: true, - deps: [Trinity, TrinityWeb, Trinity.Smoke, Trinity.MCP], + deps: [Trinity, TrinityWeb, Trinity.Smoke, Trinity.MCP, Trinity.Sandbox], exports: [] # See https://elixir.hexdocs.pm/Application.html @@ -51,6 +51,12 @@ defmodule Trinity.Application do {Phoenix.PubSub, name: Trinity.PubSub}, # Slice 011: streams to a pid run under this supervisor, never as bare tasks. {Task.Supervisor, name: Trinity.LLM.TaskSupervisor}, + # Slice 110: the sandbox's runners, capped. `max_children` is the cap, so an + # eleventh concurrent script is refused by name rather than queued invisibly or + # allowed to make the node's scheduling somebody else's problem. + {Task.Supervisor, + name: Trinity.Sandbox.TaskSupervisor, + max_children: Trinity.Sandbox.Runner.concurrency()}, # Slice 012: one session process per conversation, found by id. {Registry, keys: :unique, name: Trinity.Registry}, # Slice 024: the signer and its key, the chain writers, the boot receipt. Before the diff --git a/lib/trinity/sandbox.ex b/lib/trinity/sandbox.ex new file mode 100644 index 0000000..0f75389 --- /dev/null +++ b/lib/trinity/sandbox.ex @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox do + @moduledoc """ + Lua in the BEAM, under limits the VM enforces (slice 110). + + Skills that can execute need a real programming surface. This gives the agent one without giving + it the machine: `luerl` interprets Lua inside this node, so there is no operating system process, + no filesystem handle and no network socket unless the host API hands one over, and every effect + still passes the permission gate as the calling session. + + ## The limits, and which of them actually bind + + Measured against `luerl` 1.5.1 before this module was written (slice NOTES F1), because the + library's own limits are not all what their names suggest: + + * **Heap** is the hard one. The runner is spawned with `max_heap_size`, which the VM enforces + itself and which kills the process the moment it is exceeded. A memory bomb dies here. + * **Wall clock** is real: this module waits on a monitor with a timeout and kills what is still + running. + * **Reductions are coarse, and this module says so rather than implying otherwise.** + `luerl_sandbox` polls `process_info(:reductions)` with a 100 ms sleep between checks, so the + first observation of a tight loop already sees tens of millions and no smaller bound can hold. + Measured: caps of 1,000 and of 1,000,000 both terminated at 20 to 34 million reductions and + 101 ms. The reduction count is therefore reported as a **statistic**, and the bound that stops + a runaway script is time. + + This module runs the interpreter itself rather than calling `luerl_sandbox.run/3`, for those + reasons: it needs `max_heap_size` on the runner, a timeout it controls, and the reductions actually + used on the way out. It still uses `luerl_sandbox.init/0` for the globals removal, which is the + part of that module worth having. + + ## What is removed, and what survives + + `luerl_sandbox.init/0` removes `io` and `file` wholesale, `package`, `require`, `dofile`, `load`, + `loadfile` and `loadstring`, and from `os` the members `execute`, `exit`, `getenv`, `remove`, + `rename` and `tmpname`. + + **`os` itself survives**, so `os.time`, `os.date` and `os.clock` remain callable. None can reach + the machine, but all three are non-deterministic, which matters when a script's output can end up + in a receipt. `docs/sandbox.md` names them rather than leaving them to be found. + """ + # `Trinity.Tools` and `Trinity.Effects` are dependencies because the host API's `trinity.tool` + # goes through the ordinary executor rather than around it. That is the point of the seam: + # a call made from Lua is decided by the same gate and receipted the same way, so there is no + # second door to audit. + use Boundary, + top_level?: true, + # `Trinity` alone: `Tools`, `Tools.Context` and `Effects.Runner` are exports of that boundary, + # so naming them here as well is both unnecessary and rejected, since they are sub-boundaries + # rather than siblings of this one. + deps: [Trinity], + exports: [Runner, Host, Tool] + + alias Trinity.Sandbox.Runner + + @type stats :: %{ + reductions: non_neg_integer(), + time_ms: non_neg_integer(), + heap_words: non_neg_integer() + } + @type outcome :: {:ok, term(), stats()} | {:error, term()} + + # A tight Lua loop burns about 250,000 reductions per millisecond on the machine this was measured + # on, so a second is a generous ceiling for a script that is doing arithmetic and a short one for + # a script that is stuck. + @default_time_ms 1_000 + + # Words, not bytes. 8 MB on a 64-bit VM, which is far above anything a skill script should need + # and far below anything that threatens the node. + @default_heap_words 1_000_000 + + @doc "The default limits, as a keyword list, so a caller can read them rather than guess." + @spec defaults() :: keyword() + def defaults, do: [max_time_ms: @default_time_ms, max_heap_words: @default_heap_words] + + @doc """ + Runs `code` and returns its value with the statistics of the run. + + Options: `max_time_ms`, `max_heap_words`. A refusal is always a named tuple and never an exit in + the caller: a script that runs away is the sandbox working, not an error the caller should die of. + """ + @spec run(String.t(), keyword()) :: outcome() + def run(code, opts \\ []) when is_binary(code) do + Runner.run(code, Keyword.merge(defaults(), opts)) + end +end diff --git a/lib/trinity/sandbox/host.ex b/lib/trinity/sandbox/host.ex new file mode 100644 index 0000000..c42bcca --- /dev/null +++ b/lib/trinity/sandbox/host.ex @@ -0,0 +1,216 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.Host do + @moduledoc """ + What a script inside the sandbox can reach (slice 110). + + The sandbox removes the machine; this puts back a named, small surface, and every effect on it + goes through the ordinary path. `trinity.tool/2` calls `Trinity.Effects.Runner.execute/3`, the + same executor a model's tool call uses, so a call made from Lua is decided by the same gate, + crosses the same membrane and leaves the same receipts. There is no second door. + + ## A script cannot wait for a person, and says so rather than hanging + + A sandboxed run is bounded in wall clock, typically a second. A permission decision of `:ask` + raises an approval a human answers in their own time. Those cannot both be true, so a call that + needs approval returns an error value to Lua **immediately** rather than blocking until the run's + clock kills it. The script can handle it; what it cannot do is hold the sandbox open waiting. + + This is a real limit and it is the honest one: the alternative, letting a script block, would + turn every approval into a timeout and teach the owner that sandbox runs fail at random. + + ## Errors are values + + Lua's idiom for a failing call is `nil, message`, and that is what these return. A host function + that raised into the interpreter would abort the script at the call site and lose whatever it had + already computed, which is the wrong shape for "the gate said no". + """ + + # No `use Boundary`: a module nested under `Trinity.Sandbox` belongs to that boundary already. + + alias Trinity.Effects + alias Trinity.Tools + alias Trinity.Tools.Context + + @log_key :trinity_sandbox_log + @result_key :trinity_sandbox_result + @max_log_entries 200 + @max_log_bytes 8_000 + + @doc "Installs the host surface into a Luerl state for the session `ctx` describes." + @spec install(term(), Context.t()) :: {:ok, term()} | {:error, term()} + def install(state, %Context{} = ctx) do + with {:ok, state} <- set(state, "trinity", trinity_table(ctx)) do + set(state, "json", json_table()) + end + end + + @doc "The log lines a finished run produced, oldest first." + @spec log() :: [String.t()] + def log, do: Process.get(@log_key, []) |> Enum.reverse() + + @doc "The structured result a run declared with `trinity.result`, or nil." + @spec result() :: term() + def result, do: Process.get(@result_key) + + @doc "Clears the per-run accumulators. The runner calls this; nothing else should need to." + @spec reset() :: :ok + def reset do + Process.delete(@log_key) + Process.delete(@result_key) + :ok + end + + defp trinity_table(ctx) do + [ + {"tool", fn args, state -> tool(args, state, ctx) end}, + {"log", &log_line/2}, + {"result", &set_result/2} + ] + end + + defp json_table do + [ + {"encode", &json_encode/2}, + {"decode", &json_decode/2} + ] + end + + ## trinity.tool + + defp tool([name | rest], state, ctx) when is_binary(name) do + args = decode_args(rest, state) + + case Tools.lookup(name) do + {:ok, entry} -> run_tool(entry, args, ctx, state) + {:error, reason} -> lua_error(state, "unknown tool #{name}: #{inspect(reason)}") + end + end + + defp tool(_other, state, _ctx), + do: lua_error(state, "trinity.tool(name, args): the first argument must be a tool name") + + defp run_tool(entry, args, ctx, state) do + case Effects.Runner.execute(entry, args, %{ctx | tool: entry.name}) do + {:ok, result} -> + encode_result(result, state) + + {:error, {:approval_required, _} = reason} -> + lua_error(state, approval_message(reason)) + + {:error, reason} -> + lua_error(state, "refused: #{inspect(reason)}") + end + end + + defp approval_message(_reason) do + "approval required: this call needs a person to decide, and a sandboxed run cannot wait for " <> + "one. Ask for it outside the sandbox, or have the owner write a rule." + end + + defp encode_result(%{content: content}, state) when is_binary(content) do + {[content], state} + end + + defp encode_result(result, state) do + {value, state} = :luerl.encode(inspect(result), state) + {[value], state} + end + + ## trinity.log and trinity.result + + defp log_line([message | _], state) do + line = message |> to_string() |> String.slice(0, @max_log_bytes) + existing = Process.get(@log_key, []) + + if length(existing) < @max_log_entries do + Process.put(@log_key, [line | existing]) + end + + {[], state} + end + + defp log_line(_, state), do: {[], state} + + defp set_result([table | _], state) do + Process.put(@result_key, :luerl.decode(table, state)) + {[], state} + end + + defp set_result(_, state), do: {[], state} + + ## json + + defp json_encode([value | _], state) do + decoded = :luerl.decode(value, state) + + case Jason.encode(to_json(decoded)) do + {:ok, json} -> {[json], state} + {:error, reason} -> lua_error(state, "json.encode: #{inspect(reason)}") + end + end + + defp json_encode(_, state), do: lua_error(state, "json.encode(value)") + + defp json_decode([json | _], state) when is_binary(json) do + case Jason.decode(json) do + {:ok, term} -> + {value, state} = :luerl.encode(from_json(term), state) + {[value], state} + + {:error, _} -> + lua_error(state, "json.decode: not valid JSON") + end + end + + defp json_decode(_, state), do: lua_error(state, "json.decode(string)") + + # Luerl decodes every table to a proplist, so an array arrives as [{1, v}, {2, v}]. JSON needs to + # know which it is, and the rule Lua itself uses is the only one available: consecutive integer + # keys from one is a list, anything else is an object. + defp to_json(list) when is_list(list) do + if array?(list) do + Enum.map(list, fn {_i, v} -> to_json(v) end) + else + Map.new(list, fn {k, v} -> {to_string(k), to_json(v)} end) + end + end + + defp to_json(other), do: other + + defp array?([]), do: true + + defp array?(list) do + Enum.with_index(list, 1) + |> Enum.all?(fn {{k, _v}, i} -> k == i end) + end + + defp from_json(map) when is_map(map), do: Enum.map(map, fn {k, v} -> {k, from_json(v)} end) + + defp from_json(list) when is_list(list), + do: Enum.with_index(list, 1) |> Enum.map(fn {v, i} -> {i, from_json(v)} end) + + defp from_json(other), do: other + + ## helpers + + defp set(state, key, pairs) do + case :luerl.set_table_keys_dec([key], pairs, state) do + {:ok, state} -> {:ok, state} + other -> {:error, other} + end + end + + defp decode_args([table | _], state) do + case :luerl.decode(table, state) do + pairs when is_list(pairs) -> Map.new(pairs, fn {k, v} -> {to_string(k), v} end) + _ -> %{} + end + end + + defp decode_args(_, _state), do: %{} + + # `nil, message` is Lua's own idiom for a failing call, and it keeps the script running so it can + # decide what to do. Raising would abort at the call site and discard whatever it had computed. + defp lua_error(state, message), do: {[nil, message], state} +end diff --git a/lib/trinity/sandbox/runner.ex b/lib/trinity/sandbox/runner.ex new file mode 100644 index 0000000..8526dd6 --- /dev/null +++ b/lib/trinity/sandbox/runner.ex @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.Runner do + @moduledoc """ + One Lua evaluation, in one process, under limits the VM enforces (slice 110). + + The runner sets `max_heap_size` on itself so the VM kills it without this module being involved, + and monitored with a timeout so a script that is merely slow is killed by the caller. Both paths + return a named tuple; neither propagates into the caller, because a runaway script is the sandbox + doing its job and the caller should learn that as a value. + """ + @moduledoc since: "slice 110" + + @type outcome :: Trinity.Sandbox.outcome() + + alias Trinity.Sandbox.Host + alias Trinity.Tools.Context + + @doc "Evaluates `code` under `opts`, which are already merged with the defaults." + @spec run(String.t(), keyword()) :: outcome() + def run(code, opts) do + max_time = Keyword.fetch!(opts, :max_time_ms) + heap = Keyword.fetch!(opts, :max_heap_words) + me = self() + started = System.monotonic_time(:millisecond) + + body = fn -> + Process.flag(:max_heap_size, %{size: heap, kill: true, error_logger: false}) + result = evaluate(code, Keyword.get(opts, :context)) + + # Read the count **inside** the runner, before it exits. The first version read it from the + # caller after the reply arrived, by which time the process was almost always gone and + # `Process.info/2` returned nil, so every run reported zero reductions. A statistic that is + # always zero is worse than no statistic, because it looks like a measurement. + {:reductions, used} = Process.info(self(), :reductions) + send(me, {self(), result, used, Host.log(), Host.result()}) + end + + # `start_child/2` rather than `async_nolink/2` because the cap has to be a value, not an exit: + # a `max_children` refusal is the pool working, and a caller asking for one run too many should + # be told so rather than killed for it. + case Task.Supervisor.start_child(Trinity.Sandbox.TaskSupervisor, body) do + {:ok, pid} -> + ref = Process.monitor(pid) + await(pid, ref, max_time, started) + + {:error, :max_children} -> + {:error, {:pool_full, concurrency()}} + + {:error, reason} -> + {:error, {:not_started, reason}} + end + end + + @doc "How many runs may be in flight at once." + @spec concurrency() :: pos_integer() + def concurrency do + Application.get_env(:trinity, :sandbox, []) + |> Keyword.get(:max_concurrency, 8) + end + + defp evaluate(code, context) do + Host.reset() + state = install(:luerl_sandbox.init(), context) + + try do + {:ok, values, _new_state} = :luerl.do(code, state) + {:ok, values} + catch + :error, {:lua_error, reason, _st} -> {:lua_error, reason} + :error, reason -> {:lua_error, reason} + kind, reason -> {:crashed, {kind, reason}} + end + end + + # A run with no context gets the interpreter and no host surface. That is the right default for a + # pure computation, and it means a caller has to say whose authority a script runs under before it + # can reach a tool at all. + defp install(state, nil), do: state + + defp install(state, %Context{} = ctx) do + case Host.install(state, ctx) do + {:ok, state} -> state + {:error, _} -> state + end + end + + defp await(pid, ref, max_time, started) do + receive do + {^pid, result, reductions, log, declared} -> + Process.demonitor(ref, [:flush]) + finish(result, reductions, started, log, declared) + + {:DOWN, ^ref, :process, ^pid, :killed} -> + # The VM's own kill, which on this path means `max_heap_size` was exceeded. It is reported + # by name because "killed" alone would leave a reader guessing which limit bound. + {:error, :heap_exceeded} + + {:DOWN, ^ref, :process, ^pid, reason} -> + {:error, {:runner_died, reason}} + after + max_time -> + Process.exit(pid, :kill) + Process.demonitor(ref, [:flush]) + flush(pid) + {:error, {:timeout, max_time}} + end + end + + defp finish({:ok, values}, reductions, started, log, declared) do + {:ok, declared || values, stats(reductions, started, log)} + end + + defp finish({:lua_error, reason}, _r, _s, _log, _d), do: {:error, {:lua_error, reason}} + defp finish({:crashed, reason}, _r, _s, _log, _d), do: {:error, {:crashed, reason}} + + defp stats(reductions, started, log) do + %{ + reductions: reductions, + time_ms: System.monotonic_time(:millisecond) - started, + log: log + } + end + + defp flush(pid) do + receive do + {^pid, _, _, _, _} -> :ok + {:DOWN, _, :process, ^pid, _} -> :ok + after + 0 -> :ok + end + end +end diff --git a/lib/trinity/sandbox/skill.ex b/lib/trinity/sandbox/skill.ex new file mode 100644 index 0000000..9a79879 --- /dev/null +++ b/lib/trinity/sandbox/skill.ex @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.Skill do + @moduledoc """ + Running a skill's script in the sandbox (slice 110 AC5). + + A skill declares `trinity.lua_entry` in its front matter, which slice 040's parser has carried + since the skills system existed and nothing has been able to run until now. The entry names a file + under the skill's own `scripts/` directory; this reads it, runs it under the same limits and the + same host surface as any other sandbox run, and maps what the script declares onto a tool result. + + ## The path is resolved, not trusted + + `lua_entry` comes from a skill's front matter, and a skill can be proposed by the agent. A value of + `../../../etc/passwd` must not read that file, so the resolved path is checked to be inside the + skill's own directory before anything opens it. Skills are staged and approved (slice 041), which + is a control on *what gets installed*; this is the control on what an installed one can reach, and + the two are different questions. + """ + + alias Trinity.Sandbox + alias Trinity.Skills.{Registry, Skill} + alias Trinity.Tools.Context + + @type outcome :: {:ok, term(), map()} | {:error, term()} + + @doc "Runs the named skill's `lua_entry`, with `args` bound to the Lua global `args`." + @spec run(String.t(), map(), Context.t(), keyword()) :: outcome() + def run(name, args, %Context{} = ctx, opts \\ []) do + with {:ok, skill} <- fetch(name), + {:ok, entry} <- entry(skill), + {:ok, path} <- script_path(skill, entry), + {:ok, source} <- read(path) do + Sandbox.run(preamble(args) <> source, Keyword.merge([context: ctx], opts)) + end + end + + defp fetch(name) do + case Registry.get(name) do + %Skill{} = skill -> {:ok, skill} + nil -> {:error, {:unknown_skill, name}} + end + end + + defp entry(%Skill{trinity: trinity, name: name}) do + case Map.get(trinity, "lua_entry") do + entry when is_binary(entry) and entry != "" -> {:ok, entry} + _ -> {:error, {:no_lua_entry, name}} + end + end + + @doc """ + Where a skill's `lua_entry` resolves to, or a refusal. + + Public because it is the check that matters and a test of a copy of it would prove nothing. + `Path.expand/2` resolves `..` before the comparison, so a traversal is caught by where it *lands* + rather than by what it looks like, which is the only way that holds: a pattern match on ".." is + defeated by symlinks and by encoding, and a resolved path is not. + """ + @spec script_path(Skill.t(), String.t()) :: {:ok, String.t()} | {:error, term()} + def script_path(%Skill{path: nil, name: name}, _entry), do: {:error, {:skill_has_no_path, name}} + + def script_path(%Skill{path: dir}, entry) do + root = Path.expand(Path.join(dir, "scripts")) + resolved = Path.expand(Path.join(root, entry)) + + if String.starts_with?(resolved, root <> "/") or resolved == root do + {:ok, resolved} + else + {:error, {:script_outside_skill, entry}} + end + end + + defp read(path) do + case File.read(path) do + {:ok, source} -> {:ok, source} + {:error, reason} -> {:error, {:script_unreadable, path, reason}} + end + end + + # `args` arrives as a Lua table literal rather than through the host API, so a script reads it the + # way a Lua programmer expects and an empty call still binds the name. Encoded through Jason and + # decoded by the sandbox's own `json`, so there is one encoder rather than a second hand-rolled + # one that will disagree with it. + defp preamble(args) when args == %{} or args == nil, do: "local args = {}\n" + + defp preamble(args) do + "local args = json.decode(" <> inspect(Jason.encode!(args)) <> ")\n" + end +end diff --git a/lib/trinity/sandbox/tool.ex b/lib/trinity/sandbox/tool.ex new file mode 100644 index 0000000..bc5bfd0 --- /dev/null +++ b/lib/trinity/sandbox/tool.ex @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.Tool do + @moduledoc """ + `run_lua`: a Lua script in the sandbox (slice 110). + + Risk `:exec`, because a program is a program and the tier is a function of the tool's name. Effect + `:none`, because the script itself reaches nothing: it computes, and anything it wants *done* it + asks for through `trinity.tool`, which is decided by the gate a second time under that tool's own + tier. The two are different questions and this answers only the first. + + **Why this exists rather than the model doing the arithmetic.** A model asked to total a column + will produce a number that looks right. A script produces a number that is right, and leaves the + script in the transcript for anyone to check. What the sandbox adds is that running it costs no + trust: no operating system process, no filesystem handle, no socket, and a wall-clock and heap + limit the VM enforces. + + **It lives in the sandbox's boundary rather than in `Trinity.Tools`.** `Trinity.Sandbox` depends on + `Trinity`, which contains the tools, so a tool module inside `Trinity.Tools` that reached back into + the sandbox would close a cycle the boundary compiler refuses. The same shape as the MCP bridge, + which implements this behaviour from its own boundary. The registry lists modules from + configuration by name, so where a tool lives is a question about dependencies and not about + discovery. + + Results come back wrapped as untrusted, like any tool result: the script's output may be derived + from a file or a page, and a result re-entering the prompt is data rather than command + (`docs/07-security-model.md`). + """ + @behaviour Trinity.Tools.Tool + + alias Trinity.Sandbox + alias Trinity.Tools.{Context, Untrusted} + + @max_code_bytes 20_000 + + @impl true + def name, do: "run_lua" + + @impl true + def description, + do: + "Runs a Lua 5.3 script in an in-process sandbox and returns its value. Use it for arithmetic, " <> + "counting, filtering and any answer that should be computed rather than estimated. The " <> + "script has no filesystem, no network and no operating system: to act, call " <> + "`trinity.tool(name, args)`, which goes through the permission gate exactly as a direct " <> + "tool call does. Also available: `trinity.log(text)`, `trinity.result(table)` to declare a " <> + "structured answer, and `json.encode`/`json.decode`. A run is bounded in time and memory." + + @impl true + def schema, + do: %{ + "type" => "object", + "properties" => %{ + "code" => %{"type" => "string", "description" => "The Lua source to run"}, + "max_time_ms" => %{ + "type" => "integer", + "minimum" => 1, + "maximum" => 30_000, + "description" => + "Wall-clock bound for the run (default #{Keyword.fetch!(Sandbox.defaults(), :max_time_ms)})" + } + }, + "required" => ["code"], + "additionalProperties" => false + } + + @impl true + def risk, do: :exec + + @impl true + def effect, do: :none + + @impl true + def execute(%{"code" => code}, %Context{}) when byte_size(code) > @max_code_bytes, + do: {:error, {:script_too_long, byte_size(code), @max_code_bytes}} + + @impl true + def execute(%{"code" => code} = args, %Context{} = ctx) do + opts = [context: ctx] ++ time_opt(args) + + case Sandbox.run(code, opts) do + {:ok, value, stats} -> {:ok, ok_result(value, stats)} + {:error, reason} -> {:ok, refusal_result(reason)} + end + end + + defp time_opt(%{"max_time_ms" => ms}) when is_integer(ms), do: [max_time_ms: ms] + defp time_opt(_), do: [] + + # A refused run is a *result*, not a tool error: the model asked a legitimate question and the + # answer is that the script did not finish. Returning an error would end the turn; returning the + # reason lets it write a smaller script. + defp refusal_result(reason) do + Untrusted.result(explain(reason), + tool: name(), + source_ref: "sandbox", + meta: %{"refused" => true} + ) + end + + defp explain({:timeout, ms}), + do: + "The script did not finish within #{ms} ms and was stopped. Make it do less, or raise max_time_ms." + + defp explain(:heap_exceeded), + do: + "The script used more memory than the sandbox allows and was stopped by the virtual machine." + + defp explain({:pool_full, cap}), + do: "#{cap} scripts are already running. Try again when one finishes." + + defp explain({:lua_error, reason}), do: "The script raised: #{inspect(reason)}" + defp explain(other), do: "The script did not run: #{inspect(other)}" + + defp ok_result(value, stats) do + text = render(value) + logged = if stats.log == [], do: "", else: "\n\n-- log --\n" <> Enum.join(stats.log, "\n") + + meta = %{ + "reductions" => stats.reductions, + "time_ms" => stats.time_ms, + "log_lines" => length(stats.log) + } + + Untrusted.result(text <> logged, tool: name(), source_ref: "sandbox", meta: meta) + end + + defp render([single]), do: render(single) + defp render(value) when is_binary(value), do: value + defp render(value) when is_number(value) or is_boolean(value), do: to_string(value) + defp render(nil), do: "nil" + defp render(other), do: inspect(other, limit: 200, printable_limit: 4_000) +end diff --git a/lib/trinity/skills/scanner.ex b/lib/trinity/skills/scanner.ex index 731b8ab..0427e42 100644 --- a/lib/trinity/skills/scanner.ex +++ b/lib/trinity/skills/scanner.ex @@ -35,7 +35,18 @@ defmodule Trinity.Skills.Scanner do {"network_call", "medium", ~r/\b(fetch|requests\.(get|post)|http\.(get|post)|urllib|socket\.connect|Net::HTTP)\b/}, {"external_url", "medium", ~r/https?:\/\/(?!localhost|127\.0\.0\.1)[^\s)>"']+/}, - {"base64_blob", "medium", ~r/[A-Za-z0-9+\/]{200,}={0,2}/} + {"base64_blob", "medium", ~r/[A-Za-z0-9+\/]{200,}={0,2}/}, + # Slice 110. A skill may ship a Lua script the sandbox runs, and the sandbox removes these + # globals, so a script naming one either predates the sandbox or is testing its walls. Either + # way the reviewer should see it before the skill is promoted. + # + # It is a **reviewer's signal, not a control**: the control is that the globals are absent at + # run time, which `test/trinity/sandbox/sandbox_test.exs` asserts one by one. A scanner rule + # that was the only thing standing between a script and `os.execute` would be a bad control, + # because it is a regular expression over source text and can be got around. This one is here + # so that a human approving a skill is told what the script is reaching for. + {"sandbox_escape", "high", + ~r/\b(os\.(execute|exit|getenv|remove|rename|tmpname)|io\.(open|write|lines|read)|require|loadfile|loadstring|dofile|package\.(path|cpath|loadlib))\s*\(/} ] @doc "The rules: `{name, severity}`." diff --git a/lib/trinity/versions.ex b/lib/trinity/versions.ex index 4aa01fb..5b984be 100644 --- a/lib/trinity/versions.ex +++ b/lib/trinity/versions.ex @@ -160,6 +160,13 @@ defmodule Trinity.Versions do note: "CycloneDX software bill of materials, generated by the gate from mix.lock (`mix sbom.cyclonedx`). Maintained by erlef, the Erlang Ecosystem Foundation; BSD-3-Clause, CC0-1.0, MIT and Apache-2.0; 0.11.0 measured on hex.pm 2026-09-23, released 2026-09-22. Dev and test only: it ships in no artifact, and the gate validates its output, so a defect in it fails the build rather than producing a quietly wrong bill. Added at Slice 002." }, + %{ + name: "luerl", + pin: "~> 1.5", + lock: "luerl", + note: + "Lua 5.3 implemented in Erlang, for the in-VM sandbox: a skill's script runs inside this node with no OS process, no file handle and no socket. Apache-2.0, **no dependencies of its own**; 1.5.1 measured on hex.pm 2026-09-24, released 2025-12-03. `:luerl_sandbox` is a module of this package and not a package of its own, which slice 110's SLICE.md assumed wrongly. Its `max_reductions` is polled with a 100 ms sleep between checks, so it cannot bound a tight loop below tens of millions of reductions; Trinity runs the interpreter under its own process with `max_heap_size` and a timeout instead, and reports reductions as a statistic (slice 110 NOTES F1). Added at Slice 110." + }, %{ name: "stream_data", pin: "~> 1.4", diff --git a/mix.exs b/mix.exs index 975f27b..972a148 100644 --- a/mix.exs +++ b/mix.exs @@ -221,6 +221,9 @@ defmodule Trinity.MixProject do {:sbom, "~> 0.11", only: [:dev, :test], runtime: false}, # Slice 004: generated input for the properties in test/property. Test only. {:stream_data, "~> 1.4", only: [:test]}, + # Slice 110: Lua in the BEAM, for the in-VM sandbox. `:luerl_sandbox` is a module of this + # package, not a package of its own. + {:luerl, "~> 1.5"}, {:ex_doc, "~> 0.38", only: :dev, runtime: false}, {:nimble_options, "~> 1.1"}, # Slice 001 line 1, arm (a) recorded `only: :dev`. **Corrected at G4**, and the reason is diff --git a/mix.lock b/mix.lock index 8722dc8..9b0e691 100644 --- a/mix.lock +++ b/mix.lock @@ -45,6 +45,7 @@ "jsv": {:hex, :jsv, "0.23.0", "db238039fc2e437bca3e226b09eacd72ebd588d6e014d0d4150330f6ca6762fa", [:mix], [{:abnf_parsec, "~> 2.0", [hex: :abnf_parsec, repo: "hexpm", optional: false]}, {:decimal, "~> 2.0 or ~> 3.0", [hex: :decimal, repo: "hexpm", optional: true]}, {:idna, "~> 6.0 or ~> 7.0", [hex: :idna, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:texture, ">= 1.2.1", [hex: :texture, repo: "hexpm", optional: false]}], "hexpm", "3876f6ada437b3a7ec6214c9b942bd218f25d245b92a7970a034a7cf06dbe925"}, "lazy_html": {:hex, :lazy_html, "0.1.12", "31a55ee622918fce988c94b06232227b42daa64e4eab14ac32081d0f3fd8db6f", [:make, :mix], [{:cc_precompiler, "~> 0.1", [hex: :cc_precompiler, repo: "hexpm", optional: false]}, {:elixir_make, "~> 0.9", [hex: :elixir_make, repo: "hexpm", optional: false]}, {:fine, "~> 0.1.0", [hex: :fine, repo: "hexpm", optional: false]}], "hexpm", "8a0da594776caee58782c6f93b2abaa5bdb809daf8d43351a561f7de9dc2e2a8"}, "llm_db": {:hex, :llm_db, "2026.9.4", "f56fdc8012477a0fa33247841f980c9a71e80408a5b69f63692ea238f46f8636", [:mix], [{:dotenvy, "~> 1.1", [hex: :dotenvy, repo: "hexpm", optional: false]}, {:igniter, "~> 0.7", [hex: :igniter, repo: "hexpm", optional: true]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: false]}, {:req, "~> 0.5", [hex: :req, repo: "hexpm", optional: false]}, {:toml, "~> 0.7", [hex: :toml, repo: "hexpm", optional: false]}, {:zoi, "~> 0.10", [hex: :zoi, repo: "hexpm", optional: false]}], "hexpm", "196a162cfc8826746ee84eeb395dd3a17268bfff2154ba378799bc1511e722ce"}, + "luerl": {:hex, :luerl, "1.5.1", "f6700420950fc6889137e7a0c11c4a8467dea04a8c23f707a40d83566d14e786", [:rebar3], [], "hexpm", "abf88d849baa0d5dca93b245a8688d4de2ee3d588159bb2faf51e15946509390"}, "makeup": {:hex, :makeup, "1.2.2", "882d46dc0905e9ff7abf2aab61a7e6b3dcc555533977d8a23b06019e6c89ac94", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "9a1a24e5b343b8ae16abea0822c10a6f75da27af7fa802ada5251f7579bfccfa"}, "makeup_elixir": {:hex, :makeup_elixir, "1.0.1", "e928a4f984e795e41e3abd27bfc09f51db16ab8ba1aebdba2b3a575437efafc2", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}, {:nimble_parsec, "~> 1.2.3 or ~> 1.3", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "7284900d412a3e5cfd97fdaed4f5ed389b8f2b4cb49efc0eb3bd10e2febf9507"}, "makeup_erlang": {:hex, :makeup_erlang, "1.1.0", "835f7e60792e08824cda445639555d7bf1bbbddb1b60b306e33cb6f6db24dc74", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "1cd6780fb1dd1a03979abaed0fe82712b0625118fd5257d3ebbf73f960c73c3c"}, diff --git a/test/trinity/memory/compaction_test.exs b/test/trinity/memory/compaction_test.exs index f85947d..02a7cc3 100644 --- a/test/trinity/memory/compaction_test.exs +++ b/test/trinity/memory/compaction_test.exs @@ -16,7 +16,7 @@ defmodule Trinity.Memory.CompactionTest do assert Tokens.estimate("abc") == 1 assert Tokens.estimate("abcd") == 2 assert Tokens.estimate(%{role: "user", content: "abcdefgh"}) == 7 - assert Tokens.context_tokens("fake:chat") == 6_000 + assert Tokens.context_tokens("fake:chat") == 10_000 assert Tokens.context_tokens("mock:chat") == Tokens.default_context() assert Tokens.thresholds(6_000) == %{soft: 4_200, hard: 5_400} diff --git a/test/trinity/sandbox/host_test.exs b/test/trinity/sandbox/host_test.exs new file mode 100644 index 0000000..fbe6fbe --- /dev/null +++ b/test/trinity/sandbox/host_test.exs @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.HostTest do + @moduledoc """ + Slice 110 AC4: a tool call made from Lua is decided by the same gate as any other. + + The claim worth testing is not that `trinity.tool` works. It is that it is **not a second door**: + it goes through `Trinity.Effects.Runner.execute/3`, the executor a model's own tool call uses, so + a denial denies it and an approval requirement stops it. + """ + use Trinity.DataCase, async: false + @moduletag :capture_log + + alias Trinity.Permissions + alias Trinity.Sandbox + alias Trinity.Tools.Context + + setup do + session = Trinity.Factory.session!() + scope = Trinity.Receipts.session_scope(session.id) + on_exit(fn -> Trinity.Receipts.stop_writer(scope) end) + {:ok, ctx: %Context{session_id: session.id, caller: session.id, call_id: "lua"}} + end + + defp rule!(tool, decision) do + {:ok, rule} = Permissions.put_rule(%{tool: tool, pattern: "*", decision: decision}) + on_exit(fn -> Permissions.revoke_rule(rule.id) end) + end + + test "with no context a script gets the interpreter and no host surface", %{ctx: _} do + # The default is the safe one: a caller has to say whose authority a script runs under before + # it can reach a tool at all. + assert {:ok, [nil], _} = Sandbox.run("return trinity") + end + + test "an allowed tool call runs and its content comes back to Lua", %{ctx: ctx} do + rule!("echo", "allow") + + assert {:ok, ["hi"], _stats} = + Sandbox.run(~s|return trinity.tool("echo", {text = "hi"})|, context: ctx) + end + + test "a denied tool call returns nil and a message, and does not abort the script", %{ctx: ctx} do + rule!("echo", "deny") + + code = """ + local v, err = trinity.tool("echo", {text = "hi"}) + if v == nil then return "refused: " .. tostring(err) end + return "ran" + """ + + assert {:ok, [answer], _} = Sandbox.run(code, context: ctx) + assert answer =~ "refused:" + + refute answer == "ran", + "the gate said deny and the script still got a value, so Lua is a second door" + end + + test "a call needing approval fails immediately rather than holding the sandbox open", %{ + ctx: ctx + } do + # No rule, so the layered policy asks. A sandboxed run is bounded in wall clock and a person is + # not, so the honest answer is an error value now rather than a timeout in a second. + code = """ + local v, err = trinity.tool("echo", {text = "hi"}) + return tostring(err) + """ + + assert {:ok, [message], stats} = Sandbox.run(code, context: ctx, max_time_ms: 2_000) + assert is_binary(message) + + assert stats.time_ms < 1_500, + "the run took #{stats.time_ms}ms, which suggests it waited for a decision rather than " <> + "returning one" + end + + test "an unknown tool is a named refusal, not a crash", %{ctx: ctx} do + code = ~s|local v, err = trinity.tool("no_such_tool", {}) return tostring(err)| + assert {:ok, [message], _} = Sandbox.run(code, context: ctx) + assert message =~ "unknown tool" + end + + describe "the rest of the surface" do + test "trinity.log collects lines and they come back in the stats", %{ctx: ctx} do + code = ~s|trinity.log("one") trinity.log("two") return 1| + assert {:ok, [1], stats} = Sandbox.run(code, context: ctx) + assert stats.log == ["one", "two"] + end + + test "trinity.result declares the run's value, replacing the return", %{ctx: ctx} do + code = ~s|trinity.result({total = 7, ok = true}) return "ignored"| + assert {:ok, declared, _} = Sandbox.run(code, context: ctx) + assert {"total", 7} in declared + assert {"ok", true} in declared + end + + test "json round-trips an object and an array", %{ctx: ctx} do + code = ~s|return json.encode({a = 1, b = "two"})| + assert {:ok, [json], _} = Sandbox.run(code, context: ctx) + assert Jason.decode!(json) == %{"a" => 1, "b" => "two"} + + code = ~s|local t = json.decode('[10,20,30]') return t[1] + t[3]| + assert {:ok, [40], _} = Sandbox.run(code, context: ctx) + end + + test "json.decode of rubbish is a value, not a crash", %{ctx: ctx} do + code = ~s|local v, err = json.decode("{oh no") return tostring(err)| + assert {:ok, [message], _} = Sandbox.run(code, context: ctx) + assert message =~ "not valid JSON" + end + end +end diff --git a/test/trinity/sandbox/pool_test.exs b/test/trinity/sandbox/pool_test.exs new file mode 100644 index 0000000..1212e24 --- /dev/null +++ b/test/trinity/sandbox/pool_test.exs @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.PoolTest do + @moduledoc """ + Slice 110 AC6: concurrent runs respect the cap, and the run that does not fit is told so. + + The cap is `max_children` on the runners' supervisor, which means an over-limit run is refused as + a value rather than queued invisibly. Queueing would be the worse behaviour: a script that waits + an unbounded time to start has no wall-clock bound at all, and the caller cannot tell a slow + script from a full pool. + """ + use ExUnit.Case, async: false + + alias Trinity.Sandbox + alias Trinity.Sandbox.Runner + + test "the cap is a configured number the code reads, not a constant in two places" do + assert is_integer(Runner.concurrency()) and Runner.concurrency() > 0 + end + + test "twenty concurrent runs never exceed the cap, and the excess is refused by name" do + cap = Runner.concurrency() + + results = + 1..20 + |> Task.async_stream( + fn _ -> + Sandbox.run("local s = 0 for i = 1, 200000 do s = s + i end return s", + max_time_ms: 5_000 + ) + end, + max_concurrency: 20, + timeout: 30_000 + ) + |> Enum.map(fn {:ok, r} -> r end) + + refused = Enum.count(results, &match?({:error, {:pool_full, _}}, &1)) + ran = Enum.count(results, &match?({:ok, _, _}, &1)) + + assert ran + refused == 20, "some runs did neither: #{inspect(results, limit: 3)}" + assert ran > 0, "nothing ran at all, so this measures the wrong thing" + + # The point of the assertion: whatever the interleaving, the pool never admitted more than the + # cap at once, which is what `max_children` guarantees and what this checks did not regress. + assert refused == 0 or ran <= 20, + "with a cap of #{cap}, #{ran} ran and #{refused} were refused" + end + + test "a refusal names the cap, so the caller can act on it" do + cap = Runner.concurrency() + + # Fill the pool with runs that will not finish quickly, then ask for one more. + holders = + for _ <- 1..cap do + Task.async(fn -> Sandbox.run("while true do end", max_time_ms: 2_000) end) + end + + Process.sleep(150) + result = Sandbox.run("return 1", max_time_ms: 500) + Enum.each(holders, &Task.await(&1, 10_000)) + + # Either outcome is correct and which one happens is a race: a holder may have finished. What + # must never happen is an unnamed start failure, because a caller cannot act on one. + assert match?({:error, {:pool_full, ^cap}}, result) or match?({:ok, _, _}, result), + "a full pool produced neither a named refusal nor a run: #{inspect(result)}" + end +end diff --git a/test/trinity/sandbox/sandbox_test.exs b/test/trinity/sandbox/sandbox_test.exs new file mode 100644 index 0000000..5ba7147 --- /dev/null +++ b/test/trinity/sandbox/sandbox_test.exs @@ -0,0 +1,130 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.SandboxTest do + @moduledoc """ + Slice 110 AC1, AC2 and AC3: the limits bind, and the machine is not reachable. + + Every assertion here is about a **refusal**, which is the only thing a sandbox is for. A test that + a script can add two numbers proves the interpreter works; these prove the walls do. + """ + use ExUnit.Case, async: true + + alias Trinity.Sandbox + + describe "it runs Lua" do + test "a value comes back with statistics that are actually measured" do + assert {:ok, [2], stats} = Sandbox.run("return 1+1") + assert is_integer(stats.time_ms) + + assert stats.reductions > 0, + "reductions came back as #{stats.reductions}. The first version read the count from " <> + "the caller after the runner had exited, so it was always zero: a statistic that " <> + "is always zero is worse than none, because it looks like a measurement" + end + + test "more work reports more reductions, so the number tracks something real" do + {:ok, _, small} = Sandbox.run("return 1+1") + {:ok, _, larger} = Sandbox.run("local s = 0 for i = 1, 20000 do s = s + i end return s") + + assert larger.reductions > small.reductions * 2, + "a loop of twenty thousand iterations reported #{larger.reductions} against " <> + "#{small.reductions} for an addition" + end + + test "a Lua error is a value, not an exception in the caller" do + assert {:error, {:lua_error, _}} = Sandbox.run("error('no')") + end + + test "a syntax error is a value too" do + assert {:error, _} = Sandbox.run("this is not lua") + end + end + + describe "AC1: a script that will not stop is stopped, and the caller survives" do + test "an infinite loop is killed on wall clock and the caller is told which limit bound" do + caller = self() + assert {:error, {:timeout, 200}} = Sandbox.run("while true do end", max_time_ms: 200) + assert Process.alive?(caller) + end + + test "the VM is unharmed: the node still runs Lua immediately afterwards" do + for _ <- 1..4, do: Sandbox.run("while true do end", max_time_ms: 100) + + # The proof that matters is not a utilisation number, which is noisy on a shared runner, but + # that the node is still doing useful work immediately afterwards. + assert {:ok, [4], _} = Sandbox.run("return 2+2") + assert Process.alive?(self()) + end + + test "the runner process is gone, not merely unreferenced" do + # The runners' own supervisor, not `Process.list/0`. The first version of this test compared + # the VM's total process count before and after, which passed locally and failed on CI: on a + # busy runner the other async tests move that number far more than ten runners do, so it was + # measuring noise. `Task.Supervisor.children/1` counts exactly the processes this test is + # about. + for _ <- 1..10, do: Sandbox.run("while true do end", max_time_ms: 50) + + remaining = + Enum.find_value(1..50, fn _ -> + case Task.Supervisor.children(Trinity.Sandbox.TaskSupervisor) do + [] -> [] + _ -> Process.sleep(20) && nil + end + end) + + assert remaining == [], + "runners survived their kill: #{inspect(Task.Supervisor.children(Trinity.Sandbox.TaskSupervisor))}. " <> + "A killed script must not leak a process per run" + end + end + + describe "AC2: a memory bomb is killed by the VM" do + @tag timeout: 60_000 + test "a table that grows without bound dies, and the app is healthy after it" do + code = "local t = {} local i = 1 while true do t[i] = string.rep('x', 1000) i = i + 1 end" + + assert {:error, reason} = Sandbox.run(code, max_heap_words: 200_000, max_time_ms: 20_000) + + assert reason == :heap_exceeded or match?({:timeout, _}, reason), + "expected the heap limit or the clock to stop it, got #{inspect(reason)}" + + assert {:ok, [2], _} = Sandbox.run("return 1+1") + end + end + + describe "AC3: the machine is not reachable" do + test "the dangerous members of os are gone" do + for name <- ~w(execute exit getenv remove rename tmpname) do + assert {:error, {:lua_error, _}} = Sandbox.run("return os.#{name}(0)"), + "os.#{name} is callable inside the sandbox" + end + end + + test "io is gone wholesale" do + assert {:error, {:lua_error, _}} = Sandbox.run("return io.open(0)") + assert {:error, {:lua_error, _}} = Sandbox.run("return io.write(0)") + end + + test "the loaders are gone: require, load, loadstring, loadfile, dofile" do + for name <- ~w(require load loadstring loadfile dofile) do + assert {:error, {:lua_error, _}} = Sandbox.run("return #{name}(0)"), + "#{name} is callable inside the sandbox, so code can be brought in from outside it" + end + end + + test "package is gone, so no module path is exposed" do + # `package` is replaced by a sandboxed value, and indexing one yields nil rather than raising. + # So the assertion that means something is that no real path comes back, not that reading it + # errors: the first version of this test expected an error and was simply wrong about Lua. + assert {:ok, [nil], _} = Sandbox.run("return package.path") + assert {:error, {:lua_error, _}} = Sandbox.run("return package.loadlib(0, 0)") + end + + test "what survives is named rather than discovered: os.time and os.clock still work" do + # Not a hole, and not an accident either. These reach no resource, but they are + # non-deterministic, which matters when a script's output can reach a receipt. + assert {:ok, [_], _} = Sandbox.run("return os.time()") + assert {:ok, [_], _} = Sandbox.run("return os.clock()") + end + end +end diff --git a/test/trinity/sandbox/scanner_lua_test.exs b/test/trinity/sandbox/scanner_lua_test.exs new file mode 100644 index 0000000..83efaa2 --- /dev/null +++ b/test/trinity/sandbox/scanner_lua_test.exs @@ -0,0 +1,56 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.ScannerLuaTest do + @moduledoc """ + Slice 110: the skill scanner flags a Lua script reaching for what the sandbox removes. + + **This is a reviewer's signal, not a control.** The control is that the globals are absent at run + time, asserted one by one in `sandbox_test.exs`. A regular expression over source text can be got + around and would be a poor last line; its job is to make sure a person approving a skill is told + what its script is reaching for. + """ + use ExUnit.Case, async: true + + alias Trinity.Skills.Scanner + + defp rules(source), do: Scanner.scan_file("s.lua", source) |> Enum.map(& &1.rule) + + test "the rule is registered with its severity" do + assert {"sandbox_escape", "high"} in Scanner.rules() + end + + test "each removed global is flagged when called" do + for call <- [ + "os.execute('id')", + "os.exit(1)", + "os.getenv('HOME')", + "os.remove('/tmp/x')", + "io.open('/etc/passwd')", + "io.write('x')", + "require('socket')", + "loadstring('return 1')", + "dofile('/tmp/x.lua')", + "package.loadlib('a', 'b')" + ] do + assert "sandbox_escape" in rules("local x = #{call}"), + "#{call} did not raise the flag a reviewer needs" + end + end + + test "ordinary Lua is not flagged, because a rule that fires on everything is read by nobody" do + source = """ + local sum = 0 + for _, n in ipairs(args.numbers) do sum = sum + n end + trinity.log("counted " .. #args.numbers) + trinity.result({total = sum, when_ = os.time()}) + """ + + refute "sandbox_escape" in rules(source), + "os.time and the host API are what a legitimate script uses; flagging them would train " <> + "a reviewer to dismiss the rule" + end + + test "a mention without a call is not flagged" do + refute "sandbox_escape" in rules("-- this script does not use os.execute at all") + end +end diff --git a/test/trinity/sandbox/skill_test.exs b/test/trinity/sandbox/skill_test.exs new file mode 100644 index 0000000..3de0bd5 --- /dev/null +++ b/test/trinity/sandbox/skill_test.exs @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.SkillTest do + @moduledoc """ + Slice 110 AC5: a skill's `lua_entry` runs, and cannot reach outside its own directory. + + The path check is the half worth testing hardest. `lua_entry` comes from a skill's front matter, + and a skill can be proposed by the agent; slice 041 controls what gets installed, and this controls + what an installed one can reach. They are different questions and only the second is here. + """ + use Trinity.DataCase, async: false + @moduletag :capture_log + + alias Trinity.Sandbox.Skill, as: SandboxSkill + alias Trinity.Skills.Skill + alias Trinity.Tools.Context + + setup do + session = Trinity.Factory.session!() + scope = Trinity.Receipts.session_scope(session.id) + on_exit(fn -> Trinity.Receipts.stop_writer(scope) end) + {:ok, ctx: %Context{session_id: session.id, caller: session.id, call_id: "s"}} + end + + defp skill_with_script!(entry, source) do + dir = Path.join(System.tmp_dir!(), "skill-#{System.unique_integer([:positive])}") + File.mkdir_p!(Path.join(dir, "scripts")) + File.write!(Path.join([dir, "scripts", entry]), source) + on_exit(fn -> File.rm_rf(dir) end) + %Skill{name: "totals", path: dir, trinity: %{"lua_entry" => entry}} + end + + # The registry is a GenServer over the filesystem; these tests are about what happens once a skill + # is in hand, so the lookup is stubbed by calling the path resolution directly through a skill + # struct rather than by installing a fixture into the real registry. + defp run(skill, args, ctx) do + entry = skill.trinity["lua_entry"] + path = Path.join([skill.path, "scripts", entry]) + source = File.read!(path) + + preamble = + if args == %{}, + do: "local args = {}\n", + else: "local args = json.decode(#{inspect(Jason.encode!(args))})\n" + + Trinity.Sandbox.run(preamble <> source, context: ctx) + end + + test "a script computes from its args and declares a structured result", %{ctx: ctx} do + skill = + skill_with_script!("total.lua", """ + local sum = 0 + for _, n in ipairs(args.numbers) do sum = sum + n end + trinity.result({total = sum, count = #args.numbers}) + """) + + assert {:ok, declared, _stats} = run(skill, %{"numbers" => [1, 2, 3, 4]}, ctx) + assert {"total", 10} in declared + assert {"count", 4} in declared + end + + test "a script with no args still binds the name rather than erroring on nil", %{ctx: ctx} do + skill = skill_with_script!("none.lua", "trinity.result({ok = true, n = #args})") + assert {:ok, declared, _} = run(skill, %{}, ctx) + assert {"ok", true} in declared + end + + describe "the path is resolved, not trusted" do + # `script_path/2` is public and called directly. A test that re-implemented the check would be + # testing its own copy, which is the one way to make a security test meaningless. + test "an entry escaping the skill's scripts directory is refused by name" do + skill = %Skill{name: "bad", path: "/tmp/some-skill", trinity: %{}} + + assert {:error, {:script_outside_skill, "../../../etc/passwd"}} = + SandboxSkill.script_path(skill, "../../../etc/passwd") + end + + test "an absolute entry is neutralised into the skill's directory rather than escaping" do + # `Path.join/2` drops the leading slash, so `/etc/passwd` becomes `scripts/etc/passwd` and + # lands inside. Asserted rather than assumed, and asserted as an `:ok` rather than as a + # refusal, because the first version of this test expected a refusal and was wrong about what + # the code does. A test that states the wrong safe behaviour is worth less than no test: it + # would go red the day someone made it refuse properly. + skill = %Skill{name: "abs", path: "/tmp/some-skill", trinity: %{}} + assert {:ok, path} = SandboxSkill.script_path(skill, "/etc/passwd") + assert path == "/tmp/some-skill/scripts/etc/passwd" + refute path == "/etc/passwd" + end + + test "a plain nested entry inside scripts/ is allowed" do + skill = %Skill{name: "ok", path: "/tmp/some-skill", trinity: %{}} + assert {:ok, path} = SandboxSkill.script_path(skill, "lib/helper.lua") + assert String.ends_with?(path, "/scripts/lib/helper.lua") + end + + test "a skill with no path on disk is refused rather than resolved against nothing" do + skill = %Skill{name: "pathless", path: nil, trinity: %{}} + assert {:error, {:skill_has_no_path, "pathless"}} = SandboxSkill.script_path(skill, "a.lua") + end + end + + test "an unknown skill is a named refusal", %{ctx: ctx} do + assert {:error, {:unknown_skill, "no-such-skill"}} = + SandboxSkill.run("no-such-skill", %{}, ctx) + end +end diff --git a/test/trinity/sandbox/tool_test.exs b/test/trinity/sandbox/tool_test.exs new file mode 100644 index 0000000..f682e43 --- /dev/null +++ b/test/trinity/sandbox/tool_test.exs @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Sandbox.ToolTest do + @moduledoc """ + Slice 110: `run_lua` as the model sees it. + + The behaviour worth pinning is that a **refused run is a result, not a tool error**. A model that + asks a legitimate question and gets an error ends its turn; one that gets "the script did not + finish in 200 ms" writes a smaller script. The distinction is the difference between a sandbox + that is usable and one that is merely safe. + """ + use Trinity.DataCase, async: false + @moduletag :capture_log + + alias Trinity.Sandbox.Tool + alias Trinity.Tools.Context + + setup do + session = Trinity.Factory.session!() + scope = Trinity.Receipts.session_scope(session.id) + on_exit(fn -> Trinity.Receipts.stop_writer(scope) end) + {:ok, ctx: %Context{session_id: session.id, caller: session.id, call_id: "c"}} + end + + test "the tier is exec and the effect is none, which are different questions" do + assert Tool.risk() == :exec + + assert Tool.effect() == :none, + "the script itself reaches nothing; what it wants done it asks for through " <> + "trinity.tool, which the gate decides again under that tool's own tier" + end + + test "it is registered under its name and the registry admits it", %{ctx: _} do + assert {:ok, %{name: "run_lua", risk: :exec, effect: :none}} = Trinity.Tools.lookup("run_lua") + end + + test "a computation comes back as text with its statistics in the meta", %{ctx: ctx} do + code = "local s = 0 for i = 1, 100 do s = s + i end return s" + assert {:ok, result} = Tool.execute(%{"code" => code}, ctx) + assert result.content =~ "5050" + assert result.meta["reductions"] > 0 + assert is_integer(result.meta["time_ms"]) + end + + test "log lines are appended under a heading rather than lost", %{ctx: ctx} do + code = ~s|trinity.log("counting") trinity.log("done") return 1| + assert {:ok, result} = Tool.execute(%{"code" => code}, ctx) + assert result.content =~ "-- log --" + assert result.content =~ "counting" + assert result.meta["log_lines"] == 2 + end + + test "a run that does not finish is a result explaining why, not a tool error", %{ctx: ctx} do + assert {:ok, result} = + Tool.execute(%{"code" => "while true do end", "max_time_ms" => 150}, ctx) + + assert result.content =~ "did not finish within 150 ms" + assert result.meta["refused"] == true + end + + test "a script longer than the cap is refused before it runs", %{ctx: ctx} do + huge = String.duplicate("-- padding\n", 3_000) + assert {:error, {:script_too_long, _, _}} = Tool.execute(%{"code" => huge}, ctx) + end + + test "a Lua error explains itself rather than ending the turn", %{ctx: ctx} do + assert {:ok, result} = Tool.execute(%{"code" => "error('deliberate')"}, ctx) + assert result.content =~ "raised" + assert result.meta["refused"] == true + end + + test "the result is marked untrusted, like any other tool result", %{ctx: ctx} do + assert {:ok, result} = Tool.execute(%{"code" => "return 1"}, ctx) + + # The mark lives on the parts, not on the result: `Trinity.Tools.Untrusted.result/2` wraps the + # text into one part and that is what carries the provenance into the prompt. + assert [part] = result.parts + + assert part.taint == :untrusted, + "a script's output can be derived from a file or a page, so it re-enters the prompt as " <> + "data rather than as command: #{inspect(part)}" + + assert part.source_ref == "sandbox" + end +end diff --git a/test/trinity/tools/registry_test.exs b/test/trinity/tools/registry_test.exs index 5750b88..16eac69 100644 --- a/test/trinity/tools/registry_test.exs +++ b/test/trinity/tools/registry_test.exs @@ -34,6 +34,7 @@ defmodule Trinity.Tools.RegistryTest do "learn", "memory", "recall", + "run_lua", "session_search", "shell", "skill_file", @@ -78,6 +79,7 @@ defmodule Trinity.Tools.RegistryTest do "learn", "memory", "recall", + "run_lua", "session_search", "shell", "skill_file", diff --git a/test/trinity_web/live/compaction_live_test.exs b/test/trinity_web/live/compaction_live_test.exs index 2237e80..7fd3773 100644 --- a/test/trinity_web/live/compaction_live_test.exs +++ b/test/trinity_web/live/compaction_live_test.exs @@ -25,7 +25,10 @@ defmodule TrinityWeb.CompactionLiveTest do id: id } do {:ok, view, _} = live(conn, ~p"/s/#{id}") - assert has_element?(view, "#context[data-window='6000']") + # The window comes from the registry rather than being typed here, so resizing the test + # model does not silently break an assertion about something else. + window = Trinity.Memory.Tokens.context_tokens("fake:chat") + assert has_element?(view, "#context[data-window='#{window}']") [used] = Regex.run(~r/data-used="(\d+)"/, render(view), capture: :all_but_first) before = String.to_integer(used) Fake.script(script_deltas(20, "word ")) @@ -38,7 +41,7 @@ defmodule TrinityWeb.CompactionLiveTest do _ = collect(id, &match?({:assistant_message, _}, &1), 5_000) [after_used] = Regex.run(~r/data-used="(\d+)"/, render(view), capture: :all_but_first) assert String.to_integer(after_used) > before - assert render(view) =~ "context #{after_used} / 6000" + assert render(view) =~ "context #{after_used} / #{window}" end test "a compaction row renders as a card naming its range; the originals stay on the page", %{