Merge pull request #94 from ScriptKittyOS/slice/004-property-based-testing

Slice 004: property-based testing, and a planted defect it finds

Author: HackTuah

CHANGELOG.md

@@ -24,6 +24,32 @@ evidence for each increment is retained by the maintainers and summarised here.
 
 ## 2026-09-24
 
+### `slice/004` — Property-based testing, and the assertions it enables
+The canonicalisers, the argument validator, the tighten-only state modifier and the receipt chain are
+now exercised by generated input rather than only by chosen input. That matters most where the code
+answers "is this the same thing": a fingerprint decides whether an approval still matches the call
+being made, and a definition digest decides whether a server's tool is the one you approved. Both are
+wrong in the same two ways, saying different about two spellings of one value or same about two
+different values, and an example test covers the spellings somebody thought of, which is exactly the
+set an attacker will not use.
+
+The suite proves it can fail. A defect is planted and committed rather than described: a
+canonicaliser that concatenates its fields with no separator, which is stable, order-independent,
+looks right, and quietly identifies two different definitions whenever a boundary moves between
+adjacent fields. The example-style assertions all pass on it; generated input finds the collision.
+
+Running the properties wider than the build does immediately found a defect in one of them. It
+passed at the hundred cases the build runs and failed at three thousand, because it filtered for an
+absent key instead of constructing one. A property that has only ever run at the build's width is a
+property nobody has tested, and the convention now says so along with the command and the timeout a
+wide run needs.
+
+Chosen over two larger slices on the published criteria rather than on preference: the Open Source
+Security Foundation's gold tier asks for dynamic analysis enabling many assertions, Elixir has no
+focused fuzzing library, and property-based testing is the recognised equivalent. Statement coverage
+is unchanged at 80.31% and this work does not claim otherwise, since properties exercise existing
+lines harder rather than reaching new ones.
+
 ### `slice/120` — Open-source hygiene and governance, audited
 Every one of the 613 files in this repository now carries copyright and licence information, and
 `reuse lint` exits 0 against version 3.3 of the REUSE Specification. It did not before: there was no

THIRD_PARTY_LICENSES.md

@@ -4,12 +4,19 @@
      hand: the gate regenerates it and fails when this file and the bill disagree. -->
 # Third-party licences
 
-132 components, derived from `sbom.cdx.json`, the CycloneDX bill this project's
+133 components, derived from `sbom.cdx.json`, the CycloneDX bill this project's
 quality gate generates on every commit. This file and that bill are the same data rendered
 twice, so they cannot disagree.
 
 Trinity itself is Apache-2.0 (`LICENSE`). Everything below is somebody else's work.
 
+**What this list covers.** Every component of the bill, which is every dependency the lock file
+resolves, including the ones that exist only to build and test this project and ship in no
+artifact. That is deliberate: a list of only the shipped subset would be the more flattering
+document and the less useful one, because a licence obligation can attach to a build-time
+dependency too. Which environment each one belongs to is in `mix.exs`; this file does not repeat
+it, and a reader should not infer from an entry here that the component is in a release.
+
 | Component | Version | Licence | Source of the licence |
 |---|---|---|---|
 | `abnf_parsec` | `2.1.0` | MIT | bom |
@@ -124,6 +131,7 @@ Trinity itself is Apache-2.0 (`LICENSE`). Everything below is somebody else's wo
 | `ssh` | `5.5.2.4` | Apache-2.0 | bom |
 | `ssl` | `11.6.0.4` | Apache-2.0 | bom |
 | `stdlib` | `7.3.0.1` | Apache-2.0 | bom |
+| `stream_data` | `1.4.0` | Apache-2.0 | bom |
 | `tailwind` | `0.5.1` | MIT | bom |
 | `telemetry` | `1.4.2` | Apache-2.0 | bom |
 | `telemetry_metrics` | `1.2.0` | Apache-2.0 | bom |

VERSIONS.md

@@ -90,6 +90,7 @@ never pin a version hex marks as retired or vulnerable.
 | `pgvector` | ~> 0.4.1 | ✅ in `mix.lock` | Vectors on the Postgres path: `memories.embedding_vector vector(384)` under an HNSW cosine index, searched by `Trinity.Memory.VectorStores.Pgvector` (Slice 032). The postgres job runs on the `pgvector/pgvector:pg17` image. Split from the postgrex row at Slice 010. |
 | `oban` | ~> 2.24 | ✅ in `mix.lock` | Uses `Oban.Engines.Lite` on SQLite and `Oban.Engines.Basic` on Postgres (the adapter is compile time, ADR-0002, so the engine is too). ⚠️ Oban Pro Workflows/Smart engine are Postgres-only. Added at Slice 050 (2.24.1 measured on hex.pm 2026-09-22, Apache-2.0). |
 | `sbom` | ~> 0.11 | ✅ in `mix.lock` | CycloneDX software bill of materials, generated by the gate from mix.lock (`mix sbom.cyclonedx`). Maintained by erlef, the Erlang Ecosystem Foundation; BSD-3-Clause, CC0-1.0, MIT and Apache-2.0; 0.11.0 measured on hex.pm 2026-09-23, released 2026-09-22. Dev and test only: it ships in no artifact, and the gate validates its output, so a defect in it fails the build rather than producing a quietly wrong bill. Added at Slice 002. |
+| `stream_data` | ~> 1.4 | ✅ in `mix.lock` | Property-based testing: generated input for `test/property`. Apache-2.0; 1.4.0 measured on hex.pm 2026-09-24, released 2026-07-14. Test only, so it ships in no artifact. Elixir has no focused fuzzing library and property-based testing is what the OpenSSF gold criterion `dynamic_analysis` accepts in its place, which is why this is a dependency rather than a habit. Added at Slice 004. |
 | `jose` | ~> 1.11 | ✅ in `mix.lock` | JOSE (JWK, JWS, JWKS) over OTP's crypto, for the MCP resource server's token validation and the personal profile's embedded authorization server. MIT; 1.11.12 measured on hex.pm 2026-09-22. Not joken, a layer over it this tree does not need. Added at Slice 062. |
 | `oban_web` | ~> 2.13 | ✅ in `mix.lock` | The Oban dashboard, a LiveView mounted at /oban (dev, and behind `config :trinity, :oban_web, true` elsewhere). Apache-2.0 on hex.pm since the 2.12 line (2.13.0 measured 2026-09-22); the plan of 2026-09-05 wrote it in when it was commercial. Added at Slice 050. |
 | `req` | ~> 0.5 | ✅ in `mix.lock` | HTTP client. |

config/test.exs

@@ -239,3 +239,19 @@ config :trinity, :mcp_client, backoff_ms: 50, max_backoff_ms: 400, connect_timeo
 config :trinity, Oban, testing: :manual
 # Slice 050: the dashboard route is mounted in the suite so its mount is a test.
 config :trinity, :oban_web, true
+
+# Slice 004: how many generated cases each property runs. The default of 100 is what the gate runs
+# on every commit, because the gate has to stay fast enough that nobody is tempted to skip it. A
+# property is only worth as much as the space it covered, so before trusting one, run it wider:
+#
+#     TRINITY_PROPERTY_RUNS=20000 mix test test/property --timeout 900000
+#
+# The `--timeout` is not optional at that width: ExUnit's per-test limit is 60 s and a property is
+# one test however many cases it runs, so a wide run without it fails with TimeoutError, which
+# looks exactly like a property violation in the summary and is not one. Measured: 20,000 runs of
+# the validator properties take about 356 s.
+#
+# ExUnit prints the seed on every run and `stream_data` shrinks a failure to a minimal case, so a
+# counter-example found at 20,000 runs is reproducible with that seed at any width.
+config :stream_data,
+  max_runs: String.to_integer(System.get_env("TRINITY_PROPERTY_RUNS") || "100")

coverage.tsv

@@ -30,3 +30,4 @@ slice_id	percent	sha	date
 028	80.31	48e9b6c	2026-09-24
 029	80.31	16e42ad	2026-09-24
 120	80.31	78ee950	2026-09-24
+004	80.31	91de943	2026-09-24

docs/03-conventions.md

@@ -235,3 +235,44 @@ a consistent template, numbered sequentially, status `proposed` / `accepted` / `
 - `docs/01-architecture.md` supervision tree and context table are updated in the slice that changes them.
 - `docs/05-data-model.md` is updated in the slice that adds a migration.
 - `VERSIONS.md` is updated in the slice that adds/updates a dependency.
+
+## Properties, and when to write one instead of an example (Slice 004)
+
+An example test states that one input gives one answer. A property states that a rule holds over a
+space of inputs, and `stream_data` generates from that space and shrinks any failure to the smallest
+case that still fails.
+
+**Write a property when the code answers "is this the same thing".** Canonicalisers, digests,
+fingerprints, parsers and validators are all of this shape, and they are wrong in two ways an
+example rarely reaches: saying *different* about two spellings of one value, and saying *same* about
+two different values. An example test picks the spellings somebody thought of, which is exactly the
+set an attacker will not use.
+
+**Write a property when the rule is about an order or a combination.** The tighten-only state
+modifier has to hold for every subset of states in every order; asserting that by example means
+asserting it for the orders someone listed.
+
+**Keep the example tests.** They document intent and they name the case that mattered. A property
+saying "no two definitions collide" does not tell a reader that the description is in the digest on
+purpose. Slices 028 and 029 have both, and the properties in `test/property` complement those rather
+than replacing them.
+
+**A property that has never failed proves less than it looks.** `test/property/finds_what_examples_miss_test.exs`
+plants the classic canonicaliser defect, concatenating fields with no separator, and shows the
+example-style assertions passing on it while a generated search finds a collision. Anything added to
+`test/property` should be able to say what it would have caught.
+
+**Running wider.** The gate runs 100 cases per property so that it stays fast enough that nobody
+skips it. Before trusting a property, run it wider:
+
+```
+TRINITY_PROPERTY_RUNS=20000 mix test test/property --timeout 900000
+```
+
+The `--timeout` is not optional at that width. ExUnit's per-test limit is 60 seconds and a property
+is one test however many cases it runs, so a wide run without it fails with `TimeoutError`, which in
+the summary looks exactly like a property violation and is not one. Measured while writing this:
+20,000 runs of the validator properties take about 356 seconds.
+
+ExUnit prints the seed and `stream_data` shrinks a failure to a minimal case, so a counter-example
+found at any width is reproducible.

lib/mix/tasks/trinity.third_party_licenses.ex

@@ -183,6 +183,13 @@ defmodule Mix.Tasks.Trinity.ThirdPartyLicenses do
 
     Trinity itself is Apache-2.0 (`LICENSE`). Everything below is somebody else's work.
 
+    **What this list covers.** Every component of the bill, which is every dependency the lock file
+    resolves, including the ones that exist only to build and test this project and ship in no
+    artifact. That is deliberate: a list of only the shipped subset would be the more flattering
+    document and the less useful one, because a licence obligation can attach to a build-time
+    dependency too. Which environment each one belongs to is in `mix.exs`; this file does not repeat
+    it, and a reader should not infer from an entry here that the component is in a release.
+
     | Component | Version | Licence | Source of the licence |
     |---|---|---|---|
     #{Enum.map_join(rows, "\n", &line/1)}

lib/trinity/versions.ex

@@ -160,6 +160,13 @@ defmodule Trinity.Versions do
       note:
         "CycloneDX software bill of materials, generated by the gate from mix.lock (`mix sbom.cyclonedx`). Maintained by erlef, the Erlang Ecosystem Foundation; BSD-3-Clause, CC0-1.0, MIT and Apache-2.0; 0.11.0 measured on hex.pm 2026-09-23, released 2026-09-22. Dev and test only: it ships in no artifact, and the gate validates its output, so a defect in it fails the build rather than producing a quietly wrong bill. Added at Slice 002."
     },
+    %{
+      name: "stream_data",
+      pin: "~> 1.4",
+      lock: "stream_data",
+      note:
+        "Property-based testing: generated input for `test/property`. Apache-2.0; 1.4.0 measured on hex.pm 2026-09-24, released 2026-07-14. Test only, so it ships in no artifact. Elixir has no focused fuzzing library and property-based testing is what the OpenSSF gold criterion `dynamic_analysis` accepts in its place, which is why this is a dependency rather than a habit. Added at Slice 004."
+    },
     %{
       name: "jose",
       pin: "~> 1.11",

mix.exs

@@ -219,6 +219,8 @@ defmodule Trinity.MixProject do
         # Slice 002: the CycloneDX bill of materials the gate generates. Dev and test only:
         # it produces a record about the build and ships in no artifact.
         {:sbom, "~> 0.11", only: [:dev, :test], runtime: false},
+        # Slice 004: generated input for the properties in test/property. Test only.
+        {:stream_data, "~> 1.4", only: [:test]},
         {:ex_doc, "~> 0.38", only: :dev, runtime: false},
         {:nimble_options, "~> 1.1"},
         # Slice 001 line 1, arm (a) recorded `only: :dev`. **Corrected at G4**, and the reason is

mix.lock

@@ -94,6 +94,7 @@
   "sourceror": {:hex, :sourceror, "1.12.3", "f58eebef0765c7a369a49a755ab2a5ee88d92777f403ea5a08b722cefcc37f51", [:mix], [], "hexpm", "d5f2f37099de794840f08c54ae546d7f6e4ea015e397be64aebe4996fa9f7da7"},
   "spitfire": {:hex, :spitfire, "0.4.2", "5c719208d4eeb810e5b2a2aa1024d1e4b2974b7ca422c274b2486666e5159735", [:mix], [], "hexpm", "9bbbbffe93e6f88ccf193487ef56b83c2646a6dc3875bb0975d4654bfe96c5bb"},
   "splode": {:hex, :splode, "0.3.2", "7716b6b2260a98a6f018c65cc0393da2cdf17314202eb351a0956e8762190dff", [:mix], [], "hexpm", "08fd658f80da7f1cd254b149164dcff8acd44b22f032001d5416b97223d32bc9"},
+  "stream_data": {:hex, :stream_data, "1.4.0", "026f929db613aabea6208012ae9b8970d3fd5f88b3bdf26831bc536f98c42036", [:mix], [], "hexpm", "2b0ee3a340dcce1c8cf6302a763ee757d1e01c54d6e16d9069062509d68b1dc9"},
   "tailwind": {:hex, :tailwind, "0.5.1", "35435b13158c90d37da11e1cfc808755fca1d7b6c5ab87b1b19c5de87e2f0a10", [:mix], [], "hexpm", "c4e26302a59fec72abc5610ecb6ad2116d9aa31f31aab2d4b8eb6e95d25a689c"},
   "telemetry": {:hex, :telemetry, "1.4.2", "a0cb522801dffb1c49fe6e30561badffc7b6d0e180db1300df759faa22062855", [:rebar3], [], "hexpm", "928f6495066506077862c0d1646609eed891a4326bee3126ba54b60af61febb1"},
   "telemetry_metrics": {:hex, :telemetry_metrics, "1.2.0", "7632c19c01d88d8aaca5da1a0e8912f5af39b79e7c08a2c253aeb3c14c2c957e", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "71dde12fc29b58b9c77ec17ec319109e5ca848d010fc1965ed4463bba1837c07"},