Repository navigation
feat: a regulated profile that refuses to boot on a configuration it cannot support - #110
Merged
Merged
Conversation
…cannot support
Five refusals, gated on TRINITY_PROFILE=regulated, or :trinity, :profile set to
:regulated. Under the default profile nothing here does anything: every check
takes the profile as its first argument and answers :ok for :default without
looking at the value.
AC1 the MCP authorization profile must be :production
AC2 the personal issuer must not be the one in force
AC3 every model must name an endpoint on TRINITY_REGULATED_LLM_ENDPOINTS
AC4 receipts that cannot append at boot stop the node
AC5 the authority in force must not be Trinity.Authority.Local
AC5 supplies no adapter and invents none. If TRINITY_AUTHORITY is unset or
resolves to Local, the boot is refused and the operator has to supply one. The
refusal is the point.
AC3 refuses a model with no base_url rather than letting it through. Such a
model goes to its provider's default endpoint, which the configuration does not
state, and an endpoint that is not stated cannot be shown to be on the list.
Trinity.Profile is pure. Nothing in it starts, stops or touches a process, and
it reads the environment in two named functions only. The caller reads the world
and hands the values in, so every refusal is unit testable and the Trinity
boundary grows no dependency on Authority or Receipts to ask a question about a
value.
Two things deliberately not changed, because the owner's rules forbade it and
the rules were right:
- lib/trinity/receipts/supervisor.ex keeps its fail-open shape for every profile
that is not :regulated. A missing signer still logs, sets the alarm, and lets
the tree start, because every effect is denied until a signer returns. AC4 is
a separate call that :default never makes.
- lib/trinity/mcp/boot.ex keeps its rescue. AC2 is decided before any child
starts, which is before MCP.Boot runs, rather than by narrowing a rescue so a
warning could become a refusal.
Neither file appears in this diff.
One export added: Trinity exported Receipts.KeyCustody already but not
Authority.Selection, and the application needs the selection RULE before the
selection PROCESS exists. Adding both boundaries to the application's deps was
the first attempt and the compiler refused it, correctly.
The mistake worth recording: the first version of AC4's tests called
KeyCustody.boot!/1 from an async file. That function writes its answer to
persistent_term on both paths including the failure, so inducing a failure
poisoned the signer for the whole run and 48 unrelated tests failed with
{:signer_unavailable, {:key_file, :enotdir}}. Moved to its own async: false
file, which ExUnit runs only after every async module has finished, restoring
the selection in on_exit.
Signed-off-by: Ayla Croft <aylacroft@proton.me>
The profile suite proves the decisions Trinity.Profile makes. It does not prove a
boot, and start/2 cannot be re-run in a VM where the application is already up,
so calling that suite AC1 to AC5 proves the easy half.
Four cases, each a separate OS process with its own environment:
1. default + a broken key store -> starts. This is rule 1 as a test: the
receipts supervisor stays fail-open for every profile that is not
:regulated, and if this case ever goes red that shape has been changed.
2. regulated + production mcp_auth + a non-Local authority + no endpoint
allow-list -> does not start, naming regulated_llm_endpoints_unset.
3. the same, with the authority left to resolve to Local -> does not start,
naming regulated_refuses_local_authority.
4. every condition met -> starts, then stops. Without this case, 2 and 3 pass
for a profile that refuses everything and prove nothing.
Isolation: XDG_DATA_HOME per case, so each child gets its own data directory,
lock and database files. MIX_ENV=dev rather than test, because the test
environment puts both repos on the Ecto sandbox and a child with no owner blocks
on checkout instead of booting. mix run --no-start, so the script can set
application environment and define the non-Local authority fixture before the
code under test runs; that fixture stays in the child rather than going into
lib/, because no adapter is supplied by this tree.
Nothing here writes :persistent_term in this VM, and there is no collect/3.
Verified by planting three violations and watching the right case go red:
returning :ok from check_receipts/2 under :default changed nothing, which is
rule 2 holding; running the receipts check for every profile turned case 1 red;
short-circuiting the endpoint check turned case 2 red; short-circuiting the
authority check turned case 3 red. lib/ is byte-identical to before the plants.
Signed-off-by: Ayla Croft <aylacroft@proton.me>
Case 5: TRINITY_PROFILE=regulated with TRINITY_MCP_AUTH_PROFILE set to personal
and then to local, everything else as case 4. Neither starts, and the printed
reason names regulated_requires_production_mcp_auth and the profile it refused.
The variable goes through config/runtime.exs rather than Application.put_env,
because that is the path an operator uses and the one that rejects a value
outside the three.
Case 6: case 4's configuration with case 1's broken key store. Does not start,
and the printed reason is
{:regulated_boot_refused, {:regulated_requires_receipts, {:key_file, :eisdir}}}.
Case 1 keeps passing, so the same key store still does not stop a default node,
which is the whole difference AC4 is about. This is also the first case to reach
the error branch of the receipts check at an actual boot: cases 1 to 4 never did,
so whether the reason survived Supervisor.stop/2 inside start/2 was untested
until now. It does.
Found while writing case 5: the report tail inspected the reason with limit: 3,
which printed BOOT_REFUSED {:trinity, {{...}, ...}} and hid it completely. The
earlier cases asserted the reason appeared somewhere in the captured output, and
matched the crash report OTP writes to stderr rather than the line the node
printed. Now inspected with limit: :infinity, and cases 5 and 6 assert against
the BOOT_REFUSED line alone via refusal_line/1.
Planted breaks, each reverted, lib/ left byte-identical:
skipping the receipts gate entirely (verify_regulated_receipts) turned case 6
red and left case 1 green, which is the pair the profile exists to create.
making AC1's check_mcp_auth/2 always pass turned case 5 red, and showed that
:personal is still caught by AC2's check_embedded_as/2 with a different reason
(regulated_refuses_personal_issuer). Case 5 catches that only because it
asserts the specific reason rather than that a refusal happened.
Signed-off-by: Ayla Croft <aylacroft@proton.me>
Cases 2 and 3 asserted the reason appeared somewhere in the captured output.
With stderr_to_stdout: true that output also carries OTP's crash report, which
repeats the same words, so those assertions passed whether or not the node ever
printed a reason. They now go through refusal_line/1, as cases 5 and 6 already
did, and there is no unscoped match left on any refusal.
Proved by planting the original inspect limit back:
with limit: 3 the node prints BOOT_REFUSED {:trinity, {:bad_return, ...}} and
all four refusing cases go red, while cases 1 and 4 (which assert BOOT_OK)
stay green.
in the same configuration, run by hand, the reason is still present in the
captured output, on the crash report line:
** (RuntimeError) TRINITY_PROFILE=regulated refuses to boot:
{:regulated_llm_endpoints_unset, "TRINITY_REGULATED_LLM_ENDPOINTS"}
so the old assertion would have passed exactly where the new one fails. That
is the difference the change is for.
Signed-off-by: Ayla Croft <aylacroft@proton.me>
The FIPS leg went red on all six cases at 05f444d, and none of them was red because a refusal failed. The child runs `mix run` under MIX_ENV=dev; the leg has no prebuilt dev tree, so the child compiled the dev dependency tree from scratch and reached the network for the `tokenizers` precompiled NIF. OTP's TLS client cannot finish that download: it fails the HelloRetryRequest middlebox assertion (erlang/otp#8470, docs/fips-leg.md finding 2, owned by slice 002). The child died at `mix compile`, so every assertion about what a node booted was really an assertion about a failed download. The module now carries :needs_dev_compile, and test_helper.exs excludes that tag only when TRINITY_FIPS_LEG=1, which is the variable the fips job already sets. gate and postgres run all six cases unchanged. No workflow change was needed: the leg already declares itself and the helper already branches on it. MIX_ENV=test was tried first, as the preferred mechanism, and is written up in the moduledoc. It does not block on the Ecto sandbox: test_helper.exs never runs in the child, so the pool stays in :auto mode and the app boots. It was rejected for a different reason. config/test.exs fixes the database path and does not derive it from XDG_DATA_HOME, so six children write into the suite's own database; measured, a child appended its boot receipt at seq 360 on the live chain. Overriding both repos' paths and migrating inside the child fixes that on SQLite, measured at 1.45 s with 25 and 4 tables created and the boot receipt back at seq 1. It does not carry to the postgres job, where :database names a database rather than a file and each case would need one provisioned and dropped. That code is not written here because it could not be tested here. Nothing under lib/. No case deleted. No product change: this is a limit of the harness on one leg. Signed-off-by: Ayla Croft <aylacroft@proton.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five refusals, gated on
TRINITY_PROFILE=regulated(orconfig :trinity, :profile, :regulated). Under the default profile nothing here does anything: every check takes the profile as its first argument and answers:okfor:defaultwithout looking at the value.:productionTRINITY_REGULATED_LLM_ENDPOINTSTrinity.Authority.LocalAC5 supplies no adapter and invents none. If
TRINITY_AUTHORITYis unset or resolves toLocal, the boot is refused and the operator has to supply one. The refusal is the point.AC3 refuses a model with no
base_url. Such a model goes to its provider's default endpoint, which the configuration does not state, and an endpoint that is not stated cannot be shown to be on the list.The two files the rules protected, and they are not in the diff
lib/trinity/receipts/supervisor.exkeeps its fail-open shape for every profile that is not:regulated. A missing signer still logs, sets the alarm, and lets the tree start. AC4 is a separate call that:defaultnever makes.lib/trinity/mcp/boot.exkeeps its rescue. AC2 is decided before any child starts, which is beforeMCP.Bootruns, rather than by narrowing a rescue so a warning could become a refusal.The commit-msg hook confirms it independently: it reports both paths as named in the message but not touched.
Rule 5's pair, raw
Both use a real failure: the setup writes a file where a directory should be and calls
KeyCustody.boot!/1beneath it, getting{:error, {:key_file, :enotdir}}, and asserts it really failed before either test runs.The mistake that rule caught
The first version of those tests lived in an async file and called
KeyCustody.boot!/1directly. That function writes to:persistent_termon both paths, including{:unavailable, reason}on failure, so inducing a failure poisoned the signer for the whole run:48 unrelated failures from one test. Moved to its own
async: falsefile, which ExUnit runs only after every async module finishes, restoring the selection inon_exit.Design
Trinity.Profileis pure. Nothing in it starts, stops or touches a process, and it reads the environment in two named functions only. The caller reads the world and hands the values in, so every refusal is unit testable and theTrinityboundary grows no dependency onAuthorityorReceiptsto ask a question about a value.One export added:
Trinityalready exportedReceipts.KeyCustodybut notAuthority.Selection, and the application needs the selection rule before the selection process exists. Adding both boundaries to the application's deps was my first attempt and the compiler refused it, correctly.Housekeeping
No public
ROADMAP.mdrow: plan_check rule 4 iterates ROADMAP rows to slices and never the reverse, so a slice with no row is never visited. Branch is not namedslice/*, because rule 11 requiresslice/NNN-. No numeric id: the private ROADMAP reserved none.Three consecutive full runs, all 938, so the new synchronous file does not flake by ordering.
AC1 to AC5 at a boot, not at a function call
The first version of this branch proved the decisions in
Trinity.Profileand called that AC1 to AC5. That was the easy half. AC1 to AC5 are about a boot, andstart/2cannot be re-run in a VM where the application is already up, sotest/trinity/regulated_boot_node_test.exsspawns a separate OS process per case and asserts on what that process did.:default, key file is a directoryBOOT_OK, status 0):regulated, production mcp_auth, non-Local authority, no allow-listregulated_llm_endpoints_unset:regulated, production mcp_auth, allow-list set,TRINITY_AUTHORITYunsetregulated_refuses_local_authority:regulated, every condition metBOOT_OK, status 0)Case 1 is rule 1 written as a test:
lib/trinity/receipts/supervisor.exstays fail-open for every profile that is not:regulated, and if case 1 ever goes red that shape has been changed. Case 4 exists so that 2 and 3 cannot be satisfied by a profile that refuses everything.Isolation:
XDG_DATA_HOMEper case, so each child has its own data directory, lock and database files.MIX_ENV=devrather thantest, deliberately: the test environment puts both repos on the Ecto sandbox, and a child with no sandbox owner blocks on checkout instead of booting.mix run --no-start, so the script sets application environment and defines the non-Local authority fixture before the code under test runs. That fixture is defined inside the child script rather than added tolib/, because no adapter other thanLocalis supplied by this tree. Nothing writes:persistent_termin the suite VM, and there is nocollect/3.Run three times in a row, 4 passed each time.
The cases were checked against planted violations
A passing test proves nothing until it has been seen to fail for its own reason.
check_receipts(:default, {:error, _})returns an error:defaultthe application never calls it, which is rule 2 holding:regulatedcheck_llm_endpoints(:regulated, nil, _)returns:okthe node started with no endpoint allow-listcheck_authority(:regulated, Local)returns:okthe node started on the local authoritygit diff --stat -- lib/is empty after the plants were removed, so the reviewed code is the code that was tested.One earlier
mix gaterun reported 941/942 with a single failure that was not captured before the output scrolled; the two full runs after it, and three runs of the new file alone, are green. The failure is unidentified rather than explained, and it is recorded here as unidentified.lib/trinity/receipts/supervisor.exandlib/trinity/mcp/boot.exare untouched on this branch:git diff origin/main...HEAD -- lib/trinity/receipts/supervisor.ex lib/trinity/mcp/boot.exis empty.Cases 5 and 6
:regulated,TRINITY_MCP_AUTH_PROFILE=personal, thenlocal, otherwise case 4regulated_requires_production_mcp_authand the profile refusedregulated_boot_refused/regulated_requires_receipts/key_fileCase 5 sets the variable and lets
config/runtime.exsmap it, rather than callingApplication.put_env/3: that is the path an operator uses, and the one that rejects a value outside the three.Case 6 is the first case to reach the error branch of the receipts check at an actual boot. Cases 1 to 4 never did: case 1 is
:defaultso the check is not called, cases 2 and 3 fail earlier at configuration, and case 4 succeeds. Whether the reason survivedSupervisor.stop/2called from insidestart/2was therefore untested. It does:Case 1 still passes, so the same broken key store does not stop a
:defaultnode. That pair is the whole of AC4.A defect in the earlier cases, found while writing case 5
The report tail inspected the reason with
limit: 3. It printed:The reason was hidden completely. Cases 2 and 3 asserted that the reason appeared somewhere in the captured output, and what they matched was the crash report OTP writes to stderr, not the line the node printed. The assertion was true; it was not testing what it claimed to test. Now inspected with
limit: :infinity, and cases 5 and 6 assert against theBOOT_REFUSEDline alone throughrefusal_line/1.Planted breaks, each reverted
verify_regulated_receiptscheck_mcp_auth(:regulated, _)always returns:ok:personalis still caught by AC2'scheck_embedded_as/2, with the different reasonregulated_refuses_personal_issuer. Case 5 catches it only because it asserts the specific reason rather than that some refusal happened. Thelocaliteration did not run, so this plant does not isolate that halfgit diff --stat -- lib/empty after each revert.Only
test/trinity/regulated_boot_node_test.exschanged for these two cases.lib/is untouched by this commit.Every refusing case now asserts on the printed line
Cases 2 and 3 have been pointed at
refusal_line/1, as 5 and 6 already were. No unscopedout =~match is left on any refusal: cases 1 and 4 matchBOOT_OK, and 2, 3, 5 and 6 match the reason on theBOOT_REFUSEDline only.Proved by planting the original
limit: 3back into the report tail:limit: 3BOOT_REFUSED {:trinity, {:bad_return, ...}}BOOT_OK)And in that same configuration, run by hand, the reason is still in the captured output:
out =~ "regulated_llm_endpoints_unset"passes there.refusal_line(out) =~ "regulated_llm_endpoints_unset"fails. That is the whole difference, and it is now the one being asserted.Only
test/trinity/regulated_boot_node_test.exschanged: 13 insertions, 6 deletions.mix gateis red on this branch, and onmain, for an unrelated reasonmix deps.auditnow reports three advisories againstmint 1.10.1, published upstream since the last green run:mintis transitive, throughfinch.mix.exsandmix.lockare byte-identical tomainon this branch, so this is amaincondition and not something this PR introduced.mix hex.outdated mintsays 1.11.0 is available and satisfies finch's~> 1.8, so a lock bump looks sufficient, but that is a dependency change and this task was scoped to one test file. Not done, and flagged for your call.prod_check, credo and the suite all pass; the gate exits at the audit step before reaching them in a single run, which is why the numbers above come frommix teston its own.Not merged.