Skip to content

Close four gold criteria, and measure the two that were guesses - #78

Merged
HackTuah merged 1 commit into
mainfrom
docs/gold-criteria
Sep 23, 2026
Merged

HackTuah merged 1 commit into
mainfrom
docs/gold-criteria

Conversation

@HackTuah

Copy link
Copy Markdown
Member

Fourteen gold criteria were open. Four needed writing rather than building, and two I had been treating as unknowns turned out to be measurable. Taken now, while the rest of the project is still being built out, because each is cheaper today than after another twenty slices of history.

build_reproducible — met, for the release

I had recorded this as not met. That was right about _build and wrong about the artifact.

./scripts/check_reproducible.sh builds the release twice from a removed _build and compares every file:

files in the release: 4561
files differing:      4

expected, by design:
  releases/COOKIE  (a per-release secret; a fixed one would be the defect)

non-deterministic dependencies (upstream's, not this project's):
  lib/llm_db-2026.9.4/ebin/Elixir.LLMDB.Model.beam
  lib/nx-0.13.1/ebin/Elixir.Nx.Defn.Kernel.beam
  lib/req_llm-1.24.0/ebin/Elixir.ReqLLM.beam

check_reproducible: PASS - every module this project compiles is bit-for-bit identical

All 243 modules this project compiles are byte-identical. The script separates the three things that are not, because they mean different things: the node cookie is a per-release secret that is supposed to differ (a build producing the same one every time would be a security defect wearing reproducibility as a disguise), and three dependency modules vary in a way nothing here can fix — the count itself varies between runs, which places the non-determinism in their compile-time code generation.

docs/packaging.md records why _build is different: 24 of 243 modules differ, entirely in the Dbgi and ExCk chunks, while the Code chunk is identical in all 243. ERL_COMPILER_OPTIONS=deterministic takes that to 18 and no further, because what remains is abstract code carrying map literals ordered by the atom table. mix release strips exactly those chunks — and the project already depended on that fact from the other direction, since Trinity.CorePolicy hashes stripped beams for the same reason.

The script is deliberately not in mix gate: two full production builds is minutes, and the property changes when the toolchain changes rather than when a line of code does. It exists so the claim can be re-run by whoever doubts it, which is the only thing that makes a reproducibility claim worth making.

test_branch_coverage80 — not applicable, and measured rather than argued

The criterion is conditional on a FLOSS tool existing for the language. Every Elixir coverage tool — excoveralls, lcov_ex, coverex, chaps — wraps Erlang's cover, and cover:analyse/3 has no clause for branch:

branch -> {'EXIT',{function_clause,[{cover,analyse,[nonexistent,coverage,branch]...
line   -> {ok,[]}

code_review_standards — written

A Code review section in docs/03-conventions.md: how review is conducted, the seven things a reviewer must form an opinion on, and the six conditions a change must meet together. Most of it is about evidence rather than style, and it says plainly that approving something a reviewer has not understood is worse than declining to review it.

small_tasks — written

A Good first tasks section in CONTRIBUTING.md. Every task in it came out of the tree rather than being invented to fill the list.

security_review — performed and recorded

docs/11-security-review.md, dated, scoped, with its method and its findings. Five resolved, four open. Every resolved finding is a defect a person found in something that was passing its tests at the time — a release-only compile failure that reached a tag, pairing codes from a non-cryptographic generator, an architectural boundary the compiler had never enforced, workflow tokens broader than the work, and a bill of materials no consumer could validate.

A review with no findings would be evidence that the review did not happen.

Evidence

mix gate exit 0; 576 passed, 18 excluded; coverage 80.31%.

Four of the fourteen open gold criteria needed writing rather than building, and
two more needed measuring rather than assuming. Taken together while the rest of
the project is still being built out, because each is cheaper now than after
another twenty slices of history.

build_reproducible: met, for the release. scripts/check_reproducible.sh builds
the release twice from a removed _build and compares every file. All 243 modules
this project compiles are byte-identical. Three things are not, and the script
separates them because they mean different things: releases/COOKIE is a
per-release secret that is supposed to differ, and three modules from llm_db, nx
and req_llm vary between builds in a way nothing here can fix. The intermediate
_build tree is not reproducible and docs/packaging.md now says why: 24 of 243
modules differ in their Dbgi and ExCk chunks while the Code chunk is identical
in all 243, ERL_COMPILER_OPTIONS=deterministic takes that to 18 and no further,
and mix release strips exactly those chunks. The project already relied on this
from the other direction, since CorePolicy hashes stripped beams for the same
reason.

test_branch_coverage80: not applicable, and measured rather than argued. The
criterion is conditional on a FLOSS tool existing for the language. Every Elixir
coverage tool wraps Erlang's cover, and cover:analyse/3 has no clause for
branch: it raises function_clause.

code_review_standards: a Code review section in docs/03-conventions.md saying
how review is conducted, the seven things a reviewer must form an opinion on,
and the six conditions a change must meet together. Most of it is about evidence
rather than style, and it says plainly that approving something a reviewer has
not understood is worse than declining to review it.

small_tasks: a Good first tasks section in CONTRIBUTING.md. Every task in it
came out of the tree rather than being invented to fill the list.

security_review: docs/11-security-review.md, dated, scoped, with its method and
its findings. Five resolved, each a defect a person found in something that was
passing its tests at the time, and four open with what would close each. A
review with no findings would be evidence that the review did not happen.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
@HackTuah
HackTuah merged commit e14b669 into main Sep 23, 2026
12 checks passed
@HackTuah
HackTuah deleted the docs/gold-criteria branch September 23, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant