Repository navigation
Close four gold criteria, and measure the two that were guesses - #78
Merged
Merged
Conversation
Four of the fourteen open gold criteria needed writing rather than building, and two more needed measuring rather than assuming. Taken together while the rest of the project is still being built out, because each is cheaper now than after another twenty slices of history. build_reproducible: met, for the release. scripts/check_reproducible.sh builds the release twice from a removed _build and compares every file. All 243 modules this project compiles are byte-identical. Three things are not, and the script separates them because they mean different things: releases/COOKIE is a per-release secret that is supposed to differ, and three modules from llm_db, nx and req_llm vary between builds in a way nothing here can fix. The intermediate _build tree is not reproducible and docs/packaging.md now says why: 24 of 243 modules differ in their Dbgi and ExCk chunks while the Code chunk is identical in all 243, ERL_COMPILER_OPTIONS=deterministic takes that to 18 and no further, and mix release strips exactly those chunks. The project already relied on this from the other direction, since CorePolicy hashes stripped beams for the same reason. test_branch_coverage80: not applicable, and measured rather than argued. The criterion is conditional on a FLOSS tool existing for the language. Every Elixir coverage tool wraps Erlang's cover, and cover:analyse/3 has no clause for branch: it raises function_clause. code_review_standards: a Code review section in docs/03-conventions.md saying how review is conducted, the seven things a reviewer must form an opinion on, and the six conditions a change must meet together. Most of it is about evidence rather than style, and it says plainly that approving something a reviewer has not understood is worse than declining to review it. small_tasks: a Good first tasks section in CONTRIBUTING.md. Every task in it came out of the tree rather than being invented to fill the list. security_review: docs/11-security-review.md, dated, scoped, with its method and its findings. Five resolved, each a defect a person found in something that was passing its tests at the time, and four open with what would close each. A review with no findings would be evidence that the review did not happen. Signed-off-by: Ayla Croft <aylacroft@proton.me>
HackTuah
force-pushed
the
docs/gold-criteria
branch
from
September 23, 2026 21:05
54bee76 to
d5d222d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourteen gold criteria were open. Four needed writing rather than building, and two I had been treating as unknowns turned out to be measurable. Taken now, while the rest of the project is still being built out, because each is cheaper today than after another twenty slices of history.
build_reproducible— met, for the releaseI had recorded this as not met. That was right about
_buildand wrong about the artifact../scripts/check_reproducible.shbuilds the release twice from a removed_buildand compares every file:All 243 modules this project compiles are byte-identical. The script separates the three things that are not, because they mean different things: the node cookie is a per-release secret that is supposed to differ (a build producing the same one every time would be a security defect wearing reproducibility as a disguise), and three dependency modules vary in a way nothing here can fix — the count itself varies between runs, which places the non-determinism in their compile-time code generation.
docs/packaging.mdrecords why_buildis different: 24 of 243 modules differ, entirely in theDbgiandExCkchunks, while theCodechunk is identical in all 243.ERL_COMPILER_OPTIONS=deterministictakes that to 18 and no further, because what remains is abstract code carrying map literals ordered by the atom table.mix releasestrips exactly those chunks — and the project already depended on that fact from the other direction, sinceTrinity.CorePolicyhashes stripped beams for the same reason.The script is deliberately not in
mix gate: two full production builds is minutes, and the property changes when the toolchain changes rather than when a line of code does. It exists so the claim can be re-run by whoever doubts it, which is the only thing that makes a reproducibility claim worth making.test_branch_coverage80— not applicable, and measured rather than arguedThe criterion is conditional on a FLOSS tool existing for the language. Every Elixir coverage tool — excoveralls, lcov_ex, coverex, chaps — wraps Erlang's
cover, andcover:analyse/3has no clause forbranch:code_review_standards— writtenA Code review section in
docs/03-conventions.md: how review is conducted, the seven things a reviewer must form an opinion on, and the six conditions a change must meet together. Most of it is about evidence rather than style, and it says plainly that approving something a reviewer has not understood is worse than declining to review it.small_tasks— writtenA Good first tasks section in
CONTRIBUTING.md. Every task in it came out of the tree rather than being invented to fill the list.security_review— performed and recordeddocs/11-security-review.md, dated, scoped, with its method and its findings. Five resolved, four open. Every resolved finding is a defect a person found in something that was passing its tests at the time — a release-only compile failure that reached a tag, pairing codes from a non-cryptographic generator, an architectural boundary the compiler had never enforced, workflow tokens broader than the work, and a bill of materials no consumer could validate.A review with no findings would be evidence that the review did not happen.
Evidence
mix gateexit 0; 576 passed, 18 excluded; coverage 80.31%.