Skip to content

Slice 004: property-based testing, and a planted defect it finds - #94

Merged
HackTuah merged 2 commits into
mainfrom
slice/004-property-based-testing
Sep 24, 2026
Merged

HackTuah merged 2 commits into
mainfrom
slice/004-property-based-testing

Conversation

@HackTuah

Copy link
Copy Markdown
Member

Generated input for the canonicalisers, the argument validator, the tighten-only state modifier and the receipt chain.

Chosen over slices 100 and 110 on the published criteria, not on preference. The OpenSSF gold tier asks for "a configuration for at least some dynamic analysis (such as testing or fuzzing) which enables many assertions"; Elixir has no focused fuzzing library and property-based testing is the recognised equivalent, which OpenSSF Scorecard also accepts as a fuzzing integration. Cheapest open gold criterion, and the only candidate with no acceptance criterion needing a person at a desktop (100 has nine).

Why these targets

Where code answers "is this the same thing" it is wrong in two ways: saying different about two spellings of one value, or same about two different values. A fingerprint decides whether an approval still matches the call being made; a definition digest decides whether a server's tool is the one you approved. An example test covers the spellings somebody thought of, which is exactly the set an attacker will not use.

The suite proves it can fail

test/property/finds_what_examples_miss_test.exs plants the classic defect — a canonicaliser concatenating fields with no separator — committed as a module rather than described as a patch someone once applied. It is stable, order-independent, looks right, and identifies two different definitions whenever a boundary moves between adjacent fields. The example-style assertions all pass on it; generated input finds the collision.

The chain property is separately mutation-tested: breaking its length assertion fails and shrinks to a one-element case.

Running wider found a defect in a property, immediately

"an added or removed argument changes the fingerprint" passed at the gate's 100 cases and failed at 3,000 with FilterTooNarrowError: it filtered for an absent key rather than constructing one. A property that has only ever run at the gate's width is a property nobody has tested, and the convention now says so.

At 20,000 cases six properties appear to fail and none does — all six are ExUnit.TimeoutError, because the per-test limit is 60s and a property is one test however many cases it runs. In the summary that is indistinguishable from six violated properties, so the documented wide run carries --timeout with the reason.

Slice 120's check caught this slice

Adding stream_data failed trinity.third_party_licenses --check until the published list was regenerated. That check is one day old and worked on its first real opportunity. It also showed the list did not say which build environments its components come from; the generated file now states its coverage.

Verification

mix gate
Result: 746 passed (14 properties, 732 tests), 18 excluded
trinity.coverage: 120 80.31% vs 029 80.31%: OK

Coverage is unchanged and this slice does not claim otherwise: properties exercise existing lines harder rather than reaching new ones. Gold's 90% bar is separate work.

🤖 Generated with Claude Code

The canonicalisers, the argument validator, the tighten-only state modifier and
the receipt chain are exercised by generated input rather than only by chosen
input. That matters most where the code answers is this the same thing: a
fingerprint decides whether an approval still matches the call being made, and a
definition digest decides whether a server's tool is the one the owner approved.
Both are wrong in the same two ways, saying different about two spellings of one
value or same about two different values, and an example test covers the spellings
somebody thought of, which is exactly the set an attacker will not use.

The suite proves it can fail. The defect is planted and committed rather than
described: a canonicaliser concatenating its fields with no separator, which is
stable, order-independent, looks right, and identifies two different definitions
whenever a boundary moves between adjacent fields. The example-style assertions
all pass on it; generated input finds the collision.

Running wider than the gate does immediately found a defect in one property. It
passed at the gate's hundred cases and failed at three thousand with
FilterTooNarrowError, because it filtered for an absent key rather than
constructing one. A property that has only ever run at the gate's width is a
property nobody has tested, and the convention now says so.

At twenty thousand cases six properties appear to fail and none does: they are
ExUnit timeouts, since the per-test limit is sixty seconds and a property is one
test however many cases it runs. In the summary a timeout and a violated property
look identical, so the documented wide run carries the timeout with the reason.

Chosen over slices 100 and 110 on the published criteria rather than on
preference. The OpenSSF gold tier asks for dynamic analysis enabling many
assertions; Elixir has no focused fuzzing library and property-based testing is
the recognised equivalent, which Scorecard also accepts. It is the cheapest open
gold criterion and the only candidate with no criterion needing a person at a
desktop.

Slice 120's own check caught this slice on its first real opportunity: adding the
dependency failed trinity.third_party_licenses --check until the published list
was regenerated. It also showed the list did not say which build environments its
components come from, which the generated file now states.

Statement coverage is unchanged at 80.31% against gold's 90%, and this slice does
not claim otherwise: properties exercise existing lines harder rather than
reaching new ones.

mix gate: 746 passed (14 properties, 732 tests), 18 excluded.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
@HackTuah
HackTuah merged commit 1a5d480 into main Sep 24, 2026
8 checks passed
@HackTuah
HackTuah deleted the slice/004-property-based-testing branch September 24, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant