Repository navigation
Slice 004: property-based testing, and a planted defect it finds - #94
Merged
Merged
Conversation
The canonicalisers, the argument validator, the tighten-only state modifier and the receipt chain are exercised by generated input rather than only by chosen input. That matters most where the code answers is this the same thing: a fingerprint decides whether an approval still matches the call being made, and a definition digest decides whether a server's tool is the one the owner approved. Both are wrong in the same two ways, saying different about two spellings of one value or same about two different values, and an example test covers the spellings somebody thought of, which is exactly the set an attacker will not use. The suite proves it can fail. The defect is planted and committed rather than described: a canonicaliser concatenating its fields with no separator, which is stable, order-independent, looks right, and identifies two different definitions whenever a boundary moves between adjacent fields. The example-style assertions all pass on it; generated input finds the collision. Running wider than the gate does immediately found a defect in one property. It passed at the gate's hundred cases and failed at three thousand with FilterTooNarrowError, because it filtered for an absent key rather than constructing one. A property that has only ever run at the gate's width is a property nobody has tested, and the convention now says so. At twenty thousand cases six properties appear to fail and none does: they are ExUnit timeouts, since the per-test limit is sixty seconds and a property is one test however many cases it runs. In the summary a timeout and a violated property look identical, so the documented wide run carries the timeout with the reason. Chosen over slices 100 and 110 on the published criteria rather than on preference. The OpenSSF gold tier asks for dynamic analysis enabling many assertions; Elixir has no focused fuzzing library and property-based testing is the recognised equivalent, which Scorecard also accepts. It is the cheapest open gold criterion and the only candidate with no criterion needing a person at a desktop. Slice 120's own check caught this slice on its first real opportunity: adding the dependency failed trinity.third_party_licenses --check until the published list was regenerated. It also showed the list did not say which build environments its components come from, which the generated file now states. Statement coverage is unchanged at 80.31% against gold's 90%, and this slice does not claim otherwise: properties exercise existing lines harder rather than reaching new ones. mix gate: 746 passed (14 properties, 732 tests), 18 excluded. Signed-off-by: Ayla Croft <aylacroft@proton.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated input for the canonicalisers, the argument validator, the tighten-only state modifier and the receipt chain.
Chosen over slices 100 and 110 on the published criteria, not on preference. The OpenSSF gold tier asks for "a configuration for at least some dynamic analysis (such as testing or fuzzing) which enables many assertions"; Elixir has no focused fuzzing library and property-based testing is the recognised equivalent, which OpenSSF Scorecard also accepts as a fuzzing integration. Cheapest open gold criterion, and the only candidate with no acceptance criterion needing a person at a desktop (100 has nine).
Why these targets
Where code answers "is this the same thing" it is wrong in two ways: saying different about two spellings of one value, or same about two different values. A fingerprint decides whether an approval still matches the call being made; a definition digest decides whether a server's tool is the one you approved. An example test covers the spellings somebody thought of, which is exactly the set an attacker will not use.
The suite proves it can fail
test/property/finds_what_examples_miss_test.exsplants the classic defect — a canonicaliser concatenating fields with no separator — committed as a module rather than described as a patch someone once applied. It is stable, order-independent, looks right, and identifies two different definitions whenever a boundary moves between adjacent fields. The example-style assertions all pass on it; generated input finds the collision.The chain property is separately mutation-tested: breaking its length assertion fails and shrinks to a one-element case.
Running wider found a defect in a property, immediately
"an added or removed argument changes the fingerprint"passed at the gate's 100 cases and failed at 3,000 withFilterTooNarrowError: it filtered for an absent key rather than constructing one. A property that has only ever run at the gate's width is a property nobody has tested, and the convention now says so.At 20,000 cases six properties appear to fail and none does — all six are
ExUnit.TimeoutError, because the per-test limit is 60s and a property is one test however many cases it runs. In the summary that is indistinguishable from six violated properties, so the documented wide run carries--timeoutwith the reason.Slice 120's check caught this slice
Adding
stream_datafailedtrinity.third_party_licenses --checkuntil the published list was regenerated. That check is one day old and worked on its first real opportunity. It also showed the list did not say which build environments its components come from; the generated file now states its coverage.Verification
Coverage is unchanged and this slice does not claim otherwise: properties exercise existing lines harder rather than reaching new ones. Gold's 90% bar is separate work.
🤖 Generated with Claude Code