docs(lessons): add lessons 86 and 87 on blind test scaffolding - #1129
Open
dekobon wants to merge 1 commit into
Open
docs(lessons): add lessons 86 and 87 on blind test scaffolding#1129dekobon wants to merge 1 commit into
dekobon wants to merge 1 commit into
Conversation
Two entries from the #1091 and #1054 fixes, plus a third sub-example for lesson 85 from the same investigation. Lesson 86: a test helper that canonicalises the value under test deletes that dimension from every caller at once. `check_ops` sorted `operators_str` and `operands_str` alongside the expectations from its first commit, so eighteen per-language tests could not observe that `Ops`'s vocabularies came off `HashMap` keys in per-instance-random order — while `bca ops` printed byte-different output on consecutive runs. Removing only the actual-side sort turned those same callers into ordering regression tests for free. Lesson 87: an assertion gets the review attention, but the selector decides what it is about. A connector test filtered group lines at three columns where groups sit six columns in, so it matched only the `metrics` header and asserted something true about that instead — it passed with every metric group rendering a dangling `|-`. Substring matching cannot work here at all, because a deeper rail ends with the shallower one. Every figure was verified against the tree rather than transcribed from the issues: the caller count is eighteen, not the "~20" issue #1091 records, and the 20 of 27 failing tests decomposes as those eighteen plus the two tests the fix added. Cites issues and PRs, never commit hashes, which this repository's rebase merges orphan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two lessons and one sub-example, from investigating #1091 and
#1054 — the two bug-labelled issues the last lessons pass left
uninvestigated.
Both turned out to be test-quality failures rather than the perf work
their titles suggest, and both describe ways a test suite can look
thorough while being structurally unable to see a defect.
Lesson 86 — a normalizing test helper blinds every caller at once
check_opssortedoperators_strandoperands_str— the valuesreturned by the code under test — alongside the expectations, from the
helper's first commit.
Ops::operatorsandOps::operandswere builtfrom
HashMapkeys and Rust'sRandomStatereseeds per map instance,so the order differed between two parses in the same process.
The consequence reached users:
bca opsprinted byte-different outputfor an unchanged input on consecutive runs. Eighteen per-language
tests parsed real source and compared real vocabularies, and not one
could observe it. The fix sorts in production and removes only the
actual-side sort, which turned those same eighteen callers into
ordering regression tests without writing one.
Lesson 87 — an assertion can be correct and still be about the wrong rows
last_emitted_metric_group_uses_closing_connectorselected group linesat three columns while metric groups sit six columns in. It matched
exactly one line — the
metricsheader — and its closing-connectorassertion was a true statement about that header. The test passed with
every metric group rendering a dangling
|-, which is the precisedefect its name claims to prevent.
The same fix records why
containscannot work on this output at all:a deeper rail ends with the shallower one, and
sum/valuerecuracross groups, so a substring search accepts exactly the
mis-indentation under test.
Lesson 85 — third sub-example
cyclomatic.modifiedis the only nested metric object, sodump_object's per-field truncate is the only thing keeping the fieldsthat follow a nested object on the right rail. Removing it mis-indents
cyclomatic's trailingsum/valuein every dump — with all 2,869lib tests still passing. An independent instance of lesson 85's thesis
from a different subsystem.
Verification
Reviewed before commit, which caught two false claims in the first
draft — both now fixed and re-verified against the tree:
The draft had dropped the tilde and asserted it flatly, which is
exactly the count-rot lesson 84 warns about.
the same commit, and the 20-of-27 figure decomposes as eighteen
callers plus those two.
Six further imprecisions were corrected: the ambiguous
sort_unstablereferent, the unreproducible
27denominator (now scoped to the testsunder
opsandoutput::dump_opsas they stood then), an overstated"deletes a property from the contract" (the order was documented as
arbitrary, so the helper accommodated a sanctioned non-guarantee), an
overstated equivalence with lesson 23, the undated
2,869, and atruncate description that implied a dedicated line.
Every figure in both entries was checked against the repository rather
than transcribed from the issue bodies. Entries cite issues and PRs,
never commit hashes — this repository rebase-merges, so branch hashes
are orphaned on landing.
rumdl checkpasses; the change is purelyadditive (two new entries appended, one sub-example inserted), so no
existing lesson was renumbered and the number-based references in
.claude/skills/and.claude/rules/are unaffected.