Skip to content

feat(conformance): publish the report-only corpus in the summary - #836

Merged
hhimanshu merged 1 commit into
mainfrom
feat/835-publish-tracked-corpus
Aug 5, 2026
Merged

feat(conformance): publish the report-only corpus in the summary#836
hhimanshu merged 1 commit into
mainfrom
feat/835-publish-tracked-corpus

Conversation

@hhimanshu

@hhimanshu hhimanshu commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Closes #835.

conformance-summary.json published only the enforced corpus. The report-only corpus went to the test log and nowhere else — so anything downstream that wanted the whole picture had to hard-code figures read out of local test output. That is already happening on the marketing site, and it goes stale the moment someone fixes a gap, silently, with nothing to catch it.

It is now published as a sibling of overall:

"overall":  { "passed": 10646, "total": 10649 },
"tracked":  { "passed": 2025,  "total": 2236  }

Why not fold it into overall

Because overall is what the README badge asserts — the enforced suite passes — and folding these rows in would take that badge from 100.0% to 98.3% as a side effect of a refactor. That may well be a change worth making, but it should be a decision, not something that falls out of a plumbing PR. A consumer that wants the full picture adds the two numbers; the point here is only that both come from one file.

Say the word and I'll do the badge change as a one-line follow-up.

Why the collector, not the runner

The two paths count differently. The report-only runner prints 2055/2235; the collector gives 2025/2236, because the runner evaluates volatile formulas against a pinned clock where the collector skips them.

overall is collector-derived — its 10646/10649 matches the published badge exactly — so tracked has to be too. Publishing a runner-derived figure next to a collector-derived one would be mixing methods in a single object, and any consumer adding them would get a number that means nothing.

Verification

  • Full suite green: 21 passed, 0 failed.
  • overall byte-identical to before, so the badge and the published report page are untouched.
  • Simulated the CI summary step against the regenerated report: overall 10646/10649, tracked 2025/2236, combined 12671/12885 = 98.34%.
  • tracked is Option, and the CI step is guarded, so a report without it still produces a valid summary.

Follow-up on my side once this lands: the marketing site drops its hard-coded constant and reads this file.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

conformance-summary.json published only the enforced corpus. The
report-only corpus went to the test log and nowhere else, so anything
downstream that wanted the whole picture had to hard-code figures read
out of local test output — which goes stale the moment someone fixes a
gap, silently, with nothing to catch it.

It is now collected into its own bucket and published as a sibling of
`overall`:

  "overall": { "passed": 10646, "total": 10649 },
  "tracked": { "passed": 2025,  "total": 2236  }

Deliberately not folded into `overall`. `overall` is what the README
badge asserts — the enforced suite passes — and folding these rows in
would take that badge from 100.0% to 98.3% as a side effect of a
refactor. A consumer that wants everything adds the two.

Collected through the same path as every enforced category, so the two
halves are comparable. The report-only runner's console figure differs
because it evaluates volatile formulas against a pinned clock where the
collector skips them; publishing that number beside a collector-derived
`overall` would be mixing methods.

Closes #835

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hhimanshu hhimanshu self-assigned this Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Test Coverage by Category

Category Unit Tests Google Sheets Conformance Property Cases Total
Array 42 552/552 ✓ 1,000 (2×500) 1,594
Database 35 182/182 ✓ 3,500 (7×500) 3,717
Date 369 418/418 ✓ 2,500 (5×500) 3,287
Engineering 245 886/888 ⚠ 5,500 (11×500) 6,633
Filter 11 81/81 ✓ 4,500 (9×500) 4,592
Financial 149 1,208/1,208 ✓ 2,000 (4×500) 3,357
Info 0 256/256 ✓ 4,500 (9×500) 4,756
Logical 121 267/267 ✓ 3,500 (7×500) 3,888
Lookup 69 393/393 ✓ 1,000 (2×500) 1,462
Math 536 2,006/2,006 ✓ 8,000 (16×500) 10,542
Operator 87 251/251 ✓ 7,500 (15×500) 7,838
Parser 83 93/93 ✓ 4,000 (8×500) 4,176
Query 37 37
Statistical 516 3,191/3,191 ✓ 5,000 (10×500) 8,707
Text 309 803/804 ⚠ 4,000 (8×500) 5,113
Timezone 47 47
Volatile 0 3,500 (7×500) 3,500
Web 29 59/59 ✓ 6,000 (12×500) 6,088
Total 2,941 10,646/10,649 66,000 (132×500) ~79,590

✓ = 100% passing · ⚠ = known deviation · The ~79,590 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 3,797 Rust test functions: 2,941 unit + 159 property functions (shown as cases above) + 697 conformance/integration.

@hhimanshu
hhimanshu merged commit 3201dae into main Aug 5, 2026
9 checks passed
@hhimanshu
hhimanshu deleted the feat/835-publish-tracked-corpus branch August 5, 2026 20:18
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish the report-only corpus in conformance-summary.json

1 participant