Repository navigation
[finding] measure-test-shard-timings: a file-sharded package bypasses the median merge rule — one run wins, and slices from different runs are summed as one measurement #16473
Description
scripts/measure-test-shard-timings.mjs merges repeated measurements of a package by median across summaries — that is its stated rule, it is spelled in provenance.mergeRule, and it is why feeding several runs is supposed to be more trustworthy than feeding one. For a file-sharded package (today: @objectstack/cli, the heaviest suite in the workspace) the median rule does not apply. That package gets exactly one sample no matter how many runs are fed, and the sample is whichever run was read last.
Where
buildDataset in scripts/measure-test-shard-timings.mjs. Slices are held back in a ledger keyed name -> sliceCount -> sliceIndex -> seconds, and the innermost value is overwritten by each successive summary that carries that index. The completed set is then summed and pushed into the median pool as ONE sample. Nothing keys the ledger by the run a summary came from, and a run summary carries no run identifier for it to key on.
Measured, with a repro
Driving the exported buildDataset directly with three runs' worth of slices for one sliced package, plus an unsliced control fed the same three values:
run A slices 100 + 100 = 200s
run B slices 300 + 300 = 600s
run C slices 500 + 500 = 1000s
sliced package = 1000 (median across the three runs would be 600)
unsliced control = 300 (median across the same three values, correct)
The control proves the median rule is alive and that the sliced path alone bypasses it.
The sharper half: a cross-run splice, unreported
The ledger's indices are overwritten independently, so a set completed from different runs is summed as though it were one measurement:
run B slices 1/2 = 300, 2/2 = 300
run C slice 1/2 = 500 (run C's 2/2 not fed, or fed earlier)
recorded whole-package weight = 800s
skippedIncompleteSlices = [] (empty — nothing warns)
800s is a duration no run observed. The file's own skippedIncompleteSlices guard does not fire, because from the ledger's point of view the set is complete — it just is not from one run.
Why this matters more than it looks
The script's header already names this exact hazard for the other direction ("a wrong number that reads exactly like a right one... n times too small") and refuses cache hits and incomplete slice sets to prevent it. The same hazard survives on the axis of which run, undefended, on the one package the slicing machinery exists for and the one whose mis-weighting killed a shard twelve times in a day.
It is also live for the current refresh procedure rather than hypothetical: the refresh ruling on #16173 directs feeding eighteen summaries from three fully green runs. Under that plan every ordinary package is a median of three observations and @objectstack/cli is a single observation from whichever run sorts last — the least robust reading for the package that most needs a robust one, with no line of output saying so.
Not a duplicate
Checked against the open cards on this dataset: #16173 (the stale dataset and its refresh), #16222 (the refresh path is unreachable from an agent container), #16464 (a scheduled refresh workflow — it feeds ONE run's six artifacts, so this defect is outside its completion scope), #16465 (the drift alarm). None of them touches the merge rule. Channel note: the repo-scoped REST search path answers 403 for this session ("sessions are bound to their configured repositories"), so the dedup sweep was one targeted MCP search_issues over this repo's open issues.
Shapes, not a ruling
- A. Key the ledger per input group. Treat each summary as belonging to a run and complete slice sets within a run, so a sliced package contributes one sample per run to the median pool. Needs a grouping key the summaries do not carry today; the caller knows it (the six files of one run are named and fetched together), so it could be passed in rather than inferred.
- B. Refuse the ambiguity. If more than one value is ever seen for the same package, slice count and slice index, fail loudly instead of taking the last. Cheapest, keeps the file honest, and turns a silent wrong number into a named refusal — at the cost of making multi-run refreshes of a sliced package impossible until A lands.
- C. Report it. Keep last-wins but name the sliced package in the generator's stderr summary and in the dataset, the way
skippedIncompleteSlicesalready names its case. Weakest: it documents the defect rather than removing it.
Filed unassigned by the domain:devx execution seat while working #16173; #16173 is not addressed by this card and remains open.
Generated by Claude Code
Activity
claude commentedon Sep 7, 2026
Claim: PM loop round 9 — flight C2a, maintainer-directed (「同意你的建议，你负责执行派发所有可行的优化」, 2026-09-07): the scheduled shard-timings refresh workflow, with #16473 folded in as its prerequisite — flight C1 (#16173) measured at 03:2xZ that the artifact host (
productionresultssa*.blob.core.windows.net) is denied to every agent container by egress policy, so the documented refresh cannot run from a dev worktree; the refresh therefore runs IN-RUNNER, where.turbo/runs/*.jsonalready sits, and this workflow's firstworkflow_dispatchis the refresh itself — the PR it opens is #16173's deliverable. Rider #16473:buildDatasetinscripts/measure-test-shard-timings.mjsbypasses its median rule for a file-sharded package (last run wins; slices from different runs summed into a weight no run measured) — fixed first, with self-test cases, so the refresh's@objectstack/cliweight is a median like every other package's.Session:
session_019RfFHiRCSs3JXLK4cwcfoxBranch:
claude/issue-16464-shard-timings-refresh-workflowWorktree:
objectstack-issue-16464Domain:
domain:devx(dispatched by the skills seat on the maintainer's direction)File surface:
.github/workflows/shard-timings-refresh.yml(new),scripts/measure-test-shard-timings.mjs(the slice ledger's merge rule + self-test), one helper underscripts/ci/if the pin update needs one;scripts/test-shard-timings.jsonis written only by the workflow run, never by hand. ⛔ci.ymluntouched; notimeout-minutes, matrix or bound change.Container & model: S,
mode:subagent, opus —dispatch-gates.mjs --tier: no path-derived mandate.Clause-②: no
Serial: no open PR touches the measure script or the JSON (C1 produced no PR); flights V and X hold
ci.yml, which this card does not touch. Not governed ⇒ in-seat review, then the seat flips ready and arms SQUASH auto-merge;skip-changeset. The PR carriesFixes #16464andFixes #16473; #16173 and #16222 areRefs(they close when the workflow's first refresh PR merges).Generated by Claude Code
claude commentedon Sep 7, 2026
Implemented as the prerequisite rider of flight C2a, on branch
claude/issue-16464-shard-timings-refresh-workflow. Draft PR: #16498 (it carries the closing keyword for this card). The fullos-dev-reportfor both cards is #16464 (comment); this is the pointer, so the fix is discoverable from the card that describes it.Shape taken: A, with B folded in as the guard that makes A's default safe. The card offered three. A alone (key the ledger per input group) leaves the ungrouped call resolving multiple runs by last-wins exactly as before; B alone (refuse any repeat) makes a multi-run refresh of a sliced package impossible, which the card names as its cost. Together they compose without either cost: slices are summed within a run, the per-run sums are medianed across runs, and a repeat within one declared group — which can only mean two runs sharing a group — is a named refusal with
--rungiven as the remedy. C was not taken: it documents the defect rather than removing it, and the dataset it would annotate is machine-generated input to a scheduling decision, not something a human reads before each split.The grouping key comes from the caller, as the card's shape A anticipated, because a run summary carries no run identifier to infer one from.
--runopens a group; summaries before any--runshare one implicit group, which is the ordinary single-run refresh the scheduled workflow performs.Measured, red then green. The red leg reproduces flight C1's scratch driver of the exported
buildDataset, and the numbers are the card's own:The
AFTERsplice row readscli in run C (missing 2/2)— the partial set is attributed to the run that could not assemble it, rather than silently completed from another run's slice.Also corrected:
provenance.mergeRulesaidmedian across summaries, which was only ever true of unsliced packages, since a sliced package's summaries are PARTS of one measurement rather than repeats of it. It now states both composed rules, andprovenance.runsrecords the declared ids.Seven self-test cases, battery floor 34 to 41, and the whole path exercised through the real CLI on synthetic summaries as well as through
buildDatasetdirectly.Load-bearing for #16464 rather than incidental: the refresh workflow feeds one run today, but the moment anyone feeds more than one — which the #16173 refresh recipe explicitly directs — the heaviest package in the workspace would have taken a single observation while every other package took a median, with no line of output saying so.
Generated by Claude Code
ci: accumulate runs until the workspace is covered, instead of huntin…
claude commentedon Sep 7, 2026
Pointer correction: the report link in my comment above now points at a superseded report. The current one is #16464 (comment) (status
needs_decision).Generated by Claude Code
claude commentedon Sep 7, 2026
Final pointer: the current report is #16464 (comment) (status
done), superseding the two before it. PR #16498 binds this card's closing keyword and is green on head94eeaa19b1.Nothing about this card changed across the three reports — it was repaired, self-tested and proven live in the first round. What changed is how load-bearing it turned out to be: the refresh lane was ruled to merge rather than replace, so it now feeds the generator several runs' summaries under per-run
--rungroups as a matter of course, and the live green run did exactly that with no ambiguity refusal. The per-run assembly this card added is not hygiene on a path nobody walks; it is the reason that path is correct.Generated by Claude Code
ci: regenerate the shard-timings dataset on a timer, and assemble a s…
fix(ci): accumulate shard-timings runs until every package has three …