Repository navigation
cargo-test: perry-runtime's suite fails a different number of tests on every run #7365
Description
Summary
cargo test -p perry-runtime --lib fails a different number of tests on every run, on clean main, with no changes applied. It is a required check (cargo-test), so this is not cosmetic: a suite whose failure count is nondeterministic can hide a real regression indefinitely — a genuine new failure looks exactly like the noise.
Measurement
Ten consecutive runs on 8f05d65f9, unmodified:
3 failed 1 failed 1 failed 3 failed
4 failed 2 failed 2 failed (earlier baseline, same tree)
Seven distinct tests are involved, by frequency across six sampled runs:
| count | test |
|---|---|
| 5 | gc::tests::teardown::map_set_owner_records_follow_growth |
| 3 | gc::tests::teardown::map_set_side_allocations_release_exactly_once |
| 2 | promise::keyed_table::tests::settling_many_keys_is_not_quadratic |
| 1 | url::node_compat::tests::path_to_file_url_posix_preserves_relative_trailing_slash |
| 1 | url::node_compat::tests::path_to_file_url_posix_does_not_add_slash_without_input_slash |
| 1 | object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush |
| 1 | gc::tests::teardown::map_set_side_allocations_release_on_thread_exit |
Every one of them passes in isolation. Verified individually:
gc::tests::teardown::map_set_owner_records_follow_growth ok
gc::tests::teardown::map_set_side_allocations_release_exactly_once ok
gc::tests::teardown::map_set_side_allocations_release_on_thread_exit ok
object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush ok
So it is interference under parallel execution, not a defect in any individual test.
Two distinct causes, probably
1. Shared process-global state. The gc::tests::teardown::map_set_* family is 9 of the 14 observed failures. Those tests assert on side-table ownership records and release counts — process-global structures that other tests mutate concurrently. Related prior art: PERRY_GC_FORCE_EVACUATE's test-only mode override is deliberately thread-local "so one test enabling the instrument cannot change the collector's behaviour for any other test" (arena/quarantine.rs), and #7251 found a test made vacuous by libtest ordering, where array::tests built the intrinsic towers first. Same family of problem.
2. A timing assertion. settling_many_keys_is_not_quadratic asserts on complexity, which under a loaded 10-core box running 1600+ tests in parallel is a coin flip.
Why it matters now
cargo-test is a required status check. Today its signal is: "somewhere between 0 and 4 tests failed, and which ones varies." Nobody can distinguish a regression from the baseline noise, which is the same failure mode as a gate that cannot fail — it just fails uselessly instead of passing uselessly.
Suggested fix
- The
map_set_*teardown tests need serialisation against each other and anything else touching the side tables (a shared test lock, or#[serial]), or the global they assert on needs to be per-thread the way the quarantine's mode override already is. - The quadratic assertion should measure operation counts rather than wall time, or be marked
#[ignore]under parallel runs.
Found while establishing a baseline for #7341 — I nearly filed a false regression against my own change before checking clean main gave the same varying counts.
Activity
perf(gc): nursery cap + scavenge on by default — peak RSS -69%
perf(gc): nursery cap + scavenge on by default — peak RSS -69% (#7377)
fix(gc): refresh obj before the class-static mirror on both own-data-…
fix(gc): root the callee across the instance allocation in `new` (#7391)
proggeramlug commentedon Aug 10, 2026
A datum from running the suite repeatedly while doing unrelated work on this machine.
Four back-to-back runs of
cargo test -p perry-runtime --lib --no-fail-fast, on an otherwise-busy host (a fulltest-files/parity sweep was saturating the CPU):promise::keyed_table::tests::draining_a_heavily_populated_key_ahead_of_another_is_not_quadraticgc::tests::inline_generation_gate_contract::sabotaged_parent_gate_strands_a_young_child_the_shipped_gate_keeps,promise::keyed_table::tests::settling_many_keys_is_not_quadraticpromise::keyed_table::tests::settling_many_keys_is_not_quadraticThat is the issue's symptom exactly — "a different number of tests on every run" — and it points at a mechanism: the two
*_is_not_quadratictests assert a wall-clock ratio, which is a property of how much CPU the test got, not of the code. Under load the ratio moves. On an idle host the same four runs are green, which is why this reads as random rather than as a load correlation.If that holds up, the fix shape for those two is to assert on an operation COUNT (or an algorithmic invariant) rather than elapsed time — the same "assert the subject, not a proxy for it" move CLAUDE.md's gate-hazard section describes. I have not looked at whether the
inline_generation_gate_contractone has the same cause; it appeared once in four runs.Noting rather than fixing — it came up while validating #7752 and none of it is that PR's subject.
proggeramlug commentedon Aug 10, 2026
Named it. On
main@db44b31b7,perry-dev, macOS arm64, the intermittent failure incargo test -p perry-runtime --libis a single test:Rate: 2 of 10 consecutive runs on a clean tree (no local changes, archives rebuilt in between). Every failing run reports exactly
2051 passed; 1 failed, and every passing run2052 passed; 0 failed— so on this host the "different number of tests every run" is one flaky test toggling, not a varying set.Found while validating an unrelated Intl change: it failed once in my working tree, and the only way to tell whether I had caused it was to revert to clean
main, rebuild, and run ten times. That is the cost this issue describes — every unrelated PR pays it — so the name above is worth having even before the fix.Two things that make it easy to miss when you go looking:
cargo test's summary block is buried in a lot of output and my first three attempts atgrep -E "^failures|FAILED"on the captured log came back empty; capturing each run to its own file and breaking on the first^test result: FAILEDis what surfaced it.The test's own name states the property it is asserting — a bare address in a shadow slot surviving a real collection — so a 20% failure rate is a real intermittent liveness result, not a harness artifact. It should probably be triaged as a GC-correctness intermittency in its own right rather than only as test-suite noise.
proggeramlug commentedon Aug 10, 2026
Second flaky test, and this one is order-dependent, which changes how it should be hunted:
Measured on clean
main@db44b31b7,perry-dev, macOS arm64:maincargo test -p perry-runtime --lib obj_dispatch_ic_tests(isolated)cargo test -p perry-runtime --libSo it mostly passes in the suite and mostly fails alone — the opposite of the usual flake, and evidence it depends on state some other test leaves behind (an inline-cache table primed by an earlier test, given the name). That matters for this issue in two ways:
--exact/ filtered reruns are misleading here. The standard triage move — rerun just the failing test to confirm — makes this one look reliably broken rather than flaky, and the same move on the other known offender (gc::tests::root_words::bare_address_in_shadow_slot_survives_a_real_collection, 2/10 in-suite) makes it look fine. Neither reproduces its in-suite rate in isolation.main10/12. The change made it less likely to fail, and only measuring both arms showed that. Every PR touching this area pays that same cost.Both tests together mean the suite has at least two independent intermittencies with different mechanisms — one timing, one ordering — so a single "retry until green" policy will not characterise either.
test(runtime): let the dispatch-IC generation retry actually fire (#7365
test(runtime): let the dispatch-IC generation retry actually fire (#7365
proggeramlug commentedon Aug 14, 2026
Closing this as superseded/resolved after reconciling every named failure against freshly fetched
origin/mainat08173f8b403be848f2218746b22a462fd883042f.Exact failure → fix map
Original issue-body set
gc::tests::teardown::{map_set_owner_records_follow_growth,map_set_side_allocations_release_exactly_once,map_set_side_allocations_release_on_thread_exit}→ test(runtime): make the --lib suite order-independent under default-parallel cargo test (#6965) #7445 (fa11b0e700a7b39d114d20cdd2d82be6c9e64487). The module is serialized; the exact ownership/exactly-once assertions remain inside the probe thread's owned window, while only the outer cross-thread deltas became lower bounds.url::node_compat::{path_to_file_url_posix_preserves_relative_trailing_slash,path_to_file_url_posix_does_not_add_slash_without_input_slash}→ test(runtime): make the --lib suite order-independent under default-parallel cargo test (#6965) #7445. Readers and the temporary-CWD writer now share the crate-wide process-CWD test lock.object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush→ test(runtime): make the --lib suite order-independent under default-parallel cargo test (#6965) #7445. The positive record→read window retries across legitimate process-global epoch/vtable invalidation; the negative invalidation assertions remain single-shot. The bounded retry was sabotage-checked and still fails when recording is disabled.promise::keyed_table::tests::settling_many_keys_is_not_quadratic→ test(promise): replace wall-clock scaling gates with operation counts #8059 (a793b4178a341e617a918da0e5a0b6d5dc8c38df). The elapsed-time ratio is gone. The normal, unignored test now counts actual HashMap hash/equality probes and displaced-slot comparisons.Failures named in later #7365 comments
promise::keyed_table::tests::draining_a_heavily_populated_key_ahead_of_another_is_not_quadratic→ test(promise): replace wall-clock scaling gates with operation counts #8059 as well: deterministic operation-count and relocation invariants replace the wall clock.gc::tests::inline_generation_gate_contract::sabotaged_parent_gate_strands_a_young_child_the_shipped_gate_keeps→ fix(test-isolation): the GC guards' clear can no longer reach another test's data (#7672) #7674 (5296e3cae…). The observed failure was another test guard zeroingGENERATED_WRITE_BARRIERS_EMITTEDmid-test; that flag is per-test in test builds, with a deterministic cross-thread isolation witness.gc::tests::root_words::{bare_address_in_shadow_slot_survives_a_real_collection,bare_address_in_global_root_survives_a_real_collection}→ test(runtime): stop five process-globals leaking across libtest threads (#7946) #7954 (56f30b8ff…). These were not intermittent root-liveness failures: another test moved process-globalGC_UNSAFE_ZONES, socomplete_budgeted_gc_cyclereturnedSKIPPED. Unsafe-zone tests now pin the answer per thread.object::class_registry::dispatch::obj_dispatch_ic_tests::a_hit_requires_matching_name_bytes_not_a_matching_address→ test(runtime): let the dispatch-IC generation retry actually fire (#7365) #7822 (29f786b683f13c0bf9f70ecbfece580125fbab29). A generation-straddled setup miss now retries before the subject assertions; the recorded isolated rate moved from 10/12 failing to 0/20.Later runtime-suite members were split out and closed explicitly
bf5ed9aad…) closed test(runtime): prototype_addr_cache flake — the memoized prototype addresses are a process-global cache of a thread-local heap address #7955's prototype-cache sibling race by giving mutating tests owned atomic cells, and closed test(runtime): telemetry_verifier asserts a wall-clock pause budget and fails ~2 runs in 100 on a loaded host #7956 by replacing the telemetry verifier's host-load-dependent elapsed-time postcondition with deterministic work-unit/coherence checks.No #7365 test was quarantined. The only four current
perry-runtimeignores are two explicit dyn-eval benchmarks, one manual Promise performance probe, and an abort-on-purpose GC sabotage child; none belongs to this issue.Separate from #7365: #8060/#8061
The six Map/Set indexed-read failures seen immediately before #8061 were a deterministic regression introduced by #8041, not a resurfacing of the flaky Map/Set teardown family. Strict array cleaning rejected live collections before their registry dispatch and returned
NaNevery time. #8061 (118f49d41192c248f36436b7e020acc291aebaff) restored tag-gated, authoritative-registry-confirmed collection routing before array-only cleaning. Its reported full-suite keyed-table failure came from a pre-#8059 branch/base and is covered by #8059 above.Scope of this closure
This closes the finite evidence in #7365: every failure named in the issue and its comments has a merged causal fix on current
main, and the later independently catalogued suite flakes were also resolved rather than retried or ignored. It is not an open-ended claim that every present or futureperry-runtimetest is forever safe under arbitrary parallelism.CLAUDE.mdstill carries a blanket single-thread warning whose concrete proxy/element-shape examples predate and were subsequently fixed by #7954. That wording should be reconciled separately; it is not evidence that one of the named #7365 failures remains live, and I did not edit it here.This was a source/history/status audit only: no CI result was used as proof, no timing-performance conclusion was drawn, and no broad local build/soak was run.