[wmfeht]

Description

Rust tests ran one GTK test at a time per machine. Every GTK test runs in a child process, and every child of a test process waits on one lock because children on one display share its clipboard and pointer grabs. So the 844 GTK tests (about 924 s of serial time) queue on a single Xvfb, whatever the core count. Locally, ./scripts/quality.sh test took about 16 minutes of test time on 24 CPUs, and each CI shard used about one of its four vCPUs for 230–265 s.

• Lanes. scripts/quality_ci.py run now splits a shard's tests across lanes. Each lane runs its libtest processes under its own Xvfb server and private D-Bus session, and all lanes run at once. The shard's receipt, the plan checks, the isolated deferred-scroll process and verify's exactly-once check are unchanged. A failed lane fails the shard after the others finish, and lane logs are written while tests run.
• Local runs. ./scripts/quality.sh test builds one shard and runs it with one lane per two CPUs. The new --jobs N option sets the lane count (--jobs 1 gives the old single display). A failing run prints the failed tests' panics, without the expected panics that passing tests print.
• CI. Shards run ./scripts/quality.sh test --jobs 3. Shard logs are uploaded with always(), so a timed-out shard still has its logs.
• Balancing. ./scripts/quality.sh test --refresh-timings (opt-in) measures every test with libtest's --report-time (behind RUSTC_BOOTSTRAP=1), using one thread per process so GTK lock waits don't count. It refuses to write unless it measured exactly the runnable tests, then rewrites scripts/quality-durations.json for all 3712 runnable tests. Before, the file had hints for only 35. Tests in a GTK child keep their full time; other tests share libtest's threads, so they count for a quarter of their time.
• Load-sensitive tests, found by running lanes at higher load:
  • The transfer "slow destination" test assumed that, after a cancel, the next paste's unblocked conflict check finishes within the 350 ms progress delay. It failed 12 times in 192 under load, and 0 in 320 after the fix.
  • The go-prompt test recorded visits through NavigationHistory::shared(), which the keyboard fixture never holds. So each call, and then the search, reloaded the history from a file the background writer might not have saved yet. It failed 2 times in 96 at 2× load, and 0 in 96 after the fix.
  • The merge test for a folder kept by an open original failed on CI's first run at three lanes (Merge cleanup reports a generic failure instead of a kept folder when slow (CIFS test flakes under load) #1964). The cause was in the product: the delete queue stops every worker at its first error. So when a folder was kept only by an open file, whatever the other workers hadn't reached yet stayed behind, depending on scheduling. In a merge's stage, a replaced original left that way turned "kept, still open" into a generic failure. A folder refused only for its leftovers now records its error and lets the rest of the delete finish. Its parents stay, and any other error still stops at once. A new deletion test leaves 5–10 of 200 sibling files behind with the old queue and none with the new one. The merge test failed 4 times in 144 at 2× load before, and 0 in 288 after.
  • One rare abort remains, seen only at one lane per CPU, so this PR stays below that: a GLib worker finalizes a context holding local tasks (Rust test process aborts when a GLib worker finalizes a MainContext holding local tasks #1963).

Measurements, all in the pinned image:

	Before	After
Local quality.sh test, 24 CPUs	945 s tests (single display)	about 90 s (12 lanes, lanes within 84–92 s of each other)
One CI shard, container pinned to 4 CPUs	241 s	87–94 s with 3 lanes, all four shards twice

Shard times on 4 CPUs were 127 s with 2 lanes and 70 s with 4. At 6 lanes on 4 CPUs (and 24 on 24), load-sensitive tests failed.

Visual evidence

N/A: test harness, CI and a deletion change visible only as which items remain after an error, described above.

How to test

1. ./scripts/quality.sh test. It prints one line per lane, then Verified 3712 passed tests exactly once; 619 explicitly ignored by libtest.
2. ./scripts/quality.sh test --jobs 1. It runs everything on one display and gives the same verified counts.
3. ./scripts/quality.sh fmt --jobs 2 and ./scripts/quality.sh test --jobs 0. Both exit 2 with the usage line, without starting a container.
4. ./scripts/quality.sh test --refresh-timings. It writes scripts/quality-durations.json, and the diff only changes values.
5. In CI, each Rust tests shard N job prints three lane lines, and Format, lint, and test verifies the receipts as before.
6. On an NFS or SMB mount, open a file inside tree/open/ in another program, then permanently delete tree, which also contains other files and folders. Everything except tree/open and tree is deleted, and the error names open as still open.

Expected result: every run verifies all runnable tests exactly once, and local and CI shard times drop to about 90 s.

Related issue

Closes #1962
Closes #1964

[wmfeht]

Code review findings and fixes

Fixed in 8330769.

1. A deferred "still open" error could hide the error that stopped the delete (bug, low–medium; NFS/SMB only)

WorkQueue::defer and WorkQueue::fail both kept only the first error. Before this PR, a still-open folder always stopped the queue itself, so the first error always explained what was left. Now it might not:

1. Worker A finds tree/open held by an open file and calls defer. The still-open message is stored and the queue keeps going.
2. Worker B then hits a real error elsewhere (for example EIO or EACCES) and calls fail. The queue stops, but B's error is dropped because an error is already stored.

The user saw only "Could not delete “open”: A file in it is still open…", while other items were also left for a reason nobody reported. That contradicts the new paragraph in docs/filesystem-compatibility.md.

Fix: work_queue.rs keeps deferred errors in their own slot, and result() returns the stopping error if there is one, otherwise the deferred one. fail behaves as before, and the copy engine queues never call defer.

Test: an_error_that_stops_a_delete_is_reported_over_an_earlier_open_folder uses one worker, so the held folder is refused before an injected EIO unlink. It fails without the fix (the result names open) and passes with it.

2. quality.sh options weren't checked against STRATA_QUALITY_TASK (nit)

--refresh-timings with STRATA_QUALITY_TASK=shard started the container and then failed in quality_ci.py's argument parsing. With STRATA_QUALITY_TASK=build, both options were ignored silently. Now both cases exit 2 with the usage line before the container starts, and --jobs with shard (what CI uses) is still passed through. test_quality_runner.py covers all three cases.

3. CI shard console no longer shows test output (behaviour change, informational)

That's intended, since lane output goes to the uploaded logs. The shard's first line now also gives the log path (target/quality-reports/shard-N-lane-*.log), so a shard that hangs or times out points to the artifact.

Checked, no change needed

• A deferred folder never completes its parent's group, so parent folders stay and the workers still finish.
• gtk_test starts its child without --report-time, so test_times takes the parent's time and correctly marks the test as GTK.
• Lane display numbers can't collide: lane L uses 99+L+k·lanes, with a new display for each batch.
• The transfer test's delay override is reset at the end of every iteration.

Validation

• ./scripts/test-headless.py adapters::local_operations::tests::deletion::: 34 passed. The new test also failed as expected with the fix reverted.
• ./scripts/test-headless.py adapters::local_operations::tests::merge::: 66 passed.
• python3 -m unittest test_quality_ci test_quality_runner (from scripts/): 27 passed.
• ./scripts/quality.sh fmt and ./scripts/quality.sh clippy: clean.
• I didn't run a full ./scripts/quality.sh test or any QA scenarios. The change only touches the delete queue's error reporting and the runner's option checks, and CI runs the full suite.

[l0gicgate]

Review

Merged main (9b032bf) into the branch and reviewed the diff against it. The lanes, the runner tests, the delete queue's deferred error and the two deletion tests all look right; no comments or tests needed trimming.

One blocker, fixed in 371d5ce: on a rootless Podman host, ./scripts/quality.sh test at its default lane count never finished. Podman caps a container at 2048 tasks by default, and 16 lanes on a 32-CPU machine exceed that, so tests failed with settings writer thread: Os { code: 11, kind: WouldBlock } and Could not start delete worker: Resource temporarily unavailable, and seven lanes then hung on GTK children waiting for threads that never started. quality.sh now runs the container with --pids-limit=-1, which Docker already does by default, so CI is unchanged; test_quality_runner.py asserts the option.

Validation

• ./scripts/quality.sh all in the pinned image: fmt and clippy clean; the test phase at 16 lanes failed and hung as above before the fix.
• ./scripts/quality.sh test after the fix: 16 lanes within 58.8–67.1 s of each other, Verified 3718 passed tests exactly once; 624 explicitly ignored by libtest, 69 s in total.
• python3 -m unittest test_quality_ci test_quality_runner: 27 passed.
• git diff --check clean.

Not changed: scripts/qa.sh runs under the same default Podman limit, which is likely why multi-scenario QA runs on large hosts needed a lower --test-threads. That is outside this PR.