Skip to content

chore: run the ui tests in parallel, each with directories of its own - #579

Merged
Tiaxi merged 1 commit into
mainfrom
chore/parallel-ui-tests
Nov 10, 2026
Merged

Tiaxi merged 1 commit into
mainfrom
chore/parallel-ui-tests

Conversation

@Tiaxi

@Tiaxi Tiaxi commented Nov 10, 2026

Copy link
Copy Markdown
Owner

The UI tests ran one after another: about 450 s of test time, mostly spent waiting on animations and timers, with the machine nearly idle. They now run in parallel, and ./scripts/run-tests.sh --ui-only spends about 83 s in tests, as long as its longest test (ferrous_qml_smoke_transport_input). The tests and their checks are the same.

Isolation

Before, one CTest run shared a single TMPDIR and one set of XDG config, data and cache directories. Now:

  • ui/tests/own_directories.sh is the tests' CMAKE_TEST_LAUNCHER. It gives every test program its own empty TMPDIR, XDG_CONFIG_HOME, XDG_DATA_HOME and XDG_CACHE_HOME, inside the run's TMPDIR, which run-tests.sh removes when the run ends. It execs the test, so a CTest timeout still kills the test itself. This applies to running ctest directly too, which before used the real user directories.
  • The script, CMake and appstreamcli tests don't go through the launcher (CMake applies it only to built programs). They already work in a build-tree directory of their own (script-tests/<name>, release-version-check, and so on) and use mktemp for anything in TMPDIR.
  • The app-icon install already runs before the app-icon test through a fixture, which -j respects.
  • D-Bus: no CTest test registers a well-known name except ferrous_bridge_client_tests. Its MPRIS tests can register org.mpris.MediaPlayer2.ferrous once the event loop runs, and only that one process does, so tests can't collide over it. Running the script inside dbus-run-session keeps that name off the desktop session bus; this PR doesn't change that.
  • No test needed RUN_SERIAL or RESOURCE_LOCK.
  • ferrous_test_launcher_own_directories tests the launcher: separate, empty directories per run; arguments, standard input and exit status passed through; and that CTest really launches a built test program through it.
  • Tests now need CMake 3.29 or newer for CMAKE_TEST_LAUNCHER; configuring with tests on older CMake fails with a message (docs/INSTALL.md says so). Configuring with tests now also requires Bash, which the script tests already used when it was found.

Parallelism

  • run-tests.sh runs ctest --parallel N: CTEST_PARALLEL_LEVEL if set (checked before the Rust phase starts), otherwise half the CPUs, at most 8.
  • -V is now --verbose, because the output of tests running at the same time is interleaved. --output-on-failure stays, so a failed test still shows all its output.
  • The longest tests carry a COST, so a first run with no CTest timing history (CI, a fresh build directory) starts them first. Later runs use CTest's recorded times.
  • CI runs ctest --parallel 2 (half the runner's 4 CPUs) without -V.

Timings (16 CPU threads, test phase only)

Run Load Test time Result
Before: serial, -V none 447.2 s 104/104
run-tests.sh --ui-only none 82.8 s 105/105
--parallel 8 --schedule-random none 82.9 s 105/105
run-tests.sh, no CTest timing history stress-ng, 6 CPU workers 90.8 s 105/105
--parallel 8 --schedule-random stress-ng, 6 CPU workers 101.9 s 105/105
run-tests.sh stress-ng, 16 CPU workers 124.7 s 104/105, see below
run-tests.sh stress-ng, 6 CPU workers 96.0 s 105/105
CTEST_PARALLEL_LEVEL=2 run-tests.sh (as CI) none 219.7 s 105/105
full ./scripts/run-tests.sh (Rust: fmt, check, Clippy pedantic, 1,827 tests, audit; then UI) none UI 96.1 s, all 267 s all passed

All runs were under dbus-run-session.

A timing failure that predates this PR

AccessibilityTest::seekTouchShowsNoHoverTime failed once under the 16-worker load. It isn't caused by parallel runs or the new directories:

  • It fails every time when run alone, without the launcher, on this branch's unchanged test code (ferrous_accessibility_tests seekTouchShowsNoHoverTime).
  • Under the same load it fails 2 of 10 times in a single process run with only its preceding function, with no other test running.

A Qt trace shows Qt Quick delivering a hover from the primary mouse device at the touch point after a touch press. The seek bar's HoverHandler (mouse and touchpad only) then accepts it as hovered. Whether that synthetic hover arrives before the check depends on frame timing and on the state the earlier test functions leave. That is a touch-input issue in TransportBar.qml or the test, not test isolation, so it's left for its own issue; marking the test serial wouldn't change it.

@Tiaxi
Tiaxi merged commit b61406e into main Nov 10, 2026
5 checks passed
@Tiaxi
Tiaxi deleted the chore/parallel-ui-tests branch November 10, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant