Repository navigation
chore: run the ui tests in parallel, each with directories of its own - #579
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The UI tests ran one after another: about 450 s of test time, mostly spent waiting on animations and timers, with the machine nearly idle. They now run in parallel, and
./scripts/run-tests.sh --ui-onlyspends about 83 s in tests, as long as its longest test (ferrous_qml_smoke_transport_input). The tests and their checks are the same.Isolation
Before, one CTest run shared a single
TMPDIRand one set of XDG config, data and cache directories. Now:ui/tests/own_directories.shis the tests'CMAKE_TEST_LAUNCHER. It gives every test program its own emptyTMPDIR,XDG_CONFIG_HOME,XDG_DATA_HOMEandXDG_CACHE_HOME, inside the run'sTMPDIR, whichrun-tests.shremoves when the run ends. Itexecs the test, so a CTest timeout still kills the test itself. This applies to runningctestdirectly too, which before used the real user directories.script-tests/<name>,release-version-check, and so on) and usemktempfor anything inTMPDIR.-jrespects.ferrous_bridge_client_tests. Its MPRIS tests can registerorg.mpris.MediaPlayer2.ferrousonce the event loop runs, and only that one process does, so tests can't collide over it. Running the script insidedbus-run-sessionkeeps that name off the desktop session bus; this PR doesn't change that.RUN_SERIALorRESOURCE_LOCK.ferrous_test_launcher_own_directoriestests the launcher: separate, empty directories per run; arguments, standard input and exit status passed through; and that CTest really launches a built test program through it.CMAKE_TEST_LAUNCHER; configuring with tests on older CMake fails with a message (docs/INSTALL.md says so). Configuring with tests now also requires Bash, which the script tests already used when it was found.Parallelism
run-tests.shrunsctest --parallel N:CTEST_PARALLEL_LEVELif set (checked before the Rust phase starts), otherwise half the CPUs, at most 8.-Vis now--verbose, because the output of tests running at the same time is interleaved.--output-on-failurestays, so a failed test still shows all its output.COST, so a first run with no CTest timing history (CI, a fresh build directory) starts them first. Later runs use CTest's recorded times.ctest --parallel 2(half the runner's 4 CPUs) without-V.Timings (16 CPU threads, test phase only)
-Vrun-tests.sh --ui-only--parallel 8 --schedule-randomrun-tests.sh, no CTest timing history--parallel 8 --schedule-randomrun-tests.shrun-tests.shCTEST_PARALLEL_LEVEL=2 run-tests.sh(as CI)./scripts/run-tests.sh(Rust: fmt, check, Clippy pedantic, 1,827 tests, audit; then UI)All runs were under
dbus-run-session.A timing failure that predates this PR
AccessibilityTest::seekTouchShowsNoHoverTimefailed once under the 16-worker load. It isn't caused by parallel runs or the new directories:ferrous_accessibility_tests seekTouchShowsNoHoverTime).A Qt trace shows Qt Quick delivering a hover from the primary mouse device at the touch point after a touch press. The seek bar's
HoverHandler(mouse and touchpad only) then accepts it as hovered. Whether that synthetic hover arrives before the check depends on frame timing and on the state the earlier test functions leave. That is a touch-input issue inTransportBar.qmlor the test, not test isolation, so it's left for its own issue; marking the test serial wouldn't change it.