Skip to content

fix(shell): the catalog tests read the build's apps, not other tests' probes - #293

Merged
ymote merged 2 commits into
mainfrom
fix/flaky-bundled-apps-catalog
Oct 2, 2026
Merged

ymote merged 2 commits into
mainfrom
fix/flaky-bundled-apps-catalog

Conversation

@ymote

@ymote ymote commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Fixes the shell test that fails beside the glance tile tests, apps::tests::bundled_apps_open_without_catalog_files_or_child_processes, and its sibling apps::tests::the_system_apps_ship_as_card_apps, which compares the same list.

assertion `left == right` failed
  left: ["rinx", "reference", "sheets", "apphub", "news", "photos", "maps", "camera", "mail", "ai-providers", "youtube", "glancesweep", "glanceinput", "glancetile", "glancequiet"]
 right: ["rinx", "reference", "sheets", "apphub", "news", "photos", "maps", "camera", "mail", "ai-providers", "youtube"]

Cause

The glance tile tests register probe system apps (os.glancetile, os.glanceinput, os.glancequiet, os.glancesweep) with octosense_appstore::system::register_system_app. That registry is process-wide, has no removal, and is exactly what system_card_apps(), and with it bundled_catalog(), lists. Any catalog test that ran after a glance test in the same process therefore saw the probes as launcher rows. Which test ran first decided the outcome.

Fix

Only test code changes; production code is untouched.

  • apps::test_system_apps (cfg(test)): register notes a test's system-app id, then registers it. is_test_app answers whether an id was noted.
  • The glance tests' register_test_app registers through it.
  • The tests of the build's own catalog (bundled_apps_open_without_catalog_files_or_child_processes, bundled_apps_receive_same_base_theme_without_recreation, the_system_apps_ship_as_card_apps) read the catalog, then drop rows whose manifest id a test noted (build_rows). The note comes before the registration and the filter after the read, so a probe the catalog lists has always been noted. Anything else unexpected still fails the comparison.

The scenario tests' os.news fixture is left as it is: it runs only with OCTOS_SCENARIO_TEST_KERNEL, in its own filtered run.

Evidence

All runs were on the Linux build host. "main" is the shell test binary built at b221f7b; "fix" is this branch.

Run main fix
apps::tests::bundled_apps_open_without_catalog_files_or_child_processes apps::tests::the_system_apps_ship_as_card_apps glance_card::, 50× 11 of 50 failed (the bundled test 8×, the system-apps test 11×, always with the glance probes in the list) 0 of 50
app_chat:: apps::tests:: glance_card::, 50× 10 of 50 failed (all the catalog test) 0 of 50
Whole octosense-shell --lib suite with the real kernel, 50× the catalog tests failed in 6 runs 0 runs (also 10 runs through cargo test --locked --features mobile-apps -p octosense-shell --lib)

Not part of this flake, and seen in both columns of the whole-suite runs:

  • app_chat::tests::the_panel_opens_a_sharing_context_and_sends_person_turns_there (1 of 50 on main, 5 of 50 with the fix) is an existing race on the process-wide approvals store. approvals::tests::the_relay_gets_decisions_made_before_it_was_installed calls init_memory() unconditionally, which drops app_chat's consent mid-test. Beside approvals:: alone it fails 50 of 50 on main.
  • mobile_island::tests::a_fourth_activity_docks_the_oldest_finished_else_the_least_important (1 of 30 on both).
  • The Linux-only sandbox::tests::a_program_reached_through_a_link_outside_its_roots_starts (ETXTBSY).
  • clients::tests::every_registered_app_names_a_real_crate failed in every run on both binaries. It needs cargo's [env], which a test binary run directly does not have.

🤖 Generated with Claude Code

ymote and others added 2 commits October 2, 2026 13:57
The supervisor saw the kernel end (its stdout closing, or its exit
status) and read the stderr tail at once, but stderr is copied into the
tail by a task of its own. The line the kernel wrote just before it
exited could still be in the pipe, so the reason lost it: CI's
`a_kernel_that_dies_closes_its_connections_with_its_last_words` failed
with "the kernel closed its output: fake kernel up", without the
"asked to exit" the fake kernel wrote last.

The tail is now read after the kernel is stopped and the stderr task has
reached the end of the pipe (bounded at 2 s, for a process the kernel
started that still holds the pipe). Failed reasons get the same complete
tail. The test asks for the last words themselves, not just "exit".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… probes

The glance tile tests register probe system apps (os.glancetile,
os.glanceinput, os.glancequiet, os.glancesweep) in App Hub's
system-app registry. That registry is process-wide, has no removal, and
is the one `system_card_apps` lists, so a catalog test running after a
glance test in the same process saw the probes as launcher rows:
`bundled_apps_open_without_catalog_files_or_child_processes` failed its
catalog comparison 7 of 10 times beside `glance_card::` on main, and
`the_system_apps_ship_as_card_apps` compares the same list.

Tests now register their system apps through
`apps::test_system_apps::register`, which notes the id before it
registers it, and the tests of the build's own catalog leave noted ids
out. Production code is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ymote
ymote force-pushed the fix/flaky-bundled-apps-catalog branch from 11f34b4 to 57d36f3 Compare October 2, 2026 20:57
@ymote
ymote merged commit 67b76af into main Oct 2, 2026
3 of 5 checks passed
@ymote

ymote commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Merged in a merge train; local CI (Mac with rom toolchain + --linux-host) passed on the train head 72e5d55, which contains this PR.

  • tools/ci-local.sh --only all --linux-host, started 2026-10-02T14:00:45: passed. Desktop, phone and apps passed on the Mac; apps services, kernel-security and linux-host / sandbox passed on the Linux build host. Its rom.yml steps were skipped on the Linux host, and the train doesn't trigger rom.yml.
  • tools/ci-local.sh --only all on the Mac with the rom toolchain (JDK 17, Android SDK, node 24), started 2026-10-02T14:51:21: 42/42 steps passed, none skipped.

The train was #291, #293, #297, #294, #298, #295, #296, #299, #300 and #292, on main f6678b5. After the last merge, main 301be0c has the tested head's tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant