Skip to content

task-monitor: a_large_snapshot_does_not_deadlock_the_pipe flakes on Linux CI (likely ETXTBSY on freshly written test scripts) #283

Closed

Description

@martin-conur

Part of #272. This was found during #276 (PR #282) and is now on main.

What happened

refresh::tests::a_large_snapshot_does_not_deadlock_the_pipe failed once on Rust (ubuntu-latest), in run 39476771572, for the superseded commit cf16684. The same test passed on both OSes for the merged head bffad34 (run 39477329001), and locally every time. The whole suite took 0.31s, so this was a fast error, not the 10s timeout. The assert is a bare .is_ok(), so the log doesn't say which error it was.

Likely cause (unconfirmed)

ETXTBSY ("Text file busy"). Each collector test writes an executable shell script with fs::write and runs it immediately. With tests in parallel, another test's fork can briefly inherit that file's still-open write fd, and on Linux exec then fails with os error 26. macOS doesn't do this, which fits a Linux-only failure.

Fix

  • Make the failing asserts print the error (.unwrap() / expect), so a recurrence is diagnosable.
  • Remove the race in the test helpers: run the script as /bin/sh <path> so it is never exec'd directly. Alternatively, write it to a temp name, close it, and rename it before any spawn, or retry the spawn on ETXTBSY. Production never writes the executable it runs, so only the tests are affected.
  • If possible, reproduce first: loop the suite on Linux with high --test-threads and confirm that the error is ETXTBSY and that the fix removes it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions