Skip to content

[BUG] Rust: default parallel cargo test SIGSEGVs on GPU tests; passes with --test-threads=1 #2233

Description

@jamie8johnson

Describe the bug

Running the Rust crate's test suite with cargo's default parallel test threads SIGSEGVs intermittently on GPU tests. Observed on unmodified main (78135be): ivf_flat tests crash under the default runner; the same tests pass reliably with --test-threads=1. We reproduced the identical pattern while developing the IVF-SQ (#2229) and refine (#2230) bindings — single-threaded always green, default runner intermittently fatal.

Steps/Code to reproduce bug

cd rust
cargo test -p cuvs              # intermittent SIGSEGV (ivf_flat and others)
cargo test -p cuvs -- --test-threads=1   # passes

Expected behavior

Either the tests are safe to run concurrently, or the harness enforces serialization so contributors don't chase phantom crashes.

Recommended fix

Near-term: enforce serial execution at the harness level rather than relying on contributor knowledge — either a .cargo/config.toml in rust/ setting [env] RUST_TEST_THREADS = "1", or the serial_test crate's #[serial] on GPU-touching tests; plus one line in the rust README. Root-cause follow-up: tests each create their own Resources, so the crash suggests shared global state in the C layer (stream pool, memory resource, or RMM current-device resource) isn't safe under concurrent Resources creation/destruction from multiple host threads — an audit of cuvsResourcesCreate/Destroy thread-safety would pin it down. We're happy to submit the harness-level guard as a small PR; the thread-safety audit likely needs maintainer eyes.

Environment details

Rust crate at main (78135be), conda libcuvs 26.06, CUDA 13.1, RTX 4000 (8GB), Rust 1.96.0.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions