Skip to content

Rust prepare backend rejects a POD5 directory source; Python fallback OOMs under multiprocessing #338

Description

@jayhesselberth

Summary

`leech data prepare --pod5` accepts either a single `.pod5` file or a directory of them per `leech/io/pod5_reader.py`'s own docstring (`escapepod.DatasetReader` scans the directory and does per-read-ID random-access lookup via a lazily-built index). In practice, only the Python backend can actually use that capability -- the default/fast Rust backend fails outright on a directory source, and the Python fallback is impractical at real pool sizes.

Reproduction

Pointed `--pod5` directly at a raw MinKNOW run-pool directory (46 `.pod5` files, ~213k mapped reads in the BAM) instead of a single pre-filtered file, to see whether the extra `escpod bam-filter` extraction step (used upstream to collapse a pool into one file) is actually necessary given `DatasetReader`'s documented directory + random-access support.

```
leech data prepare --pod5 <pool_dir> --bam <cognate.bam> --output-dir


--motif ... --motif-offset 8 --motif-reference fasta --reference-fasta ...
--label Asp --workers 8 --chunk-size 100 --feature-start 0 --feature-end 20
--base-justify end --anchor reference --no-split --no-compress
--signal-context-bases 0 24
```

Default (`--backend auto`, resolves to Rust): every batch fails identically:

```
leech.preparation.parallel - WARNING - Rust batch failed, skipping: Failed to open POD5 <pool_dir>: I/O error: Is a directory (os error 21)
...
RuntimeError: prepare failed [Rust (rayon)]: 2131 of 2132 batch(es) failed outright rather than rejecting individual reads.


**\`--backend python\`:** doesn't fail, but never produced a single progress log line during a ~9-minute run (the Rust backend logs \`Progress [Rust (rayon)]: ...\` periodically; the Python backend logged nothing after "Streaming BAM reads and processing in parallel..."), and was ultimately OOM-killed under an 8GB allocation (\`--workers 8\`):

\`\`\`
error: Detected 44 oom_kill events in StepId=<job>.batch. Some of the step tasks have been OOM Killed.
\`\`\`

44 kill events across 8 workers suggests each Python worker process opens its own full \`DatasetReader\`/index over the whole pool rather than sharing one, multiplying memory use by worker count.

## Why this matters

Our pipeline currently always pre-extracts a matching-reads-only \`.pod5\` via \`escpod bam-filter\` before calling \`leech data prepare\`, specifically to work around the Rust backend's single-file requirement -- an extra full linear pass over the source pool that our own project notes already flag as slow and page-fault-bound on our filesystem (BeeGFS). If the Rust backend supported directory sources the way the Python backend's underlying \`DatasetReader\` already does (per-read-ID index, no full scan), that extraction step could be skipped entirely for any pipeline whose BAM already names the reads it needs.

## Ask

- Rust backend: support a POD5 directory source (or document clearly that it's single-file-only, so the docstring in \`pod5_reader.py\` doesn't read as a promise the CLI itself doesn't keep).
- Python backend: share one \`DatasetReader\`/index across worker processes for a given \`--pod5\` source instead of one per worker, or otherwise bound memory when \`--workers N\` > 1 against a large directory source.

## Environment

- leech 0.12.1
- Rust backend requested via \`auto\` (default)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions