Summary
`leech data prepare --pod5` accepts either a single `.pod5` file or a directory of them per `leech/io/pod5_reader.py`'s own docstring (`escapepod.DatasetReader` scans the directory and does per-read-ID random-access lookup via a lazily-built index). In practice, only the Python backend can actually use that capability -- the default/fast Rust backend fails outright on a directory source, and the Python fallback is impractical at real pool sizes.
Reproduction
Pointed `--pod5` directly at a raw MinKNOW run-pool directory (46 `.pod5` files, ~213k mapped reads in the BAM) instead of a single pre-filtered file, to see whether the extra `escpod bam-filter` extraction step (used upstream to collapse a pool into one file) is actually necessary given `DatasetReader`'s documented directory + random-access support.
```
leech data prepare --pod5 <pool_dir> --bam <cognate.bam> --output-dir
--motif ... --motif-offset 8 --motif-reference fasta --reference-fasta ...
--label Asp --workers 8 --chunk-size 100 --feature-start 0 --feature-end 20
--base-justify end --anchor reference --no-split --no-compress
--signal-context-bases 0 24
```
Default (`--backend auto`, resolves to Rust): every batch fails identically:
```
leech.preparation.parallel - WARNING - Rust batch failed, skipping: Failed to open POD5 <pool_dir>: I/O error: Is a directory (os error 21)
...
RuntimeError: prepare failed [Rust (rayon)]: 2131 of 2132 batch(es) failed outright rather than rejecting individual reads.
**\`--backend python\`:** doesn't fail, but never produced a single progress log line during a ~9-minute run (the Rust backend logs \`Progress [Rust (rayon)]: ...\` periodically; the Python backend logged nothing after "Streaming BAM reads and processing in parallel..."), and was ultimately OOM-killed under an 8GB allocation (\`--workers 8\`):
\`\`\`
error: Detected 44 oom_kill events in StepId=<job>.batch. Some of the step tasks have been OOM Killed.
\`\`\`
44 kill events across 8 workers suggests each Python worker process opens its own full \`DatasetReader\`/index over the whole pool rather than sharing one, multiplying memory use by worker count.
## Why this matters
Our pipeline currently always pre-extracts a matching-reads-only \`.pod5\` via \`escpod bam-filter\` before calling \`leech data prepare\`, specifically to work around the Rust backend's single-file requirement -- an extra full linear pass over the source pool that our own project notes already flag as slow and page-fault-bound on our filesystem (BeeGFS). If the Rust backend supported directory sources the way the Python backend's underlying \`DatasetReader\` already does (per-read-ID index, no full scan), that extraction step could be skipped entirely for any pipeline whose BAM already names the reads it needs.
## Ask
- Rust backend: support a POD5 directory source (or document clearly that it's single-file-only, so the docstring in \`pod5_reader.py\` doesn't read as a promise the CLI itself doesn't keep).
- Python backend: share one \`DatasetReader\`/index across worker processes for a given \`--pod5\` source instead of one per worker, or otherwise bound memory when \`--workers N\` > 1 against a large directory source.
## Environment
- leech 0.12.1
- Rust backend requested via \`auto\` (default)
Summary
`leech data prepare --pod5` accepts either a single `.pod5` file or a directory of them per `leech/io/pod5_reader.py`'s own docstring (`escapepod.DatasetReader` scans the directory and does per-read-ID random-access lookup via a lazily-built index). In practice, only the Python backend can actually use that capability -- the default/fast Rust backend fails outright on a directory source, and the Python fallback is impractical at real pool sizes.
Reproduction
Pointed `--pod5` directly at a raw MinKNOW run-pool directory (46 `.pod5` files, ~213k mapped reads in the BAM) instead of a single pre-filtered file, to see whether the extra `escpod bam-filter` extraction step (used upstream to collapse a pool into one file) is actually necessary given `DatasetReader`'s documented directory + random-access support.
```
leech data prepare --pod5 <pool_dir> --bam <cognate.bam> --output-dir
--motif ... --motif-offset 8 --motif-reference fasta --reference-fasta ...
--label Asp --workers 8 --chunk-size 100 --feature-start 0 --feature-end 20
--base-justify end --anchor reference --no-split --no-compress
--signal-context-bases 0 24
```
Default (`--backend auto`, resolves to Rust): every batch fails identically:
```
leech.preparation.parallel - WARNING - Rust batch failed, skipping: Failed to open POD5 <pool_dir>: I/O error: Is a directory (os error 21)
...
RuntimeError: prepare failed [Rust (rayon)]: 2131 of 2132 batch(es) failed outright rather than rejecting individual reads.