Problem
When running BRCore workflows (such as multi_rarefy() or core selection) on large or complex microbiome datasets, especially with parallel processing, it is easy to exceed 200GB+ RAM and encounter out-of-memory (OOM) kills even on high-performance compute clusters. This restricts scalability for large datasets and limits reproducibility in cluster and containerized environments.
Impacted Functions/Areas
multi_rarefy()identify_core() / extract_core()- Parallelization via
furrr, parallel, or future
- Data aggregation (e.g.,
bind_rows, storing multiple full-size matrices in memory simultaneously)
Example
A rarefaction task with ~2000 samples × ~25,000 taxa at 100 iterations and 8 threads can consume >250GB RAM, with the process killed via OOM before completion.
Potential Solutions
- Process rarefaction and aggregation in smaller batches or via disk-backed temporary storage
- Use memory profiling to identify and optimize bottlenecks (e.g., avoid unnecessary copying of large matrices)
- Provide clear documentation and warnings for expected memory requirements based on dataset size, iteration count, and thread number
- Consider chunked or streaming computation for intermediate results
- Catch OOM errors and fail gracefully with user guidance
Related Discussions
Desired Outcome
- Substantially reduce RAM usage for key BRCore workflows on large datasets
- Documentation outlining expected resource usage and memory-saving best practices
- Improved support for running on cloud, container, or cluster nodes with constrained RAM
Originally reported after repeated OOM kills running multi_rarefy() and core selection tasks on a 256GB RAM compute node (InterBRC Core project).
Problem
When running BRCore workflows (such as
multi_rarefy()or core selection) on large or complex microbiome datasets, especially with parallel processing, it is easy to exceed 200GB+ RAM and encounter out-of-memory (OOM) kills even on high-performance compute clusters. This restricts scalability for large datasets and limits reproducibility in cluster and containerized environments.Impacted Functions/Areas
multi_rarefy()identify_core()/extract_core()furrr,parallel, orfuturebind_rows, storing multiple full-size matrices in memory simultaneously)Example
A rarefaction task with ~2000 samples × ~25,000 taxa at 100 iterations and 8 threads can consume >250GB RAM, with the process killed via OOM before completion.
Potential Solutions
Related Discussions
Desired Outcome
Originally reported after repeated OOM kills running
multi_rarefy()and core selection tasks on a 256GB RAM compute node (InterBRC Core project).