From a814ade214b87913ffafa5843dc43de1688397f0 Mon Sep 17 00:00:00 2001 From: rob-p Date: Sat, 15 Aug 2026 11:06:28 -0400 Subject: [PATCH 1/4] docs: add temporary alevin-fry 0.18 handoff --- .github/alevin-fry-v0.18-integration.md | 45 +++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 .github/alevin-fry-v0.18-integration.md diff --git a/.github/alevin-fry-v0.18-integration.md b/.github/alevin-fry-v0.18-integration.md new file mode 100644 index 0000000..6f775c6 --- /dev/null +++ b/.github/alevin-fry-v0.18-integration.md @@ -0,0 +1,45 @@ +# Temporary alevin-fry 0.18 integration checklist + +This tracked note is the release handoff from alevin-fry 0.18 to simpleaf +0.28. Delete it only after every row is implemented, documented, and tested; +the commit history will retain the context without making this permanent user +documentation. + +## Argument forwarding + +| Status | simpleaf command | simpleaf option | alevin-fry destination | Inherited default | +|---|---|---|---|---| +| [ ] | `quant`, `multiplex-quant` | `--cell-bc-correction` | GPL same name | Unique | +| [ ] | `quant`, `multiplex-quant` | `--cell-bc-neighborhood` | GPL same name | Protocol/filter-specific | +| [ ] | `quant`, `multiplex-quant` | `--cell-bc-confidence` | GPL same name | 97.5% | +| [ ] | `quant`, `multiplex-quant` | `--collate-memory-limit` | collate `--memory-limit` | 2 GiB | +| [ ] | `multiplex-quant` | `--sample-bc-correction` | GPL same name | Exact | +| [ ] | `multiplex-quant` | `--sample-bc-neighborhood` | GPL same name | Hamming-1 | +| [ ] | `multiplex-quant` | `--sample-bc-confidence` | GPL same name | 97.5% | +| [ ] | `multiplex-quant` | `--gpl-memory-limit` | GPL `--memory-limit` | 512 MiB | +| [ ] | `multiplex-quant` | `--gpl-tmp-dir` | GPL `--tmp-dir` | GPL output directory | +| [ ] | `atac process` | cell correction, neighborhood, confidence | ATAC GPL same names | Unique, Hamming-1, 90% | + +Unspecified values must be omitted from child commands so alevin-fry remains +the source of truth for resolved protocol defaults. New controls are visible +under advanced barcode-correction or resource help headings. + +## Compatibility and validation + +- [ ] Keep `--sample-correction-mode` accepted but hidden and deprecated. +- [ ] Translate legacy `exact` to sample Exact correction. +- [ ] Translate legacy `1-edit` to sample Unique plus + substitution-or-shift-1 without forwarding the deprecated spelling. +- [ ] Keep sample-barcode orientation precedence: CLI override, chemistry + preset, then the alevin-fry default. +- [ ] Do not expose `--max-records`, `--collation-mode`, or the `edit-1` + neighbourhood spelling. +- [ ] Validate exact-fraction/decimal confidence and human-readable memory + values before mapping starts. +- [ ] Resolve the pipeline thread count once. Values below two warn, become + two, and continue; an available-parallelism result of one also warns and + attempts two. +- [ ] Update CLI snapshots and live quant, multiplex, and ATAC documentation. +- [ ] Test omitted defaults, every explicit forwarding route, legacy aliases, + stage-specific memory controls, and thread-floor edge cases. +- [ ] Require alevin-fry `>=0.18.0, <1.0.0`. From e406aa975d310573f7656508116351aee48a45ea Mon Sep 17 00:00:00 2001 From: rob-p Date: Sat, 15 Aug 2026 11:34:56 -0400 Subject: [PATCH 2/4] feat: integrate alevin-fry 0.18 correction controls --- .github/alevin-fry-v0.18-integration.md | 45 +- .github/workflows/test_simpleaf.yml | 6 + CHANGELOG.md | 37 ++ CONTRIBUTING.md | 14 + docs-site/astro.config.mjs | 1 + .../src/content/docs/atac-process-command.md | 62 +++ .../src/content/docs/flex-quant-command.md | 105 +++- docs-site/src/content/docs/installation.md | 2 +- docs-site/src/content/docs/quant-command.md | 90 +++- .../content/docs/threads-and-decompression.md | 7 + release.sh | 284 ----------- src/atac/commands.rs | 11 +- src/atac/process.rs | 40 +- src/core/runtime.rs | 69 ++- src/main.rs | 3 +- src/simpleaf_commands.rs | 447 +++++++++++++++++- src/simpleaf_commands/multiplex_quant.rs | 141 +++++- src/simpleaf_commands/quant.rs | 57 ++- src/simpleaf_commands/workflow.rs | 3 +- src/utils/prog_utils.rs | 9 +- src/utils/workflow_utils.rs | 2 +- tests/cli_help_snapshots.rs | 22 +- .../cli-help/simpleaf_atac_index___help.txt | 2 +- .../cli-help/simpleaf_atac_process___help.txt | 54 ++- .../simpleaf_multiplex_quant___help.txt | 89 ++-- .../cli-help/simpleaf_quant___help.txt | 55 ++- 26 files changed, 1173 insertions(+), 484 deletions(-) create mode 100644 docs-site/src/content/docs/atac-process-command.md delete mode 100755 release.sh diff --git a/.github/alevin-fry-v0.18-integration.md b/.github/alevin-fry-v0.18-integration.md index 6f775c6..c54a19a 100644 --- a/.github/alevin-fry-v0.18-integration.md +++ b/.github/alevin-fry-v0.18-integration.md @@ -9,16 +9,16 @@ documentation. | Status | simpleaf command | simpleaf option | alevin-fry destination | Inherited default | |---|---|---|---|---| -| [ ] | `quant`, `multiplex-quant` | `--cell-bc-correction` | GPL same name | Unique | -| [ ] | `quant`, `multiplex-quant` | `--cell-bc-neighborhood` | GPL same name | Protocol/filter-specific | -| [ ] | `quant`, `multiplex-quant` | `--cell-bc-confidence` | GPL same name | 97.5% | -| [ ] | `quant`, `multiplex-quant` | `--collate-memory-limit` | collate `--memory-limit` | 2 GiB | -| [ ] | `multiplex-quant` | `--sample-bc-correction` | GPL same name | Exact | -| [ ] | `multiplex-quant` | `--sample-bc-neighborhood` | GPL same name | Hamming-1 | -| [ ] | `multiplex-quant` | `--sample-bc-confidence` | GPL same name | 97.5% | -| [ ] | `multiplex-quant` | `--gpl-memory-limit` | GPL `--memory-limit` | 512 MiB | -| [ ] | `multiplex-quant` | `--gpl-tmp-dir` | GPL `--tmp-dir` | GPL output directory | -| [ ] | `atac process` | cell correction, neighborhood, confidence | ATAC GPL same names | Unique, Hamming-1, 90% | +| [x] | `quant`, `multiplex-quant` | `--cell-bc-correction` | GPL same name | Unique | +| [x] | `quant`, `multiplex-quant` | `--cell-bc-neighborhood` | GPL same name | Protocol/filter-specific | +| [x] | `quant`, `multiplex-quant` | `--cell-bc-confidence` | GPL same name | 97.5% | +| [x] | `quant`, `multiplex-quant` | `--collate-memory-limit` | collate `--memory-limit` | 2 GiB | +| [x] | `multiplex-quant` | `--sample-bc-correction` | GPL same name | Exact | +| [x] | `multiplex-quant` | `--sample-bc-neighborhood` | GPL same name | Hamming-1 | +| [x] | `multiplex-quant` | `--sample-bc-confidence` | GPL same name | 97.5% | +| [x] | `multiplex-quant` | `--gpl-memory-limit` | GPL `--memory-limit` | 512 MiB | +| [x] | `multiplex-quant` | `--gpl-tmp-dir` | GPL `--tmp-dir` | GPL output directory | +| [x] | `atac process` | cell correction, neighborhood, confidence | ATAC GPL same names | Unique, Hamming-1, 90% | Unspecified values must be omitted from child commands so alevin-fry remains the source of truth for resolved protocol defaults. New controls are visible @@ -26,20 +26,25 @@ under advanced barcode-correction or resource help headings. ## Compatibility and validation -- [ ] Keep `--sample-correction-mode` accepted but hidden and deprecated. -- [ ] Translate legacy `exact` to sample Exact correction. -- [ ] Translate legacy `1-edit` to sample Unique plus +- [x] Keep `--sample-correction-mode` accepted but hidden and deprecated. +- [x] Translate legacy `exact` to sample Exact correction. +- [x] Translate legacy `1-edit` to sample Unique plus substitution-or-shift-1 without forwarding the deprecated spelling. -- [ ] Keep sample-barcode orientation precedence: CLI override, chemistry +- [x] Keep sample-barcode orientation precedence: CLI override, chemistry preset, then the alevin-fry default. -- [ ] Do not expose `--max-records`, `--collation-mode`, or the `edit-1` +- [x] Do not expose `--max-records`, `--collation-mode`, or the `edit-1` neighbourhood spelling. -- [ ] Validate exact-fraction/decimal confidence and human-readable memory +- [x] Validate exact-fraction/decimal confidence and human-readable memory values before mapping starts. -- [ ] Resolve the pipeline thread count once. Values below two warn, become +- [x] Resolve the pipeline thread count once. Values below two warn, become two, and continue; an available-parallelism result of one also warns and attempts two. -- [ ] Update CLI snapshots and live quant, multiplex, and ATAC documentation. -- [ ] Test omitted defaults, every explicit forwarding route, legacy aliases, +- [x] Update CLI snapshots and live quant, multiplex, and ATAC documentation. +- [x] Test omitted defaults, every explicit forwarding route, legacy aliases, stage-specific memory controls, and thread-floor edge cases. -- [ ] Require alevin-fry `>=0.18.0, <1.0.0`. +- [x] Require alevin-fry `>=0.18.0, <1.0.0`. + +Validation completed with 71 unit tests, CLI snapshots, CLI smoke/regression +tests, strict rustdoc, strict clippy, formatting, `git diff --check`, and the +Astro production build. Final end-to-end smoke runs remain release gates against +the published alevin-fry 0.18.0 executable. diff --git a/.github/workflows/test_simpleaf.yml b/.github/workflows/test_simpleaf.yml index 7cc773a..5487054 100644 --- a/.github/workflows/test_simpleaf.yml +++ b/.github/workflows/test_simpleaf.yml @@ -47,6 +47,12 @@ jobs: - name: Run cargo tests run: cargo test --release --verbose + - name: Strict API documentation + if: matrix.os == 'ubuntu-latest' + env: + RUSTDOCFLAGS: -D warnings + run: cargo doc --no-deps + - name: Install conda env uses: conda-incubator/setup-miniconda@v3 with: diff --git a/CHANGELOG.md b/CHANGELOG.md index 0768a78..7c3d1f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,43 @@ > (2024-07-01); releases 0.18.0 through 0.26.2 are not recorded here. See the > git history and the GitHub releases page for that range. +## [0.28.0](https://github.com/COMBINE-lab/simpleaf/compare/v0.27.0...v0.28.0) (2026-08-15) + +### Features + +* Require alevin-fry 0.18.0 and expose its deterministic cell-barcode + correction policy, neighborhood, and confidence controls across ordinary + RNA, multiplex RNA, and ATAC processing. +* Expose independent sample-barcode correction controls for multiplex assays, + including Exact, Unique, and Frequency policies. +* Add stage-specific `--gpl-memory-limit`, `--gpl-tmp-dir`, and + `--collate-memory-limit` resource overrides. Omitted values remain omitted + from child commands so alevin-fry owns protocol defaults. +* Validate confidence fractions/decimals and human-readable memory sizes before + mapping begins. + +### Compatibility and execution + +* Keep the deprecated `--sample-correction-mode` accepted but hidden. Its + `exact` and `1-edit` values translate to the corresponding new alevin-fry + arguments; deprecated child spellings are never emitted. +* Use two threads as the practical minimum. Requests of zero or one warn, are + raised to two, and continue; a host report of one available thread also warns + and still attempts two. +* Resolve the effective thread count once per pipeline and pass it consistently + to mapping, permit-list generation, collation or ATAC sorting, and + quantification. +* Retire the obsolete `release.sh`; `bump_and_publish.sh` is now the sole + documented release entry point. + +### Documentation and testing + +* Document barcode correction for standard RNA, multiplex/Flex, and ATAC, and + add strict rustdoc to CI. +* Add typed command-building helpers and coverage for omitted defaults, + explicit forwarding, legacy translation, stage-specific resources, and the + thread floor. + ## [0.27.0](https://github.com/COMBINE-lab/simpleaf/compare/v0.26.2...v0.27.0) ### Breaking changes diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a20d346..f24958d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,3 +10,17 @@ on _dev_. Before you make a PR, please check that: * you've run `cargo fmt` on the relevant code. * any non-obvious code is documented (we don't yet have formal documentation guidelines, so use common sense) * you've run `cargo clippy` on the relevant code and any issues are either resolved or the PR describes why they were ignored. + +## Cutting a release + +`bump_and_publish.sh` is the sole supported release entry point. It validates +the requested semantic version and clean upstream state, checks the crate, +verifies its crates.io package when `--publish` is selected, updates both the +manifest and lockfile, commits, tags, pushes, and optionally publishes. Preview +the release without changing repository or registry state with: + +```sh +./bump_and_publish.sh 0.28.0 --publish --dry-run +``` + +Do not create a release commit or tag with a separate version-bump script. diff --git a/docs-site/astro.config.mjs b/docs-site/astro.config.mjs index 1a6e3d3..1805653 100644 --- a/docs-site/astro.config.mjs +++ b/docs-site/astro.config.mjs @@ -37,6 +37,7 @@ export default defineConfig({ { label: "index", slug: "index-command" }, { label: "quant", slug: "quant-command" }, { label: "flex-quant", slug: "flex-quant-command" }, + { label: "atac process", slug: "atac-process-command" }, { label: "chemistry", slug: "chemistry-command" }, { label: "inspect", slug: "inspect-command" }, ], diff --git a/docs-site/src/content/docs/atac-process-command.md b/docs-site/src/content/docs/atac-process-command.md new file mode 100644 index 0000000..69f535c --- /dev/null +++ b/docs-site/src/content/docs/atac-process-command.md @@ -0,0 +1,62 @@ +--- +title: "atac process command" +description: "Map and process single-cell ATAC-seq data, including deterministic barcode correction." +--- + +`simpleaf atac process` maps a single-cell ATAC-seq library, generates a +corrected cell permit list, and asks alevin-fry to sort and deduplicate the +result into BED records. It can optionally run MACS3 peak calling. + +The command requires a piscem ATAC index, barcode reads, an ATAC chemistry, an +output directory, and either paired-end `--reads1`/`--reads2` inputs or +single-end `--reads` input. The supported chemistry names are `10x-v1`, +`10x-v2`, and `10x-multi`. + +## Barcode correction + +ATAC uses the same deterministic correction controls as RNA: + +- `--cell-bc-correction {unique,frequency}` selects how collisions between + possible corrected targets are handled. The inherited default is `unique`. +- `--cell-bc-neighborhood {hamming-1,substitution-or-shift-1}` selects the + one-error candidate neighborhood. The inherited ATAC default is `hamming-1`. +- `--cell-bc-confidence ` overrides the Frequency acceptance + threshold. The inherited ATAC default is `0.90`; decimal and exact-fraction + forms are accepted. + +These options are validated before mapping and then forwarded only to the ATAC +permit-list stage. If they are omitted, simpleaf emits no corresponding flags, +so the protocol defaults remain owned by alevin-fry. The compiled correction +plan produced there is then consumed by ATAC sorting; sorting does not choose a +different correction. + +```sh +simpleaf atac process \ + --index /path/to/atac-index \ + --reads1 sample_R1.fastq.gz \ + --reads2 sample_R2.fastq.gz \ + --barcode-reads sample_R3.fastq.gz \ + --chemistry 10x-v2 \ + --cell-bc-correction frequency \ + --cell-bc-confidence 9/10 \ + --output atac_out +``` + +## Threads + +`--threads` is resolved once and passed consistently to mapping, permit-list +generation, and ATAC sorting. Values below two are not rejected: simpleaf logs +a prominent warning, raises the effective value to two, and continues. If the +host reports only one available execution slot, simpleaf warns and still +attempts two threads. + +Mapping's gzip decoder shares this thread budget. See [Threads and +decompression](/simpleaf/threads-and-decompression/) for `--decoder` and +`--thread-policy`. + +## Outputs + +Mapping output is written beneath `af_map/`; corrected and sorted ATAC output is +written beneath `af_process/`. The process log records the exact child commands +and stage timings. `--compress` requests compressed BED output and +`--call-peaks` enables the optional MACS3 stage. diff --git a/docs-site/src/content/docs/flex-quant-command.md b/docs-site/src/content/docs/flex-quant-command.md index 1609a46..697d505 100644 --- a/docs-site/src/content/docs/flex-quant-command.md +++ b/docs-site/src/content/docs/flex-quant-command.md @@ -29,6 +29,35 @@ If the chemistry registry contains the needed metadata, `simpleaf` can automatic The default output is the standard Matrix Market directory under `af_quant/alevin`. If you pass `--anndata-out`, `simpleaf` will additionally write an AnnData `.h5ad` file at `af_quant/alevin/quants.h5ad`. +## Barcode-correction controls + +Correction decisions are made during `alevin-fry generate-permit-list` and +reused by collation through its compiled correction plan. `simpleaf` exposes the +same typed controls without duplicating alevin-fry's protocol defaults: + +| Option | Inherited default | Destination | +| --- | --- | --- | +| `--cell-bc-correction {unique,frequency}` | `unique` | permit-list generation | +| `--cell-bc-neighborhood {hamming-1,substitution-or-shift-1}` | protocol/filter-specific | permit-list generation | +| `--cell-bc-confidence ` | `0.975` | permit-list generation | +| `--sample-bc-correction {exact,unique,frequency}` | `exact` | permit-list generation | +| `--sample-bc-neighborhood {hamming-1,substitution-or-shift-1}` | `hamming-1` for non-exact correction | permit-list generation | +| `--sample-bc-confidence ` | `0.975` | permit-list generation | +| `--gpl-memory-limit ` | `512 MiB` | deferred sample-Frequency buffers | +| `--gpl-tmp-dir ` | permit-list output directory | compressed deferred runs | +| `--collate-memory-limit ` | `2 GiB` | collation buffers | + +Confidence accepts decimals or exact fractions such as `39/40`. Memory accepts +human-readable values such as `512MiB`, `1GiB`, and `4GB`. Values are validated +before mapping begins but forwarded in their original spelling. If an override +is omitted, simpleaf emits no corresponding child flag and alevin-fry remains +the source of truth for the default. + +The old hidden `--sample-correction-mode` option remains accepted for script +compatibility. `exact` becomes `--sample-bc-correction exact`; `1-edit` becomes +`--sample-bc-correction unique --sample-bc-neighborhood +substitution-or-shift-1`. New scripts should use the explicit controls above. + For multiplex output, the resulting AnnData object is intended to preserve the extra sample-level structure of the experiment: - `obs_names` are sample-qualified cell identifiers @@ -57,7 +86,7 @@ Options: --organism Target organism for automatic probe set selection - + [possible values: human, mouse] --cell-bc-list @@ -65,7 +94,7 @@ Options: --expected-ori Expected read orientation: fw, rc, or both - + [default: both] --sample-bc-ori @@ -74,7 +103,7 @@ Options: `sample_bc_ori` when set. Useful for cycle-plan variants (e.g. 10x Flex Configuration B) where the sample BC is read off the opposite strand from the canonical preset. Vocabulary matches the preset JSON and alevin-fry's `--sample-bc-ori` - + [possible values: forward, reverse] -o, --output @@ -82,7 +111,7 @@ Options: -t, --threads Number of threads to use - + [default: 16] -h, --help @@ -91,16 +120,51 @@ Options: -V, --version Print version +Advanced Barcode Correction Options: + --cell-bc-correction + Cell-barcode collision policy (inherited default: unique) + + [possible values: unique, frequency] + + --cell-bc-neighborhood + One-error cell-barcode neighbourhood. When omitted, alevin-fry chooses the + protocol/filter-specific default + + [possible values: hamming-1, substitution-or-shift-1] + + --cell-bc-confidence + Frequency confidence as a decimal or exact fraction. The inherited default is 97.5% for + RNA and 90% for ATAC + + --sample-bc-correction + Sample-barcode correction policy (inherited default: exact) + + [possible values: exact, unique, frequency] + + --sample-bc-neighborhood + Sample-barcode neighbourhood for non-exact correction (inherited default: Hamming-1) + + [possible values: hamming-1, substitution-or-shift-1] + + --sample-bc-confidence + Sample-Frequency confidence as a decimal or exact fraction (inherited default: 97.5%) + +Advanced Resource Options: + --gpl-memory-limit + Deferred GPL buffer budget (inherited default: 512 MiB) + + --gpl-tmp-dir + Temporary directory for compressed GPL correction runs (inherited default: the GPL output + directory) + + --collate-memory-limit + Collation buffer budget (inherited default: 2 GiB) + Permit List Options: - --sample-correction-mode - Sample barcode correction mode - - [default: exact] - [possible values: exact, 1-edit] --min-reads Minimum read count threshold for unfiltered permit list - + [default: 10] Mapping Options: @@ -120,7 +184,7 @@ Probe Set Options: --kmer-length k-mer length for probe index building - + [default: 23] Reference Options: @@ -141,7 +205,7 @@ Reference Options: Quantification Options: -r, --resolution UMI resolution mode - + [default: cr-like] [possible values: cr-like, cr-like-em, parsimony, parsimony-em, parsimony-gene, parsimony-gene-em] @@ -150,14 +214,13 @@ Quantification Options: Cells with fewer than this many reads are resolved by alevin-fry's tiny-cell fast path, which applies `cr-like` (winner-take-all) semantics regardless of `--resolution`. Pass 0 to resolve every cell with the requested strategy. - - Left unset, alevin-fry's own default applies. Requires alevin-fry >= 0.17.1; earlier - versions parse the option and ignore it. + + Left unset, alevin-fry's own default applies Piscem Mapping Options: --skipping-strategy The skipping strategy to use for k-mer collection - + [default: permissive] [possible values: permissive, strict] @@ -166,29 +229,29 @@ Piscem Mapping Options: --max-ec-card Maximum cardinality equivalence class to examine - + [default: 4096] --dict Piscem dictionary backend: `auto` (default), `sshash`, or `tiny`. Applies both to the auto-built probe index (build time) and to map-sc (map time) - + [default: auto] [possible values: auto, sshash, tiny] --decoder Gzip decoder selection passed to piscem: `auto`, `serial`, `parallel`, or `parallel=N`. - + `auto` lets piscem adapt the mapping/decode split while the run proceeds. `serial` gives mapping the whole budget. `parallel` forces the parallel decoder where the input allows it; `parallel=N` fixes N decode slots per gzip input and stops the adaptation. Inputs that cannot be read positionally (FIFOs, process substitution) stay serial regardless. - + [default: auto] --thread-policy JSON file overriding piscem's thread and decoder policy. - + Every field is optional and defaults to a measured value; an unrecognised field is an error rather than a silent no-op. Currently understood: `{"parallel_decode": {"min_threads_per_stream": 8}}`, the number of threads that must be free per gzip input diff --git a/docs-site/src/content/docs/installation.md b/docs-site/src/content/docs/installation.md index fce5731..997414c 100644 --- a/docs-site/src/content/docs/installation.md +++ b/docs-site/src/content/docs/installation.md @@ -5,7 +5,7 @@ title: "Installation" `Simpleaf` can be installed from source, from [crates.io](https://crates.io/crates/simpleaf), or installed via [bioconda](https://bioconda.github.io/recipes/simpleaf/README.html). `simpleaf` requires [alevin-fry](https://github.com/COMBINE-lab/alevin-fry), [piscem](https://github.com/COMBINE-lab/piscem), and `wget`. :::caution -This release requires **piscem >= 0.22.0** and **alevin-fry >= 0.17.0**. These are hard requirements, not recommendations: `simpleaf set-paths` will reject older binaries. `piscem` 0.22.0 changed the meaning of `-t` (see [Threads and decompression](/simpleaf/threads-and-decompression/)) and added the build options `simpleaf` now passes unconditionally, so an older `piscem` cannot run these commands. If you install from bioconda, check that the versions it resolves meet these floors. +This release requires **piscem >= 0.22.0** and **alevin-fry >= 0.18.0**. These are hard requirements, not recommendations: `simpleaf set-paths` will reject older binaries. `piscem` 0.22.0 changed the meaning of `-t` (see [Threads and decompression](/simpleaf/threads-and-decompression/)), while alevin-fry 0.18.0 provides the deterministic correction and resource controls forwarded by simpleaf 0.28. If you install from bioconda, check that the versions it resolves meet these floors. ::: ## Recommended: installing from conda diff --git a/docs-site/src/content/docs/quant-command.md b/docs-site/src/content/docs/quant-command.md index f40bd03..69a5b5f 100644 --- a/docs-site/src/content/docs/quant-command.md +++ b/docs-site/src/content/docs/quant-command.md @@ -10,18 +10,37 @@ The `quant` command takes as input **either**: and runs all relevant the steps of the `alevin-fry` pipeline. When processing a new dataset from scratch, the first option is the one you are likely interested in (you will provide the `--index`, `--reads1` and `--reads2` arguments). **If multiple read files are provided to the** `--reads1` **and** `--reads2` **arguments, those files must be comma (,) separated.** -On the other hand, if you have already performed quantification or have, for some other reason, already mapped the reads to produce a RAD file, you can start the process from the mapped read directory directly using the `--map-dir` argument instead. This latter approach makes it easy to test out different quantification approaches (e.g. different filtering options or UMI resolution strategies). +On the other hand, if you have already performed quantification or have, for some other reason, already mapped the reads to produce a RAD file, you can start the process from the mapped read directory directly using the `--map-dir` argument instead. This latter approach makes it easy to test out different quantification approaches (e.g. different filtering options or UMI resolution strategies). **Note**: If you use the unfiltered-permit-list `-u` mode for permit-list generation, and you are using either `10xv2` or `10xv3` chemistry, you can provide the flag by itself, and `simpleaf` will automatically fetch and apply the appropriate unifltered permit list. However, if you are using `-u` with any other chemistry, you must explicitly provide a path to the unfiltered permit list to be used. The `-d`/`--expected-ori` flag allows controlling the like-named option that is passed to the `generate-permit-list` command of `alevin-fry`. This is an "optional" option. If it is not provided explicitly, it is set to "both" (allowing reads aligning in both orientations to pass through), unless the chemistry is set as `10xv2` or `10xv3`, in which case it is set as "fw". Regardless of the chemistry, if the user sets this option explicitly, this choice is respected. The default output format is a Matrix Market format sparse matrix with the relevant counts. However, if you pass the `--anndata-out` flag to the `quant` command (in addition to the normal `-o` argument to specify the output directory), then additionally an [AnnData](https://anndata.readthedocs.io/en/stable/) file will be created, which should be directly usable in downstream workflows expecting this data type. +## Barcode correction and collation resources + +`simpleaf` normally leaves barcode-correction policy and resource defaults to +`alevin-fry`. This is intentional: an omitted option is not written into the +child command, so protocol- and filter-specific defaults continue to be resolved +by the version of `alevin-fry` doing the work. The advanced controls are: + +- `--cell-bc-correction {unique,frequency}` (inherited default: `unique`) +- `--cell-bc-neighborhood {hamming-1,substitution-or-shift-1}` (inherited + protocol/filter-specific default) +- `--cell-bc-confidence ` (inherited RNA default: `0.975`; decimals + and exact fractions such as `39/40` are accepted) +- `--collate-memory-limit ` (inherited default: `2 GiB`; for example, + `768MiB` or `4GB`) + +These options are validated before mapping starts and are forwarded only when +you explicitly provide them. `simpleaf` does not expose alevin-fry's deprecated +collation-mode or maximum-record-count controls. + ## A note on the `--chemistry` flag :::note The geometry specification language has changed in `simpleaf` v0.9.0 and above. This change is to unify the geometry description language between `simpleaf` and the tools in the backend that actually perform the fragment mapping. Further, the new laguage is more general, capable and exensible, so it will be easier to add more features in the future in a backward compatible manner. However, this means that if you have a `custom_chemistries.json` file from before `simpleaf` v0.9.0, you will have to re-create that file with the new chemistries by overwriting them with the custom geometry descriptions in the new format. ::: -The `--chemistry` option can take either a string describing the specific chemisty, or a string describing the geometry of the barcode, umi and mappable read. For example, the string `10xv2` and `10xv3` will apply the appropriate settings for the 10x chromium v2 and v3 protocols respectively. However, general geometries can be provided as well, in case the chemistry you are trying to use has not been added as a pre-registered option. For example, the instead of providing the `--chemistry` flag with the string `10xv2`, you could instead provide it with the string `"1{b[16]u[10]x:}2{r:}"`, or, instead of providing `10xv3` you could provide `"1{b[16]u[12]x:}2{r:}"`. +The `--chemistry` option can take either a string describing the specific chemisty, or a string describing the geometry of the barcode, umi and mappable read. For example, the string `10xv2` and `10xv3` will apply the appropriate settings for the 10x chromium v2 and v3 protocols respectively. However, general geometries can be provided as well, in case the chemistry you are trying to use has not been added as a pre-registered option. For example, the instead of providing the `--chemistry` flag with the string `10xv2`, you could instead provide it with the string `"1{b[16]u[10]x:}2{r:}"`, or, instead of providing `10xv3` you could provide `"1{b[16]u[12]x:}2{r:}"`. The custom format is as follows; you must specify the content of read 1 and read 2 in terms of the barcode, UMI, and mappable read sequence. A specification looks like this: @@ -62,11 +81,8 @@ cells took the fast path is recorded in `af_quant/quant.json` as is auditable after the fact. :::note -This option requires **alevin-fry >= 0.17.1**. Earlier versions parse it and -silently ignore it, so on 0.17.0 passing `--small-thresh 0` will have no -effect. On a 73k-cell PBMC run, `--small-thresh 0` recovers about 0.53% of -total UMI mass and empties no cells, against 132 zero-count cells at the -default. +On a 73k-cell PBMC run, `--small-thresh 0` recovers about 0.53% of total UMI +mass and empties no cells, against 132 zero-count cells at the default. ::: ## Threads, and the shared decode budget @@ -79,6 +95,13 @@ you steer or pin that split; see and when changing them is worthwhile. The defaults adapt on their own, so neither option needs to be set for a normal run. +Two threads is the practical minimum throughout the pipeline. If you request +zero or one, `simpleaf` emits a prominent warning, resolves the count to two, +and uses that same value for mapping, permit-list generation, collation, and +quantification. It follows the same policy if the operating system reports only +one available execution slot: the discrepancy is warned about and two threads +are still attempted. + The relevant options (which you can obtain by running `simpleaf quant --help`) are below: ```sh @@ -96,7 +119,7 @@ Options: -t, --threads Number of threads to use when running - + [default: 16] -h, --help @@ -124,7 +147,7 @@ Piscem Mapping Options: --with-position Record the position of each mapped read in the RAD file. - + alevin-fry detects the positional record type from the RAD header and adapts automatically, so no downstream option needs to change. The RAD file is larger. Not available for `multiplex-quant`: there is no multi-barcode positional record type, and the @@ -141,53 +164,53 @@ Piscem Mapping Options: --skipping-strategy The skipping strategy to use for k-mer collection - + [default: permissive] [possible values: permissive, strict] --max-ec-card Determines the maximum cardinality equivalence class (number of (txp, orientation status) pairs) to examine (cannot be used with --ignore-ambig-hits) - + [default: 4096] --max-hit-occ In the first pass, consider only collected and matched k-mers of a read having <= --max-hit-occ hits - + [default: 256] --max-hit-occ-recover If all collected and matched k-mers of a read have > --max-hit-occ hits, then make a second pass and consider k-mers having <= --max-hit-occ-recover hits - + [default: 1024] --max-read-occ Threshold for discarding reads with too many mappings - + [default: 2500] --dict Piscem dictionary backend to use at map time: `auto` (default, honors the index's embedded choice), `sshash`, or `tiny` - + [default: auto] [possible values: auto, sshash, tiny] --decoder Gzip decoder selection passed to piscem: `auto`, `serial`, `parallel`, or `parallel=N`. - + `auto` lets piscem adapt the mapping/decode split while the run proceeds. `serial` gives mapping the whole budget. `parallel` forces the parallel decoder where the input allows it; `parallel=N` fixes N decode slots per gzip input and stops the adaptation. Inputs that cannot be read positionally (FIFOs, process substitution) stay serial regardless. - + [default: auto] --thread-policy JSON file overriding piscem's thread and decoder policy. - + Every field is optional and defaults to a measured value; an unrecognised field is an error rather than a silent no-op. Currently understood: `{"parallel_decode": {"min_threads_per_stream": 8}}`, the number of threads that must be free per gzip input @@ -212,22 +235,42 @@ Permit List Generation Options: -d, --expected-ori The expected direction/orientation of alignments in the chemistry being processed. If not provided, will default to `fw` for 10xv2/10xv3, otherwise `both` - + [possible values: fw, rc, both] --min-reads Minimum read count threshold for a cell to be retained/processed; only use with --unfiltered-pl - + [default: 10] +Advanced Barcode Correction Options: + --cell-bc-correction + Cell-barcode collision policy (inherited default: unique) + + [possible values: unique, frequency] + + --cell-bc-neighborhood + One-error cell-barcode neighbourhood. When omitted, alevin-fry chooses the + protocol/filter-specific default + + [possible values: hamming-1, substitution-or-shift-1] + + --cell-bc-confidence + Frequency confidence as a decimal or exact fraction. The inherited default is 97.5% for + RNA and 90% for ATAC + +Advanced Resource Options: + --collate-memory-limit + Collation buffer budget (inherited default: 2 GiB) + UMI Resolution Options: -m, --t2g-map Path to a transcript to gene map file -r, --resolution UMI resolution mode - + [possible values: cr-like, cr-like-em, parsimony, parsimony-em, parsimony-gene, parsimony-gene-em] @@ -235,9 +278,8 @@ UMI Resolution Options: Cells with fewer than this many reads are resolved by alevin-fry's tiny-cell fast path, which applies `cr-like` (winner-take-all) semantics regardless of `--resolution`. Pass 0 to resolve every cell with the requested strategy. - - Left unset, alevin-fry's own default applies. Requires alevin-fry >= 0.17.1; earlier - versions parse the option and ignore it. + + Left unset, alevin-fry's own default applies Output Options: --anndata-out diff --git a/docs-site/src/content/docs/threads-and-decompression.md b/docs-site/src/content/docs/threads-and-decompression.md index 786f59c..9a7d3fb 100644 --- a/docs-site/src/content/docs/threads-and-decompression.md +++ b/docs-site/src/content/docs/threads-and-decompression.md @@ -9,6 +9,13 @@ decompression**, rather than a mapping-thread count with decompression happening somewhere off the books. A broker inside `piscem` decides how to split that budget and re-solves the split while the run proceeds. +`simpleaf` and `alevin-fry` use two threads as a practical minimum. A request of +zero or one is accepted, accompanied by a prominent warning, and raised to two. +If the operating system reports only one available execution slot, simpleaf +warns about that discrepancy and still attempts two. The effective value is +resolved once per pipeline and reused by mapping, permit-list generation, +collation or ATAC sorting, and quantification. + This page explains the two options that steer it. Both are forwarded verbatim to `piscem`, and both are available on [quant](/simpleaf/quant-command/), [multiplex-quant](/simpleaf/flex-quant-command/), and `simpleaf atac process`. diff --git a/release.sh b/release.sh deleted file mode 100755 index b7442a1..0000000 --- a/release.sh +++ /dev/null @@ -1,284 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -usage() { - cat <<'USAGE' -Usage: release.sh [--push] [--dry-run] - -Creates a release by: -1. Validating as semantic versioning (SemVer 2.0.0) -2. Ensuring is greater than the current Cargo.toml package version -3. Ensuring tag v does not already exist -4. Updating Cargo.toml [package].version -5. Committing Cargo.toml and creating tag v -6. Optionally pushing the tag when --push is provided -7. With --dry-run, print planned actions without changing git state/files -USAGE -} - -err() { - echo "Error: $*" >&2 - exit 1 -} - -is_valid_semver() { - local version="$1" - local semver_regex='^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-((0|[1-9][0-9]*|[0-9A-Za-z-]*[A-Za-z-][0-9A-Za-z-]*)(\.(0|[1-9][0-9]*|[0-9A-Za-z-]*[A-Za-z-][0-9A-Za-z-]*))*))?(\+([0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*))?$' - printf '%s\n' "$version" | grep -Eq "$semver_regex" -} - -split_semver() { - local version="$1" - local core="$version" - local pre="" - local major minor patch - - if [[ "$core" == *+* ]]; then - core="${core%%+*}" - fi - - if [[ "$core" == *-* ]]; then - pre="${core#*-}" - core="${core%%-*}" - fi - - IFS='.' read -r major minor patch <<<"$core" - printf '%s\t%s\t%s\t%s\n' "$major" "$minor" "$patch" "$pre" -} - -compare_prerelease() { - local left="$1" - local right="$2" - - if [[ -z "$left" && -z "$right" ]]; then - echo 0 - return - fi - if [[ -z "$left" ]]; then - echo 1 - return - fi - if [[ -z "$right" ]]; then - echo -1 - return - fi - - local -a lparts=() - local -a rparts=() - IFS='.' read -r -a lparts <<<"$left" - IFS='.' read -r -a rparts <<<"$right" - - local i - local lnum rnum - for ((i = 0; i < ${#lparts[@]} || i < ${#rparts[@]}; i++)); do - if ((i >= ${#lparts[@]})); then - echo -1 - return - fi - if ((i >= ${#rparts[@]})); then - echo 1 - return - fi - - if [[ "${lparts[$i]}" == "${rparts[$i]}" ]]; then - continue - fi - - if [[ "${lparts[$i]}" =~ ^(0|[1-9][0-9]*)$ ]]; then - lnum=1 - else - lnum=0 - fi - if [[ "${rparts[$i]}" =~ ^(0|[1-9][0-9]*)$ ]]; then - rnum=1 - else - rnum=0 - fi - - if ((lnum == 1 && rnum == 1)); then - if ((10#${lparts[$i]} > 10#${rparts[$i]})); then - echo 1 - else - echo -1 - fi - return - fi - if ((lnum == 1 && rnum == 0)); then - echo -1 - return - fi - if ((lnum == 0 && rnum == 1)); then - echo 1 - return - fi - - if [[ "${lparts[$i]}" > "${rparts[$i]}" ]]; then - echo 1 - else - echo -1 - fi - return - done - - echo 0 -} - -compare_semver() { - local left="$1" - local right="$2" - - local lmaj lmin lpat lpre - local rmaj rmin rpat rpre - IFS=$'\t' read -r lmaj lmin lpat lpre <<<"$(split_semver "$left")" - IFS=$'\t' read -r rmaj rmin rpat rpre <<<"$(split_semver "$right")" - - if ((10#$lmaj > 10#$rmaj)); then - echo 1 - return - fi - if ((10#$lmaj < 10#$rmaj)); then - echo -1 - return - fi - - if ((10#$lmin > 10#$rmin)); then - echo 1 - return - fi - if ((10#$lmin < 10#$rmin)); then - echo -1 - return - fi - - if ((10#$lpat > 10#$rpat)); then - echo 1 - return - fi - if ((10#$lpat < 10#$rpat)); then - echo -1 - return - fi - - compare_prerelease "$lpre" "$rpre" -} - -push_tag=false -dry_run=false -version="" - -while [[ $# -gt 0 ]]; do - case "$1" in - --push) - push_tag=true - shift - ;; - --dry-run) - dry_run=true - shift - ;; - -h|--help) - usage - exit 0 - ;; - -*) - err "Unknown option: $1" - ;; - *) - if [[ -n "$version" ]]; then - err "Provide exactly one version argument." - fi - version="$1" - shift - ;; - esac -done - -[[ -n "$version" ]] || { usage >&2; exit 1; } - -[[ -f "Cargo.toml" ]] || err "Cargo.toml not found in current directory." -git rev-parse --is-inside-work-tree >/dev/null 2>&1 || err "Current directory is not a git repository." - -is_valid_semver "$version" || err "Version '$version' is not valid SemVer 2.0.0." - -current_version="$( - awk ' - /^\[package\]/ { in_package=1; next } - in_package && /^\[/ { in_package=0 } - in_package && /^[[:space:]]*version[[:space:]]*=/ { - match($0, /"[^"]+"/) - if (RSTART > 0) { - print substr($0, RSTART + 1, RLENGTH - 2) - exit - } - } - ' Cargo.toml -)" - -[[ -n "$current_version" ]] || err "Could not find [package].version in Cargo.toml." -is_valid_semver "$current_version" || err "Current Cargo.toml version '$current_version' is not valid SemVer." - -cmp="$(compare_semver "$version" "$current_version")" -if [[ "$cmp" != "1" ]]; then - err "Version '$version' must be greater than current version '$current_version'." -fi - -tag="v${version}" -if git rev-parse -q --verify "refs/tags/${tag}" >/dev/null 2>&1; then - err "Tag '${tag}' already exists locally." -fi -if git remote get-url origin >/dev/null 2>&1; then - if git ls-remote --exit-code --tags origin "refs/tags/${tag}" >/dev/null 2>&1; then - err "Tag '${tag}' already exists on remote 'origin'." - fi -fi - -tmp_file="$(mktemp)" -cleanup() { - rm -f "$tmp_file" -} -trap cleanup EXIT - -awk -v new_version="$version" ' - BEGIN { in_package=0; updated=0 } - /^\[package\]/ { in_package=1; print; next } - in_package && /^\[/ { in_package=0 } - in_package && !updated && /^[[:space:]]*version[[:space:]]*=/ { - print "version = \"" new_version "\"" - updated=1 - next - } - { print } - END { - if (!updated) { - exit 2 - } - } -' Cargo.toml > "$tmp_file" || err "Failed to update Cargo.toml." - -if [[ "$dry_run" == true ]]; then - echo "Dry run successful:" - echo "- Current version: ${current_version}" - echo "- New version: ${version}" - echo "- Tag to create: ${tag}" - echo "- Would update Cargo.toml [package].version to ${version}" - echo "- Would commit: Release ${tag}" - echo "- Would create tag: ${tag}" - if [[ "$push_tag" == true ]]; then - git remote get-url origin >/dev/null 2>&1 || err "No 'origin' remote configured, cannot push tag." - echo "- Would push tag to origin: ${tag}" - fi - exit 0 -fi - -mv "$tmp_file" Cargo.toml - -git add Cargo.toml -git commit -m "Release ${tag}" -- Cargo.toml -git tag "${tag}" - -if [[ "$push_tag" == true ]]; then - git remote get-url origin >/dev/null 2>&1 || err "No 'origin' remote configured, cannot push tag." - git push origin "${tag}" -fi - -echo "Released ${tag} (updated Cargo.toml from ${current_version} to ${version})." diff --git a/src/atac/commands.rs b/src/atac/commands.rs index 4998522..d490bcb 100644 --- a/src/atac/commands.rs +++ b/src/atac/commands.rs @@ -1,6 +1,8 @@ use crate::atac::defaults::{AtacIndexParams, DefaultAtacParams}; use crate::defaults::{DefaultMappingParams, DefaultParams}; -use crate::simpleaf_commands::{PiscemBuildResourceOpts, PiscemDecoderOpts}; +use crate::simpleaf_commands::{ + CellBarcodeCorrectionOpts, PiscemBuildResourceOpts, PiscemDecoderOpts, +}; use crate::utils::chem_utils::{ExpectedOri, QueryInRegistry}; use clap::{ Args, Subcommand, ValueEnum, @@ -165,7 +167,7 @@ pub enum AtacCommand { #[derive(Args, Clone, Debug)] #[command(arg_required_else_help = true)] pub struct IndexOpts { - /// number of threads to use when running + /// number of threads to use when running; values below two warn and use two #[arg(short, long, default_value_t = 16, display_order = 5)] pub threads: u32, @@ -305,7 +307,7 @@ pub struct ProcessOpts { #[arg(long = "output")] pub output: PathBuf, - /// number of threads to use when running + /// number of threads to use when running; values below two warn and use two #[arg(short, long, default_value_t = 16, display_order = 5)] pub threads: u32, @@ -349,6 +351,9 @@ pub struct ProcessOpts { )] pub min_reads: usize, + #[command(flatten)] + pub cell_correction: CellBarcodeCorrectionOpts, + /// compress the output mapping bed file. #[arg(long, help_heading = "Advanced Options")] pub compress: bool, diff --git a/src/atac/process.rs b/src/atac/process.rs index b02b31f..94f0ac9 100644 --- a/src/atac/process.rs +++ b/src/atac/process.rs @@ -1,5 +1,5 @@ use crate::atac::commands::ProcessOpts; -use crate::core::{context, exec, index_meta, io, runtime}; +use crate::core::{context, exec, index_meta, io}; use crate::utils::chem_utils::ExpectedOri; use crate::utils::chem_utils::QueryInRegistry; use crate::utils::chem_utils::get_single_custom_chem_from_file; @@ -241,14 +241,11 @@ pub(crate) fn map_reads(af_home_path: &Path, opts: &ProcessOpts) -> anyhow::Resu .arg("--bin-overlap") .arg(opts.bin_overlap.to_string()); - // if the user requested more threads than can be used - let threads = runtime::cap_threads_warned(opts.threads); - // location of output directory, number of threads let map_output = opts.output.join("af_map"); piscem_map_cmd .arg("--threads") - .arg(threads.to_string()) + .arg(opts.threads.to_string()) .arg("-o") .arg(&map_output); @@ -398,9 +395,7 @@ fn af_sort(af_home_path: &Path, opts: &ProcessOpts) -> anyhow::Result anyhow::Result (u32, Option) { ) } -/// Cap `requested` at the parallelism actually available to this process, and -/// explain the cap if one was applied. +/// Resolve `requested` against the two-thread floor and the parallelism +/// available to this process, and explain every adjustment. /// /// Every caller previously open-coded this warning, and the message was hard to /// act on: `available_parallelism` reports the CPUs *this process may use*, not /// how many the machine has, so a scheduler or container that hands out a single /// CPU produces "maximum available parallelism is 1" on a 48-core node and the -/// run silently drops to one thread (COMBINE-lab/simpleaf#135). Naming the cause -/// is the difference between a confusing log line and a fixable job script. +/// run used to silently drop to one thread (COMBINE-lab/simpleaf#135). A report +/// of one is now the sole exception to the upper cap: simpleaf warns and still +/// attempts the practical minimum of two threads. pub fn cap_threads_warned(requested: u32) -> u32 { let (effective, capped_at) = cap_threads(requested); - if let Some(max_threads) = capped_at { - warn!( - "{} threads were requested, but only {} are available to this process; \ - using {}.", - requested, max_threads, max_threads - ); + if requested < 2 { warn!( - "This reflects the CPU affinity / cgroup limit visible to simpleaf, not the \ - machine's total core count. If the machine has more cores, the limit is \ - usually set by the job scheduler (e.g. an unset or low --cpus-per-task) or \ - by the container runtime." + "{} thread(s) were requested, but 2 threads is the practical minimum for simpleaf; using 2.", + requested ); } + if let Some(max_threads) = capped_at { + if max_threads < 2 { + warn!( + "std::thread::available_parallelism() reported {} thread, but 2 threads is the practical minimum for simpleaf; attempting to use 2.", + max_threads + ); + } else { + warn!( + "{} threads were requested, but only {} are available to this process; using {}.", + requested, max_threads, max_threads + ); + warn!( + "This reflects the CPU affinity / cgroup limit visible to simpleaf, not the machine's total core count. If the machine has more cores, the limit is usually set by the job scheduler (e.g. an unset or low --cpus-per-task) or by the container runtime." + ); + } + } effective } fn cap_threads_with_limit(requested: u32, limit: Option) -> (u32, Option) { - if let Some(max_threads) = limit - && requested > max_threads - { - return (max_threads, Some(max_threads)); + let requested = requested.max(2); + if let Some(max_threads) = limit { + if max_threads < 2 { + return (2, Some(max_threads)); + } + if requested > max_threads { + return (max_threads, Some(max_threads)); + } } (requested, None) } @@ -63,6 +77,19 @@ mod tests { assert_eq!(capped_at, None); } + #[test] + fn raises_zero_and_one_to_the_practical_minimum() { + assert_eq!(cap_threads_with_limit(0, Some(8)), (2, None)); + assert_eq!(cap_threads_with_limit(1, Some(8)), (2, None)); + assert_eq!(cap_threads_with_limit(2, Some(8)), (2, None)); + } + + #[test] + fn attempts_two_when_available_parallelism_reports_one() { + assert_eq!(cap_threads_with_limit(1, Some(1)), (2, Some(1))); + assert_eq!(cap_threads_with_limit(16, Some(1)), (2, Some(1))); + } + #[test] fn reports_no_cap_when_parallelism_is_unknown() { // `available_parallelism` can fail; when it does, the user's request @@ -71,4 +98,10 @@ mod tests { assert_eq!(effective, 32); assert_eq!(capped_at, None); } + + #[test] + fn unknown_parallelism_still_applies_the_thread_floor() { + assert_eq!(cap_threads_with_limit(0, None), (2, None)); + assert_eq!(cap_threads_with_limit(1, None), (2, None)); + } } diff --git a/src/main.rs b/src/main.rs index 086ff7f..dd51f81 100644 --- a/src/main.rs +++ b/src/main.rs @@ -112,7 +112,8 @@ fn main() -> anyhow::Result<()> { } // processing for ATAC-seq data - Commands::Atac(AtacCommand::Process(process_opts)) => { + Commands::Atac(AtacCommand::Process(mut process_opts)) => { + process_opts.threads = crate::core::runtime::cap_threads_warned(process_opts.threads); // validate versions atac::process::check_progs(&af_home_path, &process_opts)?; // first we map the reads diff --git a/src/simpleaf_commands.rs b/src/simpleaf_commands.rs index 4bb56f5..695499f 100644 --- a/src/simpleaf_commands.rs +++ b/src/simpleaf_commands.rs @@ -135,6 +135,295 @@ impl PiscemBuildResourceOpts { } } +/// Collision policy for cell-barcode correction in alevin-fry. +#[derive(ValueEnum, Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellBarcodeCorrection { + Unique, + Frequency, +} + +impl CellBarcodeCorrection { + fn as_cli(self) -> &'static str { + match self { + Self::Unique => "unique", + Self::Frequency => "frequency", + } + } +} + +/// One-error barcode neighbourhood searched by alevin-fry. +#[derive(ValueEnum, Clone, Copy, Debug, PartialEq, Eq)] +pub enum BarcodeNeighborhood { + #[value(name = "hamming-1")] + HammingOne, + #[value(name = "substitution-or-shift-1")] + SubstitutionOrShiftOne, +} + +impl BarcodeNeighborhood { + fn as_cli(self) -> &'static str { + match self { + Self::HammingOne => "hamming-1", + Self::SubstitutionOrShiftOne => "substitution-or-shift-1", + } + } +} + +/// Collision policy for sample/probe-barcode correction in alevin-fry. +#[derive(ValueEnum, Clone, Copy, Debug, PartialEq, Eq)] +pub enum SampleBarcodeCorrection { + Exact, + Unique, + Frequency, +} + +impl SampleBarcodeCorrection { + fn as_cli(self) -> &'static str { + match self { + Self::Exact => "exact", + Self::Unique => "unique", + Self::Frequency => "frequency", + } + } +} + +#[derive(ValueEnum, Clone, Copy, Debug, PartialEq, Eq)] +enum LegacySampleCorrectionMode { + Exact, + #[value(name = "1-edit")] + OneEdit, +} + +fn confidence_parser(value: &str) -> Result { + let value = value.trim(); + let (numerator, denominator) = if let Some((numerator, denominator)) = value.split_once('/') { + let numerator = numerator + .parse::() + .map_err(|_| format!("invalid barcode-correction confidence {value:?}"))?; + let denominator = denominator + .parse::() + .map_err(|_| format!("invalid barcode-correction confidence {value:?}"))?; + (numerator, denominator) + } else { + let (whole, fractional) = value.split_once('.').unwrap_or((value, "")); + if whole.is_empty() || !whole.bytes().all(|byte| byte.is_ascii_digit()) { + return Err(format!("invalid barcode-correction confidence {value:?}")); + } + if !fractional.bytes().all(|byte| byte.is_ascii_digit()) || fractional.len() > 18 { + return Err(format!("invalid barcode-correction confidence {value:?}")); + } + let denominator = 10_u64 + .checked_pow(fractional.len() as u32) + .ok_or_else(|| "barcode-correction confidence is too precise".to_string())?; + let whole = whole + .parse::() + .map_err(|_| format!("invalid barcode-correction confidence {value:?}"))?; + let fractional = if fractional.is_empty() { + 0 + } else { + fractional + .parse::() + .map_err(|_| format!("invalid barcode-correction confidence {value:?}"))? + }; + let numerator = whole + .checked_mul(denominator) + .and_then(|whole| whole.checked_add(fractional)) + .ok_or_else(|| "barcode-correction confidence is too large".to_string())?; + (numerator, denominator) + }; + + if denominator == 0 || numerator > denominator { + return Err(format!( + "barcode-correction confidence must be between zero and one (got {numerator}/{denominator})" + )); + } + Ok(value.to_string()) +} + +fn memory_size_parser(value: &str) -> Result { + let value = value.trim(); + let split = value + .find(|character: char| !character.is_ascii_digit() && character != '.') + .unwrap_or(value.len()); + let (number, suffix) = value.split_at(split); + if number.is_empty() { + return Err("memory size must start with a positive number".to_string()); + } + let number = number + .parse::() + .map_err(|_| format!("invalid memory size {value:?}"))?; + if !number.is_finite() || number <= 0.0 { + return Err("memory size must be positive and finite".to_string()); + } + let multiplier = match suffix.trim().to_ascii_lowercase().as_str() { + "" | "b" => 1_u64, + "k" | "kb" => 1_000, + "ki" | "kib" => 1_u64 << 10, + "m" | "mb" => 1_000_000, + "mi" | "mib" => 1_u64 << 20, + "g" | "gb" => 1_000_000_000, + "gi" | "gib" => 1_u64 << 30, + "t" | "tb" => 1_000_000_000_000, + "ti" | "tib" => 1_u64 << 40, + other => return Err(format!("unsupported memory-size suffix {other:?}")), + }; + if number * multiplier as f64 > u64::MAX as f64 { + return Err("memory size exceeds the supported range".to_string()); + } + Ok(value.to_string()) +} + +/// Optional cell-barcode overrides shared by RNA, multiplex RNA, and ATAC. +#[derive(Args, Clone, Debug, Default)] +pub struct CellBarcodeCorrectionOpts { + /// Cell-barcode collision policy (inherited default: unique). + #[arg(long, value_enum, help_heading = "Advanced Barcode Correction Options")] + pub cell_bc_correction: Option, + + /// One-error cell-barcode neighbourhood. When omitted, alevin-fry chooses + /// the protocol/filter-specific default. + #[arg(long, value_enum, help_heading = "Advanced Barcode Correction Options")] + pub cell_bc_neighborhood: Option, + + /// Frequency confidence as a decimal or exact fraction. The inherited + /// default is 97.5% for RNA and 90% for ATAC. + #[arg( + long, + value_name = "CONFIDENCE", + value_parser = confidence_parser, + help_heading = "Advanced Barcode Correction Options" + )] + pub cell_bc_confidence: Option, +} + +impl CellBarcodeCorrectionOpts { + pub(crate) fn append_to(&self, command: &mut std::process::Command) { + if let Some(policy) = self.cell_bc_correction { + command.arg("--cell-bc-correction").arg(policy.as_cli()); + } + if let Some(neighborhood) = self.cell_bc_neighborhood { + command + .arg("--cell-bc-neighborhood") + .arg(neighborhood.as_cli()); + } + if let Some(confidence) = &self.cell_bc_confidence { + command.arg("--cell-bc-confidence").arg(confidence); + } + } +} + +/// Optional collation resource override shared by RNA workflows. +#[derive(Args, Clone, Debug, Default)] +pub struct CollationResourceOpts { + /// Collation buffer budget (inherited default: 2 GiB). + #[arg( + long, + value_name = "SIZE", + value_parser = memory_size_parser, + help_heading = "Advanced Resource Options" + )] + pub collate_memory_limit: Option, +} + +impl CollationResourceOpts { + pub(crate) fn append_to(&self, command: &mut std::process::Command) { + if let Some(memory_limit) = &self.collate_memory_limit { + command.arg("--memory-limit").arg(memory_limit); + } + } +} + +/// Optional sample/probe-barcode overrides for multiplex RNA. +#[derive(Args, Clone, Debug, Default)] +pub struct SampleBarcodeCorrectionOpts { + /// Sample-barcode correction policy (inherited default: exact). + #[arg(long, value_enum, help_heading = "Advanced Barcode Correction Options")] + pub sample_bc_correction: Option, + + /// Sample-barcode neighbourhood for non-exact correction (inherited + /// default: Hamming-1). + #[arg(long, value_enum, help_heading = "Advanced Barcode Correction Options")] + pub sample_bc_neighborhood: Option, + + /// Sample-Frequency confidence as a decimal or exact fraction (inherited + /// default: 97.5%). + #[arg( + long, + value_name = "CONFIDENCE", + value_parser = confidence_parser, + help_heading = "Advanced Barcode Correction Options" + )] + pub sample_bc_confidence: Option, + + /// Deprecated compatibility alias for sample-barcode correction. + #[arg( + long, + value_enum, + hide = true, + conflicts_with_all = ["sample_bc_correction", "sample_bc_neighborhood"] + )] + sample_correction_mode: Option, +} + +impl SampleBarcodeCorrectionOpts { + pub(crate) fn append_to(&self, command: &mut std::process::Command) { + match self.sample_correction_mode { + Some(LegacySampleCorrectionMode::Exact) => { + command.arg("--sample-bc-correction").arg("exact"); + } + Some(LegacySampleCorrectionMode::OneEdit) => { + command + .arg("--sample-bc-correction") + .arg("unique") + .arg("--sample-bc-neighborhood") + .arg("substitution-or-shift-1"); + } + None => { + if let Some(policy) = self.sample_bc_correction { + command.arg("--sample-bc-correction").arg(policy.as_cli()); + } + if let Some(neighborhood) = self.sample_bc_neighborhood { + command + .arg("--sample-bc-neighborhood") + .arg(neighborhood.as_cli()); + } + } + } + if let Some(confidence) = &self.sample_bc_confidence { + command.arg("--sample-bc-confidence").arg(confidence); + } + } +} + +/// Resource controls used only by deferred sample-Frequency GPL correction. +#[derive(Args, Clone, Debug, Default)] +pub struct GplResourceOpts { + /// Deferred GPL buffer budget (inherited default: 512 MiB). + #[arg( + long, + value_name = "SIZE", + value_parser = memory_size_parser, + help_heading = "Advanced Resource Options" + )] + pub gpl_memory_limit: Option, + + /// Temporary directory for compressed GPL correction runs (inherited + /// default: the GPL output directory). + #[arg(long, value_name = "DIR", help_heading = "Advanced Resource Options")] + pub gpl_tmp_dir: Option, +} + +impl GplResourceOpts { + pub(crate) fn append_to(&self, command: &mut std::process::Command) { + if let Some(memory_limit) = &self.gpl_memory_limit { + command.arg("--memory-limit").arg(memory_limit); + } + if let Some(tmp_dir) = &self.gpl_tmp_dir { + command.arg("--tmp-dir").arg(tmp_dir); + } + } +} + /// The type of references we might create /// to map against for quantification with /// alevin-fry. @@ -175,7 +464,7 @@ pub struct MapQuantOpts { #[arg(short, long)] pub output: PathBuf, - /// Number of threads to use when running + /// Number of threads to use when running. Values below two warn and use two. #[arg(short, long, default_value_t = 16)] pub threads: u32, @@ -332,6 +621,12 @@ pub struct MapQuantOpts { )] pub min_reads: usize, + #[command(flatten)] + pub cell_correction: CellBarcodeCorrectionOpts, + + #[command(flatten)] + pub collation_resources: CollationResourceOpts, + /// Path to a transcript to gene map file #[arg(short = 'm', long, help_heading = "UMI Resolution Options")] pub t2g_map: Option, @@ -345,8 +640,7 @@ pub struct MapQuantOpts { /// regardless of `--resolution`. Pass 0 to resolve every cell with the /// requested strategy. /// - /// Left unset, alevin-fry's own default applies. Requires alevin-fry - /// >= 0.17.1; earlier versions parse the option and ignore it. + /// Left unset, alevin-fry's own default applies. #[arg(long, value_name = "N", help_heading = "UMI Resolution Options")] pub small_thresh: Option, @@ -671,7 +965,8 @@ pub struct MultiplexQuantOpts { #[arg(short, long)] pub chemistry: Option, - /// Override the read geometry string (e.g. '1{b[16]u[12]x[0-3]hamming(f[TTGCTAGGACCG],1)s[10]x:}2{r:}') + /// Override the read geometry string (for example, + /// `1{b[16]u[12]x[0-3]hamming(f[TTGCTAGGACCG],1)s[10]x:}2{r:}`). #[arg(short, long)] pub geometry: Option, @@ -697,17 +992,23 @@ pub struct MultiplexQuantOpts { value_parser = clap::builder::PossibleValuesParser::new(["forward", "reverse"]))] pub sample_bc_ori: Option, - /// Sample barcode correction mode - #[arg(long, default_value = "exact", - value_parser = clap::builder::PossibleValuesParser::new(["exact", "1-edit"]), - help_heading = "Permit List Options")] - pub sample_correction_mode: String, + #[command(flatten)] + pub cell_correction: CellBarcodeCorrectionOpts, + + #[command(flatten)] + pub sample_correction: SampleBarcodeCorrectionOpts, + + #[command(flatten)] + pub gpl_resources: GplResourceOpts, + + #[command(flatten)] + pub collation_resources: CollationResourceOpts, /// Path to output directory #[arg(short, long)] pub output: PathBuf, - /// Number of threads to use + /// Number of threads to use. Values below two warn and use two. #[arg(short, long, default_value_t = 16)] pub threads: u32, @@ -771,8 +1072,7 @@ pub struct MultiplexQuantOpts { /// regardless of `--resolution`. Pass 0 to resolve every cell with the /// requested strategy. /// - /// Left unset, alevin-fry's own default applies. Requires alevin-fry - /// >= 0.17.1; earlier versions parse the option and ignore it. + /// Left unset, alevin-fry's own default applies. #[arg(long, value_name = "N", help_heading = "Quantification Options")] pub small_thresh: Option, @@ -996,3 +1296,126 @@ pub enum WorkflowCommands { skip_step: Option>, }, } + +#[cfg(test)] +mod barcode_forwarding_tests { + use super::*; + + fn args_of(command: &std::process::Command) -> Vec { + command + .get_args() + .map(|argument| argument.to_string_lossy().into_owned()) + .collect() + } + + #[test] + fn omitted_barcode_and_resource_overrides_emit_no_arguments() { + let mut command = std::process::Command::new("alevin-fry"); + CellBarcodeCorrectionOpts::default().append_to(&mut command); + SampleBarcodeCorrectionOpts::default().append_to(&mut command); + CollationResourceOpts::default().append_to(&mut command); + GplResourceOpts::default().append_to(&mut command); + assert!(args_of(&command).is_empty()); + } + + #[test] + fn explicit_overrides_are_forwarded_to_their_stage_spellings() { + let cell = CellBarcodeCorrectionOpts { + cell_bc_correction: Some(CellBarcodeCorrection::Frequency), + cell_bc_neighborhood: Some(BarcodeNeighborhood::HammingOne), + cell_bc_confidence: Some("39/40".to_string()), + }; + let sample = SampleBarcodeCorrectionOpts { + sample_bc_correction: Some(SampleBarcodeCorrection::Frequency), + sample_bc_neighborhood: Some(BarcodeNeighborhood::SubstitutionOrShiftOne), + sample_bc_confidence: Some("0.975".to_string()), + sample_correction_mode: None, + }; + let gpl = GplResourceOpts { + gpl_memory_limit: Some("1GiB".to_string()), + gpl_tmp_dir: Some(PathBuf::from("/tmp/gpl")), + }; + let collate = CollationResourceOpts { + collate_memory_limit: Some("4GB".to_string()), + }; + + let mut gpl_command = std::process::Command::new("alevin-fry"); + cell.append_to(&mut gpl_command); + sample.append_to(&mut gpl_command); + gpl.append_to(&mut gpl_command); + assert_eq!( + args_of(&gpl_command), + [ + "--cell-bc-correction", + "frequency", + "--cell-bc-neighborhood", + "hamming-1", + "--cell-bc-confidence", + "39/40", + "--sample-bc-correction", + "frequency", + "--sample-bc-neighborhood", + "substitution-or-shift-1", + "--sample-bc-confidence", + "0.975", + "--memory-limit", + "1GiB", + "--tmp-dir", + "/tmp/gpl", + ] + ); + + let mut collate_command = std::process::Command::new("alevin-fry"); + collate.append_to(&mut collate_command); + assert_eq!(args_of(&collate_command), ["--memory-limit", "4GB"]); + } + + #[test] + fn legacy_sample_modes_translate_without_deprecated_child_arguments() { + for (legacy, expected) in [ + ( + LegacySampleCorrectionMode::Exact, + vec!["--sample-bc-correction", "exact"], + ), + ( + LegacySampleCorrectionMode::OneEdit, + vec![ + "--sample-bc-correction", + "unique", + "--sample-bc-neighborhood", + "substitution-or-shift-1", + ], + ), + ] { + let options = SampleBarcodeCorrectionOpts { + sample_correction_mode: Some(legacy), + ..Default::default() + }; + let mut command = std::process::Command::new("alevin-fry"); + options.append_to(&mut command); + assert_eq!(args_of(&command), expected); + assert!( + !args_of(&command) + .iter() + .any(|argument| argument == "--sample-correction-mode") + ); + } + } + + #[test] + fn confidence_and_memory_validation_match_alevin_fry_grammar() { + for confidence in ["0", "0.90", "0.975", "39/40", "1"] { + assert!(confidence_parser(confidence).is_ok(), "{confidence}"); + } + for confidence in ["", "1.01", "40/39", "1/0", "NaN"] { + assert!(confidence_parser(confidence).is_err(), "{confidence}"); + } + + for memory in ["256MiB", "1GiB", "4GB", "512000000", "1.5G"] { + assert!(memory_size_parser(memory).is_ok(), "{memory}"); + } + for memory in ["", "0", "NaN", "1XB"] { + assert!(memory_size_parser(memory).is_err(), "{memory}"); + } + } +} diff --git a/src/simpleaf_commands/multiplex_quant.rs b/src/simpleaf_commands/multiplex_quant.rs index d0a384a..0ed8696 100644 --- a/src/simpleaf_commands/multiplex_quant.rs +++ b/src/simpleaf_commands/multiplex_quant.rs @@ -38,6 +38,17 @@ fn t2g_mode(opts: &MultiplexQuantOpts) -> probe_utils::ProbeT2gMode { } } +fn resolved_sample_bc_orientation<'a>( + opts: &'a MultiplexQuantOpts, + chemistry: Option<&'a CustomChemistry>, +) -> Option<&'a str> { + opts.sample_bc_ori.as_deref().or_else(|| { + chemistry + .and_then(|chemistry| chemistry.sample_bc_list.as_ref()) + .and_then(|sample_list| sample_list.sample_bc_ori.as_deref()) + }) +} + fn write_multiplex_metadata( output_dir: &Path, quant_output: &Path, @@ -228,9 +239,10 @@ fn resolve_user_supplied_index( } /// Main entry point for the multiplex-quant pipeline. -pub fn multiplex_map_and_quant(af_home: &Path, opts: MultiplexQuantOpts) -> anyhow::Result<()> { +pub fn multiplex_map_and_quant(af_home: &Path, mut opts: MultiplexQuantOpts) -> anyhow::Result<()> { let start = Instant::now(); info!("Starting multiplex quantification pipeline"); + opts.threads = crate::core::runtime::cap_threads_warned(opts.threads); // Load runtime context (program paths) let rt = context::load_runtime_context(af_home)?; @@ -408,27 +420,25 @@ pub fn multiplex_map_and_quant(af_home: &Path, opts: MultiplexQuantOpts) -> anyh .arg("-o") .arg(&quant_output) .arg("-t") - .arg(format!("{}", opts.threads.min(8))) + .arg(format!("{}", opts.threads)) .arg("--unfiltered-pl") .arg(&cell_bc_path) .arg("--sample-bc-list") .arg(&sample_bc_path) - .arg("--sample-correction-mode") - .arg(&opts.sample_correction_mode) .arg("--min-reads") .arg(format!("{}", opts.min_reads)); + opts.cell_correction.append_to(&mut gpl_cmd); + opts.sample_correction.append_to(&mut gpl_cmd); + opts.gpl_resources.append_to(&mut gpl_cmd); + // Forward the sample-barcode orientation to alevin-fry. Precedence: // 1. user-supplied --sample-bc-ori CLI override (`forward` / `reverse`, // matching alevin-fry's vocabulary and the preset JSON). // 2. the chemistry preset's declared sample_bc_ori (e.g. 10x Flex v2 // where the whitelist is the RC of what appears on the read). // 3. omit the flag (alevin-fry default). - let sbc_ori_override = opts.sample_bc_ori.as_deref().or_else(|| { - chem.as_ref() - .and_then(|c| c.sample_bc_list.as_ref()) - .and_then(|s| s.sample_bc_ori.as_deref()) - }); + let sbc_ori_override = resolved_sample_bc_orientation(&opts, chem.as_ref()); if let Some(ori) = sbc_ori_override { gpl_cmd.arg("--sample-bc-ori").arg(ori); } @@ -450,6 +460,7 @@ pub fn multiplex_map_and_quant(af_home: &Path, opts: MultiplexQuantOpts) -> anyh .arg(&map_output) .arg("-t") .arg(format!("{}", opts.threads)); + opts.collation_resources.append_to(&mut collate_cmd); let collate_cmd_str = prog_utils::get_cmd_line_string(&collate_cmd); info!("collate cmd: {}", collate_cmd_str); @@ -475,7 +486,7 @@ pub fn multiplex_map_and_quant(af_home: &Path, opts: MultiplexQuantOpts) -> anyh .arg("--use-mtx"); // Only forwarded when the user set it, so alevin-fry's own default stands - // otherwise. alevin-fry < 0.17.1 parses this and ignores it. + // otherwise. if let Some(small_thresh) = opts.small_thresh { quant_cmd .arg("--small-thresh") @@ -780,8 +791,9 @@ fn resolve_sample_bc_list( #[cfg(test)] mod tests { - use super::{resolve_user_supplied_index, t2g_mode}; + use super::{resolve_user_supplied_index, resolved_sample_bc_orientation, t2g_mode}; use crate::simpleaf_commands::MultiplexQuantOpts; + use crate::utils::chem_utils::{CustomChemistry, SampleBcListInfo}; use crate::utils::probe_utils::ProbeT2gMode; use clap::Parser; use serde_json::json; @@ -879,6 +891,30 @@ mod tests { assert_eq!(t2g_mode(&opts), ProbeT2gMode::Usa); } + #[test] + fn sample_barcode_orientation_uses_cli_then_preset_then_child_default() { + let mut opts = parse_multiplex_quant_opts(&["-o", "."]); + let mut chemistry = + CustomChemistry::simple_custom("1{b[16]u[12]x:}2{r:}").expect("valid test geometry"); + chemistry.sample_bc_list = Some(SampleBcListInfo { + plist_name: None, + remote_url: None, + sample_bc_ori: Some("reverse".to_string()), + }); + + assert_eq!( + resolved_sample_bc_orientation(&opts, Some(&chemistry)), + Some("reverse") + ); + opts.sample_bc_ori = Some("forward".to_string()); + assert_eq!( + resolved_sample_bc_orientation(&opts, Some(&chemistry)), + Some("forward") + ); + opts.sample_bc_ori = None; + assert_eq!(resolved_sample_bc_orientation(&opts, None), None); + } + #[test] fn usa_mode_without_probe_regions_returns_helpful_error() { let files = super::ResolvedProbeSetFiles { @@ -896,4 +932,87 @@ mod tests { "unexpected error: {msg}", ); } + + #[test] + fn barcode_and_stage_resource_overrides_are_forwarded() { + let opts = parse_multiplex_quant_opts(&[ + "-o", + ".", + "--cell-bc-correction", + "frequency", + "--cell-bc-neighborhood", + "hamming-1", + "--cell-bc-confidence", + "0.975", + "--sample-bc-correction", + "unique", + "--sample-bc-neighborhood", + "substitution-or-shift-1", + "--sample-bc-confidence", + "39/40", + "--gpl-memory-limit", + "1GiB", + "--gpl-tmp-dir", + "/tmp/gpl", + "--collate-memory-limit", + "4GiB", + ]); + + let mut gpl = std::process::Command::new("alevin-fry"); + opts.cell_correction.append_to(&mut gpl); + opts.sample_correction.append_to(&mut gpl); + opts.gpl_resources.append_to(&mut gpl); + let gpl_args: Vec<_> = gpl + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + assert!( + gpl_args + .windows(2) + .any(|pair| pair == ["--cell-bc-correction", "frequency"]) + ); + assert!( + gpl_args + .windows(2) + .any(|pair| pair == ["--sample-bc-correction", "unique"]) + ); + assert!( + gpl_args + .windows(2) + .any(|pair| pair == ["--memory-limit", "1GiB"]) + ); + assert!( + gpl_args + .windows(2) + .any(|pair| pair == ["--tmp-dir", "/tmp/gpl"]) + ); + + let mut collate = std::process::Command::new("alevin-fry"); + opts.collation_resources.append_to(&mut collate); + let collate_args: Vec<_> = collate + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + assert_eq!(collate_args, ["--memory-limit", "4GiB"]); + } + + #[test] + fn legacy_sample_mode_is_hidden_but_still_translates() { + let opts = parse_multiplex_quant_opts(&["-o", ".", "--sample-correction-mode", "1-edit"]); + let mut command = std::process::Command::new("alevin-fry"); + opts.sample_correction.append_to(&mut command); + let args: Vec<_> = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + args, + [ + "--sample-bc-correction", + "unique", + "--sample-bc-neighborhood", + "substitution-or-shift-1", + ] + ); + } } diff --git a/src/simpleaf_commands/quant.rs b/src/simpleaf_commands/quant.rs index 5ca9ed4..8330855 100644 --- a/src/simpleaf_commands/quant.rs +++ b/src/simpleaf_commands/quant.rs @@ -522,13 +522,13 @@ fn run_quant_stage( .exe_path .clone(); let mut alevin_gpl_cmd = std::process::Command::new(format!("{}", alevin_fry.display())); - let gpl_threads = setup.threads.min(8); alevin_gpl_cmd.arg("generate-permit-list"); alevin_gpl_cmd.arg("-i").arg(&mapping.map_output); alevin_gpl_cmd.arg("-d").arg(setup.ori.as_str()); - alevin_gpl_cmd.arg("-t").arg(format!("{}", gpl_threads)); + alevin_gpl_cmd.arg("-t").arg(format!("{}", setup.threads)); setup.filter_meth.add_to_args(&mut alevin_gpl_cmd); alevin_gpl_cmd.arg("-o").arg(&gpl_output); + opts.cell_correction.append_to(&mut alevin_gpl_cmd); let gpl_cmd_string = prog_utils::get_cmd_line_string(&alevin_gpl_cmd); info!("alevin-fry generate-permit-list cmd : {}", gpl_cmd_string); let input_files = vec![mapping.map_output.clone()]; @@ -544,6 +544,7 @@ fn run_quant_stage( alevin_collate_cmd .arg("-t") .arg(format!("{}", setup.threads)); + opts.collation_resources.append_to(&mut alevin_collate_cmd); let collate_cmd_string = prog_utils::get_cmd_line_string(&alevin_collate_cmd); info!("alevin-fry collate cmd : {}", collate_cmd_string); let input_files = vec![gpl_output.clone(), mapping.map_output.clone()]; @@ -563,7 +564,7 @@ fn run_quant_stage( alevin_quant_cmd.arg("-m").arg(setup.t2g_map_file.clone()); alevin_quant_cmd.arg("-r").arg(&opts.resolution); // Only forwarded when the user set it, so alevin-fry's own default stands - // otherwise. alevin-fry < 0.17.1 parses this and ignores it. + // otherwise. if let Some(small_thresh) = opts.small_thresh { alevin_quant_cmd .arg("--small-thresh") @@ -805,4 +806,54 @@ mod tests { assert!(!args.iter().any(|a| a == "--with-position")); assert!(!args.iter().any(|a| a == "--struct-constraints")); } + + #[test] + fn barcode_and_collation_overrides_reach_their_child_arguments() { + let opts = parse_quant_opts(&[ + "quant", + "-c", + "10xv3", + "-o", + "/tmp/out", + "-r", + "cr-like", + "--knee", + "--map-dir", + "/tmp/mapped", + "--cell-bc-correction", + "frequency", + "--cell-bc-neighborhood", + "hamming-1", + "--cell-bc-confidence", + "39/40", + "--collate-memory-limit", + "3GiB", + ]); + + let mut gpl = std::process::Command::new("alevin-fry"); + opts.cell_correction.append_to(&mut gpl); + let gpl_args: Vec<_> = gpl + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + gpl_args, + [ + "--cell-bc-correction", + "frequency", + "--cell-bc-neighborhood", + "hamming-1", + "--cell-bc-confidence", + "39/40", + ] + ); + + let mut collate = std::process::Command::new("alevin-fry"); + opts.collation_resources.append_to(&mut collate); + let collate_args: Vec<_> = collate + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + assert_eq!(collate_args, ["--memory-limit", "3GiB"]); + } } diff --git a/src/simpleaf_commands/workflow.rs b/src/simpleaf_commands/workflow.rs index 156e40c..3543627 100644 --- a/src/simpleaf_commands/workflow.rs +++ b/src/simpleaf_commands/workflow.rs @@ -267,7 +267,8 @@ pub fn list_workflows>(af_home_path: T) -> anyhow::Result<()> { /// /// ### Program Description /// This program is used for getting the source files of a pubished workflow -/// from the protocol estuary GitHub repo https://github.com/COMBINE-lab/protocol-estuary +/// from the protocol estuary GitHub repo +/// . /// /// This program takes a string representing the name of a published workflow, and copy the /// folder of that workflow in the protocol estuary to the provided output directory diff --git a/src/utils/prog_utils.rs b/src/utils/prog_utils.rs index 376fb20..a63ffe0 100644 --- a/src/utils/prog_utils.rs +++ b/src/utils/prog_utils.rs @@ -32,9 +32,9 @@ pub mod min_versions { /// it does not recognise. pub const PISCEM: &str = ">=0.22.0, <1.0.0"; - /// alevin-fry 0.17.0 carries the libradicl 0.17 reader work, the scATAC - /// collate/deduplicate fixes, and the `infer` matrix-type fix. - pub const ALEVIN_FRY: &str = ">=0.17.0, <1.0.0"; + /// alevin-fry 0.18.0 provides deterministic compiled barcode-correction + /// plans and the correction/resource controls forwarded by simpleaf 0.28. + pub const ALEVIN_FRY: &str = ">=0.18.0, <1.0.0"; /// Only consulted when peak calling is actually requested. pub const MACS3: &str = ">=3.0.2, <4.0.0"; @@ -72,7 +72,8 @@ pub fn shell>(cmd: S) -> Command { } /// NOTE: the body of the JSON object we fetch cannot exceed 10MB -/// this is a limitation put in place by `ureq` (see : https://docs.rs/ureq/3.0.0-rc4/ureq/struct.Body.html#method.read_json) +/// this is a limitation put in place by `ureq` (see +/// ). pub fn read_json_from_remote_url>(url: T) -> Result { let url = url.as_ref(); diff --git a/src/utils/workflow_utils.rs b/src/utils/workflow_utils.rs index b2a096a..2a8d159 100644 --- a/src/utils/workflow_utils.rs +++ b/src/utils/workflow_utils.rs @@ -1572,7 +1572,7 @@ pub fn get_protocol_estuary>( } /// Copy all files from the src folder to the dst folder.\ -/// Adapted from https://stackoverflow.com/a/65192210. +/// Adapted from . pub fn copy_dir_all(src: impl AsRef, dst: impl AsRef) -> anyhow::Result<()> { fs::create_dir_all(&dst)?; for entry in fs::read_dir(src)? { diff --git a/tests/cli_help_snapshots.rs b/tests/cli_help_snapshots.rs index 0892493..e8688a1 100644 --- a/tests/cli_help_snapshots.rs +++ b/tests/cli_help_snapshots.rs @@ -98,6 +98,18 @@ fn update_requested() -> bool { std::env::var_os("UPDATE_CLI_SNAPSHOTS").is_some_and(|v| !v.is_empty() && v != "0") } +fn normalize_help(output: &str) -> String { + let mut normalized = output + .lines() + .map(str::trim_end) + .collect::>() + .join("\n"); + if output.ends_with('\n') { + normalized.push('\n'); + } + normalized +} + #[test] fn cli_help_outputs_match_snapshots() { let binary = env!("CARGO_BIN_EXE_simpleaf"); @@ -120,7 +132,11 @@ fn cli_help_outputs_match_snapshots() { String::from_utf8_lossy(&output.stderr) ); - let actual = String::from_utf8(output.stdout).expect("stdout was not valid UTF-8"); + // Clap indents otherwise-empty separator lines. Keeping those spaces in + // tracked snapshots makes `git diff --check` fail while adding no + // information, so normalize trailing whitespace on both sides. + let actual = + normalize_help(&String::from_utf8(output.stdout).expect("stdout was not valid UTF-8")); let expected_path = snapshots_dir().join(snapshot_file); if updating { @@ -130,9 +146,9 @@ fn cli_help_outputs_match_snapshots() { continue; } - let expected = fs::read_to_string(&expected_path).unwrap_or_else(|e| { + let expected = normalize_help(&fs::read_to_string(&expected_path).unwrap_or_else(|e| { panic!("failed reading snapshot {}: {}", expected_path.display(), e) - }); + })); assert_eq!( actual, expected, diff --git a/tests/snapshots/cli-help/simpleaf_atac_index___help.txt b/tests/snapshots/cli-help/simpleaf_atac_index___help.txt index dddd9fa..031bb4e 100644 --- a/tests/snapshots/cli-help/simpleaf_atac_index___help.txt +++ b/tests/snapshots/cli-help/simpleaf_atac_index___help.txt @@ -11,7 +11,7 @@ Options: path to output directory (will be created if it doesn't exist) -t, --threads - number of threads to use when running + number of threads to use when running; values below two warn and use two [default: 16] diff --git a/tests/snapshots/cli-help/simpleaf_atac_process___help.txt b/tests/snapshots/cli-help/simpleaf_atac_process___help.txt index 721e6d4..30f3ed3 100644 --- a/tests/snapshots/cli-help/simpleaf_atac_process___help.txt +++ b/tests/snapshots/cli-help/simpleaf_atac_process___help.txt @@ -6,16 +6,16 @@ Usage: simpleaf atac process [OPTIONS] --index --barcode-reads chemistry - + [possible values: 10x-v1, 10x-v2, 10x-multi] -t, --threads - number of threads to use when running - + number of threads to use when running; values below two warn and use two + [default: 16] --output - + --call-peaks do peak calling after generating the bed file @@ -45,23 +45,23 @@ Mapping Options: --barcode-length the length of the barcode read from which to extract the barcode (usually this is the length of the entire read, and reads shorter than this will be discarded) - + [default: 16] Piscem Mapping Options: --decoder Gzip decoder selection passed to piscem: `auto`, `serial`, `parallel`, or `parallel=N`. - + `auto` lets piscem adapt the mapping/decode split while the run proceeds. `serial` gives mapping the whole budget. `parallel` forces the parallel decoder where the input allows it; `parallel=N` fixes N decode slots per gzip input and stops the adaptation. Inputs that cannot be read positionally (FIFOs, process substitution) stay serial regardless. - + [default: auto] --thread-policy JSON file overriding piscem's thread and decoder policy. - + Every field is optional and defaults to a measured value; an unrecognised field is an error rather than a silent no-op. Currently understood: `{"parallel_decode": {"min_threads_per_stream": 8}}`, the number of threads that must be free per gzip input @@ -83,9 +83,25 @@ Permit List Generation Options: --min-reads minimum read count threshold for a cell to be retained/processed; only used with --unfiltered-pl - + [default: 10] +Advanced Barcode Correction Options: + --cell-bc-correction + Cell-barcode collision policy (inherited default: unique) + + [possible values: unique, frequency] + + --cell-bc-neighborhood + One-error cell-barcode neighbourhood. When omitted, alevin-fry chooses the + protocol/filter-specific default + + [possible values: hamming-1, substitution-or-shift-1] + + --cell-bc-confidence + Frequency confidence as a decimal or exact fraction. The inherited default is 97.5% for + RNA and 90% for ATAC + Advanced Options: --compress compress the output mapping bed file @@ -101,17 +117,17 @@ Advanced Options: --thr threshold to be considered for pseudoalignment - + [default: 0.7] --bin-size size of virtual color intervals - + [default: 1000] --bin-overlap size for virtual color interval overlap - + [default: 300] --no-tn5-shift @@ -120,38 +136,38 @@ Advanced Options: --max-ec-card determines the maximum cardinality equivalence class (number of (txp, orientation status) pairs) to examine (cannot be used with --ignore-ambig-hits) - + [default: 4096] --max-hit-occ in the first pass, consider only k-mers having <= --max-hit-occ hits - + [default: 256] --max-hit-occ-recover if all k-mers have > --max-hit-occ hits, then make a second pass and consider k-mers having <= --max-hit-occ-recover hits - + [default: 1024] --max-read-occ reads with more than this number of mappings will not have their mappings reported - + [default: 2500] Peak Caller Options: --gsize The value to be passed to the `macs3` `--gsize` (genome size) option. Possible values are "hs", "mm", "ce", "dm" or an unsigned integer - + [default: hs] --qvalue The value to be passed to the `macs3` `--qvalue` (minimum FDR cutoff) option - + [default: 0.1] --extsize The value to be passed to the `macs3` `--extsize` option - + [default: 50] diff --git a/tests/snapshots/cli-help/simpleaf_multiplex_quant___help.txt b/tests/snapshots/cli-help/simpleaf_multiplex_quant___help.txt index f410e9a..f578e6b 100644 --- a/tests/snapshots/cli-help/simpleaf_multiplex_quant___help.txt +++ b/tests/snapshots/cli-help/simpleaf_multiplex_quant___help.txt @@ -9,12 +9,12 @@ Options: --geometry and --cell-bc-list are required -g, --geometry - Override the read geometry string (e.g. - '1{b[16]u[12]x[0-3]hamming(f[TTGCTAGGACCG],1)s[10]x:}2{r:}') + Override the read geometry string (for example, + `1{b[16]u[12]x[0-3]hamming(f[TTGCTAGGACCG],1)s[10]x:}2{r:}`) --organism Target organism for automatic probe set selection - + [possible values: human, mouse] --cell-bc-list @@ -22,7 +22,7 @@ Options: --expected-ori Expected read orientation: fw, rc, or both - + [default: both] --sample-bc-ori @@ -31,15 +31,15 @@ Options: `sample_bc_ori` when set. Useful for cycle-plan variants (e.g. 10x Flex Configuration B) where the sample BC is read off the opposite strand from the canonical preset. Vocabulary matches the preset JSON and alevin-fry's `--sample-bc-ori` - + [possible values: forward, reverse] -o, --output Path to output directory -t, --threads - Number of threads to use - + Number of threads to use. Values below two warn and use two + [default: 16] -h, --help @@ -48,17 +48,45 @@ Options: -V, --version Print version -Permit List Options: - --sample-correction-mode - Sample barcode correction mode - - [default: exact] - [possible values: exact, 1-edit] +Advanced Barcode Correction Options: + --cell-bc-correction + Cell-barcode collision policy (inherited default: unique) - --min-reads - Minimum read count threshold for unfiltered permit list - - [default: 10] + [possible values: unique, frequency] + + --cell-bc-neighborhood + One-error cell-barcode neighbourhood. When omitted, alevin-fry chooses the + protocol/filter-specific default + + [possible values: hamming-1, substitution-or-shift-1] + + --cell-bc-confidence + Frequency confidence as a decimal or exact fraction. The inherited default is 97.5% for + RNA and 90% for ATAC + + --sample-bc-correction + Sample-barcode correction policy (inherited default: exact) + + [possible values: exact, unique, frequency] + + --sample-bc-neighborhood + Sample-barcode neighbourhood for non-exact correction (inherited default: Hamming-1) + + [possible values: hamming-1, substitution-or-shift-1] + + --sample-bc-confidence + Sample-Frequency confidence as a decimal or exact fraction (inherited default: 97.5%) + +Advanced Resource Options: + --gpl-memory-limit + Deferred GPL buffer budget (inherited default: 512 MiB) + + --gpl-tmp-dir + Temporary directory for compressed GPL correction runs (inherited default: the GPL output + directory) + + --collate-memory-limit + Collation buffer budget (inherited default: 2 GiB) Mapping Options: -i, --index @@ -77,7 +105,7 @@ Probe Set Options: --kmer-length k-mer length for probe index building - + [default: 23] Reference Options: @@ -98,7 +126,7 @@ Reference Options: Quantification Options: -r, --resolution UMI resolution mode - + [default: cr-like] [possible values: cr-like, cr-like-em, parsimony, parsimony-em, parsimony-gene, parsimony-gene-em] @@ -107,14 +135,13 @@ Quantification Options: Cells with fewer than this many reads are resolved by alevin-fry's tiny-cell fast path, which applies `cr-like` (winner-take-all) semantics regardless of `--resolution`. Pass 0 to resolve every cell with the requested strategy. - - Left unset, alevin-fry's own default applies. Requires alevin-fry >= 0.17.1; earlier - versions parse the option and ignore it. + + Left unset, alevin-fry's own default applies. Piscem Mapping Options: --skipping-strategy The skipping strategy to use for k-mer collection - + [default: permissive] [possible values: permissive, strict] @@ -123,34 +150,40 @@ Piscem Mapping Options: --max-ec-card Maximum cardinality equivalence class to examine - + [default: 4096] --dict Piscem dictionary backend: `auto` (default), `sshash`, or `tiny`. Applies both to the auto-built probe index (build time) and to map-sc (map time) - + [default: auto] [possible values: auto, sshash, tiny] --decoder Gzip decoder selection passed to piscem: `auto`, `serial`, `parallel`, or `parallel=N`. - + `auto` lets piscem adapt the mapping/decode split while the run proceeds. `serial` gives mapping the whole budget. `parallel` forces the parallel decoder where the input allows it; `parallel=N` fixes N decode slots per gzip input and stops the adaptation. Inputs that cannot be read positionally (FIFOs, process substitution) stay serial regardless. - + [default: auto] --thread-policy JSON file overriding piscem's thread and decoder policy. - + Every field is optional and defaults to a measured value; an unrecognised field is an error rather than a silent no-op. Currently understood: `{"parallel_decode": {"min_threads_per_stream": 8}}`, the number of threads that must be free per gzip input before the parallel decoder is engaged at all. +Permit List Options: + --min-reads + Minimum read count threshold for unfiltered permit list + + [default: 10] + Output Options: --anndata-out Generate an anndata (h5ad format) count matrix from the standard (matrix-market format) diff --git a/tests/snapshots/cli-help/simpleaf_quant___help.txt b/tests/snapshots/cli-help/simpleaf_quant___help.txt index 97fe028..a0c7b07 100644 --- a/tests/snapshots/cli-help/simpleaf_quant___help.txt +++ b/tests/snapshots/cli-help/simpleaf_quant___help.txt @@ -11,8 +11,8 @@ Options: Path to the output directory -t, --threads - Number of threads to use when running - + Number of threads to use when running. Values below two warn and use two + [default: 16] -h, --help @@ -40,7 +40,7 @@ Piscem Mapping Options: --with-position Record the position of each mapped read in the RAD file. - + alevin-fry detects the positional record type from the RAD header and adapts automatically, so no downstream option needs to change. The RAD file is larger. Not available for `multiplex-quant`: there is no multi-barcode positional record type, and the @@ -57,53 +57,53 @@ Piscem Mapping Options: --skipping-strategy The skipping strategy to use for k-mer collection - + [default: permissive] [possible values: permissive, strict] --max-ec-card Determines the maximum cardinality equivalence class (number of (txp, orientation status) pairs) to examine (cannot be used with --ignore-ambig-hits) - + [default: 4096] --max-hit-occ In the first pass, consider only collected and matched k-mers of a read having <= --max-hit-occ hits - + [default: 256] --max-hit-occ-recover If all collected and matched k-mers of a read have > --max-hit-occ hits, then make a second pass and consider k-mers having <= --max-hit-occ-recover hits - + [default: 1024] --max-read-occ Threshold for discarding reads with too many mappings - + [default: 2500] --dict Piscem dictionary backend to use at map time: `auto` (default, honors the index's embedded choice), `sshash`, or `tiny` - + [default: auto] [possible values: auto, sshash, tiny] --decoder Gzip decoder selection passed to piscem: `auto`, `serial`, `parallel`, or `parallel=N`. - + `auto` lets piscem adapt the mapping/decode split while the run proceeds. `serial` gives mapping the whole budget. `parallel` forces the parallel decoder where the input allows it; `parallel=N` fixes N decode slots per gzip input and stops the adaptation. Inputs that cannot be read positionally (FIFOs, process substitution) stay serial regardless. - + [default: auto] --thread-policy JSON file overriding piscem's thread and decoder policy. - + Every field is optional and defaults to a measured value; an unrecognised field is an error rather than a silent no-op. Currently understood: `{"parallel_decode": {"min_threads_per_stream": 8}}`, the number of threads that must be free per gzip input @@ -128,22 +128,42 @@ Permit List Generation Options: -d, --expected-ori The expected direction/orientation of alignments in the chemistry being processed. If not provided, will default to `fw` for 10xv2/10xv3, otherwise `both` - + [possible values: fw, rc, both] --min-reads Minimum read count threshold for a cell to be retained/processed; only use with --unfiltered-pl - + [default: 10] +Advanced Barcode Correction Options: + --cell-bc-correction + Cell-barcode collision policy (inherited default: unique) + + [possible values: unique, frequency] + + --cell-bc-neighborhood + One-error cell-barcode neighbourhood. When omitted, alevin-fry chooses the + protocol/filter-specific default + + [possible values: hamming-1, substitution-or-shift-1] + + --cell-bc-confidence + Frequency confidence as a decimal or exact fraction. The inherited default is 97.5% for + RNA and 90% for ATAC + +Advanced Resource Options: + --collate-memory-limit + Collation buffer budget (inherited default: 2 GiB) + UMI Resolution Options: -m, --t2g-map Path to a transcript to gene map file -r, --resolution UMI resolution mode - + [possible values: cr-like, cr-like-em, parsimony, parsimony-em, parsimony-gene, parsimony-gene-em] @@ -151,9 +171,8 @@ UMI Resolution Options: Cells with fewer than this many reads are resolved by alevin-fry's tiny-cell fast path, which applies `cr-like` (winner-take-all) semantics regardless of `--resolution`. Pass 0 to resolve every cell with the requested strategy. - - Left unset, alevin-fry's own default applies. Requires alevin-fry >= 0.17.1; earlier - versions parse the option and ignore it. + + Left unset, alevin-fry's own default applies. Output Options: --anndata-out From 66bb20467327598b03ca3976c9369b2f728dacda Mon Sep 17 00:00:00 2001 From: rob-p Date: Sat, 15 Aug 2026 11:59:15 -0400 Subject: [PATCH 3/4] fix: keep ATAC peak calling opt-in --- CHANGELOG.md | 3 +++ src/atac/process.rs | 29 ++++++++++++++++++++--------- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c3d1f0..be29abb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,9 @@ * Resolve the effective thread count once per pipeline and pass it consistently to mapping, permit-list generation, collation or ATAC sorting, and quantification. +* Run MACS peak calling from `atac process` only when `--call-peaks` is set; + ordinary ATAC processing no longer fails after producing its BED merely + because MACS is not installed. * Retire the obsolete `release.sh`; `bump_and_publish.sh` is now the sole documented release entry point. diff --git a/src/atac/process.rs b/src/atac/process.rs index 94f0ac9..e711617 100644 --- a/src/atac/process.rs +++ b/src/atac/process.rs @@ -362,15 +362,26 @@ fn macs_call_peaks(af_home_path: &Path, opts: &ProcessOpts) -> anyhow::Result anyhow::Result<()> { let gpl = af_gpl(af_home_path, opts)?; let sort = af_sort(af_home_path, opts)?; - let macs = macs_call_peaks(af_home_path, opts)?; - info!( - "ATAC downstream stages completed (gpl: {:.2}s, sort: {:.2}s, macs: {:.2}s).", - gpl.gpl_duration_secs, sort.sort_duration_secs, macs.macs_duration_secs - ); - info!( - "ATAC commands: gpl=`{}`, sort=`{}`, macs=`{}`", - gpl.gpl_cmd, sort.sort_cmd, macs.macs_cmd - ); + if opts.call_peaks { + let macs = macs_call_peaks(af_home_path, opts)?; + info!( + "ATAC downstream stages completed (gpl: {:.2}s, sort: {:.2}s, macs: {:.2}s).", + gpl.gpl_duration_secs, sort.sort_duration_secs, macs.macs_duration_secs + ); + info!( + "ATAC commands: gpl=`{}`, sort=`{}`, macs=`{}`", + gpl.gpl_cmd, sort.sort_cmd, macs.macs_cmd + ); + } else { + info!( + "ATAC downstream stages completed (gpl: {:.2}s, sort: {:.2}s; peak calling not requested).", + gpl.gpl_duration_secs, sort.sort_duration_secs + ); + info!( + "ATAC commands: gpl=`{}`, sort=`{}`", + gpl.gpl_cmd, sort.sort_cmd + ); + } Ok(()) } From 49d6fa18058f5b67f49002d403ac77be9d9dcec3 Mon Sep 17 00:00:00 2001 From: rob-p Date: Sat, 15 Aug 2026 12:18:58 -0400 Subject: [PATCH 4/4] ci: test against alevin-fry 0.18 release --- .github/workflows/test_simpleaf.yml | 26 ++++++++++++++++++++++++++ simpleaf_conda_env.yml | 1 - 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test_simpleaf.yml b/.github/workflows/test_simpleaf.yml index 5487054..5669507 100644 --- a/.github/workflows/test_simpleaf.yml +++ b/.github/workflows/test_simpleaf.yml @@ -61,9 +61,35 @@ jobs: activate-environment: anaconda-client-env environment-file: simpleaf_conda_env.yml + # The bioconda package can lag a newly published alevin-fry release. + # Exercise simpleaf against the release artifact that satisfies the + # version contract instead of silently testing an older conda package. + - name: Install alevin-fry 0.18.0 release artifact + shell: bash -l {0} + run: | + case "$(uname -s)-$(uname -m)" in + Linux-x86_64) target=x86_64-unknown-linux-gnu ;; + Linux-aarch64) target=aarch64-unknown-linux-gnu ;; + Darwin-x86_64) target=x86_64-apple-darwin ;; + Darwin-arm64) target=aarch64-apple-darwin ;; + *) echo "unsupported CI platform: $(uname -s)-$(uname -m)" >&2; exit 1 ;; + esac + + archive="alevin-fry-${target}.tar.xz" + install_dir="${RUNNER_TEMP}/alevin-fry-0.18.0" + curl --proto '=https' --tlsv1.2 -LsSf \ + "https://github.com/COMBINE-lab/alevin-fry/releases/download/v0.18.0/${archive}" \ + -o "${RUNNER_TEMP}/${archive}" + mkdir -p "${install_dir}" + tar -xJf "${RUNNER_TEMP}/${archive}" \ + --strip-components=1 -C "${install_dir}" + "${install_dir}/alevin-fry" --version + echo "${install_dir}" >> "${GITHUB_PATH}" + - name: Test simpleaf shell: bash -l {0} run: | + export PATH="${RUNNER_TEMP}/alevin-fry-0.18.0:${PATH}" cd scripts chmod +x test_simpleaf.sh ulimit -n 2048 diff --git a/simpleaf_conda_env.yml b/simpleaf_conda_env.yml index 2c5306c..c1c13a3 100644 --- a/simpleaf_conda_env.yml +++ b/simpleaf_conda_env.yml @@ -6,5 +6,4 @@ channels: - bioconda dependencies: - - alevin-fry>=0.17.0 - piscem>=0.22.0