Skip to content

Sort peak QC plot inputs deterministically - #454

Open
NicoDeVeaux wants to merge 3 commits into
nf-core:devfrom
NicoDeVeaux:fix/deterministic-peak-qc-plot-order
Open

Sort peak QC plot inputs deterministically#454
NicoDeVeaux wants to merge 3 commits into
nf-core:devfrom
NicoDeVeaux:fix/deterministic-peak-qc-plot-order

Conversation

@NicoDeVeaux

@NicoDeVeaux NicoDeVeaux commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Fixes an intermittent nf-test snapshot failure in tests/skip_trimming.nf.test.

The flake

The same commit produced opposite results on the same Nextflow version (26.07.0-edge), with only CHANGELOG.md differing between the two pipeline states:

Run Result
30818548816 (Aug 3) passed
32331137515 attempt 1 (Aug 20) ❌ docker | latest-everything | 2/7
32331137515 attempt 2 (Aug 21) passed

Three md5s differ when it fails, all from the same source: macs3_annotatePeaks.mLb.clN.summary.txt, macs3_annotatePeaks.mLb.clN.summary_mqc.tsv, and the multiqc_mlib_peak_annotation-plot.txt derived from them.

Cause

collect emits in task-completion order, and PLOT_MACS3_QC / PLOT_HOMER_ANNOTATEPEAKS derive their sample ids from the order of the file list via -i ${annos.join(',')} / -s ....

In bin/plot_homer_annotatepeaks.r the summary rows are already deterministic — lines 99-101 sort the factor levels. The columns are not: summary.dat <- dcast(plot.feature.dat, variable ~ feature) takes its column order from the feature factor levels, which accumulate by first appearance across the input files. Samples with differing feature sets therefore produce different column orders depending on which file was read first, which is why this only trips occasionally.

Fix

Sort the staged file list by name inside the two process scripts.

Notes

  • Snapshots may need regenerating if the sorted order differs from the order baked into the current values.
  • subworkflows/local/bed_consensus_quantify_qc_bedtools_featurecounts_deseq2.nf lines 30 and 64 have the same unsorted-collect pattern feeding consensus peak and featureCounts column order. nf-core modules update #448 already fixes that call site.
  • A one-line sort of the feature factor levels before the dcast in plot_homer_annotatepeaks.r would harden the R script independently of input order. Left out to keep this change small.

PR checklist

  • This comment contains a description of changes (with reason).
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes — snapshot impact to be confirmed by CI.
  • CHANGELOG.md is updated.

NicoDeVeaux and others added 2 commits August 21, 2026 21:21
@NicoDeVeaux @claude
Sort peak QC plot inputs deterministically
7a89066 
`collect` emits in task-completion order, and plot_macs3_qc.r /
plot_homer_annotatepeaks.r derive sample ids from the order of the file
list they are handed. In plot_homer_annotatepeaks.r the summary column
order follows the `feature` factor levels, which accumulate by first
appearance across the input files, so a reshuffle changes the md5 of
macs3_annotatePeaks.*.summary.txt, its _mqc.tsv, and the MultiQC
peak annotation plot table. Observed as an intermittent snapshot
mismatch in tests/skip_trimming.nf.test on nf-core#453.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@NicoDeVeaux @claude
Add CHANGELOG entry for nf-core#454
f5a1bca 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 4.0.3.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the Synchronisation documentation.

@github-actions

github-actions Bot commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit b8039eb

+| ✅ 273 tests passed       |+
#| ❔   3 tests were ignored |#
!| ❗  38 tests had warnings |!
Details

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in CONTRIBUTING.md: Add any pipeline specific contribution guidelines here, such as coding styles, procedures, checklists etc.
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_if_empty_null - ifEmpty(null) found in /home/runner/work/atacseq/atacseq/subworkflows/local/prepare_genome.nf: _ versions = ch_versions.ifEmpty(null) // channel: [ versions.yml ]
    _
  • local_component_structure - samplesheet_check.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - frip_score.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - igv.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - tss_extract.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - bamtools_filter.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - get_autosomes.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - plot_macs3_qc.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - plot_homer_annotatepeaks.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - gtf2bed.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - deseq2_qc.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - genome_blacklist_regions.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - multiqc_custom_peaks.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - multiqc.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - bam_remove_orphans.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - star_genomegenerate.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - macs3_consensus.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - bedtools_genomecov.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - star_align.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - align_star.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - bed_consensus_quantify_qc_bedtools_featurecounts_deseq2.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - input_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - bam_bedgraph_bigwig_bedtools_ucsc.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - bam_shift_reads.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - bam_filter_bamtools.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - bigwig_plot_deeptools.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_genome.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - bam_peaks_call_qc_annotate_macs3_homer.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

  • nextflow_config - Config default ignored: params.bamtools_filter_pe_config
  • nextflow_config - Config default ignored: params.bamtools_filter_se_config
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore

✅ Tests passed:

Run details

  • nf-core/tools version 4.0.3
  • Run at 2026-08-21 19:52:37

@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

❌ nf-test failed with latest Nextflow version

Note

Tests with Nextflow's latest version failed but it will not cause a CI workflow failure.
Please check if the failure is expected with newer (edge-)releases of Nextflow or if it needs fixing.

  • ❌ docker | latest-everything | Shard 2/7

See the full run for details.

@NicoDeVeaux @claude
Sort the plot inputs inside the modules instead of the channel
b8039eb 
`toSortedList` emits an empty list when its source channel is empty,
whereas `collect` emits nothing at all, so the previous version started
PLOT_MACS3_QC with no files when no merged-replicate peaks exist and
plot_macs3_qc.r died on `flag "i" requires an argument`. Seen in the
`bowtie2 with stub` test.

Sort the staged file list in the process script instead, which leaves
the channel semantics untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@NicoDeVeaux