Repository navigation
Replace update_references_file with nf-schema param substitution - #164
Merged
Merged
Conversation
Drop the whole-YAML string rewrite that staged a copy into
workflow.workDir/tmp and rewrote a hardcoded igenomes prefix. nf-schema
3.0.0 substitutes ${params.X} placeholders in datasheet values at load
time, so datasheets can carry a base path instead of a hardcoded root.
Drop the 'schema' key from the input param: it deep-validated the
datasheet during validateParameters(), before substitution could run.
samplesheetToList validates against the same schema afterwards.
Generated by OpenCode
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 4.1.0. For more documentation on how to update your pipeline, please see the Synchronisation documentation. |
|
new File(...).absolutePath was a workaround for an error actually caused by a GString test name; projectDir resolves the same way and reads as intent. Generated by OpenCode
The placeholder now owns the leading slash, so the base path value stays clean and the join point stays visible in the datasheet. Add the --references_base_path section to usage.md, and rewrite the schema description and help_text, which still described the prefix rewriting removed by the substitution change. Convert the substitution test to the snapshot style used elsewhere in the suite. Generated by OpenCode
Fixes dumpParametersToJSON, which the utils subworkflow imports, and adds ReferenceSelection with igenomes_base substitution. Generated by OpenCode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR checklist
CHANGELOG.mdis updatednextflow_schema.jsonis updated (if needed)docs/usage.mdis updated (if needed)docs/output.mdis updated (if needed)assets/is updated (if needed)TODOcommentsDescription
Implements the #101 basepath strategy decided in the roadmap: drop the whole-YAML string rewrite and use nf-schema 3.0.0's
allowParamsSubstitutioninstead.update_references_filestaged a copy of the datasheet intoworkflow.workDir/tmp/<uuid>and rewrote a hardcoded['s3://ngi-igenomes/igenomes/']prefix list. nf-schema substitutes${params.X}placeholders in datasheet values in memory duringsamplesheetToList, so datasheets can carry a base path instead of a hardcoded root. The function and its work-dir mutation are deleted.references_base_pathnow has an unconditional default innextflow.config. Without it, an undefined param resolves the placeholder to the literal stringnull/...with only a warning, and the run fails later atfile(checkIfExists: true)with a confusing message. The default preserves the zero-config behaviour: datasheets without placeholders are unaffected.Slash convention: the placeholder carries the leading slash and the param value does not, so
${params.references_base_path}/GRCh38_chr21against the defaults3://ngi-igenomes/igenomesresolves tos3://ngi-igenomes/igenomes/GRCh38_chr21. The join point stays visible in the datasheet and the param value stays clean. A trailing slash on the param would double up.allowParamsSubstitutionis a no-op on datasheets containing no placeholders, so this is backwards-compatible with the current published datasheets. Re-authoringreferences-datasheetsto use placeholders is a separate cross-repo change and is not part of this PR.nf-core-utilsmoves 0.4.0 to 0.5.0 in the same PR. It fixesdumpParametersToJSON(), which the utils subworkflow imports, and addsReferenceSelectionwithigenomes_basesubstitution. The plugin is not template-managed, so this adds no template divergence.Docs
docs/usage.mdgains a--references_base_pathsection under Asset input: the placeholder form, the default and the path it resolves to, the slash rule, and the--inputdistinction. The broken../assets/asset.ymllink in that section now points at thenf-core/references-datasheetsrepository. The five other local links in the docs were audited and all resolve.descriptionandhelp_textforreferences_base_pathwere rewritten. Both still described the prefix rewriting this PR removes, andhelp_textis what--help_fullprints, so it was the most visible stale doc.Why nextflow_schema.json changed
The
inputparam declared"schema": "subworkflows/local/utils_references/schema_references.json", sovalidateParameters()deep-validated the datasheet contents beforesamplesheetToListran. That path does not substitute, so the literal placeholder failed validation first. The key is removed:samplesheetToListvalidates against the same schema file after substitution.Scope note
references_base_pathhere means substitution into the datasheet. Substitution to the datasheet (locating the datasheet file from a genome key) is a different job, used only by bamtofastq, and is renamed toreferences_datasheets_pathin that pipeline. This PR does not touch that.Testing
nextflow lintclean; the 23 warnings are pre-existingwhen-section deprecations in nf-core modules.nf-test test tests/ --profile=+docker: 27/27 passing, including a new test.tests/basepath_substitution.nf.testruns--tools faidxagainst a datasheet carrying${params.references_base_path}placeholders and snapshots its output in the same style as the rest of the suite. If substitution regresses, the literal placeholder reachesfile(checkIfExists: true)and the run fails, soworkflow.successis a real assertion.nf-core pipelines lint: 366 passed, 0 failed, 21 pre-existing warnings.Follow-ups (not in this PR)
references-datasheetswith placeholders. Blocked on bamtofastq renaming its param, else its URL default would be spliced into every file path.test.config's datasheet usestest-datasets/references/for fasta/gff andtest-datasets/modules/data/for VCFs, which one param cannot span.Generated by OpenCode