Advanced Nextflow configuration¶

Authors of Nextflow workflows have several different ways to set default resource usage for each of the workflow tasks.

One of the most common ways to set resource usage is using labels.

Configuring resource usage using labels¶

Here is a configuration I made for the ampliseq pipeline, as I found the defaults to be unoptimized.

nextflow.config

process {
    cpus   = { check_max( 1    * task.attempt, 'cpus'   ) }
    memory = { check_max( 6.GB * task.attempt, 'memory' ) }
    time   = { check_max( 4.h  * task.attempt, 'time'   ) }

    errorStrategy = { task.exitStatus in ((130..145) + 104) ? 'retry' : 'finish' }
    maxRetries    = 3
    maxErrors     = '-1'

    // Process-specific resource requirements
    // NOTE - Please try and re-use the labels below as much as possible.
    //        These labels are used and recognised by default in DSL2 files hosted on nf-core/modules.
    //        If possible, it would be nice to keep the same label naming convention when
    //        adding in your local modules too.
    // See https://www.nextflow.io/docs/latest/config.html#config-process-selectors
    withLabel:process_single {
        cpus   = { check_max( 1                  , 'cpus'    ) }
        memory = { check_max( 8.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 4.h  * task.attempt, 'time'    ) }
    }
    withLabel:process_low {
        cpus   = { check_max( 2     * task.attempt, 'cpus'    ) }
        memory = { check_max( 16.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 6.h   * task.attempt, 'time'    ) }
    }
    withLabel:process_medium {
        cpus   = { check_max( 24     * task.attempt, 'cpus'    ) }
        memory = { check_max( 192.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 12.h   * task.attempt, 'time'   ) }
    }
    withLabel:process_high {
        cpus   = { check_max( 96    * task.attempt, 'cpus'    ) }
        memory = { check_max( 384.GB * task.attempt, 'memory' ) }
        time   = { check_max( 36.h  * task.attempt, 'time'    ) }
    }
    withLabel:process_long {
        time   = { check_max( 48.h  * task.attempt, 'time'    ) }
    }
    withLabel:single_cpu {
        cpus   = { check_max( 1                  , 'cpus'    ) }
    }
    withLabel:process_high_memory {
        memory = { check_max( 200.GB * task.attempt, 'memory' ) }
    }
    withLabel:error_ignore {
        errorStrategy = 'ignore'
    }
    withLabel:error_retry {
        errorStrategy = 'retry'
        maxRetries    = 3
    }
}

// Function to ensure that resource requirements don't go beyond
// a maximum limit
def check_max(obj, type) {
    if (type == 'memory') {
        try {
            if (obj.compareTo(params.max_memory as nextflow.util.MemoryUnit) == 1)
                return params.max_memory as nextflow.util.MemoryUnit
            else
                return obj
        } catch (all) {
            println "   ### ERROR ###   Max memory '${params.max_memory}' is not valid! Using default value: $obj"
            return obj
        }
    } else if (type == 'time') {
        try {
            if (obj.compareTo(params.max_time as nextflow.util.Duration) == 1)
                return params.max_time as nextflow.util.Duration
            else
                return obj
        } catch (all) {
            println "   ### ERROR ###   Max time '${params.max_time}' is not valid! Using default value: $obj"
            return obj
        }
    } else if (type == 'cpus') {
        try {
            return Math.min( obj, params.max_cpus as int )
        } catch (all) {
            println "   ### ERROR ###   Max cpus '${params.max_cpus}' is not valid! Using default value: $obj"
            return obj
        }
    }
}

What are labels?¶

Labels are a way for pipeline authors to allow users to dynamically control the resources requested for each of the pipeline steps. The labels should be relatively consistent across nf-core pipelines, but each pipeline may benefit from minor adjustments to the resources available for each label.

Here are all of the changes I needed to make for the ampliseq pipeline:

diff --git a/modules/local/assignsh.nf b/modules/local/assignsh.nf
index 521e..56e8e 100644
--- a/modules/local/assignsh.nf
+++ b/modules/local/assignsh.nf
@@ -1,6 +1,6 @@
 process ASSIGNSH {
     tag "${asvtable}"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/barrnap.nf b/modules/local/barrnap.nf
index 1877d..2295 100644
--- a/modules/local/barrnap.nf
+++ b/modules/local/barrnap.nf
@@ -1,6 +1,6 @@
 process BARRNAP {
     tag "${fasta}"
-    label 'process_low'
+    label 'process_high'

     conda "bioconda::barrnap=0.9"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/combine_table.nf b/modules/local/combine_table.nf
index bfde..44ea 100644
--- a/modules/local/combine_table.nf
+++ b/modules/local/combine_table.nf
@@ -1,5 +1,5 @@
 process COMBINE_TABLE {
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-biostrings=2.58.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/cutadapt_summary.nf b/modules/local/cutadapt_summary.nf
index 4f3d..f79b 100644
--- a/modules/local/cutadapt_summary.nf
+++ b/modules/local/cutadapt_summary.nf
@@ -1,6 +1,6 @@
 process CUTADAPT_SUMMARY {
     tag "${name}"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::python=3.8.3"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/cutadapt_summary_merge.nf b/modules/local/cutadapt_summary_merge.nf
index d3ce..f667 100644
--- a/modules/local/cutadapt_summary_merge.nf
+++ b/modules/local/cutadapt_summary_merge.nf
@@ -1,6 +1,6 @@
 process CUTADAPT_SUMMARY_MERGE {
     tag "${files}"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/dada2_denoising.nf b/modules/local/dada2_denoising.nf
index 637b..a95e 100644
--- a/modules/local/dada2_denoising.nf
+++ b/modules/local/dada2_denoising.nf
@@ -1,6 +1,6 @@
 process DADA2_DENOISING {
     tag "$meta.run"
-    label 'process_medium'
+    label 'process_high'
     label 'process_long'

     conda "bioconda::bioconductor-dada2=1.30.0"
diff --git a/modules/local/dada2_err.nf b/modules/local/dada2_err.nf
index 9f46..491a 100644
--- a/modules/local/dada2_err.nf
+++ b/modules/local/dada2_err.nf
@@ -1,6 +1,6 @@
 process DADA2_ERR {
     tag "$meta.run"
-    label 'process_medium'
+    label 'process_high'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/dada2_filtntrim.nf b/modules/local/dada2_filtntrim.nf
index a2a0..e48de 100644
--- a/modules/local/dada2_filtntrim.nf
+++ b/modules/local/dada2_filtntrim.nf
@@ -1,6 +1,6 @@
 process DADA2_FILTNTRIM {
     tag "$meta.id"
-    label 'process_medium'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/dada2_merge.nf b/modules/local/dada2_merge.nf
index 8577..98e4 100644
--- a/modules/local/dada2_merge.nf
+++ b/modules/local/dada2_merge.nf
@@ -1,5 +1,5 @@
 process DADA2_MERGE {
-    label 'process_low'
+    label 'process_single'

     // https://depot.galaxyproject.org/singularity/bioconductor-dada2=1.28.0--r43hf17093f_0 doesnt contain 'digest', so keep here v1.22.0
     conda "bioconda::bioconductor-dada2=1.22.0 conda-forge::r-digest=0.6.30"
diff --git a/modules/local/dada2_quality.nf b/modules/local/dada2_quality.nf
index 5401..2aa3d 100644
--- a/modules/local/dada2_quality.nf
+++ b/modules/local/dada2_quality.nf
@@ -1,6 +1,6 @@
 process DADA2_QUALITY {
     tag "$meta"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/dada2_rmchimera.nf b/modules/local/dada2_rmchimera.nf
index 0ba5..d228 100644
--- a/modules/local/dada2_rmchimera.nf
+++ b/modules/local/dada2_rmchimera.nf
@@ -1,6 +1,6 @@
 process DADA2_RMCHIMERA {
     tag "$meta.run"
-    label 'process_medium'
+    label 'process_high'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/dada2_splitregions.nf b/modules/local/dada2_splitregions.nf
index 55ff..9444 100644
--- a/modules/local/dada2_splitregions.nf
+++ b/modules/local/dada2_splitregions.nf
@@ -1,5 +1,5 @@
 process DADA2_SPLITREGIONS {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::r-base=4.2.1"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/dada2_stats.nf b/modules/local/dada2_stats.nf
index 788d..ee59 100644
--- a/modules/local/dada2_stats.nf
+++ b/modules/local/dada2_stats.nf
@@ -1,6 +1,6 @@
 process DADA2_STATS {
     tag "$meta.run"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/filter_clusters.nf b/modules/local/filter_clusters.nf
index 5c0d..2559b 100644
--- a/modules/local/filter_clusters.nf
+++ b/modules/local/filter_clusters.nf
@@ -1,6 +1,6 @@
 process FILTER_CLUSTERS {
     tag "${meta.id}"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/filter_codons.nf b/modules/local/filter_codons.nf
index 265a..6cb2 100644
--- a/modules/local/filter_codons.nf
+++ b/modules/local/filter_codons.nf
@@ -1,6 +1,6 @@
 process FILTER_CODONS {
     tag "${fasta}"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/filter_len.nf b/modules/local/filter_len.nf
index 4964..df0a 100644
--- a/modules/local/filter_len.nf
+++ b/modules/local/filter_len.nf
@@ -1,6 +1,6 @@
 process FILTER_LEN {
     tag "${fasta}"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-biostrings=2.58.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/filter_ssu.nf b/modules/local/filter_ssu.nf
index 314a..1028 100644
--- a/modules/local/filter_ssu.nf
+++ b/modules/local/filter_ssu.nf
@@ -1,6 +1,6 @@
 process FILTER_SSU {
     tag "${fasta}"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-biostrings=2.58.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/filter_stats.nf b/modules/local/filter_stats.nf
index d83e..d2dc 100644
--- a/modules/local/filter_stats.nf
+++ b/modules/local/filter_stats.nf
@@ -1,5 +1,5 @@
 process FILTER_STATS {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_fastainput.nf b/modules/local/format_fastainput.nf
index 5f26..a30a 100644
--- a/modules/local/format_fastainput.nf
+++ b/modules/local/format_fastainput.nf
@@ -1,5 +1,5 @@
 process FORMAT_FASTAINPUT {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxonomy.nf b/modules/local/format_taxonomy.nf
index e2e6c..23d9 100644
--- a/modules/local/format_taxonomy.nf
+++ b/modules/local/format_taxonomy.nf
@@ -1,5 +1,5 @@
 process FORMAT_TAXONOMY {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxonomy_qiime.nf b/modules/local/format_taxonomy_qiime.nf
index 1fa1..1a250 100644
--- a/modules/local/format_taxonomy_qiime.nf
+++ b/modules/local/format_taxonomy_qiime.nf
@@ -1,6 +1,6 @@

 process FORMAT_TAXONOMY_QIIME {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxonomy_sidle.nf b/modules/local/format_taxonomy_sidle.nf
index f554..189c 100644
--- a/modules/local/format_taxonomy_sidle.nf
+++ b/modules/local/format_taxonomy_sidle.nf
@@ -1,5 +1,5 @@
 process FORMAT_TAXONOMY_SIDLE {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxonomy_sintax.nf b/modules/local/format_taxonomy_sintax.nf
index 06fd2..355d 100644
--- a/modules/local/format_taxonomy_sintax.nf
+++ b/modules/local/format_taxonomy_sintax.nf
@@ -1,5 +1,5 @@
 process FORMAT_TAXONOMY_SINTAX {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxresults.nf b/modules/local/format_taxresults.nf
index 5e7c..a41351 100644
--- a/modules/local/format_taxresults.nf
+++ b/modules/local/format_taxresults.nf
@@ -1,5 +1,5 @@
 process FORMAT_TAXRESULTS {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxresults_kraken2.nf b/modules/local/format_taxresults_kraken2.nf
index dd7c..e9b5 100644
--- a/modules/local/format_taxresults_kraken2.nf
+++ b/modules/local/format_taxresults_kraken2.nf
@@ -1,5 +1,5 @@
 process FORMAT_TAXRESULTS_KRAKEN2 {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::r-base=4.2.1"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/format_taxresults_sintax.nf b/modules/local/format_taxresults_sintax.nf
index eaf7..9179c 100644
--- a/modules/local/format_taxresults_sintax.nf
+++ b/modules/local/format_taxresults_sintax.nf
@@ -1,5 +1,5 @@
 process FORMAT_TAXRESULTS_SINTAX {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::python=3.9"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/merge_stats.nf b/modules/local/merge_stats.nf
index d934..07ce 100644
--- a/modules/local/merge_stats.nf
+++ b/modules/local/merge_stats.nf
@@ -1,5 +1,5 @@
 process MERGE_STATS {
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/metadata_all.nf b/modules/local/metadata_all.nf
index 770f..4d05 100644
--- a/modules/local/metadata_all.nf
+++ b/modules/local/metadata_all.nf
@@ -1,6 +1,6 @@
 process METADATA_ALL {
     tag "$metadata"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/metadata_pairwise.nf b/modules/local/metadata_pairwise.nf
index a4aa..199b 100644
--- a/modules/local/metadata_pairwise.nf
+++ b/modules/local/metadata_pairwise.nf
@@ -1,6 +1,6 @@
 process METADATA_PAIRWISE {
     tag "$metadata"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-dada2=1.30.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/phyloseq.nf b/modules/local/phyloseq.nf
index f2f5..086e0 100644
--- a/modules/local/phyloseq.nf
+++ b/modules/local/phyloseq.nf
@@ -1,6 +1,6 @@
 process PHYLOSEQ {
     tag "$prefix"
-    label 'process_low'
+    label 'process_single'

     conda "bioconda::bioconductor-phyloseq=1.46.0"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/phyloseq_inasv.nf b/modules/local/phyloseq_inasv.nf
index cd75..2623 100644
--- a/modules/local/phyloseq_inasv.nf
+++ b/modules/local/phyloseq_inasv.nf
@@ -1,5 +1,5 @@
 process PHYLOSEQ_INASV {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/phyloseq_intax.nf b/modules/local/phyloseq_intax.nf
index 6dbd..f32a 100644
--- a/modules/local/phyloseq_intax.nf
+++ b/modules/local/phyloseq_intax.nf
@@ -1,5 +1,5 @@
 process PHYLOSEQ_INTAX {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/qiime2_alphararefaction.nf b/modules/local/qiime2_alphararefaction.nf
index d531..95c1f 100644
--- a/modules/local/qiime2_alphararefaction.nf
+++ b/modules/local/qiime2_alphararefaction.nf
@@ -1,5 +1,5 @@
 process QIIME2_ALPHARAREFACTION {
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_barplot.nf b/modules/local/qiime2_barplot.nf
index 5559..def2 100644
--- a/modules/local/qiime2_barplot.nf
+++ b/modules/local/qiime2_barplot.nf
@@ -1,5 +1,5 @@
 process QIIME2_BARPLOT {
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_diversity_adonis.nf b/modules/local/qiime2_diversity_adonis.nf
index 946a..01579 100644
--- a/modules/local/qiime2_diversity_adonis.nf
+++ b/modules/local/qiime2_diversity_adonis.nf
@@ -1,6 +1,6 @@
 process QIIME2_DIVERSITY_ADONIS {
     tag "${core.baseName} - ${formula}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_diversity_alpha.nf b/modules/local/qiime2_diversity_alpha.nf
index ba755..327f 100644
--- a/modules/local/qiime2_diversity_alpha.nf
+++ b/modules/local/qiime2_diversity_alpha.nf
@@ -1,6 +1,6 @@
 process QIIME2_DIVERSITY_ALPHA {
     tag "${core.baseName}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_diversity_beta.nf b/modules/local/qiime2_diversity_beta.nf
index eef3..9011 100644
--- a/modules/local/qiime2_diversity_beta.nf
+++ b/modules/local/qiime2_diversity_beta.nf
@@ -1,6 +1,6 @@
 process QIIME2_DIVERSITY_BETA {
     tag "${core.baseName} - ${category}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_diversity_betaord.nf b/modules/local/qiime2_diversity_betaord.nf
index 9e54..9ef5 100644
--- a/modules/local/qiime2_diversity_betaord.nf
+++ b/modules/local/qiime2_diversity_betaord.nf
@@ -1,6 +1,6 @@
 process QIIME2_DIVERSITY_BETAORD {
     tag "${core.baseName}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_diversity_core.nf b/modules/local/qiime2_diversity_core.nf
index b06f..213b 100644
--- a/modules/local/qiime2_diversity_core.nf
+++ b/modules/local/qiime2_diversity_core.nf
@@ -1,5 +1,5 @@
 process QIIME2_DIVERSITY_CORE {
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_export_absolute.nf b/modules/local/qiime2_export_absolute.nf
index bf9c..341f 100644
--- a/modules/local/qiime2_export_absolute.nf
+++ b/modules/local/qiime2_export_absolute.nf
@@ -1,5 +1,5 @@
 process QIIME2_EXPORT_ABSOLUTE {
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_export_relasv.nf b/modules/local/qiime2_export_relasv.nf
index 19977..24e7 100644
--- a/modules/local/qiime2_export_relasv.nf
+++ b/modules/local/qiime2_export_relasv.nf
@@ -1,5 +1,5 @@
 process QIIME2_EXPORT_RELASV {
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_export_reltax.nf b/modules/local/qiime2_export_reltax.nf
index 6309..8f52 100644
--- a/modules/local/qiime2_export_reltax.nf
+++ b/modules/local/qiime2_export_reltax.nf
@@ -1,5 +1,5 @@
 process QIIME2_EXPORT_RELTAX {
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_featuretable_group.nf b/modules/local/qiime2_featuretable_group.nf
index 0890a5..55a8 100644
--- a/modules/local/qiime2_featuretable_group.nf
+++ b/modules/local/qiime2_featuretable_group.nf
@@ -1,6 +1,6 @@
 process QIIME2_FEATURETABLE_GROUP {
     tag "${category}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_filtersamples.nf b/modules/local/qiime2_filtersamples.nf
index 46a5..8d91 100644
--- a/modules/local/qiime2_filtersamples.nf
+++ b/modules/local/qiime2_filtersamples.nf
@@ -1,6 +1,6 @@
 process QIIME2_FILTERSAMPLES {
     tag "${filter}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_inasv.nf b/modules/local/qiime2_inasv.nf
index 4ef6..afd4 100644
--- a/modules/local/qiime2_inasv.nf
+++ b/modules/local/qiime2_inasv.nf
@@ -1,6 +1,6 @@
 process QIIME2_INASV {
     tag "${asv}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_inseq.nf b/modules/local/qiime2_inseq.nf
index 53b4..6ca34 100644
--- a/modules/local/qiime2_inseq.nf
+++ b/modules/local/qiime2_inseq.nf
@@ -1,6 +1,6 @@
 process QIIME2_INSEQ {
     tag "${seq}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_intax.nf b/modules/local/qiime2_intax.nf
index 1b1c..bfd2 100644
--- a/modules/local/qiime2_intax.nf
+++ b/modules/local/qiime2_intax.nf
@@ -1,6 +1,6 @@
 process QIIME2_INTAX {
     tag "${tax}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_intree.nf b/modules/local/qiime2_intree.nf
index 70d4..d9d9 100644
--- a/modules/local/qiime2_intree.nf
+++ b/modules/local/qiime2_intree.nf
@@ -1,6 +1,6 @@
 process QIIME2_INTREE {
     tag "${meta.id}:${meta.model}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_seqfiltertable.nf b/modules/local/qiime2_seqfiltertable.nf
index d031..4163 100644
--- a/modules/local/qiime2_seqfiltertable.nf
+++ b/modules/local/qiime2_seqfiltertable.nf
@@ -1,6 +1,6 @@
 process QIIME2_SEQFILTERTABLE {
     tag "${repseq} filter by ${table}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_tablefiltertaxa.nf b/modules/local/qiime2_tablefiltertaxa.nf
index edfb..2587 100644
--- a/modules/local/qiime2_tablefiltertaxa.nf
+++ b/modules/local/qiime2_tablefiltertaxa.nf
@@ -1,6 +1,6 @@
 process QIIME2_TABLEFILTERTAXA {
     tag "taxa:${exclude_taxa};min-freq:${min_frequency};min-samples:${min_samples}"
-    label 'process_low'
+    label 'process_single'

     container "qiime2/core:2023.7"

diff --git a/modules/local/qiime2_tree.nf b/modules/local/qiime2_tree.nf
index d8b9..bb03 100644
--- a/modules/local/qiime2_tree.nf
+++ b/modules/local/qiime2_tree.nf
@@ -1,5 +1,5 @@
 process QIIME2_TREE {
-    label 'process_medium'
+    label 'process_high'

     container "qiime2/core:2023.7"

diff --git a/modules/local/rename_raw_data_files.nf b/modules/local/rename_raw_data_files.nf
index 5237..086b 100644
--- a/modules/local/rename_raw_data_files.nf
+++ b/modules/local/rename_raw_data_files.nf
@@ -1,6 +1,6 @@
 process RENAME_RAW_DATA_FILES {
     tag "$meta.id"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::sed=4.7"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/sbdiexport.nf b/modules/local/sbdiexport.nf
index ce46..4b72 100644
--- a/modules/local/sbdiexport.nf
+++ b/modules/local/sbdiexport.nf
@@ -1,6 +1,6 @@
 process SBDIEXPORT {
     tag "${asvtable},${taxonomytable},${metadata}"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::r-tidyverse=1.2.1"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/sbdiexportreannotate.nf b/modules/local/sbdiexportreannotate.nf
index 8ebe..2167 100644
--- a/modules/local/sbdiexportreannotate.nf
+++ b/modules/local/sbdiexportreannotate.nf
@@ -1,6 +1,6 @@
 process SBDIEXPORTREANNOTATE {
     tag "${taxonomytable}"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::r-tidyverse=1.2.1"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/sidle_dbfilt.nf b/modules/local/sidle_dbfilt.nf
index 0c91..72eb 100644
--- a/modules/local/sidle_dbfilt.nf
+++ b/modules/local/sidle_dbfilt.nf
@@ -1,5 +1,5 @@
 process SIDLE_DBFILT {
-    label 'process_low'
+    label 'process_single'

     container 'nf-core/pipesidle:0.1.0-beta'

diff --git a/modules/local/summary_report.nf b/modules/local/summary_report.nf
index 309dc..a86fd 100644
--- a/modules/local/summary_report.nf
+++ b/modules/local/summary_report.nf
@@ -1,5 +1,5 @@
 process SUMMARY_REPORT  {
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::r-base=4.2.3 conda-forge::r-rmarkdown=2.22 conda-forge::r-tidyverse=2.0.0 conda-forge::r-knitr=1.43 conda-forge::r-dt=0.28 conda-forge::r-dtplyr=1.3.1 conda-forge::r-formattable=0.2.1 conda-forge::r-purrr=1.0.1 conda-forge::r-vegan=2.6_4 conda-forge::r-optparse=1.7.3 conda-forge::r-ggplot2=3.4.2 conda-forge::r-dplyr=1.1.2 conda-forge::r-data.table=1.14.8 conda-forge::r-patchwork=1.1.2"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/local/trunclen.nf b/modules/local/trunclen.nf
index abfa..c8f3 100644
--- a/modules/local/trunclen.nf
+++ b/modules/local/trunclen.nf
@@ -1,6 +1,6 @@
 process TRUNCLEN {
     tag "$meta"
-    label 'process_low'
+    label 'process_single'

     conda "conda-forge::pandas=1.1.5"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/nf-core/cutadapt/main.nf b/modules/nf-core/cutadapt/main.nf
index 3d3e..d689 100644
--- a/modules/nf-core/cutadapt/main.nf
+++ b/modules/nf-core/cutadapt/main.nf
@@ -1,6 +1,6 @@
 process CUTADAPT {
     tag "$meta.id"
-    label 'process_medium'
+    label 'process_low'

     conda "${moduleDir}/environment.yml"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
diff --git a/modules/nf-core/fastqc/main.nf b/modules/nf-core/fastqc/main.nf
index d79f1..697b 100644
--- a/modules/nf-core/fastqc/main.nf
+++ b/modules/nf-core/fastqc/main.nf
@@ -1,6 +1,6 @@
 process FASTQC {
     tag "$meta.id"
-    label 'process_medium'
+    label 'process_low'

     conda "${moduleDir}/environment.yml"
     container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?

It may be more worthwhile to use defaults unless you plan on running the pipeline very often.

Other methods to control resources¶

There are other process selectors implemented in nextflow as well. The withName selector can help you modify a single step in the workflow, but you have to know the name of each step in order to use this selector.

You can find the names of the processes in the modules/local directory, within the nextflow (.nf suffix) files.

As an example:

➜ rg -m1 process modules/local | head
modules/local/dada2_addspecies.nf:process DADA2_ADDSPECIES {
modules/local/qiime2_intree.nf:process QIIME2_INTREE {
modules/local/phyloseq.nf:process PHYLOSEQ {
modules/local/qiime2_seqfiltertable.nf:process QIIME2_SEQFILTERTABLE {
modules/local/assignsh.nf:process ASSIGNSH {
modules/local/qiime2_tree.nf:process QIIME2_TREE {
modules/local/format_taxonomy_sintax.nf:process FORMAT_TAXONOMY_SINTAX {
modules/local/sidle_taxrecon.nf:process SIDLE_TAXRECON {
modules/local/sidle_treerecon.nf:process SIDLE_TREERECON {
modules/local/qiime2_ancombc_tax.nf:process QIIME2_ANCOMBC_TAX {

So DADA2_ADDSPECIES, QIIME2_INTREE, etc. are the process names.

Developing Nextflow pipelines¶

The nextflow pull command can download nextflow pipelines using git so that users can make changes to the pipeline and either save the changes using git for themselves and/or push the changes back upstream so that the workflow author might include them.

The nextflow pull command looks at the $NXF_ASSETS environment variable to determine its location. The CQLS maintains some of these assets in /local/cqls/software/nextflow/pull, which is just adjacent to the workflows downloaded using the nf-core software. Unfortunately, these two methods are incompatible with one another, and cannot be stored in the same directory.

If you are an author of nextflow pipelines, you may want to set your NXF_ASSETS variable for yourself so that you can use nextflow pull to download workflows and work on them yourself.

Configuring your github token for use with Nextflow¶

You may get a github API error if you do not do these steps:

Login to https://github.com
Go to settings -> developer settings -> personal access tokens -> fine-grained tokens or click here
Put Nextflow for the Token name
- You can call it whatever you want, but Nextflow makes sense to me.
- Make note of the expiration date that you choose.
Click Generate token
Make a file that looks like this:

~/.nextflow/scm

providers {
  github {
    user = 'GITHUB USERNAME'
    password = 'GITHUB PAT'
  }
}

Put you github username in for the 'GITHUB USERNAME' and the access token that you just generated for the 'GITHUB PAT'.