What we’re doing:

• FASTQC: Analyzes our raw sequencing data (FASTQ file) and generates quality control reports. It checks things like read quality scores, sequence duplication levels, GC content, and potential contamination. Each dataset gets its own HTML report with pretty graphs.
  
• MultiQC: Takes all those individual FASTQC reports and aggregates them into a single, beautiful summary report. Instead of opening 10 different HTML files, you get a comprehensive view of your data quality in one place.
  

Think of it this way: FASTQC is your detailed inspector, checking each piece individually, and MultiQC is the manager who summarises everything into a single executive report.

main.nf - This is your pipeline definition

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

params {
    reads  = 's3://your-bucket-name/inputs/sample_data.fastq'
    outdir = 's3://your-bucket-name/results'
}

process FASTQC {
    container 'biocontainers/fastqc:v0.11.9_cv8'
    publishDir "${params.outdir}/fastqc", mode: 'copy'
    
    input:
    path reads
    
    output:
    tuple path("*.html"), path("*_fastqc.zip")
    
    script:
    """
    echo "Running FASTQC on ${reads}"
    fastqc ${reads}
    echo "FASTQC completed for ${reads}"
    """
}

process MULTIQC {
    container 'multiqc/multiqc:dev'
    publishDir "${params.outdir}/multiqc", mode: 'copy'
    
    input:
    path fastqc_reports
    
    output:
    path "multiqc_report.html"
    path "multiqc_data"
    
    script:
    """
    echo "Aggregating FASTQC reports with MultiQC"
    multiqc .
    echo "MultiQC report generated successfully"
    """
}

workflow {
    reads_ch = Channel.fromPath(params.reads)
    fastqc_reports_ch = FASTQC(reads_ch)
    MULTIQC(fastqc_reports_ch.collect())
}
‍

nextflow.config - This tell Nextflow how to run on batch :

plugins {
    id 'nf-wave'
    id 'nf-amazon'
}

wave {
    enabled = true
}

fusion {
    enabled = true

}

process {
    executor = 'awsbatch'
    queue    = '<job-queue-name>'
    
    cpus   = 1
    memory = '2 GB'

    // Error strategy: retry on transient AWS Batch issues
    errorStrategy = { task.exitStatus in [130, 137, 143, 151] ? 'retry' : 'finish' }
    maxRetries = 3

    withName: FASTQC {
        container = 'biocontainers/fastqc:v0.11.9_cv8'
    }

    withName: MULTIQC {
        container = 'multiqc/multiqc:v1.14'
    }
}

aws {
    region = 'us-east-1'
    batch {
        volumes = '/fusion'
    }
}

workDir = 's3://<your-bucket-name>/work'

‍Important: Replace <yourbucket-name> and <job-queue-name>with your actual name in both files.Push this to your GitHub repo.

Run the Pipeline

nextflow run https://github.com/username/repo-name.git

Outlook will look like:

N E X T F L O W  ~  version 24.04.4
Pulling username/repo-name ...
downloaded from https://github.com/username/repo-name.git
Downloading plugin nf-wave@1.16.1
Downloading plugin nf-amazon@3.4.2
Downloading plugin nf-tower@1.17.3
Launching `https://github.com/username/repo-name` [awesome_lichterman] DSL2 - revision: e4b5f72abf [main]
executor >  awsbatch (2)
[a1/b2c3] process > FASTQC   [100%] 1 of 1 ✔
[e5/f6g7] process > MULTIQC  [100%] 1 of 1 ✔
Completed at: 08-Jan-2025 10:30:15
Duration    : 3m 45s
CPU hours   : 0.4
Succeeded   : 2

What happens:

1. Nextflow submits jobs to AWS Batch
2. Batch automatically spins up EC2 instances
3. Runs FASTQC and MultiQC in containers
4. Saves results to S3
5. Shuts down instances when done