The Nextflow Pipeline

For this example, we’re running a bioinformatics quality control workflow on genomic sequencing data. Specifically:

What we’re doing:

• FASTQC: Analyzes our raw sequencing data (FASTQ file) and generates quality control reports. It checks things like read quality scores, sequence duplication levels, GC content, and potential contamination. Each dataset gets its own HTML report with pretty graphs.
• MultiQC: Takes all those individual FASTQC reports and aggregates them into a single, beautiful summary report. Instead of opening 10 different HTML files, you get a comprehensive view of your data quality in one place.

Think of it this way: FASTQC is your detailed inspector, checking each piece individually, and MultiQC is the manager who summarises everything into a single executive report.

main.nf- This is your pipeline definition:

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

params.reads = "s3://<your-bucket-name>/inputs/dataset.fastq"

process FASTQC {
    container 'biocontainers/fastqc:v0.11.9_cv8'
    publishDir "${params.outdir}/fastqc", mode: 'copy'
    
    input:
    path reads
    
    output:
    tuple path("*.html"), path("*_fastqc.zip")
    
    script:
    """
    echo "Running FASTQC on ${reads}"
    fastqc ${reads}
    echo "FASTQC completed for ${reads}"
    """
}

process MULTIQC {
    container 'multiqc/multiqc:dev'
    publishDir "${params.outdir}/multiqc", mode: 'copy'
    
    input:
    path fastqc_reports
    
    output:
    path "multiqc_report.html"
    path "multiqc_data"
    
    script:
    """
    echo "Aggregating FASTQC reports with MultiQC"
    multiqc .
    echo "MultiQC report generated successfully"
    """
}

workflow {
    reads_ch = Channel.fromPath(params.reads)
    fastqc_reports_ch = FASTQC(reads_ch)
    MULTIQC(fastqc_reports_ch.collect())
}

Important notes:

• Update <your-bucket-name>.git to match your S3 Bucket

nextflow.config- This tells Nextflow how to run on Kubernetes:

plugins {
    id 'nf-wave'
    id 'nf-amazon'
}

wave.enabled = true
fusion.enabled = true
fusion.exportStorageCredentials = true 

process {
    executor = 'k8s'
    cpus = 1
    memory = '2 GB'
    maxForks = 1

    withName: 'FASTQC' {
        container = 'biocontainers/fastqc:v0.11.9_cv8'
    }
    withName: 'MULTIQC' {
        container = 'multiqc/multiqc:dev'
    }
}

k8s {
    namespace = 'nextflow'
    serviceAccount = 'nextflow-sa'
}

Push this to your GitHub repo.

The Kubernetes Job

Now for the piece that ties everything together. But first, let’s talk about why we’re using a Kubernetes Job instead of a Deployment or a plain Pod.

Why a Job?

Kubernetes gives us several ways to run workloads, and choosing the right one matters:

• Pod: Runs once, but if it fails, it’s done. No retry logic, no cleanup guarantees. Not ideal for batch workloads.
• Deployment: Designed for long-running services that should stay up. If a pod dies, it gets restarted automatically. Great for web servers, terrible for batch jobs that need to run once and finish.
• Job: Perfect for batch workloads! It runs to completion, has built-in retry logic (controlled by backoffLimit), and can automatically clean itself up when done (via ttlSecondsAfterFinished).

For Nextflow pipelines, we want:

• Run once and finish
• Automatic cleanup after completion
• Clear success/failure status
• No accidental restarts

That’s exactly what a Job gives us.

Save this as nextflow-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  namespace: nextflow
  generateName: "nextflow-job-"
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 300
  template:
    spec:
      serviceAccountName: nextflow-sa
      restartPolicy: Never
      
      initContainers:
        - name: fetch-pipeline
          image: alpine/git:latest
          env:
            - name: BRANCH
              value: "main"  # Change to your branch name
          command:
            - sh
            - -c
            - |
              set -e
              mkdir -p /root/.ssh
              cp /var/ssh/ssh-privatekey /root/.ssh/id_rsa
              chmod 600 /root/.ssh/id_rsa
              ssh-keyscan github.com >> /root/.ssh/known_hosts
              git clone -b "${BRANCH}" git@github.com:<your-username>/<your-repo>.git /workspace
          volumeMounts:
            - name: workspace
              mountPath: /workspace
            - name: git-ssh-key
              mountPath: /var/ssh
              readOnly: true
      
      containers:
        - name: nextflow
          image: nextflow/nextflow:25.12.0-edge
          env:
            - name: NXF_PLUGINS_DEFAULT
              value: "nf-amazon,nf-wave,nf-k8s"
            - name: JOB_NAME
              value: "fastqc-analysis"
          command:
            - sh
            - -c
            - |
              set -e
              cd /workspace
              WORK_DIR="s3://<your-bucket-name>/work/${JOB_NAME}"
              RESULTS_DIR="s3://<your-bucket-name>/results/${JOB_NAME}"
              echo "Work directory: $WORK_DIR"
              echo "Results directory: $RESULTS_DIR"
              exec nextflow run main.nf \
                -c nextflow.config \
                --outdir $RESULTS_DIR \
                -work-dir "$WORK_DIR" \
                -resume
          volumeMounts:
            - name: workspace
              mountPath: /workspace

      volumes:
        - name: workspace
          emptyDir: {}
        - name: git-ssh-key
          secret:
            secretName: git-ssh-key
            defaultMode: 0400

Important notes:

• Update <your-username>/<your-repo>.git to match your GitHub repo
• Update <your-bucket-name> to match your S3 bucket
• The generateName field means each run creates a unique job

Running the Pipeline

Here’s the fun part. Launch your pipeline:

kubectl create -f nextflow-job.yaml

Note: We use kubectl create instead of kubectl apply because generateName creates a new job each time.

Monitoring Your Workflow

Watch the job spin up :

kubectl get jobs -n nextflow

Check the logs in real-time:

kubectl logs job/<job-name> -n nextflow -f

You’ll see Nextflow do its thing, spawning worker pods, running tasks, and saving results to S3.

Checking Your Results

Once everything completes, head to S3. You should see your FASTQC HTML reports and MultiQC summary. Beautiful!

Cleanup

The ttlSecondsAfterFinished: 300 Setting in our Job spec means Kubernetes automatically deletes completed jobs after 5 minutes. This keeps your cluster clean.