How We Built Scalable Bioinformatics Pipelines on AWS EKS Using Nextflow

Why EKS for Nextflow?

Nextflow is excellent for orchestrating complex computational workflows, but those workflows often require substantial and highly variable compute resources.

Running everything locally or on a single EC2 instance quickly becomes a bottleneck. Kubernetes provides:

• Automatic scaling: create pods when they are needed and remove them when work finishes.
• Cost optimization: use Spot Instances to reduce compute expenses for interruption-tolerant workloads.
• Flexibility: run multiple workflows concurrently without resource conflicts.
• Reliability: use built-in retry behavior, scheduling, and fault tolerance.

Amazon EKS provides a managed Kubernetes control plane, reducing the operational effort required to run Kubernetes infrastructure.

Configure the AWS CLI

The AWS CLI is used to interact with AWS services from the terminal.

Install the AWS CLI on Linux

# Download the AWS CLI installer
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

# Extract and install
unzip awscliv2.zip
sudo ./aws/install

Install the AWS CLI on macOS

brew install awscli

Configure AWS Credentials

aws configure

You will be prompted for:

• AWS Access Key ID: the access key associated with your IAM identity.
• AWS Secret Access Key: the corresponding secret key.
• Default Region: this guide uses us-east-1.
• Default output format: json is suitable for this setup.

When working with multiple AWS accounts, use aws configure --profile <profile-name> to keep credentials and configuration separated.

Install kubectl

kubectl is the command-line tool used to interact with Kubernetes clusters.

Install kubectl on Linux

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Install kubectl on macOS

brew install kubectl

Verify the Installation

kubectl version --client

Install eksctl

eksctl simplifies the process of creating and managing Amazon EKS clusters.

Install eksctl on Linux

curl --silent --location \
"https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar xz -C /tmp

sudo mv /tmp/eksctl /usr/local/bin

Install eksctl on macOS

brew install eksctl

Create the EKS Cluster

This example uses eksctl to create an EKS cluster with a managed Spot node group.

eksctl create cluster \
  --name nextflow-eks-cluster \
  --region us-east-1 \
  --nodegroup-name spot-nodes \
  --instance-types t3.medium \
  --managed \
  --spot \
  --nodes 1 \
  --with-oidc \
  --vpc-nat-mode Disable

The NAT Gateway is disabled here to reduce costs during testing. For production environments, evaluate private networking, NAT Gateway usage, and VPC endpoints based on your security and connectivity requirements.

The --spot option configures the node group to use Spot Instances, which can reduce compute costs for interruption-tolerant batch workloads.

Install the EKS Pod Identity Agent

EKS Pod Identity allows Kubernetes workloads to assume IAM roles without embedding long-lived AWS credentials in containers.

eksctl create addon \
  --cluster nextflow-eks-cluster \
  --name eks-pod-identity-agent \
  --region us-east-1

The add-on runs as a DaemonSet and manages credential delivery to eligible pods.

Create the S3 Bucket

Create an Amazon S3 bucket and add an inputs/ folder for the source dataset.

This example uses a FASTQ file. A public genomics dataset can be used when following the guide.

Configure IAM Permissions

The Nextflow pods need permission to:

• Read from and write to the required S3 bucket
• Run under the configured Kubernetes service account

eksctl create podidentityassociation \
  --cluster nextflow-eks-cluster \
  --namespace nextflow \
  --serviceaccount nextflow-sa \
  --role-name nextflow-pod-role \
  --permission-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
  --region us-east-1

For production, replace AmazonS3FullAccess with a least-privilege custom policy that grants access only to the required bucket and object prefixes.

Create the Namespace and Service Account

A dedicated namespace keeps Nextflow resources organized and isolated from unrelated workloads.

kubectl create namespace nextflow
kubectl create serviceaccount nextflow-sa --namespace nextflow

Configure Kubernetes RBAC

Role-Based Access Control defines what the Nextflow coordinator pod is allowed to do. It needs permission to create worker pods, inspect their status, read logs, and remove completed resources.

Save the following as rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nextflow-role
  namespace: nextflow
rules:
  - apiGroups: [""]
    resources:
      - pods
      - pods/log
      - pods/status
    verbs:
      - get
      - list
      - watch
      - create
      - delete
  - apiGroups: ["batch"]
    resources:
      - jobs
    verbs:
      - get
      - list
      - watch
      - create
      - delete

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nextflow-rolebinding
  namespace: nextflow
subjects:
  - kind: ServiceAccount
    name: nextflow-sa
    namespace: nextflow
roleRef:
  kind: Role
  name: nextflow-role
  apiGroup: rbac.authorization.k8s.io

Apply the configuration:

kubectl apply -f rbac.yaml

Create SSH Credentials for GitHub

SSH credentials are required when the Nextflow pipeline is stored in a private GitHub repository. Public repositories can be cloned over HTTPS without this setup.

Generate a dedicated SSH key:

ssh-keygen -t ed25519 -C "nextflow-eks-pipeline"

Save the key to a dedicated path such as ~/.ssh/nextflow_eks_key.

Store the private key as a Kubernetes Secret:

kubectl create secret generic git-ssh-key \
  --from-file=ssh-privatekey=/path/to/your/private/key \
  --namespace nextflow

Protect the private key carefully. Use a repository-scoped deploy key and grant only the access required by the workflow.

The Bioinformatics Quality-Control Workflow

This example runs a quality-control workflow against genomic sequencing data.

• FASTQC: analyzes raw FASTQ data and reports read quality, sequence duplication, GC content, and potential contamination.
• MultiQC: combines multiple FASTQC reports into one consolidated summary.

FASTQC acts as the detailed inspector for each file, while MultiQC produces a combined overview of the batch.

main.nf: Pipeline Definition

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

params.reads = "s3://<your-bucket-name>/inputs/dataset.fastq"

process FASTQC {
    container 'biocontainers/fastqc:v0.11.9_cv8'
    publishDir "${params.outdir}/fastqc", mode: 'copy'

    input:
    path reads

    output:
    tuple path("*.html"), path("*_fastqc.zip")

    script:
    """
    echo "Running FASTQC on ${reads}"
    fastqc ${reads}
    echo "FASTQC completed for ${reads}"
    """
}

process MULTIQC {
    container 'multiqc/multiqc:dev'
    publishDir "${params.outdir}/multiqc", mode: 'copy'

    input:
    path fastqc_reports

    output:
    path "multiqc_report.html"
    path "multiqc_data"

    script:
    """
    echo "Aggregating FASTQC reports with MultiQC"
    multiqc .
    echo "MultiQC report generated successfully"
    """
}

workflow {
    reads_ch = Channel.fromPath(params.reads)
    fastqc_reports_ch = FASTQC(reads_ch)
    MULTIQC(fastqc_reports_ch.collect())
}

Replace <your-bucket-name> with the actual S3 bucket name.

nextflow.config: Kubernetes Configuration

plugins {
    id 'nf-wave'
    id 'nf-amazon'
}

wave.enabled = true
fusion.enabled = true
fusion.exportStorageCredentials = true

process {
    executor = 'k8s'
    cpus = 1
    memory = '2 GB'
    maxForks = 1

    withName: 'FASTQC' {
        container = 'biocontainers/fastqc:v0.11.9_cv8'
    }

    withName: 'MULTIQC' {
        container = 'multiqc/multiqc:dev'
    }
}

k8s {
    namespace = 'nextflow'
    serviceAccount = 'nextflow-sa'
}

Commit main.nf and nextflow.config to the GitHub repository.

Why Use a Kubernetes Job?

Kubernetes provides multiple workload controllers, and selecting the correct one matters.

• Pod: runs a workload directly but does not provide the same completion and retry semantics as a Job.
• Deployment: is designed for long-running services that should remain available continuously.
• Job: is designed for finite batch workloads that run to completion and report success or failure.

A Kubernetes Job provides the desired behavior for this workflow:

• Run once and finish
• Expose a clear success or failure state
• Support controlled retry behavior
• Allow automatic cleanup after completion

nextflow-job.yaml: Coordinator Job

apiVersion: batch/v1
kind: Job
metadata:
  namespace: nextflow
  generateName: "nextflow-job-"
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 300

  template:
    spec:
      serviceAccountName: nextflow-sa
      restartPolicy: Never

      initContainers:
        - name: fetch-pipeline
          image: alpine/git:latest

          env:
            - name: BRANCH
              value: "main"

          command:
            - sh
            - -c
            - |
              set -e

              mkdir -p /root/.ssh
              cp /var/ssh/ssh-privatekey /root/.ssh/id_rsa
              chmod 600 /root/.ssh/id_rsa
              ssh-keyscan github.com >> /root/.ssh/known_hosts

              git clone -b "${BRANCH}" \
                git@github.com:<your-username>/<your-repo>.git \
                /workspace

          volumeMounts:
            - name: workspace
              mountPath: /workspace

            - name: git-ssh-key
              mountPath: /var/ssh
              readOnly: true

      containers:
        - name: nextflow
          image: nextflow/nextflow:25.12.0-edge

          env:
            - name: NXF_PLUGINS_DEFAULT
              value: "nf-amazon,nf-wave,nf-k8s"

            - name: JOB_NAME
              value: "fastqc-analysis"

          command:
            - sh
            - -c
            - |
              set -e
              cd /workspace

              WORK_DIR="s3://<your-bucket-name>/work/${JOB_NAME}"
              RESULTS_DIR="s3://<your-bucket-name>/results/${JOB_NAME}"

              echo "Work directory: $WORK_DIR"
              echo "Results directory: $RESULTS_DIR"

              exec nextflow run main.nf \
                -c nextflow.config \
                --outdir "$RESULTS_DIR" \
                -work-dir "$WORK_DIR" \
                -resume

          volumeMounts:
            - name: workspace
              mountPath: /workspace

      volumes:
        - name: workspace
          emptyDir: {}

        - name: git-ssh-key
          secret:
            secretName: git-ssh-key
            defaultMode: 0400

Update the following placeholders:

• Replace <your-username>/<your-repo>.git with the GitHub repository path.
• Replace <your-bucket-name> with the S3 bucket name.
• Update the Nextflow image tag to a tested and approved version for production use.

The generateName field causes Kubernetes to generate a unique name for every execution.

Run the Pipeline

kubectl create -f nextflow-job.yaml

Use kubectl create instead of kubectl apply because generateName creates a new Job for every run.

Monitor the Workflow

List Jobs in the Nextflow namespace:

kubectl get jobs -n nextflow

Follow the coordinator Job logs:

kubectl logs job/<job-name> -n nextflow -f

Nextflow will create worker pods, execute the FASTQC and MultiQC tasks, and write the outputs to Amazon S3.

Check the Results

After the workflow completes, open the configured S3 results location. It should contain the FASTQC HTML reports and the MultiQC summary.

Automatic Cleanup

The setting ttlSecondsAfterFinished: 300 instructs Kubernetes to delete the completed Job after five minutes.

This prevents completed coordinator Jobs from accumulating in the cluster.