A germline exome batch fails at 2:14 a.m. on a memory error in one task.
Nobody finds out until the morning stand-up, and by then the batch has missed its slot on the cluster. Someone reruns it, it goes through, and the samples ship a day late.
Three weeks later a CAP inspector picks one of those samples and asks a simple question: which version of the pipeline produced this result, and did every step actually run? The answer lives somewhere across a trace file, two Slack threads, a cluster log that has already rotated, and one engineer's memory.
That gap is what bioinformatics pipeline observability is meant to close. Done well, it gives your team three things from the same run data: a fast alert when something breaks, a clear diagnosis of why, and a durable record of what ran, with what, and at what cost. Done badly, it gives you dashboards that look reassuring and prove very little.
Bioinformatics pipeline observability is the practice of capturing run status, task-level failure detail, execution evidence, and cost for every pipeline run, across engines such as Nextflow, WDL on Cromwell, Snakemake, and CWL. In a clinical lab, it supports traceability and faster recovery. It does not replace analytical validation or laboratory director approval.
In this article, we'll explore:
- The five questions your run data has to answer in a regulated lab
- Where Nextflow, Cromwell, and Snakemake already keep run evidence, and where it falls short
- What pipeline monitoring can and can't prove to an inspector
- How to set up pipeline failure alerts in Slack or Teams without leaking patient data
- How to calculate cost per sample across HPC and cloud
- When to build observability yourself, rely on engine reports, or adopt StrixFlow
What Is Bioinformatics Pipeline Observability, and How Is It Different From Monitoring?
Monitoring tells you whether a run is up or down. Observability tells you why it behaved the way it did, and lets you answer questions you didn't plan for in advance.
For a genomics pipeline, that difference shows up in three layers of run data:
- Status: is the run queued, running, failed, or finished, and which task is it on right now?
- Diagnosis: when a task fails, which process failed, with what exit code, on which attempt, and what did its error output say?
- Evidence: which code, containers, parameters, and reference assets ran, which tasks executed fresh and which were reused, and what the run cost per sample.
Most labs have the first layer. Fewer have the second in one place. Very few keep the third in a form a lab director could hand to an inspector, and that's the layer that matters most once patient results are involved.
The Five Questions Clinical Pipeline Run Data Must Answer
A useful test for any genomics pipeline observability setup is whether it answers these five questions for any sample, within minutes, without someone rebuilding the story from logs. They come straight from what lab directors, quality managers, and inspectors actually ask.
1Which pipeline release and dependencies produced this sample?
CLIA requires the lab to keep records of all specimen testing, with dates and the people who performed it, under 42 CFR 493.1283. For a bioinformatics step, the practical version of that is a run record that names the workflow commit, engine version, container images by digest, parameters, and reference build. The Nextflow trace file records each task's hash, container, work directory, exit status, attempt number, wall-clock time and peak memory. It doesn't record the workflow commit or reference checksums on its own.
2Did every task execute in this run, or was something reused from cache?
Nextflow -resume reuses a task when its hash matches and its outputs still exist. Cromwell can serve a call from its call cache, and its cache matching ignores changes to memory, CPU, and disks (Cromwell call caching). Reuse is efficient and often correct. It just has to be visible, because a result built partly from an earlier run is a different claim from a result built entirely in this one.
3What failed, where, and why?
A failure alert that says the workflow failed is barely useful at 2 a.m. The on-call engineer needs the process name, the sample, the exit code, the attempt number, and the last lines of standard error. An exit code of 137 usually means the system killed the process, often because it ran out of memory, which points to a very different fix than a missing input file.
4Who was told, and how fast?
Alerts that land in a shared channel nobody owns get read late. Route each alert to a named owner or on-call rotation, by pipeline and by severity. A failed clinical batch with a turnaround commitment is a different event from a failed research run.
5What did this sample cost to process?
Cloud bills arrive monthly and by service. HPC usage shows up as core-hours by user. Neither tells you that a 30x whole genome cost a certain amount through alignment, calling, and annotation, or that reruns doubled the cost of last Tuesday's batch. Cost per sample is the number finance, lab leadership, and pricing decisions actually need.
Where Workflow Language Keep Run Evidence Today
Every major workflow engine already writes useful run data. The problem is that each one writes it in its own place and format, and none of them answers all five questions on its own.
| Engine | Native run evidence | What it doesn't give you on its own |
|---|---|---|
| Nextflow | Trace file per run (task hash, status, exit, attempt, container, work directory, realtime, peak_rss), plus an HTML execution report and timeline | Workflow commit and reference checksums in one record, cross-run cost per sample, alert routing |
| WDL on Cromwell | Workflow and call-level metadata, including execution status, the image actually used in dockerImageUsed, and call caching details when caching is on | A readable per-sample view, cost, alerting, and a link back to the approved release |
| Snakemake | Per-job logs, the benchmark directive for wall-clock time and memory per job, and an HTML report from --report | Central failure alerts, cost per sample, and a run record that spans many invocations |
| CWL | Depends on the runner in use; provenance options vary by implementation | A consistent format if the lab also runs other engines |
A lab running one engine on one cluster can get a long way with native reports and good discipline. Many clinical genomics teams run more than one engine, often across a mix of HPC and cloud, with pipelines inherited from different groups. That's where a cross-engine observability layer earns its keep: one format, one alert path, and one place to ask the five questions.
What Pipeline Monitoring Can Prove and What It Can't
This is where a lot of vendor material blurs the line. Good run data is strong evidence of what happened. It's not evidence that what happened was analytically correct.
Run data can show
- Which release, containers, parameters, and reference assets ran for a given sample, if the manifest captures them.
- Which tasks executed fresh and which were reused from cache.
- When and why a task failed, how many attempts it took, and who was alerted.
- Resource use and cost per sample over time.
- Which production runs used a release, so affected runs can be listed quickly if a defect turns up.
Run data can't show
- That the pipeline meets its performance specifications. Under 42 CFR 493.1253, validation establishes accuracy, precision, analytical sensitivity and analytical specificity, not monitoring.
- That a change was approved. Under 42 CFR 493.1251(d), the laboratory director approves, signs, and dates procedure changes before use. A green dashboard isn't a signature.
- That variant calls are right. A run can finish cleanly and still produce a wrong call if a reference, filter, or annotation source changed underneath it.
One more boundary worth getting right: 21 CFR Part 11 applies to electronic records kept under FDA record requirements or submitted to FDA (21 CFR 11.1). A pipeline used only for a CLIA laboratory-developed test is governed by CLIA, unless its records also support FDA-regulated work such as a clinical trial.
Observability really pays off in recovery. When an error in reported results is found, 42 CFR 493.1291(k) requires the lab to notify the ordering provider promptly, issue corrected reports, and keep both versions. That depends on listing every affected run fast.
With a release ID on every run record, it's a query. Without one, it's a week of archaeology.
How to Track Cost Per Sample in a Genomics Pipeline Across HPC and Cloud
Cost per sample sounds like a finance question. In practice it's an engineering one, because the data needed to calculate it is scattered across the scheduler, the cloud bill, and the workflow engine.
-
Tag work at the source. On cloud executors, Nextflow's
resourceLabelsdirective attaches labels to jobs at runtime. In AWS, a tag key appears for activation as a cost allocation tag only after it has been applied to resources, and AWS notes it can take up to 24 hours for the key to appear and up to another 24 hours to activate (AWS Billing documentation). Label by pipeline, release, batch, and sample where you can. -
Pull task-level usage on HPC. On SLURM or LSF clusters there's no bill per job. Use the scheduler's accounting data (on SLURM,
sacctreports elapsed time, CPU time, and peak memory per job) together with the engine's own trace or benchmark output, and convert core-hours and memory-hours to cost with an internal rate your finance team agrees on. - Attribute shared work fairly. Some tasks serve many samples at once, such as joint genotyping or a batch-level QC step. Split their cost across the samples in the batch, and write down the rule so the number means the same thing every month.
- Count the failures. Include failed attempts, reruns, and resumed runs. A pipeline that looks cheap per successful sample can be expensive per delivered sample once you count retries.
- Add storage and egress. BAM and CRAM retention, work directories nobody cleaned up, and cross-region data transfer all belong in the number. They're often the biggest surprise.
Doing this by hand across two schedulers and a cloud account is a spreadsheet job that rarely survives the second month. StrixFlow brings HPC and cloud spend into one cost-per-sample view, tied to runs and samples rather than the monthly bill (StrixFlow).
Pipeline Observability for Regulated Labs in the UK and EU
The engineering problem is the same in London, Amsterdam, or Boston. The rules around it differ.
ISO 15189:2022 expects a lab to control the data and information systems used to produce and report results (clause 7.6), and a bioinformatics pipeline is one of those systems. In the EU, IVDR Article 5(5) allows in-house devices without CE marking only under set conditions, including compliance with EN ISO 15189 or applicable national provisions. Great Britain sits outside IVDR; labs there work under the UK Medical Devices Regulations 2002, and UKAS accredits them against ISO 15189:2022.
For observability, two things matter most in Europe. Run records should be just as traceable as a US lab's, because an assessor will ask the same five questions. Run data should stay where the genomic data lives, which is why self-hosted deployment matters more under GDPR than a hosted dashboard that copies logs to a third party.
Build, Rely on Engine Reports, or Adopt StrixFlow: Choosing Bioinformatics Pipeline Monitoring Software
There are three honest paths, and the right one depends on how many engines you run and how much engineering time you can spare.
Rely on native engine reports
Works when the lab runs one engine, on one platform, with a small number of pipelines. It costs nothing extra. It struggles once cost per sample, cross-engine views, or alert routing come into play, because those sit outside what any single engine records.
Build your own observability layer
Gives full control and fits unusual environments. Expect to build log collection for each engine, a run-record schema, alert routing, cost attribution, retention, and access control, and then to maintain all of it as engines change. For most bioinformatics teams, that's months of work that pulls people away from the science.
Adopt StrixFlow
StrixFlow is a production-tested component that a lab customizes with NonStop and then owns. It connects to existing Nextflow DSL2, Snakemake, WDL, and CWL pipelines without a rewrite or a change to orchestration. It adds real-time failure alerts with step-level diagnosis, Slack and Teams routing, cost-per-sample tracking across HPC and cloud, and reproducibility manifests that record tool versions, container digests, and parameters per run. It deploys self-hosted, inside the lab's environment.
On the StrixFlow page, NonStop estimates manual monitoring at 4 to 8 hours a day spent debugging failed runs and around 60% lower pipeline operations cost with StrixFlow. Both are NonStop estimates and vary by engagement scope (StrixFlow).
StrixFlow shortens the build. It doesn't replace bioinformatics work, and it doesn't validate or approve a pipeline. When a lab's needs go beyond its scope, such as integrating approval with the lab's quality system or building a custom run-record schema, NonStop builds that as custom engineering through its digital product development practice.
How NonStop Builds Pipeline Observability for Clinical Genomics Labs
- Clinical bioinformatics pipeline engineering services in Nextflow DSL2, WDL on Cromwell, and Snakemake, with a full audit trail per pipeline run: tool versions, parameter sets, reference build, input checksums, and output manifests (bioinformatics pipeline engineering).
- Benchmarked against NIST Genome in a Bottle and SEQC2 reference datasets, with validation documentation and a structured handover in every engagement.
- Execution across AWS Batch, GCP, Azure Batch, Kubernetes, and on-premises SLURM and LSF clusters, which is where cost per sample usually gets complicated.
- StrixFlow for observability over pipelines already in production, self-hosted and owned by the lab (StrixFlow).
- Forward-deployed engineers who work inside the lab's environment through production, with full IP ownership staying with the client. On-site deployment is available for US-based clients (forward-deployed engineers).
The laboratory director and quality leadership own assay claims, validation scope, and approval. NonStop builds the pipelines and the run evidence that make those decisions faster to reach and easier to defend.
Bioinformatics Pipeline Observability FAQs
What is bioinformatics pipeline observability?
Bioinformatics pipeline observability is the practice of capturing run status, task-level failure detail, execution evidence, and cost for every pipeline run, across engines such as Nextflow, WDL on Cromwell, Snakemake, and CWL. It helps teams find and fix failures faster and trace any result back to the run that produced it.
Is pipeline monitoring the same as pipeline validation?
No. Monitoring and observability show what ran and how it behaved. Validation establishes that the pipeline meets its performance specifications under 42 CFR 493.1253, and the laboratory director approves changes before use under 42 CFR 493.1251(d). Run data supports both, but replaces neither.
How do I monitor Nextflow pipelines in production?
Turn on the trace file and execution report for every run, record the workflow commit, container digests, and parameters, and send task failures to a named owner with the process name, exit code, and attempt number. For several engines or a mix of HPC and cloud, a cross-engine observability layer keeps this in one place.
How do you calculate cost per sample for a genomics pipeline?
Tag cloud jobs by pipeline, batch, and sample, pull CPU time and memory per job from the HPC scheduler, split shared batch-level tasks across the samples they serve, and include failed attempts, reruns, storage, and data transfer. Cost per delivered sample is usually higher than cost per successful run.
Can pipeline failure alerts go to Slack or Teams in a HIPAA environment?
They can, if alerts include only what an engineer needs to act, such as the run ID, process name, internal sample index, and exit code, and link back to the full run record in the lab's environment. Patient names, record numbers, and dates of birth should stay out of alert text.
Does StrixFlow require rewriting Nextflow, Snakemake, WDL, or CWL pipelines?
No. StrixFlow connects to existing Nextflow DSL2, Snakemake, WDL, and CWL pipelines without rewriting them or changing the lab's orchestration.

Mahendra works as a Sr. Bioinformatics Engineer at NonStop. He builds genomic pipelines, agentic AI systems, and AI-powered products for genomics and life sciences. He has 10+ years of experience in bioinformatics and genomics software. He studied Biotechnology at Mumbai University.

