There's a failure mode every bioinformatician has hit at least once. You're running a WGS cohort through GATK HaplotypeCaller. Halfway through, the cluster runs out of memory, a node dies, or someone accidentally kills the job. You fix the issue and rerun. The pipeline "completes." Variant calls look off. You spend two days debugging before realizing the BAM you were calling variants on was a concatenation of a partial previous run and the new one.
That's an idempotency failure. And it's almost never discussed in genomics pipeline engineering conversations, even though it's responsible for a significant fraction of "mysterious" pipeline bugs.
In mathematics, a function is idempotent if applying it twice gives the same result as applying it once. In genomics pipeline engineering, the practical definition is simpler: running a step twice with the same inputs produces the same output, and doesn't corrupt anything.
A function f is idempotent if,
f(f(x)) = f(x)
This sounds obvious. It isn't. Most genomics pipelines we build are idempotent in the happy path, when everything runs clean from scratch. They break idempotency exactly when you need it most: during reruns after partial failures.
The three canonical failure modes are:
Each of these is insidious for the same reason: the pipeline doesn't error. It continues. You find out later, sometimes much later, sometimes after clinical reporting.
GATK tools have their own opinions about output handling that interact with naive pipeline logic.
The most useful way to think about idempotency in practice is to treat each pipeline step as a clean room operation; it should behave identically whether the output directory is empty or contains files from a previous run. This means three things:
Nextflow's -resume flag is powerful and it's one of the reasons Nextflow is so widely adopted in genomics. But it's frequently misunderstood as an idempotency guarantee. It isn't.
-resume reuses cached task outputs based on a hash of the task's inputs and script. What it does not account for is whether the output files were modified after caching, whether reference files were updated in place under the same path, or whether the work directory was partially cleaned by a storage quota enforcement job.
-resume is a performance optimization. It lets you skip work you've already done. Idempotency is a correctness guarantee. It ensures that work you redo produces the same result. These are different things, and conflating them is how teams end up trusting cached outputs that are quietly wrong.
The practical implication
-resume is safe when you can verify your work directories are intact and your inputs haven't changed. When you can't verify that, mid-cohort failure after a storage incident, for example, rerunning from scratch with clean output directories is often the more defensible choice than resuming.
In a research setting, idempotency failures are annoying and waste time. In a clinical setting, they carry a different kind of risk.
Consider a rare disease diagnostic pipeline running trio analysis. A partial failure during joint genotyping creates a corrupted joint VCF. The pipeline resumes, ACMG classification runs on the corrupt output, and a variant that should have been flagged as Likely Pathogenic is either missing or miscategorized. The report goes out. Nothing in the pipeline log indicates a problem.
This is not a hypothetical failure mode, it's a realistic consequence of non-idempotent pipelines operating under time pressure. The fix isn't to run more slowly or more carefully. It's to build pipelines where this class of failure is structurally impossible: where every step either completes cleanly or fails loudly, and where partial outputs never propagate downstream.
Idempotent pipelines change the failure recovery calculus entirely. When a step fails mid-cohort, the correct action becomes simple: fix the issue, rerun from that step. With genuine idempotency, you're confident the outputs are clean without auditing individual files. Without it, every rerun requires a manual review of what completed, what's partial, and what's safe to carry forward.
The 2-hour versus 2-day investigation gap isn't hypothetical. It compounds across a cohort, 50 samples, 8 pipeline steps each, a few partial failures per month. That's hundreds of hours of debugging time per year that largely disappear when your pipeline is built with idempotency as a first-class design requirement.
The patterns that accomplish this, atomic writes, explicit completion signals, output validation at step boundaries, treating GenomicsDBImport workspaces as disposable, aren't complex to implement. They're just rarely prioritized until after the first major incident. Build them in before that incident happens.
