Verification — A Pipeline You Can Re-Run

The problem — Most microscopy analysis is run once, by hand, on one workstation, and the result is trusted because it looks right. Verification is the opposite stance: a result counts only if the same inputs, re-run, reproduce the same outputs — and you can show why each output is what it is. Without that, "we changed nothing between batches" is an article of faith, not a fact, and the first time a number moves you cannot tell whether the biology changed or the software did.

What it is / how it works — A verified pipeline pins and records four things, so any run is re-executable from its provenance:

• Code and environment — exact versions of every package, container, and model weight, not "latest." The danger is silent drift: a dependency bump that quietly changes a result.
• Parameters and configuration — every threshold, seed, and tile size captured as data, because configuration debt is where reproducibility quietly dies.
• Determinism — controlled random seeds and, where GPU non-determinism matters, awareness that the same model can produce different masks across runs unless pinned.
• Provenance and acquisition metadata — the run manifest: instrument, acquisition settings, and processing lineage, distinguishing provenance metadata (how it was made) from quality metadata (how good it is), aligned to OME/NGFF and QUAREP-LiMi so a run is interpretable by someone who wasn't there.

The unifying idea from Sculley et al. is that data dependencies are more dangerous than code dependencies, and most of a system is not the model — it is glue, configuration, and undeclared consumers. Verification is the engineering practice that keeps that surface under control: declarative workflows (e.g. Snakemake), pinned environments, regression tests on fixed inputs, and a manifest emitted with every result.

Where it breaks — Verification fails quietly because nothing errors — the pipeline runs, just differently. In Cell Painting, an unpinned illumination-correction or feature-extraction version shifts profiles across plates and is misread as a batch effect. In digital pathology, a scanner firmware or stain-normalization change moves slide-level scores with no code diff to blame. In light-sheet, a deconvolution library update alters segmentation in deep volumes that no one re-checks. In spatial omics, a registration parameter that lives in someone's notebook — not in version control — makes the transcript-to-cell assignment unreproducible. The through-line is CACE: changing anything changes everything, so the only defense is to make "anything" explicit and recorded. Verification is what lets validation mean something — you cannot trust a result you cannot reproduce.

References

• Sculley et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS.
• Huisman et al. (2019). A Perspective on Microscopy Metadata: Data Provenance and Quality Control.