Lab Note · updated 2026-06-20

A Model Isn't the System

A trained model is a small fraction of a working measurement pipeline. Most of the code, and almost all of the risk, lives in the data, glue, configuration, and validation around it — which is why a model that scores well on a benchmark is not a result you can ship.

← Back to constellation

The thesis — When a microscopy team says "we have a model," they usually mean a network that segments cells, embeds a slide, or denoises a stack and scores well on a held-out set. That model is real, and it is a small fraction of what it takes to produce a number a biologist can act on. The system is everything around it: ingestion and chunking, metadata and provenance capture, QC gates, preprocessing, the model, post-processing, batch correction, the readout, and a report — each a place the result can silently go wrong. Sculley et al. made the general case in Hidden Technical Debt in ML Systems: the model code is a thin box in a much larger diagram, and most of a real ML system is glue, configuration, and data plumbing — where data dependencies are more dangerous than code dependencies because they change without a code diff to blame.

Why it matters — The cost of forgetting this is CACE — changing anything changes everything. Swap a scanner, bump a stain lot, repin a deconvolution library, retrain on a new plate, and the model behaves differently with nothing in the model to point at. Two failure surfaces compound it. The systems surface is Sculley's: entanglement, pipeline jungles, undeclared consumers, configuration debt. The methodology surface is Varoquaux & Cheplygina: a benchmark gain can be smaller than evaluation noise, so the model that "wins" may have won nothing real — and that gap is invisible if you only look at the model. A measurement is only as trustworthy as the weakest stage between the photons and the readout, and the model is rarely that stage.

How Fovea applies it — We build and instrument the whole pipeline, not the model in the middle. Quality is judged at the readout and the decision, not at the pixels — image quality is not one number. Every run is reproducible from a pinned environment and a recorded manifest — a pipeline you can re-run — and validated with metrics chosen for the biological question, not the convenient score (Metrics Reloaded), because validation without a ground truth is the normal case in microscopy. Acquisition itself is part of the system — measure where it matters — and verification and validation are tracked as two separate ledgers. The same worldview is why foundation models are not pipelines and why turning a notebook into a deployable workflow is its own discipline — from research script to production. The deliverable is never the model. It is the verified output.

A model that scores well is a component, not a result. If you can't say how the data got in, how the run reproduces, and what the output was validated against, you have a demo — not a measurement system.

References

Sculley et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS.
Varoquaux & Cheplygina (2022). Machine Learning for Medical Imaging: Methodological Failures and Recommendations for the Future. npj Digital Medicine.
Maier-Hein, Reinke et al. (2024). Metrics Reloaded: Recommendations for Image Analysis Validation. Nature Methods.

References

Related notes

Choosing a Bioimage Analysis Pipeline — Off-the-Shelf vs Custom
Most microscopy questions are answered by composing existing, validated tools — and a custom pipeline is justified only when off-the-shelf options fail on a specific, identifiable axis. This note gives the decision criteria and the signals that you have genuinely crossed into custom territory.
Foundation Models Are Not Pipelines
A foundation model is a powerful feature extractor, not a deployable measurement pipeline. It still needs ingestion, QC, preprocessing, confounder correction, validation, and provenance around it — and a frozen encoder makes the surrounding system more important, not less.
Image Quality Is Not One Number
Fovea treats microscopy quality as four nested levels — image-space, run/sample, readout, and decision — because a metric at one level says nothing about the levels above it. Full-reference scalars like SSIM and PSNR measure only the first, yet teams report as if they measured the third.
Why Interoperability Matters in Microscopy
A microscopy pipeline is always several tools in sequence, so the seams between them — file formats, metadata, coordinate conventions — decide whether the pipeline composes or quietly corrupts. Open standards like OME-Zarr/NGFF turn hand-offs from lossy conversions into clean interfaces, and avoid lock-in to any one vendor or tool.
Model Zoos for Bioimage Analysis — From a Zoo Model to a Validated Workflow
The BioImage Model Zoo and community packaging tools make pre-trained models shareable, runnable, and reproducible across tools — solving distribution and provenance. They do not solve fitness: a downloaded model is a component, and the path from a zoo entry to a trustworthy readout still runs through validation.
Verification — A Pipeline You Can Re-Run
Verification is the engineering discipline of making a microscopy pipeline deterministic, version-pinned, and re-executable from a provenance record. Without it, every result is a one-off, and "we changed nothing" is unprovable.
From Research Script to Production Pipeline
A notebook that worked once is not a pipeline, and a community model that scored well on its paper's data is not a validated component. Turning either into something deployable is the engineering work of pinning, wrapping, gating, and validating — most of which the original artifact deliberately skipped.
The Bioimage Analysis Ecosystem — Where Each Tool Fits
Bioimage analysis is not one tool but a landscape organized along two axes — interactive vs scriptable, and generalist vs specialist. Fiji, napari, CellProfiler, QuPath, ilastik, and the deep-learning segmenters each occupy a different cell of that grid, and choosing well means knowing which cell your problem lives in.
Validation Without a Ground Truth
Validation asks whether a microscopy pipeline's outputs are biologically true and fit for purpose — but biology rarely supplies a clean answer key. The discipline is choosing metrics that reflect the question, manufacturing ground truth honestly, and conditioning performance on the experiment.
Verification and Validation Are Two Different Questions
Verification asks whether the pipeline was built right — deterministic, reproducible, correct to spec. Validation asks whether it is the right pipeline — outputs that are biologically true and fit for purpose. Microscopy makes both hard, and conflating them is how silently wrong results get shipped.
Why Fovea Lab Uses a Knowledge Graph
A recommendation is only as trustworthy as what it knows a tool can't do. Fovea represents tools, models, papers, and constraints as a graph with explicit negative edges — unsupported modalities, known failure modes, hard requirements — so the system can say "no" with a reason, not just rank positive matches.
Measure Where It Matters
Adaptive, uncertainty-driven acquisition treats the microscope as part of the model — spending photons and time only where the image is uncertain or the biology is happening, instead of scanning everything uniformly.
Why Microscopy Needs Verified Outputs
A measurement you cannot reproduce is not a measurement. Why verification is a first-class pipeline stage, not an afterthought.

Local graph