Pipeline stage

Metadata

Capture and validate the structured provenance record — plate map, perturbations, acquisition geometry, controls — that makes a folder of images into a dataset you can reproduce, compare across runs, and trust as evidence.

The problem — A dataset without a validated manifest is a dataset that exists only in the memory of the person who ran it. Which compound was in well B14? Which scanner produced plate 3? Were the DMSO controls distributed uniformly? Without a structured, machine-readable record, these questions become reconstructions — and reconstructions are not provenance.

What it is / how it works — The metadata stage writes the run manifest: the structured record of what was acquired and how. Huisman et al. distinguish two complementary layers — provenance metadata (MPM: the instrument and acquisition story) and quality metadata (MQM: calibration and error). Both are captured here and encoded in OME-NGFF conventions so they travel with the pixel store rather than in a lab notebook. For HCS plates this means compound / dose / timepoint per well, control layout, and scanner/objective/exposure settings. For light-sheet it means acquisition geometry, view angles, and time-lapse indexing. For spatial omics it means marker panel, cycle/round index, and ROI definitions. The manifest is a build artifact — generated at ingest, validated against a schema, and versioned — not a post-hoc annotation.

Where it breaks — Metadata captured once, manually, at data deposition time is metadata that drifts. A plate map entered by hand after a run confuses wells; a missing exposure field leaves QC unable to flag saturation; a swapped scanner identifier makes multi-site batch correction incorrect by construction. The discipline is to make manifest creation automatic and validation mandatory before any downstream stage runs. A manifest that cannot answer "which well, which control, which scanner" is decoration, not provenance — and it makes Why Microscopy Needs Verified Outputs an aspiration rather than a contract. See also Metadata & Provenance: The Run Manifest for the full authoring pattern.

If provenance is not captured at ingest, it will be reconstructed from memory later — and reconstruction is not provenance.

References

Appears in these notes

  • Metadata & Provenance: The Run ManifestA run manifest captures the provenance and acquisition metadata that turn a folder of images into a dataset you can trust, reproduce, and compare across runs.
  • Data Standards and Scalable StorageThe OME data model and OME-NGFF/OME-Zarr exist because a microscopy image without a shared data model and a chunked, cloud-native layout is neither interpretable nor analyzable at scale. Format is not plumbing — it decides whether a petabyte dataset can be opened at all.
  • Why Interoperability Matters in MicroscopyA microscopy pipeline is always several tools in sequence, so the seams between them — file formats, metadata, coordinate conventions — decide whether the pipeline composes or quietly corrupts. Open standards like OME-Zarr/NGFF turn hand-offs from lossy conversions into clean interfaces, and avoid lock-in to any one vendor or tool.
  • Verification — A Pipeline You Can Re-RunVerification is the engineering discipline of making a microscopy pipeline deterministic, version-pinned, and re-executable from a provenance record. Without it, every result is a one-off, and "we changed nothing" is unprovable.
  • How to Prepare Microscopy Data for AIMost of the work of applying AI to microscopy is data preparation — formatting, normalization, tiling, and label hygiene — and the single most consequential decision is a split that respects the non-independence of microscopy data. Get the split wrong and every downstream metric is leaked, not earned.
  • Every Output Should Know Where It Came FromProvenance is a property of the artifact, not the pipeline — every mask, profile, and count matrix should carry the lineage that produced it. Where verification asks whether a run reproduces, provenance asks whether a single output, found alone, can still explain itself.
  • How to Make Imaging Data Ready for ReanalysisReanalysis-ready means a third party with no contact with the original lab can re-run, re-segment, or re-interpret the data correctly — which requires FAIR principles, complete metadata, and deposition in a public archive. The bar is reuse by a stranger, not retrieval by the author.

← Back to the constellation