Pipeline stage

Ingest

Convert raw, proprietary microscopy files into a harmonized, cloud-native OME-Zarr store — preserving dimension order, pixel calibration, and channel identities so every downstream stage reads the same layout regardless of instrument or modality.

The problem — Raw acquisition files arrive in dozens of proprietary formats: ND2, CZI, SVS, LIF, MRXS, raw camera streams. Without a conversion step that preserves the full data model — dimension order, pixel size, channel identities, z-spacing, time index — those files are pixels without context. A pipeline that silently misreads a channel or drops a z-dimension will produce numerically plausible outputs that are biologically wrong.

What it is / how it works — Ingest translates proprietary formats into a canonical, analysis-ready representation. Bio-Formats reads over 150 vendor formats and exposes a common OME data model; the output target is OME-NGFF / OME-Zarr — a chunked, multiscale pyramid stored in cloud-compatible object storage. Chunking is not cosmetic: a light-sheet volume or whole-slide image that cannot be lazily sliced cannot be analyzed at all. For HCS plates (Cell Painting), ingest also encodes the plate / well / site / field hierarchy so every downstream QC and profiling query can index by well without a secondary join.

Where it breaks — Ingest failures are silent. A format reader that mis-parses channel order delivers a five-channel Cell Painting plate with MitoTracker and DAPI swapped — an error that propagates through segmentation and profiling with no pipeline exception. Chunking mismatches (wrong chunk shape for the access pattern) translate into I/O thrash that looks like a compute bottleneck. The fix is a validation step immediately after ingest: check OME metadata against the acquisition manifest, assert dimension labels, and fail loudly before any downstream stage runs. This upstream contract is what makes Data Standards and Scalable Storage enforceable and feeds a trustworthy run manifest.

A format conversion that drops or reorders metadata is not an ingest — it is data corruption with a progress bar. Validate dimension labels and channel identities before touching a single downstream stage.

References

Appears in these notes

  • Data Standards and Scalable StorageThe OME data model and OME-NGFF/OME-Zarr exist because a microscopy image without a shared data model and a chunked, cloud-native layout is neither interpretable nor analyzable at scale. Format is not plumbing — it decides whether a petabyte dataset can be opened at all.
  • Why Interoperability Matters in MicroscopyA microscopy pipeline is always several tools in sequence, so the seams between them — file formats, metadata, coordinate conventions — decide whether the pipeline composes or quietly corrupts. Open standards like OME-Zarr/NGFF turn hand-offs from lossy conversions into clean interfaces, and avoid lock-in to any one vendor or tool.
  • Light-Sheet Microscopy — The Pipeline Problem Behind the Beautiful VolumeThe rendered light-sheet volume is the easy part. Behind it is a petabyte-scale systems problem — ingest and chunking, deconvolution and destriping, stitching and registration, then tracking — and every stage is where the science silently degrades, not the final render.
  • Measure Where It MattersAdaptive, uncertainty-driven acquisition treats the microscope as part of the model — spending photons and time only where the image is uncertain or the biology is happening, instead of scanning everything uniformly.

← Back to the constellation