Method / tool

Cellpose

Generalist deep-learning model for cell and nucleus segmentation across diverse microscopy modalities, using a learned flow-field representation to produce instance masks without per-dataset retraining.

The problem — Most cell segmenters are tuned for one image type. Move to a new instrument, magnification, or stain protocol and the model transfers poorly, silently degrading the masks that every downstream feature depends on.

What it is / how it works — Cellpose frames segmentation as a flow-field prediction problem. Rather than predicting a binary foreground map, the model (a modified U-Net with residual blocks and global style vectors) predicts per-pixel vector flows that point toward each object's center. At inference, pixels "follow the flows" by gradient ascent; those that converge to the same point are grouped into an instance mask. This representation handles arbitrary cell shapes and dense, touching objects more robustly than distance-transform or watershed approaches. The original model was trained on more than 70,000 segmented objects drawn from highly varied image types, giving it strong out-of-the-box generalization. Cellpose 2.0 extended this with a model zoo and a human-in-the-loop GUI: roughly 100–200 corrected annotations in one to two hours are enough to build a specialist model that approaches the accuracy of a fully supervised effort.

Cellpose appears in the segment stage across Cell Painting, spatial omics, live-cell, organoids, light-sheet, and label-free modalities — the widest tool footprint of any segmenter in the platform.

Where it breaks — Generalization has a ceiling. On simple, uniform nuclei (e.g., sparse DAPI on a clean background) the flow-field overhead buys little and a tuned classical pipeline can match it with less compute. More importantly, the generalist model degrades on images that differ substantially from its training distribution — a novel stain, a confocal with unusual optics, or densely packed 3D volumes where under-segmentation (merges) is the dominant failure mode. Those merge errors propagate directly into per-cell feature distributions and corrupt profile reproducibility, which is why Evaluating Segmentation Models in Real Microscopy Workflows insists on instance-level evaluation rather than pixel Dice: a merge barely moves overlap scores but doubles a cell's apparent morphology.

For any image type outside the training distribution, the correct operating stance is to fine-tune on 100–200 representative annotations before committing masks to a production pipeline — not to trust the benchmark number.

Generalist performance is not a substitute for domain validation. Evaluate Cellpose instance-wise on your own instrument and stain protocol; measure merge and split rates explicitly before accepting masks as production-ready.

References

Appears in these notes

  • Cell Painting Pipelines — From Images to Phenotypic ProfilesA Cell Painting profile is the output of a long pipeline — illumination correction, segmentation, feature extraction, batch correction, then a reproducibility readout — and the number that matters (percent-replicating / mAP) lives only at the end. Every stage upstream can silently degrade it.
  • Choosing a Bioimage Analysis Pipeline — Off-the-Shelf vs CustomMost microscopy questions are answered by composing existing, validated tools — and a custom pipeline is justified only when off-the-shelf options fail on a specific, identifiable axis. This note gives the decision criteria and the signals that you have genuinely crossed into custom territory.
  • Evaluating Segmentation Models in Real Microscopy WorkflowsA segmenter's benchmark Dice rarely predicts its behavior in a real workflow. Instance counting, merge/split errors, and cross-instrument generalization are the quantities that decide whether the downstream readout is correct — and standard semantic overlap scores are blind to all three.
  • Model Zoos for Bioimage Analysis — From a Zoo Model to a Validated WorkflowThe BioImage Model Zoo and community packaging tools make pre-trained models shareable, runnable, and reproducible across tools — solving distribution and provenance. They do not solve fitness: a downloaded model is a component, and the path from a zoo entry to a trustworthy readout still runs through validation.
  • The Challenge of Rare Events and MutantsWhen the phenotype you care about appears in 10 cells out of 300,000, accuracy is meaningless, validation has almost no positives to learn from, and false discovery dominates. Rare-event detection is a class-imbalance and ground-truth-scarcity problem — best answered by adaptive acquisition that goes looking for the positives.
  • From Research Script to Production PipelineA notebook that worked once is not a pipeline, and a community model that scored well on its paper's data is not a validated component. Turning either into something deployable is the engineering work of pinning, wrapping, gating, and validating — most of which the original artifact deliberately skipped.
  • The Bioimage Analysis Ecosystem — Where Each Tool FitsBioimage analysis is not one tool but a landscape organized along two axes — interactive vs scriptable, and generalist vs specialist. Fiji, napari, CellProfiler, QuPath, ilastik, and the deep-learning segmenters each occupy a different cell of that grid, and choosing well means knowing which cell your problem lives in.
  • Validation Without a Ground TruthValidation asks whether a microscopy pipeline's outputs are biologically true and fit for purpose — but biology rarely supplies a clean answer key. The discipline is choosing metrics that reflect the question, manufacturing ground truth honestly, and conditioning performance on the experiment.
  • Measure Where It MattersAdaptive, uncertainty-driven acquisition treats the microscope as part of the model — spending photons and time only where the image is uncertain or the biology is happening, instead of scanning everything uniformly.
  • Cellpose vs CellProfiler for Nuclei SegmentationWhen a generalist deep model beats a tuned classical pipeline for nuclei — and when it does not.

← Back to the constellation