Method / tool

StarDist

Deep learning segmenter that models each nucleus as a star-convex polygon (2D) or polyhedron (3D), producing non-overlapping instance masks without post-processing tuning; the first-line choice for round, densely packed nuclei in fluorescence microscopy.

The problem — Nucleus segmentation in dense fluorescence fields fails silently: classical threshold-and-watershed pipelines merge touching nuclei, and generic instance segmenters tuned on natural images inherit shape priors that do not match biology. The result is a systematic under-count that propagates into every downstream per-object measurement without triggering an obvious error.

What it is / how it works — StarDist encodes each nucleus as a star-convex polygon: a shape described by radial distances from a predicted centre to the boundary, sampled at fixed angular intervals. The network predicts both an object-probability map and a dense field of these distance vectors; non-maximum suppression on the candidate polygons resolves overlaps automatically. The geometric prior is an excellent fit for nuclei and round cells — the vast majority of nuclear morphologies satisfy the star-convex constraint — which is why pre-trained models (versatile_fluo_2d, versatile_2d_he) work reliably out of the box on most fluorescence and H&E datasets without retraining. A 3D extension (star-convex polyhedra) handles volumetric data from confocal and light-sheet acquisitions, with built-in tiling for large volumes. GPU is optional for 2D inference; strongly recommended for 3D.

StarDist sits at the segment stage and feeds masks directly into per-object evaluation and downstream feature extraction. In Cell Painting workflows it is listed alongside Cellpose and CellProfiler as a first-line nucleus segmenter; in digital pathology it handles nuclei detection in H&E tissue sections; in 3D organoid assays it segments spheroids as star-convex polyhedra.

Where it breaks — The star-convex prior is a hard geometric constraint, not a soft regulariser.

  • Elongated or concave shapes — neurons, muscle fibres, rod bacteria — violate the prior; Cellpose is the better choice there.
  • Cytoplasm segmentation — StarDist has no native cell-expansion step; pairing it with a ring-expansion heuristic or Cellpose cytoplasm model is required for whole-cell masks.
  • Highly heterogeneous sizes — very large and very small objects in the same field challenge the fixed angular sampling; the dense-cell regime with extreme size variation may need manual NMS threshold tuning.
  • Very large 3D volumes — enable the built-in tiling mode to avoid out-of-memory failures; naive inference on full volumes will exhaust GPU memory.
The star-convex constraint is the source of both StarDist's accuracy and its failure modes. Check it holds on your sample type before committing to the tool — a quick visual QC of predicted polygons against raw images catches violations before they corrupt the feature table. See [Evaluating Segmentation Models in Real Microscopy Workflows](/lab-notes/evaluating-segmentation-models-in-real-microscopy-workflows) for instance-level metrics that expose merge and split errors a pixel-Dice score will miss.

References

Appears in these notes

  • Choosing a Bioimage Analysis Pipeline — Off-the-Shelf vs CustomMost microscopy questions are answered by composing existing, validated tools — and a custom pipeline is justified only when off-the-shelf options fail on a specific, identifiable axis. This note gives the decision criteria and the signals that you have genuinely crossed into custom territory.
  • Evaluating Segmentation Models in Real Microscopy WorkflowsA segmenter's benchmark Dice rarely predicts its behavior in a real workflow. Instance counting, merge/split errors, and cross-instrument generalization are the quantities that decide whether the downstream readout is correct — and standard semantic overlap scores are blind to all three.
  • Model Zoos for Bioimage Analysis — From a Zoo Model to a Validated WorkflowThe BioImage Model Zoo and community packaging tools make pre-trained models shareable, runnable, and reproducible across tools — solving distribution and provenance. They do not solve fitness: a downloaded model is a component, and the path from a zoo entry to a trustworthy readout still runs through validation.
  • The Bioimage Analysis Ecosystem — Where Each Tool FitsBioimage analysis is not one tool but a landscape organized along two axes — interactive vs scriptable, and generalist vs specialist. Fiji, napari, CellProfiler, QuPath, ilastik, and the deep-learning segmenters each occupy a different cell of that grid, and choosing well means knowing which cell your problem lives in.
  • Cellpose vs CellProfiler for Nuclei SegmentationWhen a generalist deep model beats a tuned classical pipeline for nuclei — and when it does not.

← Back to the constellation