The problem — Evaluating a segmentation algorithm requires a reference. In clinical and biological imaging, that reference is usually produced by human annotators — and human annotators disagree. Naively picking one annotator's output as "ground truth" bakes in that individual's errors and biases; simple majority voting ignores the fact that some raters are more reliable than others. Without a principled way to fuse multiple raters, there is no honest denominator against which to measure algorithmic performance — and Ground Truth Is a Design Problem shows why this matters for every downstream metric.
What it is / how it works — STAPLE, introduced by Warfield, Zou, and Wells (IEEE Trans. Med. Imaging, 2004), models each segmentation as a noisy observation of a hidden true segmentation. Each rater is characterized by a confusion matrix (sensitivity and specificity per label), and the algorithm uses Expectation-Maximization to jointly estimate the hidden consensus segmentation and each rater's performance level. The E-step computes the probability that each pixel belongs to each class given the current rater models; the M-step updates the rater parameters to maximize the likelihood of the observed segmentations. The process iterates to convergence, yielding both a soft consensus mask (the probabilistic ground truth) and a calibrated per-rater reliability estimate.
STAPLE was originally binary (foreground/background) but has since been extended to multi-class settings and integrated into frameworks such as SimpleITK. It is widely used in radiology and pathology annotation pipelines to produce reference standards for algorithm training and validation.
Where it breaks — STAPLE's consensus is only as good as its rater pool. If all annotators share the same systematic bias — a common interpretation of an ambiguous boundary or a shared tendency to under-segment dense clusters — STAPLE will reproduce that bias as the "truth." The model also assumes that rater errors are spatially stationary and class-independent; in practice, annotators tend to be less reliable near object boundaries and more reliable in clear background, so the confusion-matrix assumption is violated exactly where the hard segmentation decisions live. Downstream metrics computed against a STAPLE reference inherit all of these limitations.
The Fovea stance: STAPLE is the right starting point whenever multiple raters annotate the same image set, but the consensus mask should be treated as a design artifact, not a ground truth — consistent with validation approaches that do not require a perfect reference. Report inter-rater agreement alongside the STAPLE-derived score, and prefer evaluating on held-out raters to verify that the fusion has not overfit the annotator pool.