Why mAP, Dice, and SSIM Are Not Enough

The problem — A pipeline is judged by a scalar: Dice or IoU for segmentation, mAP for detection, SSIM for restoration. Each is a correspondence metric — it measures how closely the output matches a reference, in pixels or boxes. None of them measure the thing the experiment exists to produce: a decision. The estimand — a cell count, a phenotype call, a slide-level diagnosis, a reportable-or-reject verdict — sits one or more transforms above the pixels, and a high correspondence score can coexist with a wrong decision. This is the level-decoupling problem stated for the readout: the metric and the decision are loosely linked, and optimizing one does not move the other.

What it is / how it works — problem-aware metric selection. The fix is not a better single number; it is choosing the metric from the question. Metrics Reloaded derives metrics from a "problem fingerprint" — the domain interest plus the properties of the target, data, and output — precisely because the convenient metric is so often the wrong one. Two failure mechanisms recur, both catalogued in the companion pitfalls and picture-story papers:

• The metric rewards the wrong thing. Overlap scores are dominated by large, easy regions; a model can post a high Dice while systematically losing the small, rare, or boundary structures that carry the biology. mAP averages over confidence thresholds no one will deploy at. SSIM rewards pixel similarity, not preserved signal.
• The metric ignores the cost structure. A false positive and a false negative are almost never equally costly, yet accuracy, Dice, and mAP weight them as if they were. Decision curve analysis makes the alternative explicit — net benefit across the threshold a user would actually act on — because a model that scores well on overlap can have negative clinical utility.

Where it breaks — The general principle: a correspondence metric measures fidelity to a reference, not utility for a decision, and the gap widens wherever the readout is a transform of the pixels. In Cell Painting, the decision is replicate reproducibility (percent-replicating / mAP over profiles), and a segmenter with excellent Dice can still yield non-replicating profiles. In digital pathology, a high patch-level F1 says nothing about slide-level net benefit at the operating threshold a pathologist uses. In light-sheet, a strong PSNR after deconvolution can accompany merged or hallucinated objects that corrupt downstream counts. In spatial omics, a good segmentation IoU still misassigns transcripts at boundaries, and the error is invisible to any image-level score. The methodological literature generalizes it: evaluation noise and misaligned metrics let benchmark gains outrun real utility. The metric that survives is the one chosen for the decision — which is why this is inseparable from validation and workflow-level evaluation.

References

• Maier-Hein, Reinke et al. (2024). Metrics Reloaded: Recommendations for Image Analysis Validation. Nature Methods.
• Reinke et al. (2024). Understanding Metric-Related Pitfalls in Image Analysis Validation. Nature Methods.
• Reinke et al. (2021). Common Limitations of Image Processing Metrics: A Picture Story.
• Vickers & Elkin (2006). Decision Curve Analysis: A Novel Method for Evaluating Prediction Models. Medical Decision Making.
• Varoquaux & Cheplygina (2022). Machine Learning for Medical Imaging: Methodological Failures and Recommendations. npj Digital Medicine.