The problem — CellProfiler outputs a single-cell measurement table. That table is not a profile. Before any biological signal can be trusted, measurements must be aggregated to well-level, annotated with plate-map metadata, normalized against controls, and stripped of features that are low-variance, highly correlated, or driven by noise rather than biology. Doing each step ad hoc — in a one-off script — guarantees that two labs running nominally identical assays produce incompatible numbers.
What it is / how it works — pycytominer is the standard Python library for the bioinformatics layer of Cell Painting and image-based profiling pipelines. It implements five composable operations:
- Aggregate — collapses single-cell rows to a well-level profile (mean or median) by metadata grouping.
- Annotate — joins profile data with platemap files, attaching compound, dose, and perturbation identifiers.
- Normalize — standardizes feature distributions relative to negative controls (DMSO-MAD, whole-plate z-score, or population-level scaling), making cross-plate comparisons valid.
- Feature select — removes low-variance features, near-zero-variance features, correlated feature pairs, and features that fail a replicate reproducibility threshold — reducing dimensionality from ~1,500 to a cleaner, higher-signal representation.
- Consensus — averages replicate profiles into a single consensus signature per perturbation.
The library is built on pandas and scikit-learn, with Apache Parquet and AnnData output support. It is the de facto upstream preprocessing step before batch correction (Harmony, sphering) and readout (% replicating, mAP).
Where it fits — pycytominer spans the features, correct, and readout stages. In the JUMP Cell Painting dataset and the CPJUMP1 benchmark, the published pipeline runs CellProfiler → pycytominer normalization + feature selection → Harmony → copairs % replicating. Any deviation from that ordering changes the numbers.
Where it breaks — Normalization against a sparse or contaminated DMSO population propagates control-plate failures silently into every well. Feature selection thresholds that work for one plate layout can be too aggressive or too permissive on another. The library's defaults are sensible for standard 384-well Cell Painting but should be validated per assay rather than assumed universal.