Pycytominer

Pycytominer is a toolkit for processing high-dimensional readouts from high-throughput image-based profiling experiments. It sits between feature extraction and downstream analysis.

The Workflow:

1. Feature Extraction: Image data flows from a microscope to segmentation and feature extraction tools like CellProfiler or DeepProfiler.
2. Harmonization: Single-cell readouts are curated into a manageable form. For CellProfiler, it is recommended to use CytoTable to harmonize outputs before Pycytominer processing. For DeepProfiler, Pycytominer provides utilities in pycytominer.cyto_utils.
3. Pycytominer Processing: Pycytominer performs five fundamental functions (Aggregate, Annotate, Normalize, Feature select, and Consensus) to process morphology features.
4. Downstream Analysis: The outputs are used for machine learning models and statistical testing to derive biological insights.

Pycytominer is built on top of pandas and supports several data formats:

• Core Formats: Apache Parquet, Parquet dataset directories, and compressed text (e.g., .csv.gz).
• AnnData Support: You can use anndata (specifically h5ad or zarr) by installing the extra dependency: pip install pycytominer[anndata].
• CellProfiler Integration: Pycytominer supports CellProfiler data structures directly, recognizing Metadata_ prefixes for experimental details and compartment prefixes (e.g., Cells_, Cytoplasm_, Nuclei_, or custom ones like Mito_) for morphology features.

Note on Image Payloads: If using Parquet-backed CytoTable or anndata with OME-Arrow image payload columns, Pycytominer supports numeric Image_* measurements. However, normalize() only accepts numeric feature columns, and non-normalized image payload columns are preserved in outputs rather than transformed.

Pycytominer can be installed using several different package managers depending on your environment:

pip

Install directly from PyPI:

pip install pycytominer

conda

Install from the conda-forge channel:

conda install -c conda-forge pycytominer

Docker

Container images are available on Docker Hub. You can run modules directly, start interactive sessions, or map local directories to run scripts inside the container.

Run a module:

docker run --platform=linux/amd64 cytomining/pycytominer:latest python -m pycytominer.<module_name>

Interactive bash session (specific commit):

docker run -it --platform=linux/amd64 cytomining/pycytominer:pycytominer-1.1.0.post16.dev0_b1bb292 bash

Run a local script (mapping current directory to /opt):

docker run -v $PWD:/opt --platform=linux/amd64 cytomining/pycytominer:pycytominer-1.1.0.post16.dev0_b1bb292_240417 python /opt/script.py

The primary way to use Pycytominer is via its Python API. Most functions accept either a pandas.DataFrame or a file path as input.

Common tasks include:

• Normalization: Use pycytominer.normalize() to standardize profiles. You can specify a method (e.g., 'standardize') and a samples string to define the control group (e.g., "Metadata_treatment == 'control'").
• Loading CytoTable profiles: Use pycytominer.cyto_utils.load_profiles() to load a profile table directory, or load_cytotable_profiles() as a convenience helper for specific table names within a warehouse root.

import pandas as pd
import pycytominer

# Example: Normalizing a dataset
df = pd.read_csv("path/to/profiles.csv.gz")
normalized_df = pycytominer.normalize(
    profiles=df,
    method="standardize",
    samples="Metadata_broad_sample == 'DMSO'"
)

# Example: Loading CytoTable profiles
from pycytominer import normalize
from pycytominer.cyto_utils import load_cytotable_profiles, load_profiles

profiles_from_helper_df = load_cytotable_profiles(
    "path/to/warehouse_root",
    table_name="joined_profiles",
)

normalized_df = normalize(
    profiles=profiles_from_helper_df.assign(
        Metadata_treatment=["control" if i % 2 == 0 else "drug" for i in profiles_from_helper_df["Metadata_ImageNumber"]]
    ),
    samples="Metadata_treatment == 'control'",
    method="standardize",
)