diffxpy

diffxpy is a tool designed for fast and scalable differential expression analysis on single-cell RNA-seq data. It is designed to integrate into scanpy workflows.

Note: This repository is unmaintained. If you require pure-python differential expression analysis, it is recommended to use PyDESeq2 instead.

In diffxpy, Generalized Linear Models (GLMs) are the primary model class used for differential expression analysis via Wald tests (de.test.wald) or Likelihood Ratio Tests (LRT) (de.test.lrt).

Users can specify models using formulas or by providing design matrices directly. The choice of GLM is driven by the noise_model argument.

Estimator objects are produced in two ways:

1. Indirectly: By running a test. Wald tests return an object with a model_estim attribute. LRTs return objects with full_estim and reduced_estim attributes.
2. Directly: By calling de.fit.model to produce an estimator object independently of a test.

diffxpy performs parameter estimation for Generalized Linear Models (GLMs) using the batchglm module.

Key concepts:

• GLMs vs. Tests: GLMs are required for performing Wald tests and likelihood ratio tests. They are not required for t-tests or Wilcoxon rank-sum tests.
• Estimation Methods: While some GLMs allow for closed-form maximum likelihood estimators, batchglm primarily uses numerical parameter estimation based on local approximations of the objective function (e.g., gradient descent). This approach supports a wider range of variance and noise models.
• Transparency: Unlike many differential expression frameworks that hide the training process, diffxpy exposes training details. This allows users to monitor training progress and optimize hyperparameters.

Depending on your hypothesis, you can choose from several statistical test types within diffxpy:

• Likelihood-ratio tests (LRT): For comparing nested models.
• Wald tests: For testing a single parameter or multiple parameters.
• t-tests: For comparing means.
• Wilcoxon (rank sum) tests: For non-parametric testing.

For complex experimental designs, diffxpy supports running multiple tests for a single gene. This includes:

• Pairwise tests: Comparing specific groups against each other.
• Group versus rest tests: Comparing one group against the combined remainder of the dataset.
• Partitioned tests: Conducting tests within specific partitions of the dataset.

You can also define data set partitions to map single tests across these partitions.

To access the high-level differential expression and enrichment functions, import the diffxpy.api module using the alias de.

import diffxpy.api as de

diffxpy requires batchglm, which in turn depends on tensorflow and tensorflow-probability. You can install the stable major releases or the nightly versions (which are more up-to-date). Using tf-nightly and tfp-nightly is recommended for the latest features and updates.

To install the nightly versions:

pip install tf-nightly
pip install tfp-nightly

To install the stable major releases:

pip install tensorflow
pip install tensorflow-probability

To use the differential expression and enrichment tools, import the API module as de:

import diffxpy.api as de

Once imported, you can access two main functional areas:

1. Differential expression analysis: Available in the de.test.* module.
2. Gene set enrichment analysis: Available in the de.enrich.* module (based on differential expression calls).

import diffxpy.api as de

# Access differential expression tests
# de.test.some_test_function(...)

# Access gene set enrichment
# de.enrich.some_enrichment_function(...)

To allow for monitoring and hyperparameter optimization without requiring deep technical knowledge of optimization algorithms, diffxpy provides "training strategies".

Use training strategies to modify optimizer defaults, including:

• Optimizer algorithms
• Learning rates
• Optimizer schedules (e.g., using multiple optimizers)
• Convergence criteria

If your model fails to converge using the default optimizer, you should use these strategies to adjust the hyperparameters or report the issue on GitHub.

To limit the number of threads used by Numpy/Scipy linalg sub-modules, you must set environment variables in the shell before starting your Python session (e.g., in the terminal from which you launch a Jupyter Notebook or a Python script).

This is primarily relevant on some Linux operating systems. Use the following variables to restrict Numpy to a single thread:

export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1

To install diffxpy, follow these steps:

1. Clone the diffxpy repository from https://github.com/theislab/diffxpy into your desired directory.
2. Navigate to the root directory of the cloned diffxpy repository.
3. Install it using pip in editable mode.

pip install -e .