DOCSEARCHbyAlgolia
Submit Docs

anndata Documentation

repository·main·Indexed 18 days ago

https://github.com/scverse/anndata

A Python package for handling annotated data matrices in memory and on disk, optimized for sparse data and lazy operations. It serves as a core data structure in the scverse ecosystem for single-cell omics analysis, providing the AnnData class to manage expression data alongside metadata for observations (obs) and variables (var). It supports native .h5ad and .zarr formats, as well as various other IO formats like CSV, Excel, and MTX.

Tokens
14.7K
Snippets
45
Records
72
Agent score
73%

What's inside anndata

  1. What is anndata?

    main

    anndata is a Python package designed for handling annotated data matrices both in memory and on disk. It is positioned between pandas and xarray and provides computationally efficient features such as:

    • Sparse data support
    • Lazy operations

    For applications requiring minibatch data loading (ranging from small in-memory matrices to terabyte-scale disk-backed datasets), use the companion package annbatch.

  2. Understand the AnnData on-disk format

    main

    AnnData objects are stored in hierarchical array stores, primarily HDF5 (via h5py) and Zarr. The on-disk structure closely mirrors the in-memory structure.

    To identify an AnnData object in a file, check the root group's metadata for the following attributes:

    • encoding-type: must be anndata
    • encoding-version: e.g., 0.1.0

    Typical top-level keys in an AnnData store include X, layers, obs, obsm, obsp, uns, var, varm, and varp.

    import h5py
# For HDF5 files
store = h5py.File("data.h5ad", mode="r")
print(list(store.keys()))

import zarr
# For Zarr stores
store = zarr.open("data.zarr", mode="r")
print(list(store.keys()))

  3. Represent scalar values as 0-dimensional arrays

    main

    In AnnData on-disk formats (HDF5 or Zarr), single values like strings, numbers, or booleans (e.g., parameters stored in uns) must be represented as 0-dimensional arrays.

    Specifications (v0.2.0):

    • Numeric scalars: Must have metadata "encoding-type": "numeric-scalar" and "encoding-version": "0.2.0". Supports boolean, unsigned/signed integers, and floating point/complex types.
    • String scalars: Must have metadata "encoding-type": "string" and "encoding-version": "0.2.0".
      • In Zarr: Must use a fixed-length unicode dtype.
      • In HDF5: Must use a variable-length UTF-8 encoded string dtype.
    >>> store["uns/neighbors/params/metric"][()]
'euclidean'
>>> dict(store["uns/neighbors/params/metric"].attrs)
{'encoding-type': 'string', 'encoding-version': '0.2.0'}

  4. How AwkwardArrays (ragged arrays) are stored on disk

    main

    Support for ragged arrays via the awkward-array library is considered experimental (under the 0.9.0 release series).

    When an AwkwardArray is saved to disk (HDF5 or Zarr), anndata breaks the array down into its constituent arrays using ak.to_buffers. These constituent arrays are then written using standard anndata methods.

    To reconstruct the array, the following metadata is stored in the element's attributes (.attrs):

    • length: The total length of the array.
    • form: A serialized JSON string describing the array structure (the 'form').
    • encoding-type: Set to 'awkward-array'.
    • encoding-version: The version of the encoding used.

    In HDF5, these appear as multiple datasets (e.g., nodeX-data, nodeX-mask, nodeX-offsets). In Zarr, they appear as sub-paths within the array group.

    >>> store["varm/transcript"].visititems(print)
# Example output showing constituent datasets/arrays
node1-mask <HDF5 dataset "node1-mask": shape (5019,), type "|u1">
node10-data <HDF5 dataset "node10-data": shape (250541,), type "<i8">
...

  5. Configure merge strategies for AnnData concatenation

    main

    When using ad.concat() to combine multiple AnnData objects, you can control how elements aligned to the alternative axis (the axis not being concatenated) are handled using the merge argument.

    Available strategies for the merge parameter:

    • None: No elements aligned to alternative axes are present in the result object.
    • "same": Only elements that are identical in each of the objects are kept.
    • "unique": Elements for which there is only one possible value across all objects are kept.
    • "first": The first element encountered for each position is kept.
    • "only": Only elements that appear in exactly one of the objects are kept.

    Note that comparisons are made after indices are aligned. If objects only share a subset of indices on the alternative axis, the strategy only requires that values for those shared indices match.

    # Example of different merge strategies on axis-aligned elements
ad.concat(adatas)               # Default behavior
ad.concat(adatas, merge="same") # Keeps only identical elements
ad.concat(adatas, merge="unique") # Keeps elements with a single unique value

  6. Use accessors and references to describe AnnData arrays

    main

    The anndata.acc.A accessor allows you to create AdRef objects, which are axis-aligned 1D or 2D references to arrays within an AnnData object. These references are independent of specific AnnData instances, making them ideal for driving plotting, validation, or mapping functions without being bound to a single object.

    Key Properties of AdRef:

    • dims: The dimensions the reference spans (e.g., {'var'}).
    • idx: The index (integer or string) used to select the data.
    • acc: The accessor used to create the reference.

    Common Usage Patterns:

    • Check if a reference exists in an object: ref in adata
    • Extract data using a reference: adata[ref]
    • Use references as mapping keys: adata[A.obs['column_name']]
    from anndata.acc import A
import scanpy as sc

adata = sc.datasets.pbmc3k_processed()

# Create a reference to a 1D vector
ref = A.X[:, 'gene-3']

# Check if the reference is valid for this object
print(ref in adata)  # True

# Extract the actual data
data = adata[ref]

# Inspect reference properties
print(ref.idx)   # 'gene-3'
print(ref.dims)  # {'var'}

  7. Understand missing value semantics (NA vs NaN)

    main

    When dealing with nullable arrays, two types of missing value semantics are defined:

    1. "NA": A comparison between a missing value and a defined value produces a missing value (e.g., "x" == NA $\rightarrow$ NA).
    2. "NaN": A comparison between a missing value and a defined value produces a binary result (e.g., "x" == NaN $\rightarrow$ false).

    Implementations should behave according to these semantics if the runtime data model allows. If an implementation encounters an unknown semantics value, it should not error and may choose its own behavior.

  8. Merge `.uns` dictionaries during concatenation

    main

    The uns_merge argument in ad.concat() controls how the .uns (unstructured metadata) dictionaries are combined. These strategies are applied recursively, meaning they respect the nested structure of the dictionaries.

    Available strategies for uns_merge:

    • None: Returns an empty dictionary {}.
    • "same": Keeps only the keys and nested values that are exactly the same across all objects.
    • "unique": Keeps keys where there is only one possible value across all objects (a superset of "same").
    • "only": Keeps only the keys/values that appear in exactly one of the input objects.
    • "first": Takes the union of all keys and uses the value from the first object that contains that key.

    This is particularly useful for preserving pipeline parameters or metadata (like spatial images) that are shared or unique to specific subsets.

    # Example of recursive uns merging
# If a = {a: 1, c: {c.a: 3, c.b: 4}}
#    b = {a: 1, c: {c.b: 4}}
#    c = {a: 1, c: {c.a: 3, c.b: 4, c.c: 5}}

# Result with uns_merge="same":
# {'a': 1, 'c': {'c.b': 4}}

# Result with uns_merge="unique":
# {'a': 1, 'c': {'c.a': 3, 'c.b': 4, 'c.c': 5}}

# Result with uns_merge="first":
# {'a': 1, 'b': 2, 'c': {'c.a': 3, 'c.b': 4, 'c.c': 5}}

  9. Use categorical arrays for discrete values

    main

    Discrete values can be efficiently stored using categorical arrays (similar to R's factors). These are stored as a group containing two arrays: codes (the integer indices) and categories (the original labels).

    Specifications (v0.2.0):

    • The group must contain metadata: "encoding-type": "categorical", "encoding-version": "0.2.0", and a boolean field "ordered".
    • codes array: An integer array where each entry is the zero-based index in categories. A code of -1 represents a missing value.
    • categories array: The array of unique labels.
    >>> categorical = store["obs"]["development_stage"]
>>> dict(categorical.attrs)
{'encoding-type': 'categorical', 'encoding-version': '0.2.0', 'ordered': False}
>>> categorical.visititems(print)
categories <HDF5 dataset "categories": shape (7,), type "|O">
codes <HDF5 dataset "codes": shape (164114,), type "|i1">

  10. Store string arrays with variable length encoding

    main

    Since NumPy lacks robust support for unicode string arrays, anndata treats strings as text-like data using variable-length encoding.

    Specifications (v0.2.0):

    • Metadata must include "encoding-type": "string-array" and "encoding-version": "0.2.0".
    • In Zarr: Must use the numcodecs VLenUTF8 codec.
    • In HDF5: Must use a variable-length string data type with UTF-8 encoding.
    >>> dict(categorical["categories"].attrs)
{'encoding-type': 'string-array', 'encoding-version': '0.2.0'}

  11. Use the public API instead of internal modules

    main

    When using anndata, always use the documented public API. The project does not guarantee the stability of internal APIs (those located in modules starting with an underscore, such as anndata._core).

    Avoid patterns like:

    from anndata._core import AnnData

    If a feature you need is missing from the public API, it is encouraged to open a GitHub issue to request its official export.

  12. Use the experimental API for batched access

    main

    The experimental module provides tools for working with collections of many AnnData objects or .h5ad files, specifically optimized for PyTorch-based models.

    Warning: These APIs are in development and subject to change.

    • experimental.AnnCollection: For batched access to collections of AnnData objects.
    • experimental.AnnLoader: Deprecated. Use annbatch.Loader instead. This will be removed in version 0.14.
    • experimental.concat_on_disk: For performing concatenation without loading everything into memory.
    • experimental.read_elem_lazy and experimental.read_lazy: Low-level methods for lazy reading of elements.
    • experimental.read_dispatched and experimental.write_dispatched: Utilities for customizing the IO process.