BPCells

The current Python implementation of BPCells focuses on providing read/write access to BPCells file formats for:

1. Integer matrices
2. scATAC fragments

Limitations and Future Plans:

• Data Processing: Future updates aim to add data-processing functions (like streaming normalization, PCA, or ATAC-seq peak/tile matrix creation) to provide Python access to the shared C++ core.
• Plotting: Plotting functionality is not planned for Python as the current implementation relies on R-specific plotting libraries.
• R Helpers: Pure R helper functions are unlikely to be ported to Python soon.

BPCells is a high-performance package designed for single-cell analysis of large RNA-seq and ATAC-seq datasets. It is optimized for scalability, capable of handling massive datasets (e.g., 44M cells) on standard hardware by using bitpacking compression and disk-backed C++ processing.

Key Features:

• Efficient Storage: Uses bitpacking compression for single-cell datasets.
• High Performance: Fast, disk-backed processing for RNA-seq and ATAC-seq.
• Analysis Capabilities: Supports marker gene identification and clustering.
• Interoperability: Works with AnnData, 10x datasets, R sparse matrices, and GRanges.
• Extensibility: Provides infrastructure to scale 3rd party tools like Seurat.

The BPCells Python bindings currently focus on providing read/write access to BPCells file formats for:

1. Integer matrices
2. scATAC fragments

Limitations and Future Plans

• Data Processing: Future updates aim to bring R-interface functions (like streaming normalization, PCA, or ATAC-seq peak/tile matrix creation) to Python via the shared C++ core.
• Plotting: Plotting functionality is not currently planned for Python as it relies on R-specific plotting libraries.
• R-only Helpers: Some helper functions implemented in pure R are unlikely to be ported to Python soon.

Fragments in BPCells are represented by several key data fields designed for efficient chromosome-based access and seeking:

• cell, start, end: Concatenated values for all chromosomes, sorted by start within each chromosome.
• chr_ptr: Indices into the cell, start, and end arrays. Chromosome i spans from chr_ptr[2*i] to chr_ptr[2*i+1]. This structure allows for efficient re-ordering of chromosome IDs and handles non-seekable input formats like fragments.tsv.gz.
• end_max: A running maximum of end values within each chromosome, stored every 128 values to enable seeking. end_max[i] stores the maximum of end[chr_start_idx : i*128].

The Python bindings for BPCells are currently experimental and the API is subject to change.

Supported Functionality:

• Read/write access to BPCells file formats for integer matrices.
• Integer matrix slicing.
• Import of integer matrices.
• Pseudobulk fragment insertion counts.

Unsupported/Planned Functionality:

• Data Processing: Streaming normalization, PCA, and ATAC-seq peak/tile matrix creation are planned for future updates to provide Python access to the C++ core.
• Plotting: Plotting functionality is not planned for Python as it relies on R-specific plotting libraries.
• R-only Helpers: Certain helper functions implemented in pure R are unlikely to be ported to Python soon.

Matrices are represented using a compressed format containing the following fields:

• val: A list of values for each non-zero entry.
• row: A list of row indices for each non-zero entry.
• col_ptr: An array of length number of columns + 1. Values in column i are located between index col_ptr[i] and col_ptr[i+1].
• row_count: An integer representing the total number of rows in the matrix.

The arrayIO system is designed to share code for fragment and matrix data across three distinct storage formats:

1. In-memory arrays: Data stored directly in RAM.
2. Binary files: Data arranged in folders on disk.
3. Single HDF5 file: Data encapsulated within a single HDF5 container.

Most logic for reading and writing is shared because the data is primarily represented as arrays of unsigned 32-bit integers.

BPCells provides two primary matrix implementations for handling large-scale data:

1. experimental.DirMatrix: A disk-backed matrix implementation designed for datasets that exceed available system memory. It stores data in a directory-based format.
2. experimental.MemMatrix: An in-memory matrix implementation suitable for datasets that fit within the system's RAM.

Note that these are currently located in the experimental namespace of the bpcells module.

from bpcells import experimental

# Example of accessing the matrix types
# dir_matrix = experimental.DirMatrix(...)
# mem_matrix = experimental.MemMatrix(...)

To develop the R component of BPCells, you need to install the following R packages:

• pkgdown (for documentation)
• devtools (for development workflows)
• testthat (for running tests)

install.packages(c("pkgdown", "devtools", "testthat"))

To reduce build times during development, you can use ccache. Install it via your package manager (e.g., brew install ccache or sudo apt install ccache), then set the following environment variables before building:

export CXX="ccache g++"
export CC="ccache gcc"

To install the development version of the BPCells Python package, use pip with the #subdirectory=python fragment to target the correct package directory within the repository.

pip install "git+https://github.com/bnprks/BPCells#subdirectory=python"

The recommended way to install BPCells in R is directly from GitHub using the remotes package.

Prerequisite: You must have the HDF5 library installed and accessible on your system before running the installation command.

Installation Command:

remotes::install_github("bnprks/BPCells/r")

Alternative Installation Methods:

• Pre-built Binaries: For Mac and Windows users, pre-built binary packages are available via R-universe.
• Conda: BPCells is available via Conda through the Conda Forge R team.