repository·main·Indexed 41 days ago
https://github.com/abdenlab/oxbowA genomic data I/O library that translates next-generation sequencing (NGS) file formats into Apache Arrow representations. It enables seamless integration of genomic data into Python, R, and Rust ecosystems, allowing conventional NGS files to be accessed as in-memory or larger-than-memory data frames. Version 0.8.1 supports sequence, alignment, variant call, interval feature, and UCSC BBI formats.
Oxbow is a genomic data I/O library designed to model and translate next-generation sequencing (NGS) file formats into Apache Arrow representations. This allows genomic data to be used within the broader ecosystem of computational tools by providing access to conventional NGS files as in-memory or larger-than-memory data frames.
Core functionality is implemented in Rust, with high-level bindings available for Python and R.
Oxbow manages genomic coordinate systems to prevent off-by-one errors. It uses a two-character notation based on the base of the start and end coordinates:
"01": 0-based start, 1-based end (0-based, half-open). Used by BED, BigBed, and BigWig. Notation: [start, end)."11": 1-based start, 1-based end (1-based, closed). Used by SAM, BAM, CRAM, VCF, BCF, GFF, and GTF. Notation: [start, end].Note that in both conventions, the end coordinate is numerically the same; only the start coordinate encoding changes. Oxbow defaults to the native convention of the file format being read.
Oxbow relies on specific compression and indexing patterns to provide efficient random access to genomic data:
FAI indexes uncompressed FASTA records.GZI is an additional file required to perform lookups in BGZF-compressed FASTA files.deflate compression with an internal index for access.Oxbow data sources are logically grouped into fragments.
.regions(), each query is mapped to a unique fragment. Each fragment generates its own independent stream of record batches.When converting an Oxbow data source to a Dask data frame using .dd() or .to_dask(), oxbow maps its logical fragments to Dask partitions. Each partition is expected to be independently accessible and fit in memory.
import ox
# Single fragment (no regions defined)
ds = ox.from_bam("data/sample.bam")
print(ds.fragments())
# Multiple fragments (one per region)
ds_regions = ox.from_bam("data/sample.bam").regions(["chr1", "chr3", "chrX"])
print(ds_regions.fragments())
# Mapping fragments to Dask partitions
df = (
ox.from_bam("data/sample.bam")
.regions(["chr1", "chrX", "chrY"])
.dd() # or .to_dask()
)
# Access a specific partition
print(df.partitions[1].compute())BatchReaderDataset class in oxbow.arrow is a high-level interface for working with datasets that can be read in batches, typically leveraging Apache Arrow. It provides a suite of methods for common data operations such as filtering, joining, sorting, and converting to different formats like Arrow Tables or batches. It is designed to handle large-scale data by allowing efficient scanning and batch iteration.Oxbow is designed to alleviate the friction of the 'bioinformatic silo' by translating streams of records from conventional Next-Generation Sequencing (NGS) files into Apache Arrow representations on demand.
Instead of using specialized, tightly-coupled bioinformatics toolsets that require complex and redundant I/O transformations, Oxbow allows you to treat NGS data as tabular/relational data. This enables you to use modern, general-purpose data technologies (which work directly on columnar Arrow representations with zero copying) for downstream analysis.
Key capabilities include:
Oxbow is organized as a multi-package monorepo with three primary components:
oxbow/): The core Rust library providing parsing and streaming functionality.py-oxbow/): Python bindings built using PyO3 and maturin.r-oxbow/): R bindings built using rextendr (currently minimal and under development).All data I/O operations are performed in Rust, while Python and R users interact with high-level features exposed through these bindings.
You can build the oxbow crate using cargo. Navigate to the oxbow directory and run the build command. Use the --release flag for optimized non-debug builds.
cd oxbow
cargo build # --release (for non-debug build)Oxbow is a hybrid project where data I/O is handled in Rust and features are exposed via Python. To develop with the Python package (py-oxbow), you need Rust and the uv package manager.
rustup.uv following the official instructions.py-oxbow and run uv sync to create a virtual environment in py-oxbow/.venv containing all PEP 735-style dependency groups.# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Setup Python dependencies
cd py-oxbow
uv syncTo build and install a local development version of the py-oxbow package into your virtual environment, use maturin via uvx.
Use the --uv flag to ensure compatibility with the uv environment. For faster, non-debug builds, include the --release flag. After building, you can verify the installation by running uv run python and attempting to import oxbow.
cd py-oxbow
uvx maturin develop --uv
# For a faster, non-debug build:
uvx maturin develop --uv --release
# Verify installation:
uv run python
>>> import oxbow as oxTo install the R interface for oxbow, you must first have a working Rust toolchain installed on your system.
Windows Users: You must manually add the following Rust targets before installation:
rustup target add x86_64-pc-windows-gnu
rustup target add i686-pc-windows-gnuOnce the Rust toolchain is configured, install the package using remotes::install_github or devtools::install_github specifying the r-oxbow subdirectory.
remotes::install_github("abdenlab/oxbow", subdir="r-oxbow")