repository·master·Indexed 25 days ago
https://github.com/man-group/arcticdbA high-performance, serverless DataFrame database for the Python Data Science ecosystem, optimized for time-series data. It enables reading and writing Pandas DataFrames and NumPy arrays to S3 or LMDB using a C++ backed implementation. The documentation covers C++ layer builds, persistent storage testing utilities, and integration of third-party libraries such as msgpack-c for serialization and Remotery for CPU/GPU profiling.
The C++ layer is organized into several specialized modules located in cpp/arcticdb/. Detailed documentation for each can be found in their respective .md files:
pybind11 bindings for Python integration.The developer documentation for ArcticDB is organized into high-level architecture overviews, C++ layer details, and Python layer details.
ARCHITECTURE.md for a comprehensive overview of the repository structure, C++ and Python layer organization, storage models, and core functionality.cpp/ directory, covering internals like caching, versioning, storage backends, entity types, codecs, column stores, pipelines, query processing, streaming, and async task scheduling.python/ directory, covering the Arctic class, the Library V2 API, query processing via QueryBuilder, DataFrame normalization, and storage adapters.ArcticDB is a high-performance, embedded analytical (OLAP) DataFrame database designed for the Python Data Science ecosystem.
Key characteristics:
The C++ layer (cpp/) is the performance-critical core responsible for data storage, compression, and query processing. Key modules include:
| Module | Purpose |
|---|---|
| storage/ | Backend abstraction for S3, Azure, LMDB, MongoDB, and Memory |
| version/ | Version chain management, symbol lists, and snapshots |
| pipeline/ | Data serialization/deserialization pipelines (read/write) |
| processing/ | Query execution including filtering, projection, and aggregation |
| codec/ | Data compression and encoding (LZ4, ZSTD, passthrough) |
| column_store/ | In-memory columnar representation and memory management |
| entity/ | Core domain types like keys, data types, and descriptors |
ArcticDB is designed for high performance using a lock-free approach for most operations:
VERSION_REF key. The last writer to update VERSION_REF wins.append() operations may result in rows appearing out of order or one append being dropped.list_symbols() operation uses a lock-free concurrent data structure. LOCK keys are only utilized during the compaction phase of the symbol list.The codec module manages the transformation of data between raw formats and compressed segments through two distinct paths.
ArcticDB organizes data into a hierarchy:
You can create a library using ac.create_library('name') or use ac.get_library('name', create_if_missing=True) to combine creation and instantiation.
ArcticDB does not support concurrent writers to a single symbol unless the data is written as staged data.
To use this pattern successfully, you must follow these rules:
staged=True: When calling library.write(), set the staged parameter to True. This prevents the written data from being immediately available for reading.library.finalize_staged_data(symbol) is called. This process merges all staged units into the symbol's main data set.This pattern is useful for high-throughput ingestion where multiple workers (e.g., Spark executors) need to write to the same symbol simultaneously.
ArcticDB is a client-side library and does not support transactions.
staged functionality (see staged documentation).A LazyDataFrame allows you to build a query pipeline without executing it immediately. This is useful for optimizing complex operations before triggering a single, efficient data fetch.
Key Differences from QueryBuilder:
LazyDataFrame uses the col("name") function for column references instead of q["name"]..collect().Example:
from arcticdb import col
# Read returns a LazyDataFrame when lazy=True
ldf = lib.read("symbol", lazy=True)
# Build query using col()
ldf = ldf[col("price") > 100]
# Execute
result = ldf.collect()from arcticdb import col
ldf = lib.read("symbol", lazy=True)
ldf = ldf[col("price") > 100]
result = ldf.collect()When developing for ArcticDB, ensure compatibility across three main areas:
ArcticDB is a client-side database. Newer clients might write data formats that older clients cannot read. Any breaking changes to the on-disk format must be documented and, if possible, include version numbers to trigger clear error messages in older clients.
Changes to API signatures, including changes to which exceptions are thrown or the types of those exceptions, are breaking changes. Always describe API changes clearly in Pull Requests, especially for the NativeVersionStore API.
Symbols are mostly decoupled, but snapshots create dependencies. When deleting keys from storage, ensure they are not required by an existing snapshot. When modifying or deleting a snapshot, ensure that any keys for which that snapshot was the last reference are also cleaned up.