Databento Binary Encoding (DBN)

The dbn-c package provides C FFI bindings for the DBN crate. It is designed for high-performance data processing with the following capabilities:

• Decoding: Supports decoding DBN data using a push-based decoder, suitable for streaming or buffered input.
• Encoding: Supports encoding DBN metadata.
• Serialization: Provides CSV and JSON serialization for DBN records.
• Zero-copy Writing: DBN records are their own wire representation. To write a record to a file, you can use standard fwrite directly with the record's length.

Note: Compression is not handled by the library; it is the responsibility of the caller.

/* Example: Writing a record directly to a file */
// fwrite(&record, record.length * 4, 1, file);

Databento Binary Encoding (DBN) is a high-performance message encoding and storage format designed specifically for normalized market data. It is used by all official Databento client libraries for data interchange and in-memory representation, and it is the default encoding for Databento APIs (live streaming, historical streaming, and batch flat files).

Key characteristics include:

• Self-describing: Includes a metadata header and fixed struct definitions to enforce standardization.
• Performant: Optimized for extremely fast binary encoding and decoding.
• Compressible: Highly compatible with Zstandard for efficient storage.
• Extensible: Supports extendable fixed-width schemas.

Use the --split-by/-S flag to define the splitting logic and --output-pattern/-O to define the naming convention for the resulting files. Supported split methods include schema, day, and symbol.

Available placeholders for --output-pattern include {schema}, {date}, and {symbol}.

# Split by schema
dbn opra-pillar-20260114.dbn.zst --split-by schema --output-pattern 'opra-pillar-2026114.{schema}.dbn.zst'

# Split by day (time)
dbn opra-pillar-202512.ohlcv-1s.dbn.zst --split-by day --output-pattern 'opra-pillar-{date}.ohlcv-1s.dbn.zst'

# Split by symbol using short flags
dbn equs-mini-20260114.dbn.zst -S symbol -O 'equs-mini-2026014-{symbol}.dbn.zst'

The dbn CLI can write compressed JSON or CSV files. You can trigger compression by providing a .zst extension in the --output path or by using the --zstd flag explicitly.

# Compress via file extension
dbn ohlcv-1d.dbn -o ohclv-1d.json.zst

# Compress via explicit flag
dbn ohlcv-1d.dbn --json --zstd -o ohlcv-1d.json.zst

DBN fragments are files that contain DBN data without the metadata header.

• Reading fragments: Use --input-fragment or --input-zstd-fragment.
• Writing fragments: Use --fragment or -F to output a file without a metadata header.

# Read a fragment and output JSON
dbn 120000_121000.mbo.dbn.frag --input-fragment --json

# Write a DBN file as a fragment
dbn ohlcv-1d.dbn -F -o ohlcv-1d.dbn.frag

To install the latest stable version of the databento-dbn Python bindings from PyPI, use pip.

pip install -U databento-dbn

Since databento-dbn is written in Rust, you must have Rust and Maturin installed to build it from source.

To build the project, clone the repository and use maturin build:

git clone https://github.com/databento/dbn
cd dbn
maturin build

To build and install the package directly into your active Python interpreter's environment, use:

maturin develop

This installs a package named databento-dbn in your current environment.

The dbn.h header is generated during the build process. To use it in your C project:

1. Locate the header in ${target_directory}/include/dbn/dbn.h.
2. Add ${target_directory}/include to your compiler's include path.
3. Use #include <dbn/dbn.h> in your source code.

To specify a custom directory for the header output, set the DBN_C_HEADER_DIR environment variable during the build.

To update legacy DBZ files to Zstandard-compressed DBN format, use dbn with an output filename ending in .dbn.zst. You can also use the --dbn flag to explicitly set the output encoding.

# Convert DBZ to Zstd-compressed DBN
dbn 20221212.mbo.dbz -o 20221212.dbn.zst

Linking against the shared library is simpler because it resolves its own dependencies. You only need to specify the library path and the library name.

Use the following command pattern (assuming include contains dbn.h and the shared library is in the current directory):

cc -o app app.c -Iinclude -L. -ldbn_c

Depending on your programming language or interface needs, you can use one of the following implementations of the DBN format:

• Rust: Use the dbn crate for high-performance library integration.
• Python: Use the databento-dbn package (available on PyPI as databento_dbn) for Python bindings.
• CLI: Use the dbn-cli crate to get the dbn binary for command-line interaction with DBN files and streams.