DOCSEARCHbyAlgolia
Submit Docs

Databento Python Client

repository·main·Indexed 43 days ago

https://github.com/databento/databento-python

Official Python client library (v0.86.0) for Databento, providing access to live and historical market data. Features include the Historical client for time-series and batch data, the Live client for real-time streaming via TCP, and the Reference client for instrument definitions and symbology. Supports DBN (Databento Binary Encoding) processing with utilities to convert data to Pandas DataFrames and numpy ndarrays.

Tokens
5.3K
Snippets
26
Records
31
Agent score
54%

What's inside Databento

  1. Configure the Databento client with an API key

    main

    The library requires an API key (a 32-character string starting with db-) found in your Databento user portal. You can provide the key in two ways:

    1. As a parameter: Pass the key directly to the db.Historical or db.Live constructor. This is not recommended for production applications.
    2. Via environment variable: Leave the parameter empty and set the DATABENTO_API_KEY environment variable in your system.
    import databento as db

# Pass as parameter (not recommended for production)
client = db.Historical('YOUR_API_KEY')

# Or, pass via `DATABENTO_API_KEY` environment variable
client = db.Historical()

  3. Access Historical client sub-APIs

    main

    The Historical client acts as a container for several specialized API modules. Once initialized, you can access these via the following properties:

    • client.batch: For batch data operations (BatchHttpAPI).
    • client.metadata: For retrieving metadata (MetadataHttpAPI).
    • client.symbology: For symbology and mapping services (SymbologyHttpAPI).
    • client.timeseries: For querying time-series data (TimeseriesHttpAPI).

  4. Access Reference data via Reference client sub-APIs

    main

    The Reference client exposes three specialized API sub-clients for different types of reference data:

    • adjustment_factors: Accesses adjustment factors via AdjustmentFactorsHttpAPI.
    • corporate_actions: Accesses corporate actions via CorporateActionsHttpAPI.
    • security_master: Accesses the security master via SecurityMasterHttpAPI.
    import databento as db

client = db.Reference('YOUR_API_KEY')

# Access sub-APIs
adj_factors = client.adjustment_factors
corp_actions = client.corporate_actions
sec_master = client.security_master

  5. Work with DBNStore data

    main

    The DBNStore object returned by time series requests contains the data and associated metadata.

    Accessing Metadata

    Properties available on the DBNStore object include:

    • dataset, schema, symbols, stype_in, stype_out, start, end, limit, compression.
    • symbology: Contains information equivalent to a symbology.resolve request.
    • mappings: Contains per-date interval mappings between stype_in and stype_out.

    Data Conversion and Export

    • Pandas DataFrame: Convert to a DataFrame using .to_df().
    • Numpy Arrays: Convert to individual record arrays using .to_ndarray().
    • File Export: Write raw DBN data using .to_file(path), or export to CSV using .to_csv(path) and JSON using .to_json(path).

    Replaying Data

    You can iterate through records by passing a callback function to the .replay(callback) method.

    # Convert to Pandas
df = data.to_df()

# Convert to Numpy
array = data.to_ndarray()

# Write to disk
data.to_file("test.dbn")
data.to_csv("my_data.csv")

# Replay records
def my_handler(record):
    print(record)

data.replay(my_handler)

  7. Submit and manage batch data requests

    main

    For large datasets, use the batch API to submit jobs that run asynchronously.

    1. Submit a job: Use client.batch.submit_job(...). This returns a job object containing an id.
    2. List jobs: Use client.batch.list_jobs(since=...) to check the status of jobs. Look for the state done before attempting download.
    3. Download results: Use client.batch.download(job_id=..., output_dir=...) to download all files for a job, or specify filename_to_download to get a specific file (e.g., metadata.json).
    # Submit
new_job = client.batch.submit_job(
    dataset="GLBX.MDP3",
    symbols=["ESH1"],
    schema="trades",
    start="2020-12-27T12:00",
    end="2020-12-29",
    encoding="dbn",
    delivery="download",
    limit=1000,
)
job_id = new_job["id"]

# List jobs
client.batch.list_jobs(since=pd.Timestamp.utcnow() - pd.Timedelta(minutes=5))

# Download
client.batch.download(job_id=job_id, output_dir="mydata")

  9. Fetch and process historical market data

    main

    You can use the db.Historical client to retrieve time-series data using client.timeseries.get_range(). Once retrieved, you can process the data using several methods:

    • .replay(callback=...): An event-driven market replay that dispatches each data event to a provided handler function.
    • .to_df(): Converts the data into a Pandas DataFrame.
    • .to_ndarray(): Converts the data into a numpy ndarray.
    import databento as db

client = db.Historical('YOUR_API_KEY')
data = client.timeseries.get_range(
    dataset='GLBX.MDP3',
    symbols='ES.FUT',
    stype_in='parent',
    start='2022-06-10T14:30',
    end='2022-06-10T14:40',
)

# Market replay with print as the event handler
data.replay(callback=print)

# Or convert to other formats
df = data.to_df()  # to DataFrame
array = data.to_ndarray()  # to ndarray

  10. Check databento-python requirements

    main

    The library is compatible with Anaconda 2023.x and above. Minimum dependencies include:

    • python = "^3.10"
    • aiohttp = "^3.8.3"
    • databento-dbn = "~0.69.0"
    • numpy = ">=1.23.5"
    • pandas = ">=1.5.3"
    • pyarrow = ">=13.0.0"
    • requests = ">=2.25.1"
    • zstandard = ">=0.21.0"
    • pip-system-certs >=4.0 (Windows only)

  11. Iterate over live records using Live client

    main

    You can consume live records by iterating over the Live client instance. The LiveIterator automatically starts the client when created and terminates it when destroyed, providing context-manager-like behavior for loops.

    Note: Do not call client.start() before iterating, as this will cause an error. The iterator handles starting the session for you.

    from databento import Live

client = Live(key="YOUR_API_KEY")
client.subscribe(dataset="glbx", schema="mbo", symbols="ES.c:0")

# Synchronous iteration
for record in client:
    print(record)

# Asynchronous iteration
async def stream_data():
    async for record in client:
        print(record)

  12. Request time series market data

    main

    Use client.timeseries.get_range(**params) to request historical time series data. The data is returned as a DBNStore object, which acts as an in-memory buffer.

    params = {
    "dataset": "GLBX.MDP3",
    "symbols": "ES.FUT",
    "stype_in": "parent",
    "schema": "mbo",
    "start": "2022-06-10T14:30",
    "end": "2022-06-11",
}

data = client.timeseries.get_range(**params, limit=1000)