DOCSEARCHbyAlgolia
Submit Docs

cryptofeed Documentation

repository·master·Indexed 25 days ago

https://github.com/bmoscon/cryptofeed

A cryptocurrency exchange websocket data feed handler (v2.4.1) that provides normalized market data—including trades, books, and tickers—from multiple exchanges via websockets or REST polling. It features a synthetic NBBO feed, support for various backends (such as Redis, Kafka, and MongoDB), and integrated REST interfaces for account management and order placement. The library supports both synchronous and asynchronous callbacks for public and authenticated data channels.

Tokens
12.4K
Snippets
13
Records
64
Agent score
84%

What's inside cryptofeed

  1. Implement Order Book (L2) support

    master

    To support order books, implement a handler (e.g., _book) that processes the exchange's book data format.

    For exchanges that send the full book in every update (rather than incremental deltas), the handler should:

    1. Convert price and amount fields to Decimal.
    2. Store the bids and asks in a sorted dictionary structure (e.g., using sd for sorted dicts).
    3. Call await self.book_callback(symbol, L2_BOOK, False, False, msg['ts']) to notify the client of the update.
    async def _book(self, msg):
    symbol = self.exchange_symbol_to_std_symbol(msg['ch'].split('.')[1])
    data = msg['tick']
    self._l2_book[symbol] = {
        BID: sd({
            Decimal(price): Decimal(amount)
            for price, amount in data['bids']
        }),
        ASK: sd({
            Decimal(price): Decimal(amount)
            for price, amount in data['asks']
        })
    }

    await self.book_callback(symbol, L2_BOOK, False, False, msg['ts'])

  2. Configure the Cryptofeed FeedHandler

    master

    Configuration for the FeedHandler is passed via the config keyword argument during instantiation. You can provide configuration in four ways:

    1. No configuration: If config is None (the default), the settings defined in config.py are used.
    2. Dictionary: Provide a dictionary where keys match the valid setting options.
    3. YAML file path: Provide a path to a .yaml file containing configuration entries.
    4. Environment Variable: Set the CRYPTOFEED_CONFIG environment variable to an absolute path pointing to a configuration file.

    Note: Configuration is automatically passed to exchange objects created by the feed handler (specified by name). If you manually instantiate Feed objects, you must pass the config via the Feed object's config kwarg to ensure settings are applied.

  3. Access and manipulate custom data types

    master

    Cryptofeed uses custom data types (e.g., Trade, Ticker, OrderBook) for callback data. These objects are read-only to prevent accidental modification.

    Accessing Fields

    You can access data members directly as attributes:

    assert trade.symbol == 'BTC-USD'

    Converting to Dictionary

    Use the to_dict() method to convert the object to a dictionary. You can use the numeric_type keyword argument to cast numeric values to a specific type (e.g., str or float).

    Inspecting Raw Data

    Every data object has a .raw member containing the original raw message from the exchange, which can be used to inspect data not explicitly mapped to the object fields.

    Supported Data Types

    • Trade
    • Ticker
    • Liquidation
    • Funding
    • Candle
    • Index
    • OpenInterest
    • OrderBook
    • OrderInfo
    • Balance
    • L1Book
    • Transaction
    • Fill

  4. Implement a message handler for websocket data

    master

    The message_handler is called by the ConnectionHandler whenever a message is received. It is responsible for:

    1. Decompression/Parsing: Unzipping (e.g., using zlib) and parsing (e.g., json.loads) the raw message.
    2. Heartbeats: Responding to exchange pings via the provided conn object to prevent disconnection.
    3. Routing: Identifying the channel type (e.g., trade or depth) and calling the appropriate internal handler (e.g., _trade or _book).
    4. Callback Invocation: Parsing the data and calling self.callback (for trades) or self.book_callback (for order books) to deliver updates to the user.
    async def message_handler(self, msg, conn, timestamp):
    # unzip message
    msg = zlib.decompress(msg, 16+zlib.MAX_WBITS)
    msg = json.loads(msg, parse_float=Decimal)

    # Handle pings
    if 'ping' in msg:
        await conn.write(json.dumps({'pong': msg['ping']}))
    elif 'status' in msg and msg['status'] == 'ok':
        return
    elif 'ch' in msg:
        if 'trade' in msg['ch']:
            await self._trade(msg)
        elif 'depth' in msg['ch']:
            await self._book(msg)
    else:
        LOG.warning("%s: Invalid message type %s", self.id, msg)

  5. Optimize callback performance

    master

    To prevent impacting the performance of the cryptofeed event loop, follow these best practices:

    • Avoid heavy computation: Do not perform computationally intensive tasks directly inside a callback.
    • Offload work: Quickly process data and pass it to another process, application, or a backend callback.
    • Use Async: Whenever possible, use asynchronous libraries within your callbacks to avoid blocking the loop.

  6. Optimize Cryptofeed performance for book data

    master

    Cryptofeed uses asyncio to multitask during I/O operations. However, CPU-intensive tasks like message parsing or heavy callback logic will block the event loop. Performance issues are most common when handling book data channels, which are high-throughput. To maintain low latency:

    • Scale horizontally with multiprocessing: A single process has limits. For large-scale setups (e.g., book data for hundreds of symbols), you must use multiple processes.
    • Distribute feeds: Instead of one massive configuration, break up book subscriptions into multiple calls to add_feed. Each call to add_feed creates at least one new asyncio task, helping distribute the load.
    • Minimize callback latency: User-defined callbacks increase latency. Keep them as lightweight as possible and use asyncio within them where applicable.
    • Manage book depth: Enforcing a max_depth on a book increases the computation time required for processing.
    • Avoid unsupported deltas: Using delta updates on exchanges that do not natively support them (e.g., Huobi) increases processing overhead.

  7. Configure channel and symbol mappings for a new exchange

    master

    Cryptofeed uses standardized channel names (e.g., TRADES, L2_BOOK) which must be mapped to exchange-specific strings.

    1. Channel Mapping: In cryptofeed.exchange.standards.py, update _feed_to_exchange_map to map standardized constants to the exchange's specific channel strings.

      • Example: TRADES: { HUOBI: 'trade.detail' } or L2_BOOK: { HUOBI: 'depth.step0' }.

    2. Symbol Mapping: Define a symbol_endpoint class variable in your Feed class pointing to the exchange's REST API for symbol information. Implement the @classmethod _parse_symbol_data(cls, data: dict, symbol_separator: str) to return a dictionary mapping normalized symbols (e.g., BTC/USD) to exchange-specific symbols (e.g., btcusd).

  8. Install Cryptofeed via Pip

    master

    The recommended way to install or upgrade the Cryptofeed library is using pip. You can install the base library without backend-specific dependencies to minimize the installation footprint.

    pip install --user --upgrade cryptofeed

  9. Install cryptofeed

    master

    Cryptofeed requires Python 3.8+. You can install it via PyPi. It is recommended to use a virtual environment.

    To install the base package:

    pip install cryptofeed

    To install cryptofeed along with all optional dependencies for various backends:

    pip install cryptofeed[all]

    To install from source:

    python setup.py install

    To install in development ('edit') mode:

    python setup.py develop
    pip install cryptofeed[all]

  10. Install Cryptofeed with optional backend dependencies

    master

    Cryptofeed supports various backends (e.g., Redis, MongoDB, Kafka) via optional dependencies. To avoid installing all dependencies at once, you can install only the specific backend you need using the bracket syntax.

    # Install all optional dependencies in one bundle
pip install --user --upgrade cryptofeed[all]

# Install specific backends
pip install --user --upgrade cryptofeed[arctic]
pip install --user --upgrade cryptofeed[gcp_pubsub]
pip install --user --upgrade cryptofeed[kafka]
pip install --user --upgrade cryptofeed[mongo]
pip install --user --upgrade cryptofeed[postgres]
pip install --user --upgrade cryptofeed[quasardb]
pip install --user --upgrade cryptofeed[rabbit]
pip install --user --upgrade cryptofeed[redis]
pip install --user --upgrade cryptofeed[zmq]

  11. Basic Usage: Subscribe to exchange data feeds

    master

    To use cryptofeed, create a FeedHandler object and add subscriptions using exchange classes. You can provide user-defined callback functions for specific data channels. The handler will then call these functions when updates are received.

    from cryptofeed import FeedHandler
# not all imports shown for clarity

fh = FeedHandler()

# ticker, trade, and book are user defined functions that
# will be called when ticker, trade and book updates are received
ticker_cb = {TICKER: ticker}
trade_cb = {TRADES: trade}
gemini_cb = {TRADES: trade, L2_BOOK: book}

fh.add_feed(Coinbase(symbols=['BTC-USD'], channels=[TICKER], callbacks=ticker_cb))
fh.add_feed(Bitfinex(symbols=['BTC-USD'], channels=[TICKER], callbacks=ticker_cb))
fh.add_feed(Poloniex(symbols=['BTC-USDT'], channels=[TRADES], callbacks=trade_cb))
fh.add_feed(Gemini(symbols=['BTC-USD', 'ETH-USD'], channels=[TRADES, L2_BOOK], callbacks=gemini_cb))

fh.run()

  12. Use Callbacks to handle asynchronous data

    master

    Cryptofeed uses asyncio to handle asynchronous events. When you call fh.run(), the main thread blocks until an exception occurs or the program is terminated. To process incoming data, you must define callbacks. Only the data types you explicitly register for will be delivered to your callbacks.

    If you call fh.run(start_loop=False), the feedhandler will not start immediately, allowing you to add more tasks or coroutines before manually starting the event loop.