Apibara Direct Node Access (DNA)

repository·main·Indexed 42 days ago

https://github.com/apibara/dna

A protocol and SDK suite for streaming onchain data directly into applications with high efficiency. It includes a Rust-based SDK featuring a StreamClient for gRPC communication, a beaconchain service for Beacon Chain data ingestion, and tools for observability via OpenTelemetry.

Tokens
41.4K
Snippets
148
Records
183
Agent score
52%

What's inside Apibara DNA

  1. Overview of Apibara Direct Node Access (DNA)

    main
    Apibara Direct Node Access (DNA) is a protocol designed to enable developers to stream onchain data directly and efficiently into their applications. The repository provides the canonical implementation of the DNA protocol along with various integrations built on top of it.
  2. Understand Quint specifications for DNA subsystems

    main
    The spec directory contains Quint specifications used to model and verify various DNA subsystems. Currently, it includes specifications for reorg_detection, which models a system for detecting chain reorganizations (reorgs) using the JSON-RPC API.
  3. Use the Apibara DNA Protocol gRPC types

    main
    The apibara-dna-protocol crate provides the necessary gRPC type definitions required to interact with the Apibara DNA Protocol. Developers should use this crate when building clients or services that need to communicate with the protocol via gRPC.
  4. Initialize observability with `init_opentelemetry`

    main

    To enable observability in your application using tracing and opentelemetry, call the init_opentelemetry function at the very beginning of your program's execution. This sets up the necessary telemetry pipelines for tracking application behavior.

    // Invoke this at the start of your program
    init_opentelemetry();
  5. Stream data using StreamDataRequestBuilder

    main

    To receive a continuous stream of DnaMessage objects, follow these steps:

    1. Use StreamDataRequestBuilder::new() to initialize a request.
    2. Configure the request using .with_starting_cursor(start_block) to define where the stream begins.
    3. Add filters using .add_filter(). Filters can be constructed using starknet::FilterBuilder and starknet::EventFilterBuilder to target specific contract addresses, event keys, or transactions.
    4. Call client.stream_data(request).await? to obtain a stream.
    5. Iterate over the stream using .try_next().await? to process each DnaMessage.
    let usdc_address = starknet::FieldElement::from_hex(
        "0x053c91253bc9682c04929ca02ed00b3e423f6710d2ee7e0d5ebb06f3ecf368a8",
    )?;
    let transfer_hash = starknet::FieldElement::from_hex(
        "0x99cd8bde557814842a3121e8ddfd433a539b8c9f14bf31ebf108d12e6196e9",
    )?;
    let request = StreamDataRequestBuilder::new()
        .with_starting_cursor(start_block)
        .add_filter(
            starknet::FilterBuilder::new()
                .add_event(
                    starknet::EventFilterBuilder::single_contract(usdc_address)
                        .with_keys(true, vec![Some(transfer_hash)])
                        .with_transaction()
                        .build(),
                )
                .build(),
        )
        .build();
    
    let mut stream = client.stream_data(request).await?;
    
    while let Some(message) = stream.try_next().await? {
        // Process the message here
    }
  6. Set up the Apibara DNA development environment

    main

    Apibara DNA is developed using stable Rust. To simplify the installation of all required dependencies, the project provides a Nix environment.

    • If you have nix installed, run nix develop to enter the development environment.
    • If you do not use nix, you must install the Rust toolchain manually using your preferred method.
    nix develop
  7. Connect to Apibara DNA services using StreamClient

    main

    To interact with Apibara DNA services, create a StreamClient using the StreamClient::builder(). You must provide a service URL. If the service requires authentication, use .with_bearer_token_provider() with a provider like BearerTokenFromEnv (which reads from the DNA_TOKEN environment variable).

    let url: Uri = my_url
        .parse()?;
    let mut client = StreamClient::builder()
        .with_bearer_token_provider(Arc::new(BearerTokenFromEnv::default()))
        .connect(url)
        .await?;
  8. Use BlockId to specify Starknet blocks

    main

    The BlockId enum is used in provider methods to specify which block to query. It maps to Starknet's native block identifiers:

    • BlockId::Head: The latest block.
    • BlockId::Pending: The pending/pre-confirmed block.
    • BlockId::Number(u64): A specific block number.
    • BlockId::Hash(FieldElement): A specific block hash.
    • BlockId::Finalized: The L1-accepted/finalized block.
    // Example usage with a specific block number
    let block_id = BlockId::Number(12345);
  9. Handle reconnection actions and reorgs

    main

    The ReconnectAction enum describes the relationship between an external state (e.g., after being offline) and the current canonical chain segment.

    • ReconnectAction::Continue: The external cursor is already part of the canonical chain.
    • ReconnectAction::OfflineReorg(Cursor): An offline reorg occurred. The provided Cursor is the point in the chain where the external state and the current canonical chain now converge.
    • ReconnectAction::Unknown: The external cursor does not match any known canonical or reorged block in this segment.

    Use helper methods to check the state:

    • is_continue()
    • is_offline_reorg()
    • is_unknown()
    • as_offline_reorg_cursor() (returns Option<Cursor>)
  10. Starknet Block Ingestion Data Structure

    main

    When ingesting Starknet blocks, the system processes a StateDiff and produces a BlockIngestionResult. This result is composed of three primary data categories, each containing a body (the raw data), an index (for efficient lookups), and a join fragment:

    1. Storage Diffs: Contains storage_diffs. It is indexed by contract_address using INDEX_STORAGE_DIFF_BY_CONTRACT_ADDRESS.
    2. Contract Changes: Contains contract_changes (including DeclaredClass, ReplacedClass, and DeployedContract). It is indexed by change type using INDEX_CONTRACT_CHANGE_BY_TYPE.
    3. Nonce Updates: Contains nonces. It is indexed by contract_address using INDEX_NONCE_UPDATE_BY_CONTRACT_ADDRESS.

    Each category uses specific fragment IDs and names (e.g., STORAGE_DIFF_FRAGMENT_ID, CONTRACT_CHANGE_FRAGMENT_ID, NONCE_UPDATE_FRAGMENT_ID) to organize the data into BodyFragment, IndexFragment, and JoinFragment structures.

  11. Configure header filtering behavior in BlockFilter

    main

    The BlockFilter struct allows you to control when headers are included in data retrieval using the HeaderFilter enum. This is useful for determining if a block should be processed based on whether it contains data or is a new block.

    Available HeaderFilter variants:

    • Always: The header is always included.
    • OnData: The header is included only if data is present.
    • OnDataOrOnNewBlock (Default): The header is included if data is present or if it represents a new block.
    use apibara_dna_common::query::{BlockFilter, HeaderFilter};
    
    let mut block_filter = BlockFilter::default();
    block_filter.set_header_filter(HeaderFilter::Always);
  12. How Starknet FilterBuilders work together

    main

    The Starknet SDK uses a hierarchical builder pattern to construct complex data filters. The FilterBuilder acts as the root, allowing you to compose multiple specialized filters into a single request.

    To build a filter, you typically:

    1. Initialize a FilterBuilder using FilterBuilder::new().
    2. Use specialized builders (like TransactionFilterBuilder or EventFilterBuilder) to create specific filter types.
    3. Add these specialized filters to the root builder using methods like add_transaction or add_event.
    4. Call .build() on the root builder to produce the final Filter object.

    This composition allows you to request transactions, events, messages to L1, storage diffs, contract changes, and nonce updates in a single unified stream.

    let filter = FilterBuilder::new()
        .add_transaction(TransactionFilterBuilder::only_invoke_v3().build())
        .add_event(EventFilterBuilder::single_contract(contract_address).build())
        .build();