Apibara Direct Node Access (DNA)
repository·main·Indexed 42 days ago
https://github.com/apibara/dnaA protocol and SDK suite for streaming onchain data directly into applications with high efficiency. It includes a Rust-based SDK featuring a StreamClient for gRPC communication, a beaconchain service for Beacon Chain data ingestion, and tools for observability via OpenTelemetry.
What's inside Apibara DNA
- Apibara Direct Node Access (DNA) is a protocol designed to enable developers to stream onchain data directly and efficiently into their applications. The repository provides the canonical implementation of the DNA protocol along with various integrations built on top of it.
Understand Quint specifications for DNA subsystems
mainThespecdirectory contains Quint specifications used to model and verify various DNA subsystems. Currently, it includes specifications forreorg_detection, which models a system for detecting chain reorganizations (reorgs) using the JSON-RPC API.Use the Apibara DNA Protocol gRPC types
mainTheapibara-dna-protocolcrate provides the necessary gRPC type definitions required to interact with the Apibara DNA Protocol. Developers should use this crate when building clients or services that need to communicate with the protocol via gRPC.Initialize observability with `init_opentelemetry`
mainTo enable observability in your application using
tracingandopentelemetry, call theinit_opentelemetryfunction at the very beginning of your program's execution. This sets up the necessary telemetry pipelines for tracking application behavior.// Invoke this at the start of your program init_opentelemetry();Stream data using StreamDataRequestBuilder
mainTo receive a continuous stream of
DnaMessageobjects, follow these steps:- Use
StreamDataRequestBuilder::new()to initialize a request. - Configure the request using
.with_starting_cursor(start_block)to define where the stream begins. - Add filters using
.add_filter(). Filters can be constructed usingstarknet::FilterBuilderandstarknet::EventFilterBuilderto target specific contract addresses, event keys, or transactions. - Call
client.stream_data(request).await?to obtain a stream. - Iterate over the stream using
.try_next().await?to process eachDnaMessage.
let usdc_address = starknet::FieldElement::from_hex( "0x053c91253bc9682c04929ca02ed00b3e423f6710d2ee7e0d5ebb06f3ecf368a8", )?; let transfer_hash = starknet::FieldElement::from_hex( "0x99cd8bde557814842a3121e8ddfd433a539b8c9f14bf31ebf108d12e6196e9", )?; let request = StreamDataRequestBuilder::new() .with_starting_cursor(start_block) .add_filter( starknet::FilterBuilder::new() .add_event( starknet::EventFilterBuilder::single_contract(usdc_address) .with_keys(true, vec![Some(transfer_hash)]) .with_transaction() .build(), ) .build(), ) .build(); let mut stream = client.stream_data(request).await?; while let Some(message) = stream.try_next().await? { // Process the message here }- Use
Set up the Apibara DNA development environment
mainApibara DNA is developed using stable Rust. To simplify the installation of all required dependencies, the project provides a Nix environment.
- If you have
nixinstalled, runnix developto enter the development environment. - If you do not use
nix, you must install the Rust toolchain manually using your preferred method.
nix develop- If you have
Connect to Apibara DNA services using StreamClient
mainTo interact with Apibara DNA services, create a
StreamClientusing theStreamClient::builder(). You must provide a service URL. If the service requires authentication, use.with_bearer_token_provider()with a provider likeBearerTokenFromEnv(which reads from theDNA_TOKENenvironment variable).let url: Uri = my_url .parse()?; let mut client = StreamClient::builder() .with_bearer_token_provider(Arc::new(BearerTokenFromEnv::default())) .connect(url) .await?;Use BlockId to specify Starknet blocks
mainThe
BlockIdenum is used in provider methods to specify which block to query. It maps to Starknet's native block identifiers:BlockId::Head: The latest block.BlockId::Pending: The pending/pre-confirmed block.BlockId::Number(u64): A specific block number.BlockId::Hash(FieldElement): A specific block hash.BlockId::Finalized: The L1-accepted/finalized block.
// Example usage with a specific block number let block_id = BlockId::Number(12345);Handle reconnection actions and reorgs
mainThe
ReconnectActionenum describes the relationship between an external state (e.g., after being offline) and the current canonical chain segment.ReconnectAction::Continue: The external cursor is already part of the canonical chain.ReconnectAction::OfflineReorg(Cursor): An offline reorg occurred. The providedCursoris the point in the chain where the external state and the current canonical chain now converge.ReconnectAction::Unknown: The external cursor does not match any known canonical or reorged block in this segment.
Use helper methods to check the state:
is_continue()is_offline_reorg()is_unknown()as_offline_reorg_cursor()(returnsOption<Cursor>)
Starknet Block Ingestion Data Structure
mainWhen ingesting Starknet blocks, the system processes a
StateDiffand produces aBlockIngestionResult. This result is composed of three primary data categories, each containing a body (the raw data), an index (for efficient lookups), and a join fragment:- Storage Diffs: Contains
storage_diffs. It is indexed bycontract_addressusingINDEX_STORAGE_DIFF_BY_CONTRACT_ADDRESS. - Contract Changes: Contains
contract_changes(includingDeclaredClass,ReplacedClass, andDeployedContract). It is indexed by change type usingINDEX_CONTRACT_CHANGE_BY_TYPE. - Nonce Updates: Contains
nonces. It is indexed bycontract_addressusingINDEX_NONCE_UPDATE_BY_CONTRACT_ADDRESS.
Each category uses specific fragment IDs and names (e.g.,
STORAGE_DIFF_FRAGMENT_ID,CONTRACT_CHANGE_FRAGMENT_ID,NONCE_UPDATE_FRAGMENT_ID) to organize the data intoBodyFragment,IndexFragment, andJoinFragmentstructures.- Storage Diffs: Contains
Configure header filtering behavior in BlockFilter
mainThe
BlockFilterstruct allows you to control when headers are included in data retrieval using theHeaderFilterenum. This is useful for determining if a block should be processed based on whether it contains data or is a new block.Available
HeaderFiltervariants:Always: The header is always included.OnData: The header is included only if data is present.OnDataOrOnNewBlock(Default): The header is included if data is present or if it represents a new block.
use apibara_dna_common::query::{BlockFilter, HeaderFilter}; let mut block_filter = BlockFilter::default(); block_filter.set_header_filter(HeaderFilter::Always);How Starknet FilterBuilders work together
mainThe Starknet SDK uses a hierarchical builder pattern to construct complex data filters. The
FilterBuilderacts as the root, allowing you to compose multiple specialized filters into a single request.To build a filter, you typically:
- Initialize a
FilterBuilderusingFilterBuilder::new(). - Use specialized builders (like
TransactionFilterBuilderorEventFilterBuilder) to create specific filter types. - Add these specialized filters to the root builder using methods like
add_transactionoradd_event. - Call
.build()on the root builder to produce the finalFilterobject.
This composition allows you to request transactions, events, messages to L1, storage diffs, contract changes, and nonce updates in a single unified stream.
let filter = FilterBuilder::new() .add_transaction(TransactionFilterBuilder::only_invoke_v3().build()) .add_event(EventFilterBuilder::single_contract(contract_address).build()) .build();- Initialize a