Carbon Framework

The Jito Shredstream gRPC Datasource provides real-time, low-latency access to shreds—fragments of Solana transaction data distributed by validators before they are assembled into full blocks.

⚠️ Critical Usage Warning

Do not use Shredstream for indexing. Shreds are distributed as they are received by the leader and lack transaction status or metadata.

Recommended alternatives for indexing:

• block-subscribe datasource
• block-crawling datasource

The StreamMessageDatasource is a generic, extensible datasource for the Carbon indexing framework. It provides an abstract interface to feed a unified stream of account and transaction messages into Carbon, decoupling the framework from specific transport layers like Kafka, PubSub, or Solana Geyser plugins.

It is designed to consume UnifiedMessage variants, which allow you to mix account updates and transaction updates in a single stream.

pub enum UnifiedMessage {
    Account(AccountUpdate),
    Transaction(Box<TransactionUpdate>),
}

Some examples include interchangeable upstream sources defined in src/variants.rs. The processor and pipeline wiring remain identical across these variants. To swap the source, you must edit the single line in main.rs that constructs the datasource.

Carbon is a Rust framework for building Solana indexers and data pipelines based on a modular three-part pipeline model:

1. Datasources: Stream Solana updates (transactions, account updates, deletions, block metadata) into the runtime. You can swap between RPC streams, Geyser streams, or historical/snapshot sources.
2. Decoders: Transform raw account and transaction data into typed Rust structures. You can use existing decoders for common programs or generate new ones from Anchor or Codama IDLs.
3. Processors: Handle the decoded output. Processors are used to build real-time indexers, historical backfills, or sinks like Postgres, GraphQL, or analytics engines.

This modularity allows you to combine different components (e.g., a Helius datasource with a custom decoder and a Postgres processor) to suit your latency and data requirements.

The gpa-rpc example provides two interchangeable datasource variants for fetching and decoding program account states via getProgramAccounts (gPA):

1. variants::rpc(...): A standard implementation that performs plain getProgramAccounts against any Solana RPC. This is the default variant.
2. variants::helius_gpa_v2(...): Uses Helius's enhanced gPA endpoint. This variant supports pagination and includes changed_since_slot for performing incremental synchronizations.

You can switch between these variants by modifying the datasource initialization in main.rs.

// To use the standard RPC variant
let datasource = variants::rpc(MARGINFI_PROGRAM_ID);

// To use the Helius enhanced variant
let datasource = variants::helius_gpa_v2(MARGINFI_PROGRAM_ID);