Substreams

Testing for Substreams is categorized into four distinct levels to ensure reliability from individual logic to full pipeline execution:

1. Unit Testing: Focuses on testing individual handlers using the substreams::testing::map! macro.
2. Integration Testing: Validates logic using real blockchain data and test fixtures (e.g., via firecore).
3. End-to-End Testing: Validates the complete pipeline using the substreams CLI.
4. Performance Testing: Benchmarks and optimizes code using Criterion in production mode.

Recommended workflow: Start with comprehensive unit tests, progress to integration tests with real data, and conclude with end-to-end validation.

Substreams is a high-performance blockchain indexing technology developed for The Graph Network. It allows developers to write Rust modules to compose data streams. It is designed for high throughput via parallelization and a streaming-first architecture.

Key features include:

• Low-cost caching and archiving of blockchain data (leveraging StreamingFast Firehose).
• High throughput processing.
• Cursor-based reorg handling.

Substreams is an indexing technology designed to extract, transform, and export blockchain data. It enables developers to:

• Extract data from multiple blockchains including Solana, Ethereum, Polygon, and BNB.
• Apply custom transformations to the raw blockchain data.
• Export data to various destinations, such as a Postgres database or local files.

You can leverage pre-built Substreams packages to target specific smart contracts or protocols without writing extraction logic from scratch. For example, the Uniswap v3 Substreams Package can be used to retrieve Uniswap v3 contract data directly.

The Injective Foundational Modules provide pre-programmed Substreams that retrieve fundamental data from the Injective blockchain. These modules can be used as inputs for your own custom Substreams.

Key modules available in the substreams.yaml manifest include:

• all_transactions: Maps raw Injective blocks (sf.cosmos.type.v2.Block) to a list of transactions (sf.substreams.cosmos.v1.TransactionList).
• all_events: Maps raw Injective blocks (sf.cosmos.type.v2.Block) to a list of events (sf.substreams.cosmos.v1.EventList).
• index_events: An index module that uses all_events to create a cache where events are sorted by their type field (e.g., coin_received, message) to improve performance.
• filtered_events: Uses the index_events cache to filter for specific event types. It accepts a string parameter to specify which event types to consume.

A Substreams manifest (substreams.yaml) is a YAML configuration file that defines the properties of a Substreams module. It serves three primary purposes:

1. Module Definition: It specifies the modules used, their types (map or store), their inputs, and their outputs.
2. Dependency Management: It identifies dependencies between module inputs and outputs, as well as third-party package imports.
3. Data Modeling: It defines the schemas (Protobuf definitions) used by the module and how the resulting data should be consumed (e.g., via SQL sinks).

Example of a basic manifest structure:

modules:
  - name: map_block
    kind: map
    initialBlock: 12287507
    inputs:
      - source: sf.ethereum.type.v2.Block
    output:
      type: proto:eth.example.MyBlock

A Foundational Store is a high-performance, multi-backend key-value storage system designed for Substreams ingestion and serving. It is typically used to store data extracted and transformed by Substreams modules so that other modules can perform efficient lookups.

Key features include:

• Fork-aware storage: Automatically handles blockchain reorganizations.
• Multiple backends: Supports Badger (embedded) and PostgreSQL.
• Block-level versioning: Every entry is tagged with a block number to support historical queries.
• High-performance serving: Provides a gRPC API for data retrieval.
• Streaming ingestion: Supports continuous processing via a Substreams sink.

Delta updates allow you to perform atomic database operations like increments, decrements, and conditional updates directly at the database level. This eliminates the need for a 'read-modify-write' pattern (fetching a value, computing a new one, and writing it back).

Benefits:

• Performance: Reduces overhead by avoiding expensive reads for every update.
• Concurrency: Prevents race conditions by letting the database handle computation atomically.

Note: Delta updates are currently a PostgreSQL-only feature. ClickHouse is not supported.

A robust sink must handle both incoming data and blockchain forks (reorganizations).

Processing Data (HandleBlockScopedData)

This required handler is called for each block's data. You are responsible for:

• Decoding and processing the data.
• Saving the data to your storage.
• Persisting the cursor: Save the provided *Cursor regularly as a checkpoint to allow recovery from interruptions.

Handling Reorganizations (HandleBlockUndoSignal)

This required handler is called when a blockchain reorganization occurs. The undoSignal contains the LastValidBlock.

CRITICAL: You must treat all data where BlockScopedData.Clock.Number > LastValidBlock.Number as invalid. You must remove or delete this data from your storage to maintain consistency with the canonical chain.