DOCSEARCHbyAlgolia
Submit Docs

SQD Documentation

website·Indexed 35 days ago

https://docs.sqd.dev

Blockchain data APIs and SDKs for streaming onchain data from 120+ networks over HTTP. Features include the Portal API for permissionless data access and the Pipes and Squid TypeScript SDKs for building indexers with support for EVM, Solana, Substrate, Bitcoin, Hyperliquid, and Tron.

Tokens
414.3K
Snippets
1.8K
Records
2.8K
Agent score
68%

What's inside SQD

  1. Overview of SQD Products and Services

    SQD provides a suite of tools for streaming and indexing onchain data from over 120 networks (including Ethereum, Solana, Base, Arbitrum, Bitcoin, and Hyperliquid). The core product offerings include:

    • Portal: An HTTP API for accessing raw blockchain data with support for arbitrary ranges, streaming, and finality handling.
    • SDK: TypeScript libraries used for decoding, transforming, and persisting blockchain data to any database.
    • Cloud: A managed hosting platform for indexers that provides monitoring, scaling, and zero-DevOps infrastructure.
    • Network: A decentralized data lake that powers Portal, offering self-hosting options.

  2. Overview of SubstrateBatchProcessor

    The SubstrateBatchProcessor is a specialized class in the Squid SDK designed for indexing Substrate-based blockchains (such as Polkadot and Kusama). It handles the ingestion, transformation, and persistence of blockchain data, specifically supporting runtime version-aware decoding of events, calls, and extrinsics.

  3. Overview of Network observability components

    The SQD Network utilizes several components to monitor health and performance:

    • Network Observer: Joins the P2P network to track peer visibility, ping counts/durations, reported storage use, missing chunk counts, and assignment timestamps.
    • Collectors: Receive worker heartbeats and query logs via Network transport and store them in ClickHouse.
    • Portal: Exposes /ready and /metrics endpoints, providing signals on requests, dependencies, worker selection, and assignments.

  4. Overview of self-hosting Subsquid Portal

    Subsquid Portal is an open-source tool that allows developers to operate their own self-hosted instance to manage data sources and infrastructure. While self-hosting changes the available datasets, the source of recent data, and the services the operator must maintain, it does not change the basic request structure used by clients to consume data from a Portal endpoint.

  5. Overview of sqd-go

    sqd-go is a compiled Go indexer developed by mev.tools (community-maintained). It streams EVM data from SQD Portal and writes decoded events directly into ClickHouse MergeTree tables. It is ideal for users who want decoded EVM events in ClickHouse without writing schema code, or those who need to derive state (like balances or positions) using compiled Go. It supports checkpoint-based resume and reorg recovery.

  6. Overview of EVM Pipes SDK pipeline components

    An EVM data pipeline built with the SQD Pipes SDK consists of several key components that move data from a blockchain source to a final destination:

    1. Source: The entry point of the pipeline, typically created using evmPortalStream(). It can produce one or more outputs.
    2. Queries: Define what data the source must retrieve. A query is constructed via a chain of calls and finalized with .build().
    3. Per-query transforms: Optional simple transformations applied to the data from a specific query using .pipe().
    4. Whole pipe transformers: Optional complex transformations that can combine data from multiple queries or access batch context (like cursors, loggers, and rollback callbacks).
    5. Termination: The end of the pipeline, which is either an async iterator or a specific data target (e.g., database or file).

  7. Overview of the Squid SDK GraphQL Core API

    The Squid SDK GraphQL server, powered by OpenReader (an implementation of OpenCRUD), provides a GraphQL API for querying indexed data. The API consists of two parts: the Core API, which is automatically generated based on the project's schema file, and optional extensions added via custom resolvers. Currently, the GraphQL server is only compatible with squids that use PostgreSQL as their target database. It is network-agnostic and works with any supported network as long as the database is PostgreSQL.

  8. Overview of SQD Firehose for Subgraphs

    SQD Firehose is an open-source lightweight adapter that runs as a side-car to a graph indexer node. It enables subgraphs to ingest and filter data directly from the SQD Network rather than an RPC endpoint, removing the need to run a full archival node. To handle real-time, unfinalized blocks, it can optionally use a complementary RPC endpoint for seamless ingestion of the most recent blocks.

  9. Overview of migrating from The Graph to Squid SDK

    Migrating from The Graph to Squid SDK involves moving from a per-log handler model to a batch-based processing model. This change typically results in up to a 10x increase in indexing speed due to fewer database hits. While schema files and GraphQL API generation are similar, Squid SDK provides additional flexibility, such as full control over the Postgres database, custom data formats (e.g., CSV), arbitrary code execution in handlers, and secret environment variables for private RPC endpoints.

  10. Overview of Hyperliquid Data API (Portal)

    The Portal is an HTTP API used to read executed data from Hyperliquid. It provides historical and real-time streams of HyperCore perpetual futures fills (including trader address, asset, price, size, side, direction, signed fee, and realized PnL) and HyperEVM blocks, transactions, logs, and traces.

    Important Constraints:

    • It is a read-only data API. You cannot place or cancel orders, access the live order book, or use it as an RPC endpoint for HyperEVM.

  11. Overview of file-store packages

    The @subsquid/file-store packages provide append-only Store implementations for saving Squid SDK generated data to files, primarily for offline analytics. Data is partitioned along the 'block height' dimension. A new partition is created when either the internal buffer (defined by chunkSizeMb) fills up or setForceFlush() is called. The system provides failover guarantees by rolling back to the last successful state recorded in status.txt after a restart.

  12. Overview of SQD Portal for on-chain data extraction

    SQD Portal is an HTTP interface designed for data pipelines, providing a scalable alternative to RPC for extracting large volumes of historical blockchain data. Unlike RPC, which is optimized for transactions, Portal is built for data extraction, handling pagination, finality tracking, and chain reorganizations (reorgs) natively. It returns data as newline-delimited JSON (NDJSON), allowing for constant-memory processing of terabytes of data.