DOCSEARCHbyAlgolia
Submit Docs

Graph Node

repository·master·Indexed 25 days ago

https://github.com/graphprotocol/graph-node

A core component of The Graph protocol used for indexing blockchain data and serving it via GraphQL. Includes documentation on running Graph Node via Docker Compose or standalone images, and the gnd (Graph Node Dev CLI) for initializing, building, testing, and deploying subgraphs.

Tokens
67.7K
Snippets
132
Records
414
Agent score
84%

What's inside graph-node

  1. Overview of gnd (Graph Node Dev) CLI

    master

    The gnd CLI is a Rust-based tool designed to be a drop-in replacement for the TypeScript-based graph-cli. It provides functionality for managing subgraphs, including scaffolding, building, deploying, and publishing. It is currently optimized for the Ethereum protocol.

    Key features include:

    • Identical CLI behavior, flags, and output format to graph-cli (v0.98.x).
    • Byte-for-byte identical AssemblyScript code generation.
    • Dev mode with file watching via gnd dev.

  2. Understand the Parquet Dump/Restore Format

    master

    The Parquet Dump/Restore mechanism allows for exporting subgraph entity data from PostgreSQL into a file-based format for backup, migration, or sharing. The dump consists of a directory containing metadata, the GraphQL schema, the subgraph manifest, and Parquet files for each entity type.

    Directory Layout

    <dump-dir>/
  metadata.json                  -- deployment metadata + per-table state
  schema.graphql                 -- raw GraphQL schema text
  subgraph.yaml                  -- raw subgraph manifest YAML (optional)
  <EntityType>/
    chunk_000000.parquet         -- rows ordered by vid
    chunk_000001.parquet         -- incremental append
    ...
  data_sources$/
    chunk_000000.parquet         -- dynamic data sources

    Key Components

    • metadata.json: Contains deployment metadata (version, network, deployment hash), manifest details, block information (start_block, head_block), health status, and a map of tables including their chunk files and max_vid.
    • schema.graphql: The raw GraphQL schema used to reconstruct the relational layout.
    • subgraph.yaml: The raw subgraph manifest.
    • Entity Tables: Each entity type has its own directory containing one or more .parquet files. Incremental dumps append new chunk files rather than rewriting existing ones.
    • data_sources$: A special table for dynamic data sources, dumped in its own directory.

  3. Use gnd as a replacement for graph-cli

    master

    gnd is a drop-in replacement for graph-cli designed for Graph Node development. While most commands are compatible, note the following differences:

    Commands Not Implemented

    • local: Instead of graph-cli local, use graph-node's integration test infrastructure.
    • node: Instead of graph-cli node, use graphman for node management operations.

    Code Generation Differences

    gnd produces slightly different AssemblyScript code compared to graph-cli:

    • Always imports Int8 for simplicity.
    • Uses trailing commas in multi-line constructs.
    • Uses toStringMatrix() for 2D array accessors (fixing a known bug in graph-cli).

    Other Differences

    • Debug logging: Uses the RUST_LOG environment variable instead of DEBUG=graph-cli:*.
    • --uncrashable flag: This flag is not implemented.

  4. Understand Time-travel queries and entity versioning

    master

    Time-travel queries allow you to query the state of a subgraph at a specific block height. Instead of overwriting data, Graph Node uses an immutable versioning approach where each entity version is stored as a separate row in the database with a block_range column.

    Key Concepts:

    • block_range: An int4range indicating the inclusive lower bound (start block) and exclusive upper bound (end block) for which an entity version is valid.
    • Current Version: The most recent version of an entity has a block range with an unlimited upper bound (e.g., [B, )).
    • Exclusion Constraint: A database constraint ensures that block ranges for any specific entity ID do not overlap.
    • Immutable Entities: Entities declared with @entity(immutable: true) use a block$ int column instead of block_range. For these, the upper bound is always infinite, and queries check if block$ <= $B.

  5. Understand SQL Query Generation for Nested GraphQL Queries

    master
    Graph Node optimizes nested GraphQL queries (e.g., fetching parents and their children) by generating exactly two SQL queries: one for the parents and one for the children. To ensure children are correctly associated with their specific parents, the second query uses filters (like parent_id) and LATERAL JOINs to apply pagination (first/skip) and sorting individually to each parent's set of children.

  6. Use the gnd CLI for subgraph development

    master

    The gnd (Graph Node Dev) CLI is a Rust-based, drop-in replacement for the TypeScript-based graph-cli. It is designed to support subgraph development workflows with identical commands, flags, output formats, and exit codes.

    Note on Debugging: Instead of using DEBUG=graph-cli:*, use the RUST_LOG environment variable to control debug output.

    gnd <command> [options] [arguments]

  7. Understand the Parquet Dump/Restore data model

    master

    The Parquet dump/restore system is designed to move subgraph data between environments.

    Key Components:

    • Metadata: Uses metadata.json containing information like Manifest, BlockPtr, Health, and Error states.
    • Data Format: Entity data is stored in Parquet chunks.
    • Schema: The schema is reconstructed from schema.graphql and the spec_version found in the metadata.
    • Vid Continuity: The system preserves original vid values during restore and resets the database sequence to max_vid + 1 to allow for seamless incremental updates.

    Implementation Note: The system uses an OidValue-based dynamic column strategy to handle typed extraction from PostgreSQL without requiring separate connections or JSON intermediate steps.

  8. Understand GraphQL to Postgres Schema Generation

    master

    Graph Node converts GraphQL schemas into relational Postgres tables. Data for a subgraph is stored in a dedicated Postgres namespace named sgdNNNN (the mapping is stored in deployment_schemas).

    Key Schema Rules:

    • Entity Types: Each entity type is stored in its own table.
    • Enums: GraphQL enums are stored as Postgres enum types.
    • Interfaces: Interfaces are not stored; only the concrete types implementing them are persisted.
    • Attributes: GraphQL attributes map to table columns.
      • ID, String, and Bytes map to text or bytea.
      • BigDecimal and BigInt map to numeric.
      • Lists (e.g., [String]) map to Postgres array types (nested arrays like [[String]] are not supported).
      • Entity references map to the id type of the referenced entity. Note: Foreign key constraints are not used to allow for out-of-order entity creation.
    -- Example of a generated table structure
create table sgd42.account(
    vid int8    serial primary key,
    id          text not null, -- or bytea
    -- .. attributes ..
    block_range int4range not null
)

  9. Understand the Subgraph Manifest structure

    master

    The subgraph manifest is the entry point for a subgraph, specifying all information required to index and query it. It can be defined using any data format with a 1:1 mapping to the IPLD Canonical Format, such as YAML or JSON. The manifest and its linked files are hashed to produce a unique subgraph ID.

    Top-Level API Fields:

    • specVersion: Semver version of the API.
    • schema: The GraphQL schema (points to a file path).
    • description: Optional description of the subgraph.
    • repository: Optional link to the source code.
    • graft: Optional Graft Base to build upon an existing subgraph.
    • dataSources: Definitions for data ingestion and transformation logic.
    • templates: Definitions for dynamic data sources.
    • features: A list of required feature names (e.g., grafting, fullTextSearch).

  10. Configure subgraph indexing log storage backends

    master

    Graph Node supports several backends for storing subgraph indexing logs (user-generated mapping logs, runtime logs, and system logs). All backends are accessible via a unified GraphQL query interface.

    Available backends:

    • File: Stores logs as JSON Lines files on the local filesystem. Best for local development.
    • Elasticsearch: Enterprise-grade search and analytics. Recommended for production.
    • Loki: Grafana's lightweight log aggregation system. Recommended for production.
    • Disabled: No log storage (default). Logs will still appear in stdout/stderr but will not be available via the _logs GraphQL query.

  11. Query entities at a specific point-in-time

    master

    To query the state of an entity at a specific block height $B, you must include a condition in your SQL query to ensure the entity's block_range contains that block.

    For standard entities: Use the @> operator to check if the range contains the block.

    For immutable entities (@entity(immutable: true)): Use the block$ column to check if the block is greater than or equal to the entity's creation block.

  12. Set up test dependencies manually

    master

    If you are not using Docker Compose, follow these steps to configure a local Postgres instance for testing:

    1. Create User: Create a database user graph with the password graph using createuser -W graph.
    2. Configure Authentication: Ensure psql and related tools can connect automatically as the postgres user.
      • Add local all postgres peer map=admin as the first non-comment line to $PGDATA/pg_hba.conf.
      • Add the following to data/pg_ident.conf:
        admin           postgres                postgres
admin           <your username>         postgres
    3. Enable SQL Logging (Optional): To debug failing tests, set log_statement = 'all' in $PGDATA/postgresql.conf to log all SQL statements sent by tests.
    4. Restart: Restart Postgres to apply changes.

    Resetting Databases: Use the db-reset script to (re)create databases. Note that db-reset will completely delete the graph-test and graph-sgd databases in the cluster that your psql connects to.