Shovel Indexer

Use the @indexsupply/shovel-config package to programmatically define and generate Shovel JSON configuration files using TypeScript. This allows you to use strongly-typed objects for Source, Table, and Integration definitions instead of writing raw JSON.

To generate the final JSON string, use makeConfig to assemble your sources and integrations, and toJSON to serialize the resulting configuration object.

import { makeConfig, toJSON } from "@indexsupply/shovel-config";
import type { Source, Table, Integration } from "@indexsupply/shovel-config";

const table: Table = {
  name: "transfers",
  columns: [
    { name: "log_addr", type: "bytea" },
    { name: "from", type: "bytea" },
    { name: "to", type: "bytea" },
    { name: "amount", type: "numeric" },
  ],
};

const mainnet: Source = {
  name: "mainnet",
  chain_id: 1,
  url: "https://ethereum.publicnode.com",
};

let integrations: Integration[] = [
  {
    enabled: true,
    name: "transfers",
    sources: [{ name: mainnet.name, start: 0n }],
    table: table,
    block: [
      {
        name: "log_addr",
        column: "log_addr",
        filter_op: "contains",
        filter_arg: ["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"],
      },
    ],
    event: {
      type: "event",
      name: "Transfer",
      inputs: [
        { indexed: true, name: "from", type: "address", column: "from" },
        { indexed: true, name: "to", type: "address", column: "to" },
        { indexed: false, name: "amount", type: "uint256", column: "amount" },
      ],
    },
  },
];

const config = makeConfig({
  pg_url: "postgres:///shovel",
  sources: [mainnet],
  integrations: integrations,
});

console.log(toJSON(config));

An Integration defines how data from specific sources is mapped to a PostgreSQL table.

Key components include:

• table: A Table object defining the name, columns (with PGColumnType), and optional index (an array of IndexStatment strings like ['column_name', 'DESC']).
• event: An Event object describing the ABI-decoded event data to index.
• block: An array of BlockData objects for indexing non-event data (like tx_hash or block_num).
• notification: An optional Notification object specifying which columns to notify on.

const integration = {
  name: 'my-event-integration',
  enabled: true,
  sources: [{ name: 'main-rpc', start: 12345n }],
  table: {
    name: 'my_events',
    columns: [
      { name: 'id', type: 'bigint' },
      { name: 'user_address', type: 'text' }
    ],
    index: [['id', 'ASC']]
  },
  event: {
    name: 'Transfer',
    type: 'event',
    inputs: [
      { name: 'from', type: 'address', column: 'from_addr' },
      { name: 'value', type: 'uint256', column: 'amount' }
    ]
  }
};

Shovel uses a JSON configuration file to define database connections (pg_url), blockchain sources (eth_sources), and data extraction pipelines (integrations).

Key configuration sections include:

• pg_url: The connection string for the target PostgreSQL database.
• eth_sources: An array of blockchain sources, each requiring a name, chain_id, and one or more urls (or a single url).
• integrations: A list of extraction tasks. Each integration defines:
  • sources: Which eth_sources to pull data from (referenced by name).
  • table: The destination table name and its schema (columns).
  • block: Filters applied to block data (e.g., filtering by log_addr using filter_op: "contains").
  • event: The specific EVM event to watch, including its name, type, and mapping of event inputs to database columns.

{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {
        "name": "mainnet",
        "chain_id": 1,
        "urls": ["https://ethereum-rpc.publicnode.com"]
    }
  ],
  "integrations": [{
    "name": "usdc-transfer",
    "enabled": true,
    "sources": [{"name": "mainnet"}],
    "table": {
      "name": "usdc",
      "columns": [
        {"name": "log_addr", "type": "bytea"},
        {"name": "value", "type": "numeric"}
      ]
    },
    "block": [
      {
        "name": "log_addr",
        "column": "log_addr",
        "filter_op": "contains",
        "filter_arg": ["0x...", "0x..."]
      }
    ],
    "event": {
      "name": "Transfer",
      "type": "event",
      "inputs": [
        {"indexed": true, "name": "from", "type": "address", "column": "from"}
      ]
    }
  }]
}