DOCSEARCHbyAlgolia
Submit Docs

Polymarket Data Toolkit

repository·main·Indexed 20 days ago

https://github.com/sii-wangzj/polymarket_data

A comprehensive toolkit and dataset for Polymarket prediction markets. It enables developers to fetch trading data from the Polygon blockchain and Gamma API, process it into analysis-ready Parquet formats, and perform quantitative research. The toolkit includes a Python API, a CLI for data management, and a 107GB pre-processed dataset available on HuggingFace containing over 1.1 billion records across orderfilled, trades, markets, quant, and users datasets.

Tokens
5.9K
Snippets
22
Records
27
Agent score
72%

What's inside polymarket_data

  1. Understand the data directory structure

    main

    The data/ directory is the central repository for all fetched and processed Polymarket data. It is organized into the following subdirectories:

    • dataset/: Contains the primary high-volume data stored in Parquet format (e.g., orderfilled_*.parquet, trades_*.parquet, quant_*.parquet, users_*.parquet).
    • latest_result/: Contains human-readable CSV previews of the most recent 1000 records for each data type.
    • data_clean/: Stores processed and cleaned versions of the datasets.
    • state.json: A checkpoint file used to track the last processed block number for resuming operations.
    data/
├── dataset/              # Main dataset files
│   ├── orderfilled_*.parquet
│   ├── trades_*.parquet
│   ├── quant_*.parquet
│   └── users_*.parquet
├── latest_result/        # Latest 1000 records (CSV preview)
│   ├── orderfilled.csv
│   ├── trades.csv
│   ├── quant.csv
│   └── users.csv
├── data_clean/           # Cleaned data files
└── state.json            # Checkpoint state file

  2. Analyze market statistics and price evolution with Python

    main

    You can perform quantitative analysis on the exported Parquet files using pandas.

    Common analysis tasks include:

    • Market Statistics: Aggregating usd_amount, price, and transaction_hash grouped by market_id using quant.parquet.
    • Price Evolution: Filtering quant.parquet by a specific market_id and sorting by datetime to plot price trends over time.
    • User Behavior: Analyzing users.parquet to calculate net positions (token_amount) and total volume (usd_amount) per user and market.
    • Volume Analysis: Merging quant.parquet with markets.parquet on market_id to associate trade volume with market questions.
    import pandas as pd

# Example: Calculate Market Statistics
df = pd.read_parquet('quant.parquet')
market_stats = df.groupby('market_id').agg({
    'usd_amount': ['sum', 'mean'],
    'price': ['mean', 'std', 'min', 'max'],
    'transaction_hash': 'count'
}).round(4)
print(market_stats.head())

  3. Run Continuous Real-time Data Fetching

    main

    To keep your data synchronized with the Polygon blockchain 24/7, use the continuous mode. This mode automatically switches between batch mode (fetching 100 blocks at once when behind) and real-time mode (fetching 1 block every 2 seconds when caught up).

    # Start continuous fetching
./scripts/continuous_start.sh

# View logs
tail -f logs/continuous_fetch.log

# Stop gracefully
./scripts/continuous_stop.sh

    Features:

    • Auto-data cleaning: Generates 4 parquet files in real-time.
    • Graceful shutdown: Ensures files are properly closed on exit.
    • Resume support: Auto-saves progress to prevent data loss.

  4. Install Polymarket Data toolkit

    main

    To use the toolkit, clone the repository and install the dependencies using pip. You can install it as a regular dependency or in editable mode for development.

    # Clone repository
git clone https://github.com/SII-WANGZJ/Polymarket_data.git
cd Polymarket_data

# Install dependencies
pip install -r requirements.txt

# Or install as package
pip install -e .
    git clone https://github.com/SII-WANGZJ/Polymarket_data.git
cd Polymarket_data

pip install -r requirements.txt

# Or install as package
pip install -e .

  5. Run the Full Data Pipeline

    main

    The full pipeline follows the workflow: Fetch markets $\rightarrow$ Fetch on-chain $\rightarrow$ Process data.

    You can run the entire sequence with one script:

    ./scripts/update_all.sh

    Or execute the steps individually:

    ./scripts/fetch_markets.sh        # Fetch market metadata
./scripts/fetch_onchain.sh 5000   # Fetch on-chain data (argument is block count)
./scripts/clean_data.sh           # Clean and process data
    # Run full pipeline
./scripts/update_all.sh

# Or step by step
./scripts/fetch_markets.sh        # Fetch market metadata
./scripts/fetch_onchain.sh 5000   # Fetch on-chain data
./scripts/clean_data.sh           # Clean and process data

  6. Download the Polymarket dataset from HuggingFace

    main

    You can download the pre-processed 107GB dataset from HuggingFace. It is recommended to install the huggingface_hub CLI first.

    To download a specific file (e.g., quant.parquet):

    hf download SII-WANGZJ/Polymarket_data quant.parquet --repo-type dataset

    To download the entire dataset:

    hf download SII-WANGZJ/Polymarket_data --repo-type dataset
    pip install huggingface_hub

# Download specific file
hf download SII-WANGZJ/Polymarket_data quant.parquet --repo-type dataset

# Download all files
hf download SII-WANGZJ/Polymarket_data --repo-type dataset

  7. Manage and back up data files

    main
    All files within the data/ directory (excluding .gitkeep and the README.md) are ignored by Git. Because these files are not tracked in version control, you must manually back them up or upload them to external data repositories such as HuggingFace for persistence and sharing.

  8. Configure RPC endpoints and API keys

    main

    To optimize data fetching, you can configure RPC endpoints and API keys.

    Alchemy API Key Set the ALCHEMY_API_KEY environment variable for faster RPC access:

    export ALCHEMY_API_KEY=your_key_here

    Custom RPC Endpoints To use custom Polygon RPC nodes, edit the RPC_ENDPOINTS list in polymarket/config.py:

    RPC_ENDPOINTS = [
    "https://polygon-rpc.com",
    "your_custom_endpoint",
]

  9. Use the poly_onchain CLI to manage data

    main

    The poly_onchain CLI tool provides several commands to fetch, update, and process Polymarket data. You can run these commands using the Python module interface.

    Available Commands:

    • fetch-onchain: Fetches on-chain event data incrementally.
    • fetch-markets: Fetches new market data from the Gamma API.
    • process: Processes existing data (standard mode).
    • update: Updates the status of unclosed markets.
    • process-historical: Processes large historical event files in batches to avoid memory issues.
    # Fetch on-chain data for the last 1000 blocks
python -m poly_onchain.cli fetch-onchain --blocks 1000

# Fetch new markets
python -m poly_onchain.cli fetch-markets

# Process data
python -m poly_onchain.cli process

# Update markets
python -m poly_onchain.cli update

# Process historical data in batches
python -m poly_onchain.cli process-historical --batch-size 1000000

  10. Use the Polymarket Python API

    main

    You can integrate the toolkit directly into your Python applications using the polymarket package.

    from polymarket import LogFetcher, EventDecoder, extract_trades
from polymarket import load_token_mapping

# 1. Fetch on-chain logs
fetcher = LogFetcher()
logs = fetcher.fetch_range_in_batches(start_block, end_block)

# 2. Decode events
decoder = EventDecoder()
decoded = decoder.decode_batch(logs)
events = decoder.format_batch(decoded)

# 3. Load token mapping and extract trades
token_mapping = load_token_mapping()
trades_df = extract_trades(events, token_mapping)

# 4. Save to parquet
trades_df.to_parquet('trades.parquet')

  11. Reference the available data file formats

    main

    The project utilizes three main file formats for data storage and state management:

    1. Parquet Files: Compressed columnar files used for the main datasets. These are generated via continuous fetching or batch processing and use timestamped filenames in the format <type>_YYYYMMDD_HHMMSS.parquet.
    2. CSV Preview Files: Human-readable files located in latest_result/ that provide a snapshot of the latest 1000 records. These are updated in real-time during continuous fetching.
    3. State File (state.json): A JSON file that tracks the last processed block number, enabling the system to resume from a checkpoint after an interruption.

  12. Reference: Dataset File Overview

    main

    The toolkit produces or provides 5 primary analysis-ready datasets in Parquet format:

    FileSizeRecordsDescription
    orderfilled.parquet31GB293.3MRaw blockchain events from OrderFilled logs
    trades.parquet32GB293.3MProcessed trades with market metadata linkage
    markets.parquet68MB268,706Market information and metadata
    quant.parquet21GB170.3MClean market data with unified YES perspective
    users.parquet23GB340.6MUser behavior data split by maker/taker roles

    Total: 107GB, 1.1 billion records