Jetstreamer

Jetstreamer is a high-throughput Solana backfilling and research toolkit. It streams historical chain data from the Project Yellowstone Old Faithful archive (which contains all Solana blocks and transactions from genesis to the current tip) to local Jetstreamer or Geyser plugins.

With high-end hardware (e.g., 64-core CPU, 30 Gbps+ network), it can achieve speeds exceeding 2.7M TPS. Note that Jetstreamer replays transactions in their 'already-executed' state as they originally appeared to Geyser; it does not execute them directly.

The jetstreamer-firehose threaded firehose uses several mechanisms to manage HTTP connections and cope with CDN throttling:

• Health-gated launch: Threads are spawned in batches. If a thread is stalled (marked as red), the ramp process freezes until the thread recovers.
• Exponential restart backoff: When a thread fails, it waits for an increasing duration (1s → 32s) before reconnecting to prevent reconnect storms during throttling events.
• Connection recycling (♻️): Threads that perform significantly below the fleet's p90 rate reconnect to shed throughput-clamped connections. This is done gradually to avoid uniform clamping.
• Work stealing (🥷): If a thread finishes its assigned slot range, it can request half of the remaining work from the least-progressed thread via a steal inbox. This ensures all connections remain busy until the entire range is complete.

When planning replay windows, be aware that metadata availability in Old Faithful ledger snapshots depends on the epoch.

• Geyser Compatibility: Epochs 157+ are compatible with modern Geyser plugins. Epochs 0-156 are incompatible with modern Geyser but work with the firehose interface and JetstreamerRunner.
• Compute Unit (CU) Tracking: CU tracking is available from epoch 450+ (slot 194184611+). For epochs 0-449, CU tracking is reported as 0.

Jetstreamer is organized into four crates. The main jetstreamer crate re-exports the others for convenience:

• jetstreamer: The primary facade that connects firehose ingestion to plugins using JetstreamerRunner.
• jetstreamer-firehose: Async helpers for downloading, compacting, and replaying Old Faithful CAR archives at scale.
• jetstreamer-plugin: A trait-based framework for building structured observers with ClickHouse-friendly batching and runtime metrics.
• jetstreamer-utils: Utility functions used across the ecosystem.

To access sub-crate functionality through the main crate, use these re-exports:

• jetstreamer::firehose
• jetstreamer::plugin
• jetstreamer::utils

jetstreamer-firehose implements two primary mechanisms to ensure data completeness during replay:

1. Premature-EOF detection: If an HTTP stream closes unexpectedly, the reader checks the slot-ranges index. If any slots in the thread's assigned slice are still missing, the stream is considered truncated and the range is restarted.
2. End-of-run coverage audit: Once all threads finish, the tool verifies the union of all processed intervals against the requested range. It checks for gaps against the slot index (allowing for genuine leader-skip regions). A successful run logs coverage audit passed; otherwise, it reports the exact slot range containing holes.

Jetstreamer's data availability is limited by the contents of the Old Faithful archive:

• Account Updates: Not currently supported because Old Faithful does not contain account updates.
• Transaction Logs: Available in transaction_status_meta.log_messages for all epochs.
• Execution: Jetstreamer replays the 'already-executed' state of transactions. It is a streaming and processing tool, not a transaction execution engine.

Install dependencies using Homebrew and configure your environment variables.

Note: Add the export commands to your ~/.zshrc to make them persistent.

brew install llvm@16 zlib openssl libtool

# Environment variables (add to ~/.zshrc)
export CC=/opt/homebrew/opt/llvm@16/bin/clang
export CXX=/opt/homebrew/opt/llvm@16/bin/clang++
export LIBCLANG_PATH=/opt/homebrew/opt/llvm@16/lib/libclang.dylib
export LDFLAGS="-L/opt/homebrew/opt/llvm@16/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm@16/include"

The recommended way to set up Jetstreamer is using Nix to ensure a reliable environment.

nix-shell
cargo build --release

Jetstreamer requires Clang 16 (specifically not 17) due to RocksDB dependencies. Follow these steps to install dependencies and configure environment variables.

Note: Add the export commands to your ~/.bashrc to make them persistent.

# Install Clang 16
wget -qO- https://apt.llvm.org/llvm.sh | sudo bash -s -- 16
sudo apt update && sudo apt install -y gcc-13 g++-13 zlib1g-dev libssl-dev libtool

# Set as default
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-16 100
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-16 100

# Environment variables (add to ~/.bashrc)
export CC=clang
export CXX=clang++
export LIBCLANG_PATH=/usr/lib/llvm16/lib/libclang.so

To view a live terminal dashboard instead of plain logs, add the --tui flag to your command. The TUI provides:

• TPS graph: Shows real-time transaction throughput with clickable time ranges (5m, 12h, or all).
• Progress bar: Displays slot progress, ETA, and elapsed time.
• Thread grid: Visualizes firehose thread health (Green = healthy, Red = approaching stall, Cyan = finished).
• System chart: Monitors CPU, memory, and NIC download rates.
• Stats panel: Detailed metrics including TPS spread, connection recycles, timeouts, and ClickHouse write retries.
• Log pane: A scrollable pane for viewing logs.

Controls: Use mouse to drag pane dividers. Press q, Esc, or Ctrl-C for a graceful shutdown.

cargo run --release -- 950 --tui

Use these commands for maintaining the Jetstreamer codebase:

• Format code: cargo fmt --all
• Lint code: cargo clippy --workspace
• Run tests: cargo test --workspace
• Regenerate documentation: cargo doc --workspace --open