PixelRAG Documentation

The Synthetic Data Generation Pipeline is an end-to-end process for generating the screenshot-training-natural-filtered-v2 dataset. This dataset is used to fine-tune Qwen3-VL-Embedding-2B for visual document retrieval. The pipeline produces approximately 115K high-quality query→screenshot-chunk pairs including hard negatives, starting from raw Wikipedia screenshot tiles.

There are two main pipelines:

1. Visual Query Pipeline

Processes Wikipedia screenshot tiles (kiwix_tiles/) through several stages:

• Generation: Gemini reads screenshot chunks to generate Q/A pairs.
• Filtering: GPT-4o removes queries that are not self-contained.
• Hard Negative Mining: A Search API retrieves confusable chunks.
• VQA Filtering: A VLM removes false negatives (chunks that actually answer the query).
• Cleaning: Gemini scores queries for naturalness and factoid style.
• Export & Splitting: Final data is filtered by score, split into train/eval/test sets, and packaged for Hugging Face.

2. Text Warmup Pipeline

Processes text passages (text_baseline.db) to create a text-qa-pair dataset used for --text-warmup-steps:

• Generation: Gemini reads text passages to generate Q/A pairs.
• Filtering: Removes non-self-contained queries.
• Mining: Text search API retrieves confusable passages.
• LLM Filtering: LLM removes false negatives.

Use filter_hard_negatives_vqa.py to remove false negatives from mined retrieval candidates. The script uses a VLM to verify if a candidate image actually contains the answer to the query. If the VLM determines the answer is CORRECT for a candidate, that candidate is treated as a false negative and skipped. If the verdict is WRONG or CANNOT_ANSWER, it is kept as a hard negative.

Key Arguments:

• --input: Path to the input JSONL file (must contain retrieve_top20 field).
• --output: Path to the filtered JSONL file.
• --reviews-output: Path to save candidate reviews.
• --summary-output: Path to save the summary JSON.
• --candidate-k: Maximum number of non-positive candidates to check (default 10).
• --num-hard-negatives: Target number of hard negatives to collect per example.
• --concurrency: Number of concurrent VLM requests.

OPENAI_API_KEY=... python filter_hard_negatives_vqa.py \
    --input /tmp/sample_100_hn.jsonl \
    --output /tmp/sample_100_hn_v2.jsonl \
    --reviews-output /tmp/sample_100_hn_v2.reviews.jsonl \
    --summary-output /tmp/sample_100_hn_v2.summary.json \
    --candidate-k 10 \
    --num-hard-negatives 2 \
    --concurrency 8

Before running reproduction scripts, you must ensure the necessary services are running. The topology typically includes a reader (serving the LLM) and various retrieval services (serving FAISS indexes).

Reader Service

• Model: Qwen/Qwen3.5-4B using vLLM 0.19.0.
• Setup: Install via uv sync --extra reader, then run: CUDA_VISIBLE_DEVICES=0 HF_HOME=… .venv/bin/vllm serve Qwen/Qwen3.5-4B --port 8010.

Retrieval Services

Note: All pixel/text services are direct_gpu. The reader sends the raw query, and the service performs the encoding. Do not POST precomputed embeddings.

The pixelshot turbo capture path (fast_cdp) requires a custom headless_shell binary containing specific CDP additions for high-throughput screenshotting. Without this binary, pixelshot falls back to a standard portable capture path (in-Chrome JPEG over CDP) which is slower.

Prerequisites:

• depot_tools must be on your PATH.
• A Chromium 150 checkout (the patch is a git diff from upstream commit 4deaeccb7c).
• ~100 GB free disk space.
• Platform toolchain: Linux (clang via runhooks), macOS (Xcode), or Windows (VS/MSVC).

Build Steps:

1. Fetch dependencies:

   gclient sync --with_branch_heads --with_tags --delete_unversioned_trees -j 32
   gclient runhooks

2. Apply the patch:

   git -C src apply ../render/chrome-build/pixelrag-chrome.patch

3. Configure and build:

   mkdir -p src/out/Headless
   cat > src/out/Headless/args.gn <<'EOF'
   import("//build/args/headless.gn")
   is_official_build = true
   is_debug = false
   symbol_level = 0
   blink_symbol_level = 0
   chrome_pgo_phase = 0
   EOF
   ( cd src && gn gen out/Headless && autoninja -C out/Headless headless_shell )

# 1. fetch deps for the checkout
gclient sync --with_branch_heads --with_tags --delete_unversioned_trees -j 32
gclient runhooks

# 2. apply the patch
git -C src apply ../render/chrome-build/pixelrag-chrome.patch     # or: git am / patch -p1

# 3. configure (headless, optimized) and build
mkdir -p src/out/Headless
cat > src/out/Headless/args.gn <<'EOF'
import("//build/args/headless.gn")
is_official_build = true
is_debug = false
symbol_level = 0
blink_symbol_level = 0
chrome_pgo_phase = 0
EOF
( cd src && gn gen out/Headless && autoninja -C out/Headless headless_shell )