SkillRouter Retrieval System

The evaluation protocol uses a benchmark consisting of tasks from tasks.jsonl and ground-truth relevance from relevance.json.

Scoring Logic:

• Tasks with task_type == "generic_only" are dropped.
• relevance labels from relevance.json are used for nDCG calculation.
• The benchmark includes 75 scored tasks: 24 single-skill tasks and 51 multi-skill tasks.

Protocol Tiers:

• Retrieval Protocol: Uses top-50 cosine candidates.
• Reranking Protocol: Uses retrieval top-20 followed by reranking.
• Tiers: Easy and Hard.

When using the released benchmark for evaluation, follow these requirements:

• Query set: Use tasks.jsonl.
• Ground truth: Use relevance.json for ground truth and graded relevance.
• Scoring: Skip generic_only tasks during the scoring process.
• Prediction format: Report predictions as a JSON map where keys are task_id and values are a ranked list of skill_ids.

SkillRouter uses sharded jsonl.gz files for its evaluation benchmarks. This format is chosen because:

• Raw JSONL files exceed GitHub's per-file limits.
• Line-oriented shards allow for efficient streaming.
• The evaluation scripts in src/data_io.py are designed to handle either a single file or an entire shard directory automatically.

The evaluation benchmark is composed of several key files and directories:

• tasks.jsonl: Contains the task descriptions that need to be routed.
• relevance.json: Contains the ground truth skill IDs and graded relevance scores.
• easy/: A directory containing a specific tier of the benchmark skill pool.
• hard/: A directory containing a different tier of the benchmark skill pool.
• manifest.json: Metadata describing the benchmark contents.

Scoring Logic:

• Tasks marked generic_only are excluded from scoring.
• Scoring is performed against the ground-truth set in relevance.json.
• Graded nDCG is calculated using the relevance values.
• The default evaluation produces 75 scored tasks (24 single-skill and 51 multi-skill) out of 87 total tasks.

To evaluate only the first-stage retrieval (encoder) of a custom model, follow these two steps:

1. Export predictions: Use src.export_retrieval to generate candidate lists. The default is the top-50 candidates.
2. Score the tier: Use scripts/evaluate_predictions.sh to calculate metrics for a specific tier (e.g., easy or hard).

# 1. Export predictions
python3 -m src.export_retrieval \
  --encoder_model_or_path /path/to/your/encoder \
  --data_root data/eval_core \
  --output_dir outputs/custom_eval \
  --tiers easy hard

# 2. Score one tier
bash scripts/evaluate_predictions.sh \
  --predictions outputs/custom_eval/retrieval/easy.json \
  --tier easy

To evaluate a generated prediction file against a specific benchmark tier, use the evaluate_predictions.sh script. You must specify the path to your predictions file and the target tier (e.g., easy or hard).

bash scripts/evaluate_predictions.sh \
  --predictions outputs/custom_eval/retrieval/easy.json \
  --tier easy

The data/eval_core/ directory contains only lightweight metadata. To download the full benchmark, including the easy/ and hard/ skill-pool shards from Hugging Face, run the provided download script. This is necessary if you are not using Git LFS to clone the repository.

bash scripts/download_eval_data.sh

The SkillRouter evaluation benchmark metadata and skill-pool shards are hosted on Hugging Face. To download the full benchmark into the local data/eval_core directory, run the provided download script. The data is stored as sharded jsonl.gz files located in data/eval_core/easy/ and data/eval_core/hard/.

bash scripts/download_eval_data.sh

You can evaluate the released 0.6B embedding and reranker models using a single command. This runs the models over both easy and hard benchmark tiers. Results are written to outputs/open_model_eval/.

bash scripts/evaluate_open_models.sh
# OR
make eval-open-models

To run evaluations, you need the benchmark metadata and skill-pool shards. The metadata is located in ../data/eval_core/, while the large skill-pool shards are hosted on Hugging Face. You can download the necessary easy/ and hard/ gzip-sharded skill pools using the provided shell script.

bash scripts/download_eval_data.sh

To evaluate a full pipeline consisting of an encoder and a reranker, use src.run_open_model_eval. By default, this pipeline reranks the top-20 retrieval candidates using flat-full prompts.

python3 -m src.run_open_model_eval \
  --data_root data/eval_core \
  --encoder_model_or_path /path/to/your/encoder \
  --reranker_model_or_path /path/to/your/reranker \
  --tiers easy hard \
  --output_dir outputs/custom_pipeline_eval

To set up the SkillRouter environment, create a virtual environment and install the required dependencies. Note that the scripts assume a CUDA-capable local machine. For multi-GPU setups, you can set the CUDA_VISIBLE_DEVICES environment variable before running the scripts.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt