DOCSEARCHbyAlgolia
Submit Docs

RouterBench

repository·main·Indexed 41 days ago

https://github.com/withmartian/routerbench

A benchmark for multi-LLM routing systems that provides a pipeline to convert data formats, evaluate routing strategies, and visualize trade-offs between performance and cost. It utilizes Martian as a unified gateway for model access and supports extensibility via AbstractRouter and AbstractConvertor classes.

Tokens
723
Snippets
4
Records
5
Agent score
2%

What's inside RouterBench

  1. Extend RouterBench with new routers or convertors

    main

    RouterBench is designed to be extensible. To add new functionality, implement the provided abstract classes:

    • New Routers: Inherit from AbstractRouter located in the routers/ directory.
    • New Data Formats: Inherit from AbstractConvertor located in the convertors/ directory.

  2. Run the RouterBench pipeline

    main

    The pipeline is executed in three main stages using Python scripts. You can specify configurations via command line arguments or by passing a YAML configuration file (examples are located in the configs/ directory).

    Important: Ensure you update the data file paths in your YAML configurations to match your local environment. If you do not have a MongoDB instance, ensure local_cache: true is set in your configuration to use local files for caching.

    Pipeline Stages:

    1. Data Conversion: Process raw data formats (including those from the martian-evals repo) into a common format.
    2. Router Evaluation: Evaluate different routers using the processed data. This generates a long-format CSV file and an EvaluationCollection containing the results.
    3. Visualization: Generate performance-vs-cost plots using the EvaluationCollection.
    # 1. Convert data
python convert_data.py --config=configs/convert_data.yaml

# 2. Evaluate routers
python evaluate_routers.py --config=configs/evaluate_routers.yaml

# 3. Visualize results
python visualize_results.py --config=configs/visualize.yaml

  3. Format code for contributions

    main

    Before submitting a Pull Request, ensure your code adheres to the project's style guidelines by running flake8, black, and isort on all git-tracked Python files.

    flake8 $(git ls-files '*.py')
black $(git ls-files '*.py')
isort $(git ls-files '*.py')

  4. Setup RouterBench

    main

    To set up RouterBench, follow these steps:

    1. Configure Environment Variables: Create a .env file in the root directory. If you intend to use MongoDB as an embedding cache, include the CONNECTION_STRING variable.
    2. Install Packages: Install the project in editable mode using pip.

    Note: This project uses Martian as a unified gateway for model access. You will need a Martian account to proceed.

    # .env file content
CONNECTION_STRING='your mongodb connection string'

# Installation
pip install -e .