RouteLLM

To route queries between a strong model and a weak model using the Python SDK, initialize a Controller with your desired routers and model pairs. Note that OPENAI_API_KEY is required for embeddings even when using other providers.

import os
from routellm.controller import Controller

os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"
# Example using Anyscale's Mixtral
os.environ["ANYSCALE_API_KEY"] = "esecret_XXXXXX"

client = Controller(
  routers=["mf"],
  strong_model="gpt-4-1106-preview",
  weak_model="anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1",
)

# To use the router, specify the model name in the format: router-[ROUTER_NAME]-[THRESHOLD]
response = client.chat.completions.create(
  model="router-mf-0.11593",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

The cost threshold controls the tradeoff between cost and quality. You can calibrate this threshold using a sample of your queries to achieve a specific percentage of calls to the strong model.

Use the routellm.calibrate_threshold module. For example, to calibrate the mf router so that 50% of calls go to the strong model:

python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yaml

The output will provide the recommended threshold value (e.g., 0.11593) to use in your model field.

To add a new router to RouteLLM, follow these steps:

1. Implement the abstract Router class in routers.py.
2. You must implement the calculate_strong_win_rate method. This method takes a user prompt and returns the win rate for the strong model conditioned on that prompt. If the returned win rate is greater than the user-specified cost threshold, the request is routed to the strong model; otherwise, it goes to the weak model.
3. Add the new router to the ROUTER_CLS dictionary in routers.py.

Once implemented, the router can be used immediately in the server or evaluation framework.

You can run an OpenAI-compatible server that allows any existing OpenAI client to use RouteLLM routing. Set your provider API keys as environment variables before launching.

export OPENAI_API_KEY=sk-XXXXXX
export ANYSCALE_API_KEY=esecret_XXXXXX
python -m routellm.openai_server --routers mf --strong-model gpt-4-1106-preview --weak-model anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1

Once running, clients can request routing by setting the model field to router-[ROUTER_NAME]-[THRESHOLD] (e.g., router-mf-0.5).

To add a new benchmark to RouteLLM:

1. Implement the abstract Benchmark class in benchmarks.py.
2. Update the evaluate.py module to properly initialize your new benchmark class.

Best Practice: Ideally, benchmark results should be precomputed to avoid regenerating them during every evaluation run. Refer to existing benchmarks in the repository for implementation patterns.

You can install RouteLLM either via PyPI or from the source code.

From PyPI

pip install "routellm[serve,eval]"

From source

git clone https://github.com/lm-sys/RouteLLM.git
cd RouteLLM
pip install -e .[serve,eval]

Router configurations can be specified in two ways:

1. Via the config argument when initializing a Controller.
2. Via a YAML file using the --config flag in the CLI.

The configuration is a top-level mapping where the keys are the router names and the values are the keyword arguments used to initialize those routers. Example configurations for routers trained on Arena data can be found in config.example.yaml.

The RouteLLM server uses the model field in the Chat Completion request to determine routing parameters. The model name must follow the format:

router-[router name]-[threshold]

For example, router-bert-0.7 tells the server to use the bert router with a cost/quality threshold of 0.7. The router type and threshold are extracted from this string to process that specific request.

To use Unify AI as a router in an OpenAI-compatible workflow, set the base_url to https://api.unify.ai/v0/ and provide your UNIFY_API_KEY. You must specify the desired router and the supported models in the model string.

client = openai.OpenAI(
	base_url="https://api.unify.ai/v0/",
	api_key="UNIFY_API_KEY"
)
response = client.chat.completions.create(
        model="router@q:1|c:1.71e-03|t:1.10e-05|i:1.09e-03|models:gpt-4-turbo,mixtral-8x7b-instruct-v0.1",
		...
    )

To use Martian as a router, set the base_url to https://withmartian.com/api/openai/v1 and provide your MARTIAN_API_KEY. When calling the router model, use the extra_body parameter to pass a list of models and a max_cost_per_million_tokens value to control the routing cost threshold.

client = openai.OpenAI(
    base_url="https://withmartian.com/api/openai/v1",
    api_key="MARTIAN_API_KEY",
)
response = client.chat.completions.create(
        model="router",
        extra_body={
            "models": ["gpt-4-turbo-128k", "llama-2-70b-chat"],
            "max_cost_per_million_tokens": 10.45,
        },
		...
    )

RouteLLM supports the following trained routers (optimized for the gpt-4-1106-preview and mixtral-8x7b-instruct-v0.1 pair, but generalizable to others):

1. mf: Matrix factorization model trained on preference data (recommended).
2. sw_ranking: Weighted Elo calculation based on prompt similarity.
3. bert: BERT classifier trained on preference data.
4. causal_llm: LLM-based classifier tuned on preference data.
5. random: Randomly routes to either model.