DOCSEARCHbyAlgolia
Submit Docs

LLMRouter

repository·main·Indexed 24 days ago

https://github.com/ulab-uiuc/llmrouter

A unified framework for LLM routing and evaluation (package llmrouter-lib v0.3.1). It optimizes inference by dynamically selecting the most suitable model based on complexity, cost, and performance. The system supports custom router implementation via MetaRouter, trainable difficulty-based routing with ThresholdRouter, and baseline RandomRouter. It includes a registry-based system for custom tasks, prompt templates, and evaluation metrics, as well as a CLI for inference and training.

Tokens
96.6K
Snippets
267
Records
436
Agent score
80%

What's inside llmrouter

  1. Overview of GMTRouter

    main

    GMTRouter is a graph-based, multi-turn personalized LLM router. Unlike standard routers that use simple classifiers or rankers, GMTRouter uses Heterogeneous Graph Neural Networks (HeteroGNN) to learn specific user preferences and optimize model selection across multi-turn conversation sessions.

    Key Characteristics:

    • Architecture: Uses a Heterogeneous GNN with 5 node types (User, Session, Query, LLM, Response) and 21 edge types.
    • Personalization: Learns per-user preference embeddings.
    • Multi-turn: Built-in conversation tracking via session nodes.
    • Learning Method: Uses pairwise preference learning rather than simple classification.

  2. Overview of LLMRouter

    main

    LLMRouter is an open-source intelligent routing system designed to optimize LLM inference. It dynamically selects the most suitable model for each query based on task complexity, cost, and performance requirements.

    Key features include:

    • Smart Routing: Automatic selection of optimal LLMs.
    • Diverse Router Models: Over 16 routing models across five categories: Single-Round, Multi-Round, Multimodal, Personalized, and Agentic routers.
    • Unified CLI: A command-line interface for training, inference, and interactive chat (including a Gradio-based UI).
    • Data Generation Pipeline: A pipeline to generate training data from 11 benchmark datasets with automatic API calling and evaluation.

  3. Overview of TSRouter

    main

    TSRouter is a router designed for time-series queries that selects the optimal (modality, model) pair. Unlike standard routers that only select models, TSRouter jointly models three modalities (text, visual, and mix) and multiple candidate models using a 4-partite Heterogeneous Graph Transformer (HGT). This allows routing decisions to be sensitive to both the task category and the specific content of the query.

    Key Capabilities:

    • Output: Returns a (modality, model) pair.
    • Modalities: Supports text, visual, and mix.
    • Graph Structure: Uses a 4-partite HGT (Task, Query, Modality, Model) to model relationships.
    • Training: Uses temperature-scaled soft labels via KL divergence for more nuanced learning.

  4. What is OpenClaw Router?

    main
    OpenClaw Router is an OpenAI-compatible API server that routes requests to various backend LLMs (such as Together, NVIDIA, or other OpenAI-compatible endpoints). It acts as a centralized routing layer that integrates with OpenClaw to provide a Slack-native UX. Instead of clients holding sensitive upstream API keys, the Router manages them and selects the best model per request based on configured strategies.

  5. Supported Multimodal Datasets in LLMRouter

    main

    LLMRouter supports multimodal reasoning by integrating Vision-Language Models (VLMs) to describe visual content, which is then used to augment text queries. Currently, the following three datasets are supported:

    1. Geometry3K (Geometry Problem Solving): Focuses on solving geometry problems using diagram descriptions (e.g., shapes, chords, segment lengths, and relationships).
    2. MathVista (Visual Math Reasoning): Focuses on visual mathematical reasoning, where the VLM describes scenes involving physical elements, labels, and symbols (e.g., springs, masses, and forces).
    3. Charades-Ego (Video Understanding): Focuses on video-based tasks including:
      • Activity Recognition: Identifying high-level activities (e.g., "Cooking").
      • Object Recognition: Identifying objects being interacted with.
      • Verb Recognition: Identifying action verbs.

  6. Project Directory Structure

    main

    The OpenClaw Router project is organized into the following structure:

    • scripts/: Contains lifecycle management scripts (start-openclaw.sh, stop-openclaw.sh).
    • configs/: Contains example configuration files.
    • openclaw_router/: The core Python package containing the FastAPI server (server.py), configuration logic (config.py), and routing strategies (routers.py).
    • custom_routers/: A directory for implementing and storing custom routing logic (e.g., randomrouter/).

  7. What is RouterDC (Dual-Contrastive Router)?

    main

    RouterDC is a query-based routing method that uses dual-contrastive learning to select the most suitable Large Language Model (LLM) for a given query. It employs a pre-trained mDeBERTa encoder to represent queries and learnable embeddings for each LLM.

    It uses three complementary contrastive learning objectives:

    1. Sample-LLM Contrastive Loss: Aligns queries with well-performing LLMs (positive samples) and pushes them away from poorly-performing ones (negative samples).
    2. Sample-Sample Contrastive Loss (Task-Level): Groups queries from the same task together to learn task-specific patterns.
    3. Cluster Contrastive Loss: Uses K-means clustering to learn cluster-aware representations, improving generalization.

    The 'dual' aspect refers to the combination of Query-Model Contrast (aligning queries to models) and Query-Query Contrast (grouping similar queries via tasks and clusters).

  8. What is the Automix Router?

    main
    The Automix Router is a cost-effective routing mechanism designed to decide when to escalate queries from a small, inexpensive language model (SLM) to a larger, more capable, and expensive model. It uses a self-verification process where the small model assesses its own confidence by generating multiple verification samples. If the confidence score is low, the query is escalated to the large model to ensure quality; otherwise, the small model's response is used to save costs.

  9. Overview of the MLP Router

    main

    The MLP Router (Multi-Layer Perceptron Router) is a supervised learning-based routing method. It uses a neural network classifier (specifically scikit-learn's MLPClassifier) to predict the most suitable LLM for a given query by learning patterns from historical training data.

    How it works:

    1. Query Embedding: Input queries are converted into fixed-size vectors using Longformer embeddings.
    2. Feature Learning: The MLP learns non-linear patterns in the embedding space.
    3. Classification: The trained network predicts the LLM most likely to perform best.
    4. Selection: The router selects the LLM with the highest predicted probability.

  10. Understand the Embedding Mapping System

    main

    The pipeline uses a unified embedding mapping system to ensure efficient storage and retrieval.

    How it works

    1. Unique Identification: Queries are identified by the tuple (task_name, query, ground_truth, metric).
    2. Sequential IDs: All unique queries (from both train and test sets) are assigned a sequential embedding_id starting from 0.
    3. Unified Storage: All embeddings are stored in a single PyTorch .pt file, mapping embedding_id (int) $\rightarrow$ embedding tensor.
    4. Consistency: The same query will always have the same embedding_id regardless of whether it appears in the training or test set. Multiple routing records (for different LLM models) will share the same embedding_id for the same query.

    Retrieving Embeddings in Python

    import torch
import json

# Load embeddings
embeddings = torch.load("query_embeddings_longformer.pt")

# Load routing data
with open("default_routing_train_data.jsonl", "r") as f:
    for line in f:
        record = json.loads(line)
        embedding_id = record["embedding_id"]
        query_embedding = embeddings[embedding_id]
        
        # Now you have the embedding for this query
        print(f"Query: {record['query']}")
        print(f"Embedding shape: {query_embedding.shape}")
    import torch
import json

# Load embeddings
embeddings = torch.load("query_embeddings_longformer.pt")

# Load routing data
with open("default_routing_train_data.jsonl", "r") as f:
    for line in f:
        record = json.loads(line)
        embedding_id = record["embedding_id"]
        query_embedding = embeddings[embedding_id]  # Returns torch.Tensor
        
        # Now you have the embedding for this query
        print(f"Query: {record['query']}")
        print(f"Embedding shape: {query_embedding.shape}")

  11. Compare Automix with other LLM Routers

    main

    Automix is part of a suite of routing strategies. Choose based on your specific requirements:

    • Automix Router: Balances cost and quality using POMDP or Threshold methods.
    • Hybrid LLM Router: Similar cost-quality trade-off but utilizes a learned MLP predictor.
    • Smallest LLM Router: Always selects the smallest model to maximize cost savings.
    • Largest LLM Router: Always selects the largest model to maximize quality.
    • MLP/SVM/KNN Routers: Designed to route among multiple models rather than just two.

  12. How the LLM Multi-Round Router works

    main

    The LLM Multi-Round Router implements a zero-shot routing strategy using LLM-based reasoning for query decomposition and routing decisions. Unlike KNN-based routers, it requires no training data and relies on model descriptions provided in the configuration to intelligently delegate tasks.

    The Pipeline

    1. Decomposition + Routing: A single LLM call decomposes the original query into multiple sub-queries and simultaneously selects the best model for each sub-query based on provided model descriptions.
    2. Execution: Sub-queries are executed via API using the selected routed models.
    3. Aggregation: A base LLM collects all sub-query responses and combines them into a single final answer.

    When to use it

    • Use when: You have no training data, need zero-shot routing, or have complex queries that require decomposition.
    • Avoid when: You are highly cost-sensitive (it requires multiple LLM calls), latency-sensitive (sequential generations), or have sufficient training data (use the KNN Multi-Round Router instead).
    Query → LLM Decomposition+Routing → [(Sub-Query 1, Model A), (Sub-Query 2, Model B), ...]
                                            ↓                          ↓
                                     Execute via API            Execute via API
                                            ↓                          ↓
                                    LLM Aggregation ← [Response 1, Response 2, ...]
                                            ↓
                                     Final Answer