DOCSEARCHbyAlgolia
Submit Docs

OptiLLM

repository·main·Indexed 26 days ago

https://github.com/algorithmicsuperintelligence/optillm

An optimizing inference proxy and plugin system for LLMs designed to enhance capabilities through reasoning scaling, memory management, and structured outputs. It wraps providers like OpenAI, Cerebras, and Azure to implement advanced techniques including AutoThink for adaptive reasoning, DeepConf for confidence-aware filtering, MARS (Multi-Agent Reasoning System) for complex mathematical problems, and a Deep Research plugin for iterative, citation-backed academic reporting.

Tokens
36K
Snippets
66
Records
185
Agent score
88%

What's inside optillm

  1. Understand the System Prompt Learning (SPL) Plugin

    main

    The System Prompt Learning (SPL) plugin for OptiLLM is designed to bridge the 'System Prompt Gap' by automatically learning and applying effective problem-solving strategies. Instead of relying on static or empty system prompts, SPL allows the LLM to accumulate human-readable knowledge from experience.

    Key benefits include:

    • Cumulative Learning: The model improves on specific problem types over time.
    • Explicit Knowledge: Learned strategies are human-readable, allowing for inspection, manual editing, and debugging of reasoning.
    • Efficiency: Successful approaches are reused for similar problem types.
    • Adaptability: The system develops specialized strategies for different problem subtypes (e.g., probability vs. basic arithmetic).

  2. Use AutoThink for adaptive LLM reasoning

    main
    AutoThink is an adaptive thinking approach that enhances LLM reasoning by combining query complexity classification, dynamic token budget allocation, and steering vector guidance. It manages an explicit thinking phase using <think> and </think> tokens, adjusting the depth of reasoning based on whether a query is classified as HIGH or LOW complexity.

  3. Quick Start with OptiLLM

    main

    OptiLLM is an OpenAI API-compatible proxy that improves LLM reasoning accuracy by applying optimization techniques at inference time. To get started, install the package, start the server with your API key, and then point your OpenAI client to the OptiLLM local endpoint.

    1. Install: pip install optillm
    2. Start Server: export OPENAI_API_KEY="your-key-here" followed by optillm
    3. Use Client: Update your OpenAI client base_url to http://localhost:8000/v1 and use the moa- prefix for Mixture of Agents optimization.
    # 1. Install OptiLLM
pip install optillm

# 2. Start the server
export OPENAI_API_KEY="your-key-here"
optillm

# 3. Use with any OpenAI client - just change the model name!
    from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")

# Add 'moa-' prefix for Mixture of Agents optimization
response = client.chat.completions.create(
    model="moa-gpt-4o-mini",  # This gives you GPT-4o performance from GPT-4o-mini!
    messages=[{"role": "user", "content": "Solve: If 2x + 3 = 7, what is x?"}]
)

  4. Optimize MARS for code generation vs. numerical vs. proofs

    main

    Based on benchmark results, use the following guidance for MARS configuration:

    • Code Generation: Enable thinking tags to allow agents to reason through logic before writing code.
    • Numerical Competition Problems: Use multi-agent reasoning with diverse temperatures for effective solving.
    • Proof-based Problems: Disable thinking tags (use_thinking_tags=False) and set answer_extraction_mode="none" to ensure full visibility of the proof steps.

  5. Configure the Local Inference Server with HuggingFace models and LoRAs

    main

    To use the built-in inference server, set OPTILLM_API_KEY to any value. You can load HuggingFace models directly in the model field. To add LoRAs, use the + separator between model names. Use the active_adapter parameter in extra_body to specify which adapter to use (defaults to the last one specified).

    If using private models, set the HF_TOKEN environment variable.

    OPENAI_BASE_URL = "http://localhost:8000/v1"
OPENAI_KEY = "optillm"
response = client.chat.completions.create(
  model="meta-llama/Llama-3.2-1B-Instruct+patched-codes/Llama-3.2-1B-FastApply+patched-codes/Llama-3.2-1B-FixVulns",
  messages=messages,
  temperature=0.2,
  logprobs = True,
  top_logprobs = 3,
  extra_body={"active_adapter": "patched-codes/Llama-3.2-1B-FastApply"},
)

  6. Model the impact of Commodity Hedging on COGS

    main

    To model how hedging strategies affect Cost of Goods Sold (COGS), financial models must account for gains or losses on derivative contracts.

    There are two primary types of hedges to model:

    • Fair Value Hedges: Changes in the derivative's fair value are recognized in earnings, and the hedged item's carrying amount is adjusted accordingly. Both are recognized in the same income statement line item.
    • Cash Flow Hedges: Changes in the derivative's fair value are recorded in Other Comprehensive Income (OCI) and reclassified into earnings as the hedged item affects earnings. This defers the impact to match the timing of the hedged risk.

    Note: Entities typically require an offset of at least 80% to assess hedging effectiveness.

  7. Start the OptiLLM Server with Proxy

    main

    You can start the OptiLLM server in two ways:

    1. Default Proxy Mode (Recommended): Use --approach proxy to make the proxy the default for all incoming requests.
    2. Normal Mode: Start without the flag, then use model prefixes or extra_body to trigger the proxy on a per-request basis.
    # Option A: Use proxy as default for ALL requests (recommended)
optillm --approach proxy

# Option B: Start server normally
optillm

# With custom port
optillm --approach proxy --port 8000

  9. Run OptiLLM using Docker Compose

    main

    To run OptiLLM with Docker Compose:

    1. Ensure Docker and Docker Compose are installed.
    2. Configure environment variables (like OPENAI_API_KEY) in a .env file in the project root.
    3. Start the service using docker compose up -d.

    OptiLLM will be available at http://localhost:8000.

  11. Run OptiLLM tests

    main

    OptiLLM includes a test suite for verifying reliability and compatibility. You can run the main test suite from the project root to test all approaches with default cases, specific approaches, or a single test case.

    # Test all approaches with default test cases
python tests/test.py

# Test specific approaches
python tests/test.py --approaches moa bon mcts

# Run a single test
python tests/test.py --single-test "Simple Math Problem"

  12. Control OptiLLM optimization techniques

    main

    You can specify optimization techniques (slugs) using three different methods. Note that these methods require the OptiLLM server to be started with the inference approach set to auto.

    1. Model Name Prefix: Prepend the slug to the model name: {slug}-model-name (e.g., moa-gpt-4o).
    2. Extra Body Field: Pass the slug in the optillm_approach field within the extra_body parameter.
    3. Prompt Tags: Include the approach within <optillm_approach> </optillm_approach> tags in the system or user prompt.

    Combining Techniques:

    • Use & for a pipeline: techniques are processed left-to-right, where the response from one stage is the request for the next.
    • Use | for parallel execution: all requests run in parallel and return a list of responses.
    # Method 2: extra_body
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{ "role": "user","content": "" }],
  temperature=0.2,
  extra_body={"optillm_approach": "bon|moa|mcts"}
)

# Method 3: Prompt tags
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{ "role": "user","content": "<optillm_approach>re2</optillm_approach> How many r's are there in strawberry?" }],
  temperature=0.2
)