repository·main·Indexed 21 days ago
https://github.com/groq/openbenchA provider-agnostic, open-source evaluation infrastructure for language models. openbench allows developers to run standardized benchmarks across dozens of model providers, including Groq, OpenAI, Anthropic, and Google, as well as local providers like Ollama and vLLM. It features a CLI for running evaluations, an interactive UI for viewing results, and support for custom benchmark plugins and private evaluation scripts.
The openbench project provides a variety of benchmarks to evaluate language models across different dimensions such as knowledge, reasoning, tool calling, and coding.
LiveMCPBench utilizes the LiveMCPEval framework (an LLM-as-a-judge approach) to score models. The evaluation focuses on several key capabilities:
The Exercism benchmark provides agents with a sandboxed Docker environment containing a complete project workspace. Each task follows this structure:
/workspace/{language}/{task_name}/
├── INSTRUCTIONS.md # Exercise description and requirements
├── {source_files} # Implementation files to modify
└── {test_files} # Test suite to verify solutionAgent Workflow:
INSTRUCTIONS.md.openbench is provider-agnostic and supports over 30 model providers via Inspect AI. To use a provider, set the corresponding environment variable.
Common providers include:
GROQ_API_KEY (Model: groq/model-name)OPENAI_API_KEY (Model: openai/model-name)ANTHROPIC_API_KEY (Model: anthropic/model-name)GOOGLE_API_KEY (Model: google/model-name)ollama/model-name)vllm/model-name)Refer to the full list in the documentation or the provider table for specific environment variables like AI21_API_KEY, COHERE_API_KEY, or HF_TOKEN (for Hugging Face).
openbench supports over 15 model providers out of the box. When specifying models in your evaluations, use the provider/model-name format.
Examples of supported providers and their model strings:
groq/llama-3.3-70bopenai/gpt-4oanthropic/claude-3-5-sonnetgoogle/gemini-2.5-proopenrouter/deepseek/deepseek-chat-v3.1Other supported providers include AWS Bedrock, Azure, Cohere, and Together.
# Example model identifiers
groq/llama-3.3-70b
openai/gpt-4o
anthropic/claude-3-5-sonnet
google/gemini-2.5-pro
openrouter/deepseek/deepseek-chat-v3.1openbench supports a plugin system using Python entry points. This allows you to package benchmarks as independent Python packages and distribute them. Once installed via pip, they will automatically appear in bench list.
To register a benchmark, add an entry point in your pyproject.toml:
[project.entry-points."openbench.benchmarks"]
my_benchmark = "my_pkg.metadata:get_benchmark_metadata"MMLU scoring is based on accuracy, calculated as the percentage of questions answered correctly across all 57 subjects.
Results include:
openbench supports configuration through three methods, applied in the following order of precedence (highest to lowest):
bench command.BENCH_ in your shell..env files. These are automatically loaded if present in the current directory, or can be sourced manually using source .env.If a parameter is not provided via any of these methods, the system uses its internal default values.
# 1. CLI Flag
bench mmlu --max_tokens=10000 --temperature=0.5
# 2. Environment Variable
export BENCH_MAX_TOKENS=10000
export BENCH_TEMPERATURE=0.5
# 3. .env File
# Create a .env file with:
BENCH_MAX_TOKENS=10000
BENCH_TEMPERATURE=0.5
# Run (automatically loads if in current directory)
bench eval mmlucategory (e.g., core, math, global-mmlu, domain-specific) and tags (e.g., multiple-choice, reasoning, coding, medical). This metadata allows you to understand the nature of the test and filter for specific capabilities like multilingual or long-context reasoning.SciCode evaluates generated code across several dimensions to provide a comprehensive view of model performance in scientific contexts:
GPQA Diamond is a benchmark for testing deep scientific understanding and reasoning.