DOCSEARCHbyAlgolia
Submit Docs

MCP-Bench Evaluation Framework

repository·main·Indexed 19 days ago

https://github.com/accenture/mcp-bench

An evaluation framework for assessing the tool-use capabilities of LLMs using the Model Context Protocol (MCP). It provides an end-to-end pipeline to test how models discover, select, and use tools across 28 diverse real-world MCP servers, including specialized implementations like the Bibliomantic MCP Server, Call For Papers MCP, and MedCalc-Bench.

Tokens
173.5K
Snippets
649
Records
832
Agent score
66%

What's inside MCP-Bench

  1. Overview of BioMCP Backend Services

    main

    BioMCP integrates with multiple biomedical databases to provide research capabilities across several categories. The system orchestrates calls from BioMCP tools to backend APIs to return unified responses to the user.

    Service Categories

    • Literature and Publications: Includes PubTator3 (biomedical literature with entity annotations) and Europe PMC (preprints from bioRxiv and medRxiv).
    • Clinical Trials: Includes ClinicalTrials.gov (U.S. and international registry) and NCI CTS API (National Cancer Institute's enhanced search).
    • Biomedical Annotations (BioThings Suite): Includes MyGene.info (genes), MyVariant.info (variants), MyDisease.info (diseases), and MyChem.info (drugs/chemicals).
    • Cancer Genomics: Includes cBioPortal (mutation data) and TCGA (via MyVariant.info).
    • Variant Effect Prediction: Includes AlphaGenome (Google DeepMind's AI for regulatory predictions).

  2. Overview of BioMCP interfaces

    main

    BioMCP provides three primary interfaces for accessing biomedical data:

    1. Python SDK: An async/await interface including a high-level Client API, domain-specific APIs (articles, trials, variants), a Streaming API, and a Batch API.
    2. MCP Protocol: Implements the Model Context Protocol for AI assistants, providing 24 specialized tools, unified search, sequential thinking, and streaming responses.
    3. HTTP REST API: Available when running in HTTP mode, exposing search and fetch endpoints, health monitoring, and WebSocket support for streaming.

    Refer to specific documentation for detailed implementation of each interface.

  4. BioMCP Command Line Interface Overview

    main

    BioMCP provides a CLI for biomedical data retrieval across several domains. The general syntax is:

    biomcp [OPTIONS] COMMAND [ARGS]...

    Available Domains and Commands

    DomainCommandsPurpose
    articlesearch, getSearch and retrieve biomedical literature
    trialsearch, getFind and fetch clinical trial information
    variantsearch, get, predictAnalyze genetic variants and predict effects
    genegetRetrieve gene information and annotations
    druggetLook up drug/chemical information
    diseasegetGet disease definitions and synonyms
    organizationsearchSearch NCI organization database
    interventionsearchFind interventions (drugs, devices, procedures)
    biomarkersearchSearch biomarkers used in trials
    healthcheckMonitor API status and system health

  5. NASA-MCP Capabilities and Use Cases

    main

    NASA-MCP exposes tools to query various NASA APIs. Once configured, you can use natural language to perform the following tasks:

    • Astronomy Picture of the Day (APOD): Request the daily astronomical image and its explanation.
    • Asteroids & Near-Earth Objects: Find asteroids passing near Earth within specific timeframes.
    • Space Weather (DONKI): Retrieve data on solar flares, geomagnetic storms, and other space weather events.
    • Earth Imagery (Landsat 8): Obtain satellite images for specific coordinates.
    • Earth View (EPIC): Access full-disk Earth images from the EPIC camera.
    • Exoplanet Archive: Query information about planets outside our solar system, such as those in habitable zones.
    Examples of queries:
- "Muéstrame la imagen astronómica del día de hoy"
- "Encuentra asteroides que pasarán cerca de la Tierra en la próxima semana"
- "Obtén información sobre erupciones solares de enero de 2023"
- "Muéstrame imágenes de la Tierra para las coordenadas 29.78, -95.33"
- "Encuentra exoplanetas en la zona habitable"

  6. Task Synthesis Core Components

    main

    The task synthesis process is managed by several key files:

    • generate_benchmark_tasks.py: The primary CLI script used for batch generation of tasks.
    • benchmark_generator.py: A unified task generator that handles both single-server and multi-server logic.
    • task_synthesis.py: Contains the core task generation logic and fuzzy conversion capabilities.
    • split_combinations/: A directory containing pre-defined JSON files for multi-server combinations.

    Generated tasks are saved in the tasks/ directory in JSON format.

  7. Understand the MCP-Bench project structure

    main

    MCP-Bench is organized into several functional modules that handle the lifecycle of benchmarking tool-using LLM agents:

    • agent/: Contains the executor.py (multi-round task execution with retry logic) and execution_context.py.
    • benchmark/: The evaluation framework, including runner.py (orchestrator), evaluator.py (LLM-as-judge metrics), and results processing (results_aggregator.py, results_formatter.py).
    • config/: Manages settings via benchmark_config.yaml and config_loader.py.
    • llm/: Provides a unified interface for different LLM providers through provider.py and a factory.py.
    • mcp_modules/: Handles MCP server orchestration, connection management (connector.py, server_manager.py), and tool call caching.
    • synthesis/: Responsible for generating benchmark tasks via task_synthesis.py and benchmark_generator.py.
    • mcp_servers/: Contains the implementations for 28 different MCP servers, including installation scripts and command configurations.
    • tasks/: Stores the benchmark task files in various runner formats (single, 2-server, and 3-server combinations).
    • run_benchmark.py: The primary entry point to execute the benchmark.

  9. Overview of the BioThings Suite in BioMCP

    main

    BioMCP integrates with the BioThings suite of APIs to provide consistent biomedical data retrieval. The suite includes:

    • MyGene.info: Gene information and annotations.
    • MyDisease.info: Disease ontology and synonyms.
    • MyVariant.info: Genetic variant annotations.
    • MyChem.info: Drug/chemical information and annotations.

    All services share a common infrastructure via the BioThings client module, ensuring consistent error handling, rate limiting, and response parsing across all domains.

  10. What is BioMCP?

    main
    BioMCP is an open-source implementation of the Model Context Protocol (MCP) specifically designed for biomedical research. It acts as a bridge between AI assistants (or autonomous agents) and specialized biomedical databases. By using the MCP standard, BioMCP allows users to query complex scientific data using natural language, translating conversational input into the precise syntax required by various scientific data sources.

  11. Overview of BioThings Suite APIs

    main

    The BioThings Suite provides unified, RESTful JSON access to biomedical annotations across genes, variants, diseases, and drugs. BioMCP integrates with four specific APIs:

    • MyGene.info: Gene annotations and functional information.
    • MyVariant.info: Genetic variant annotations and clinical significance.
    • MyDisease.info: Disease ontology and terminology mappings.
    • MyChem.info: Drug/chemical properties and mechanisms.

    Key Characteristics:

    • No authentication required (public access).
    • Elasticsearch-based queries.
    • Comprehensive data aggregation from multiple sources (e.g., NCBI, Ensembl, ClinVar, DrugBank).

  12. Optimize documentation retrieval with Library IDs

    main

    If you already know the exact Context7 ID for a library, include it directly in your prompt. This allows the MCP server to skip the resolve-library-id step and proceed directly to documentation retrieval.

    Example prompt: supabase로 기본 인증을 구현해줘. API와 문서는 /supabase/supabase 라이브러리를 사용해줘 (Implement basic auth with Supabase. Use the /supabase/supabase library for API and docs.)