Anemoi Documentation

GAIA (General AI Assistants Benchmark) is a benchmark designed to evaluate next-generation Large Language Models (LLMs) that possess augmented capabilities, such as access to tools, efficient prompting, and web search.

The dataset consists of over 450 non-trivial questions with unambiguous answers, categorized into three difficulty levels:

• Level 1: Achievable by high-performing LLMs.
• Level 3: Represents a significant jump in required model capabilities.

Each level includes a fully public dev set for validation and a test set containing private answers and metadata to prevent data leakage.

Anemoi is a semi-centralized multi-agent system (MAS) that utilizes an Agent-to-Agent (A2A) communication MCP server.

Unlike traditional centralized paradigms that rely heavily on context-engineering and a single planner, Anemoi enables direct inter-agent communication. This allows agents to collaborate like a real-world team by monitoring progress, assessing results, identifying bottlenecks, and proposing refinements in real time. This architecture reduces dependency on a single planner and improves scalability and cost-efficiency through more efficient context management.

The framework is built around three main components:

1. Dispatcher (mock_web.py): The central entry point. It manages the lifecycle of a benchmark by downloading assets from Hugging Face (camel-ai/mock_websites), reading task.json, launching the project's Flask server, and polling the API to detect when the agent has met the ground_truth_cart requirements.
2. Projects: Self-contained Flask applications (like shopping_mall/) that simulate specific website types. They include their own app.py and logic.
3. Task Configuration (task.json): The source of truth for the environment's state (products) and the agent's goal (the target cart state).

The GAIA dataset questions are stored in metadata.jsonl.

If a question requires an additional supporting file, the filename is provided in the file_name field within the JSONL entry. These files are located in the same directory as the metadata.

To install the necessary dependencies and set up the Python virtual environment, follow these steps:

1. Navigate to the Anemoi directory.
2. Create a virtual environment using Python 3.12.
3. Activate the environment.
4. Install requirements via pip.
5. Important: Anemoi requires a modified version of camel (v0.2.70). You must replace the installed camel package with the version provided in the utils/camel directory of this repository.

cd Anemoi
/usr/bin/python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Replace standard camel with the modified version from utils
rm -rf venv/lib/python3.12/site-packages/camel
cp -r utils/camel venv/lib/python3.12/site-packages/

Once the environment is configured and the modified camel package is in place, you can execute the experiments using the Gradle wrapper from the project root.

cd ..
./gradlew run --console=plain

To set up the environment for testing web agents, install the required Python packages using pip.

This installs Flask, huggingface-hub, requests, and other necessary dependencies required by the dispatcher and the mock projects.

pip install -r requirements.txt

Before running experiments, you must configure several API keys and paths in your environment. Ensure the following variables are set in your shell (e.g., ~/.bashrc):

• FIRECRAWL_API_KEY: Your Firecrawl API key.
• GOOGLE_API_KEY: Your Google API key.
• HF_HOME: Path to your Hugging Face home directory.
• OPENROUTER_API_KEY: Your OpenRouter API key.
• SEARCH_ENGINE_ID: Your search engine ID.
• CHUNKR_API_KEY: Your Chunkr API key.

export FIRECRAWL_API_KEY="your_firecrawl_api_key"
export GOOGLE_API_KEY="your_google_api_key"
export HF_HOME="your_hf_home_path"
export OPENROUTER_API_KEY="your_openrouter_api_key"
export SEARCH_ENGINE_ID="your_search_engine_id"
export CHUNKR_API_KEY="your_chunkr_api_key"

The task.json file defines the environment state and the success criteria for the agent. You must edit this file to specify available products and the target state of the shopping cart.

Key fields:

• products: A list of product objects available in the environment.
• ground_truth_cart: A list of items defining the target state of the shopping cart. The task is considered complete when the cart matches this state.

{
    "products": [
        {
            "id": 1,
            "name": "Gaming Laptop",
            "price": 1200,
            "image": "assets/img/products/laptop.jpg",
            "category": "Electronics",
            "rating": 4.5,
            "description": "High-performance gaming laptop with latest specs."
        }
    ],
    "ground_truth_cart": [
        {
            "id": 1,
            "quantity": 1
        }
    ]
}

The mock_web.py dispatcher accepts the following command-line arguments:

• --project: Specifies which mock website project to run (e.g., shopping_mall).
• --port: Specifies the port to run the server on (default: 5001).

The framework generates logs at two levels:

• dispatcher.log: Contains high-level information from the dispatcher, including setup steps, status updates, and final benchmark results.
• [project_name]/app.log: (e.g., shopping_mall/app.log) Contains detailed internal logs from the specific Flask application being tested.