DOCSEARCHbyAlgolia
Submit Docs

Finance Agent Benchmark

repository·main·Indexed 41 days ago

https://github.com/vals-ai/finance-agent

A tool for evaluating LLMs on complex financial research tasks. It provides a codebase for benchmarking models using specialized tools such as SEC EDGAR database access, Tavily web search, HTML parsing, and information retrieval. The package includes a CLI for executing benchmarks, a `get_agent` function for instantiating agents, and a `KeyRotator` for managing API key rotation and concurrency.

Tokens
6.2K
Snippets
21
Records
30
Agent score
49%

What's inside Finance Agent Benchmark

  1. Install and set up the Finance Agent Benchmark

    main

    The Finance Agent benchmark evaluates LLMs on their ability to use tools (web search, EDGAR search, HTML parsing, and information retrieval) to answer financial questions.

    To set up the environment:

    1. Install uv for dependency management.
    2. Run the installation command and activate the virtual environment.
    3. Create a .env file in the project root for API keys.
    4. Add your 'Test Suite IDs' to suites.json. These IDs can be found in the Vals platform under the 'Test Suites' page sidebar.
    make install
source .venv/bin/activate

  2. Configure environment variables for LLMs and Tools

    main

    Create a .env file in the root directory to manage API keys. The .env file takes precedence over system environment variables.

    Required and optional keys include:

    • VALS_API_KEY: Your Vals platform API key.
    • OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, ETC_API_KEY: LLM provider keys.
    • TAVILY_API_KEY: For the web_search tool.
    • SEC_EDGAR_API_KEY: For the edgar_search tool. This supports semicolon-separated keys for round-robin rotation (e.g., key1;key2;key3).
    VALS_API_KEY=<api_key>

# LLM API Keys
OPENAI_API_KEY=<openai_api_key>
ANTHROPIC_API_KEY=<anthropic_api_key>
GOOGLE_API_KEY=<google_api_key>
ETC_API_KEY=<etc_api_key>

# Tool API Keys
TAVILY_API_KEY=<tavily_api_key>
SEC_EDGAR_API_KEY=<sec_api_key>

  3. How the data storage lifecycle works

    main

    The agent uses a state-based data storage system to handle large documents. The workflow is:

    1. Ingest: Use parse_html_page with a specific key to scrape a website and save its text content into the agent's state.
    2. Store: The content is now associated with that key in the conversation state.
    3. Retrieve: Use retrieve_information with a prompt containing {{key}}. The tool injects the stored text into the prompt and sends it to an LLM for analysis.

    This pattern allows the agent to 'read' large amounts of information without exceeding context window limits by only retrieving specific keys or character ranges when needed.

  4. Understand the Finance Agent loop and exit behavior

    main

    The Finance Agent follows a specific execution lifecycle that differs from the default model library behavior:

    Loop Exit Conditions

    • Clean Exit: The loop terminates when the submit_final_result tool is called with done=True. The agent returns the tool's output as the final answer.
    • Max Turns: The loop terminates if max_turns is exceeded. This results in a final_error of MaxTurnsExceeded and returns an empty string "" as the answer.
    • Query Errors: If a query error is re-raised by the before_query hook, the loop stops and sets a final_error.

    Key Behaviors

    • Context Window Management: If a MaxContextWindowExceededError occurs, the _before_query hook automatically calls truncate_oldest to manage history. This does not stop the loop.
    • Text-Only Responses: Unlike the default behavior which might stop on a text-only response (no tool calls), the Finance Agent is configured via _should_stop to always continue (return False). This ensures the agent keeps trying until it either calls submit_final_result or hits the turn limit.
    • Automatic Continuation: If the last response in history is a RawResponse (meaning no tool was called), the _before_query hook injects a TextInput(text="Continue.") to prompt the model to proceed.

  6. Access benchmark logs

    main

    Detailed logs are automatically generated in the logs/ directory. Each execution creates a new timestamped directory. Inside, you will find per-question log files that include:

    • Tool usage details
    • Token counts
    • Error tracking

  7. Run the finance-agent CLI

    main

    Use the finance-agent command to execute the benchmark. You can provide questions via direct arguments, multiple arguments, or a text file.

    Common flags:

    • --questions: A single question string or multiple question strings.
    • --question-file: Path to a text file containing one question per line.
    • --model: The specific model identifier to use (e.g., openai/gpt-5.2-2025-12-11).

    Run finance-agent --help for a full list of command line options.

    # Run a single question with a specific model
finance-agent --questions "What was Apple's revenue in 2023?" --model openai/gpt-5.2-2025-12-11

# Run multiple questions
finance-agent --questions "What was Apple's revenue in 2023?" "What was NFLX's revenue in 2024?"

# Run questions from a file
finance-agent --question-file data/public.txt

  8. Search the internet with `web_search`

    main

    The web_search tool (implemented via TavilyWebSearch) allows searching the public internet. Each result includes a URL, title, and an excerpt.

    Parameters:

    • search_query (required): The search string.
    • start_date (optional): Format YYYY-MM-DD.
    • end_date (optional): Format YYYY-MM-DD. Note that any date later than 2025-04-07 will be capped at 2025-04-07.
    • number_of_results (optional): Integer between 1 and 20 (default: 10).
    {
  "search_query": "NVIDIA Q3 earnings report",
  "start_date": "2024-01-01",
  "number_of_results": 5
}

  10. Parse and store HTML content with `parse_html_page`

    main

    The parse_html_page tool (implemented via ParseHtmlPage) retrieves an HTML page, converts it to plain text (removing <script> and <style> tags), and saves it to the agent's internal data storage.

    Parameters:

    • url (required): The URL of the page to parse.
    • key (required): The unique identifier to use when saving the text in the conversation's data storage. You can later use this key with retrieve_information to access the content.

    Note: If the key already exists, the new content will overwrite the old content.

    {
  "url": "https://example.com/report",
  "key": "annual_report_2023"
}

  11. Create LLM configuration overrides with create_override_config

    main

    Use create_override_config to generate an LLMConfig object from a dictionary of parameters. This is useful for overriding default model settings like token limits. If you provide max_output_tokens, it will automatically be mapped to max_tokens if max_tokens is not explicitly provided.

    from finance_agent.custom_model import create_override_config

# Example: Overriding max tokens
config = create_override_config(model_name="gpt-4", max_output_tokens=512)

  12. Initialize a finance agent with get_agent()

    main

    Use get_agent to instantiate a finance agent. This function requires a Parameters object to configure the agent's behavior and capabilities. The agent is typically configured with a set of tools and instructions defined in the package.

    from finance_agent import get_agent, Parameters

# Example usage (requires specific Parameters configuration)
params = Parameters(
    # ... configuration fields as defined in Parameters class
)
agent = get_agent(params)