DOCSEARCHbyAlgolia
Submit Docs

FinanceBench

repository·main·Indexed 43 days ago

https://github.com/patronus-ai/financebench

A benchmark for evaluating Large Language Models (LLMs) on open-book financial question answering using publicly traded company documents. It includes a dataset of annotated examples, source PDFs, and an evaluation playground supporting various retrieval modes such as singleStore, sharedStore, inContext, and oracle.

Tokens
2.3K
Snippets
8
Records
9
Agent score
7%

What's inside FinanceBench

  1. Understand the FinanceBench dataset structure

    main

    FinanceBench is a benchmark for evaluating LLMs on open-book financial question answering. The open-source sample contains 150 annotated examples. The dataset is composed of two primary JSONL files located in the /data/ directory and corresponding source PDF documents in /pdfs/.

    financebench_open_source.jsonl

    This file contains the questions and gold answers. Each record includes:

    • financebench_id (int): Unique identifier.
    • question (str): The question to be answered.
    • answer (str): Human-annotated gold answer.
    • dataset_subset_label (str): Identifies if the question is "OPEN_SOURCE" or "CLOSED_SOURCE".
    • evidence (list[dict]): A list of EvidenceDict objects containing:
      • evidence_text (str): Extracted text (sentence, paragraph, or page).
      • evidence_doc_name (str): Identifier of the document containing the evidence.
      • evidence_page_num (int): Zero-indexed page number.
      • evidence_text_full_page (str): The full page extract containing the evidence.
    • justification (str): Human-annotated reasoning for the gold answer.
    • question_type (str): One of 'metrics-generated', 'domain-relevant', or 'novel-generated'.
    • question_reasoning (str): The reasoning type required.
    • domain_question_num (str): ID for domain-relevant questions (dg01 to dg25), or "None".
    • company (str): The company of interest.
    • doc_name (str): Unique Document Identifier (Format: {COMPANY}_{PERIOD}_{TYPE}).

    financebench_document_information.jsonl

    This file provides metadata for the source documents:

    • doc_name (str): Unique Document Identifier.
    • doc_type (str): One of "10K", "10Q", "8K", "EARNINGS", or "10K_ANNUAL".
    • doc_period (int): The period of the document.
    • doc_link (str): URL to the document.
    • company (str): The company name.
    • comany_sector_gics (str): The company's GICS sector.

  2. Set up the FinanceBench Evaluation Playground

    main

    To use the evaluation playground, you must configure your environment with specific API keys and a required directory structure.

    1. API Requirements

    Add the following keys to your .env file:

    OPENAI_API_KEY = 'INSERT API KEY HERE'
ANTHROPIC_API_KEY = 'INSERT API KEY HERE'
REPLICATE_API_TOKEN = 'INSERT API KEY HERE'

    2. Required Folder Structure

    Ensure your project directory follows this layout:

    |-- /
|    |-- data/
|    |      | -- financebench_open_source.jsonl
|    |      | -- financebench_document_information.jsonl
|    |-- pdfs/
|    |      | -- <... provided filings as PDF documents ...>
|    |-- results/
|    |-- vectorstores/
|    |-- evaluation_playground.ipynb

    3. Dependencies

    Install pymupdf to handle PDF parsing:

    pip install pymupdf
    pip install pymupdf

  3. Load and join FinanceBench data using pandas

    main

    To create a complete dataset for analysis, you can load the question file and the metadata file using pandas and join them on the doc_name key.

    import pandas as pd

df_questions = pd.read_json("data/financebench_open_source.jsonl", lines=True)
df_meta = pd.read_json("data/financebench_document_information.jsonl", lines=True)
df_full = pd.merge(df_questions, df_meta, on="doc_name")

  4. Configure Model and Evaluation Modes

    main

    The evaluation playground uses a configuration list to define which LLM providers, models, and evaluation strategies to use.

    Evaluation Modes (eval_mode)

    • singleStore: Retrieval-only mode using a vector store specific to a single document.
    • sharedStore: Retrieval-only mode using a shared vector store.
    • inContext: Provides the full document text as context to the model.
    • inContext_reverse: Provides context before the question.
    • oracle: Provides specific evidence text as context.
    • oracle_reverse: Provides context before the question in oracle mode.
    • closedBook: The model answers the question without any provided context.

    Model Providers

    Supported providers include openai, anthropic, and replicate.

    configs = [
    {"provider": "openai", "model_name":"gpt-4o-2024-05-13", "eval_mode":"singleStore", "temp":0.01, "max_tokens":2048},
    {"provider": "anthropic", "model_name":"claude-2", "eval_mode":"inContext", "temp":0.01, "max_tokens":2048},
    {"provider": "replicate", "model_name":"llama2", "eval_mode":"sharedStore", "temp":0.01, "max_tokens":2048},
]

# Note: For retrieval-only mode, provider and model_name can be empty strings
{"provider": "", "model_name":"", "eval_mode":"singleStore", "temp":None, "max_tokens":None}

  6. Cite FinanceBench in research

    main

    If you use the open-source dataset or refer to the results in your work, please use the following BibTeX citation:

    @misc{islam2023financebench,
      title={FinanceBench: A New Benchmark for Financial Question Answering}, 
      author={Pranab Islam and Anand Kannappan and Douwe Kiela and Rebecca Qian and Nino Scherrer and Bertie Vidgen},
      year={2023},
      eprint={2311.11944},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

  7. Build a Vector Store Retriever

    main

    The build_vectorstore_retriever function creates a Chroma vector store from PDF documents.

    Parameters:

    • docs: Either the string "all" to build a shared vector store for all documents, or a list containing a specific doc_name to build a document-specific store.
    • embeddings: The embedding function to use (defaults to OpenAIEmbeddings()).

    Behavior:

    • It uses PyMuPDFLoader to load PDFs.
    • It uses RecursiveCharacterTextSplitter with VS_CHUNK_SIZE (1024) and VS_CHUNK_OVERLAP (30) to chunk text.
    • It persists the database in the vectorstores/ directory.
    retriever, vectordb = build_vectorstore_retriever(docs=["document_name"], embeddings=OpenAIEmbeddings())

  8. Generate Answers via `get_answer`

    main

    The get_answer function is the core execution unit for the evaluation. It handles different eval_mode strategies to prompt the model.

    Arguments:

    • model: The LLM instance.
    • eval_mode (str): The strategy (e.g., "closedBook", "oracle", "inContext", "singleStore").
    • question (str): The question to answer.
    • context (str): The text context (used for oracle and inContext modes).
    • retriever: The LangChain retriever (used for singleStore and sharedStore modes).
    • retriever_only (bool): If True, skips LLM calls and only returns retrieved documents.

    Returns:

    • A tuple of (answer, retrieved_documents) where answer is a string and retrieved_documents is a list of source documents.
    (answer, retrieved_documents) = get_answer(
    model=model, 
    eval_mode=model_config["eval_mode"], 
    question=row["question"], 
    context=context, 
    retriever=retriever
)

  9. Get Model Instances via `get_model`

    main

    The get_model function returns a LangChain-compatible model wrapper based on the provider.

    Arguments:

    • provider (str): "openai", "anthropic", or "replicate".
    • model_name (str): The specific model identifier (e.g., "gpt-4o-2024-05-13", "claude-2", or "llama2").
    • temp (float): Sampling temperature.
    • max_tokens (int): Maximum tokens to generate.

    Special Case: For replicate, the function uses an internal replicate_model_mapping to resolve model names to specific Replicate repository strings (e.g., llama2 maps to meta/llama-2-70b-chat:...).

    model = get_model(provider="openai", model_name="gpt-4o-2024-05-13", temp=0.01, max_tokens=2048)