repository·main·Indexed 18 days ago
https://github.com/openbmb/infinitebenchA benchmark designed to evaluate Large Language Models (LLMs) on their ability to process and reason over extremely long contexts (100k+ tokens). It covers five key domains—Retrieval, Mathematics, Code, Question Answering (QA), and Summarization—using both real-world and synthetic data in English and Chinese. The benchmark includes 12 tasks with an average context length of 195K tokens and provides tools for inference and score computation using metrics such as Accuracy, ROUGE F1, and rougeLsum.
InfiniteBench evaluates models on 12 tasks designed for 100k+ token contexts. The evaluation metrics vary by task type:
Math.Find, Math.Calc, Code.Run, Code.Debug, En.Dia, En.MC, Retrieve.KV, Retrieve.Number, and Retrieve.PassKey.Zh.QA and En.QA.En.Sum.Task Details Overview:
| Task Name | Context | Avg Input Tokens | Description |
|---|---|---|---|
| En.Sum | Fake Book | 171.5k | Summarization of a fake book. |
| En.QA | Fake Book | 192.6k | Free-form QA on fake book. |
| En.MC | Fake Book | 184.4k | Multiple choice on fake book. |
| En.Dia | Script | 103.6k | Identifying talkers in scripts. |
| Zh.QA | New Book | 2068.6k | QA on newly collected books. |
| Code.Debug | Code Document | 114.7k | Finding crashing functions in code. |
| Code.Run | Synthetic | 75.2k | Simulating synthetic function execution. |
| Math.Calc | Synthetic | 43.9k | Long arithmetic equations. |
| Math.Find | Synthetic | 87.9k | Finding integers in a long list. |
| Retrieve.PassKey | Synthetic | 122.4k | Retrieving hidden keys in noise. |
| Retrieve.Number | Synthetic | 122.4k | Locating repeated numbers in noise. |
| Retrieve.KV | Synthetic | 89.9k | Finding dictionary values from keys. |
To evaluate a model (e.g., GPT-4) on a specific task, navigate to the src directory and run the evaluation script with the --task flag.
Available tasks for --task flag:
passkey (Retrieve.PassKey)number_string (Retrieve.Number)kv_retrieval (Retrieve.KV)longbook_sum_eng (En.Sum)longbook_qa_eng (En.QA)longbook_qa_chn (Zh.QA)longbook_choice_eng (En.MC)longdialogue_qa_eng (En.Dia)math_calc (Math.Calc)math_find (Math.Find)code_debug (Code.Debug)code_run (Code.Run)cd src
python eval_gpt4.py --task passkeyAfter running the inference scripts, use the compute_scores.py script to calculate the final performance metrics.
python compute_scores.pyInstall the required Python packages using the provided requirements file.
pip install -r requiremnets.txtTo run evaluations, ensure the data folder is in your current directory (or specify the location using the --data_dir argument). The data folder must contain the .jsonl files corresponding to the tasks.
Navigate to the src folder and execute the evaluation script for your target model and task.
Example commands:
python eval_yarn_mistral.py --task kv_retrieval
python eval_gpt4.py --task longbook_sum_qa
python eval_rwkv.py --task passkeyInstall the necessary dependencies for InfiniteBench using pip:
pip install -r requirements.txtpip install -r requirements.txtYou can download the InfiniteBench data using one of the following three methods:
Download directly from Hugging Face: https://huggingface.co/datasets/xinrongzhang2022/InfiniteBench
Use the datasets Python library to load the data with specific features:
from datasets import load_dataset, Value, Sequence
# Define the schema
ft = Features({
"id": Value("int64"),
"context": Value("string"),
"input": Value("string"),
"answer": Sequence(Value("string")),
"options": Sequence(Value("string"))
})
# Load the dataset
dataset = load_dataset("xinrongzhang2022/InfiniteBench", features=ft)Run the provided download script to dump data directly into a data folder:
cd InfiniteBench
bash scripts/download_dataset.shfrom datasets import load_dataset, Value, Sequence
ft = Features({"id": Value("int64"), "context": Value("string"), "input": Value("string"), "answer": Sequence(Value("string")), "options": Sequence(Value("string"))})
dataset = load_dataset("xinrongzhang2022/InfiniteBench", features=ft)The library implements specialized scoring logic for different evaluation paradigms:
| Task Category | Function | Logic Description |
|---|---|---|
| Retrieval | get_score_one_kv_retrieval | Checks if the label exists as a word in the prediction after cleaning punctuation. |
| Passkey/Number | get_score_one_passkey | Uses first_int_match to extract the first integer from the prediction and compares it to the label. |
| Code Execution | get_score_one_code_run | Extracts the last integer from the prediction and compares it to the label. |
| Code Debugging | get_score_one_code_debug | Uses regex to find the last option (A-J) or checks for specific answer prefixes (e.g., "answer is:"). |
| Math (Find) | get_score_one_math_find | Extracts the first integer or float from the prediction and compares it to the label. |
| Math (Calc) | get_score_one_math_calc | Compares the sequence of integers in the prediction against the sequence in the label. |
| Longbook QA | get_score_one_longbook_qa_eng | Uses token-based F1 score via qa_f1_score on normalized text. |
| Longbook Sum | get_score_one_longbook_sum_eng | Uses ROUGE-Lsum via the evaluate library. |
The eval_gpt4.py script allows you to evaluate models on InfiniteBench tasks using the OpenAI GPT-4 API. It processes examples from a specified task, generates predictions, and saves them to a JSONL file. The script supports resuming from where it left off if an output file already exists in the target directory.
To use this script, you must ensure your OpenAI API key and Organization ID are configured within the script (or via the environment if modified), and provide the necessary command-line arguments for the task and data directory.
python src/eval_gpt4.py --task <task_name> --data_dir <path_to_data> --output_dir <path_to_output> [--verbose]
# Example usage:
python src/eval_gpt4.py --task long_context_qa --data_dir ./data --output_dir ./results --verboseThe eval_yi_200k.py script is a specialized evaluation entrypoint designed for models with long context windows (up to 200k tokens), specifically optimized for the Yi-200k model architecture using vLLM.
It automates the process of loading a model, loading InfiniteBench datasets, generating predictions with specific truncation logic, and saving results to JSONL files. The script handles input truncation to ensure the context stays within the model's limits (using a 'middle' truncation strategy) before performing inference.
To use this script, you must provide arguments via the CLI (parsed by args.py). Key parameters include:
--model_path: Path to the model weights.--model_name: Name of the model (used for directory organization).--task: The InfiniteBench task name.--data_dir: Directory containing the dataset.--output_dir: Directory where results will be saved.--start_idx / --stop_idx: Range of examples to evaluate.--verbose: Enable detailed logging of inputs and predictions.# Example CLI invocation (assuming args.py defines these flags)
python src/eval_yi_200k.py \
--model_path /path/to/yi-200k \
--model_name yi-200k \
--task task_name \
--data_dir ./data \
--output_dir ./results \
--start_idx 0 \
--stop_idx 100The project uses a config.txt file to manage credentials and service endpoints. The load_and_log_configs() function parses this file into a structured dictionary.
Config Sections and Keys:
[API]Used for cloud providers:
anthropic_api_key, openai_api_key, cohere_api_key, groq_api_key, huggingface_api_key, openrouter_api_key, deepseek_api_key, mistral_api_keyanthropic_model, openai_model, etc.[Local-API]Used for self-hosted inference servers:
kobold_api_IP, llama_api_IP, ooba_api_IP, tabby_api_IP, vllm_api_IP, ollama_api_IP, aphrodite_api_IPkobold_api_key, llama_api_key, etc.[Paths]output_path: Where results are saved (default: results).[Processing]processing_choice: Hardware preference (e.g., cpu).