ToolBench Documentation

To evaluate your own model, you must organize your predictions into a specific directory structure and then preprocess them.

1. Directory Structure: Create a directory named after your model/method (e.g., chatgpt_cot) containing subdirectories for the six test sets (G1_instruction, G1_category, G1_tool, G2_category, G2_instruction, G3_instruction). Each subdirectory should contain the prediction JSON files.

2. Preprocessing: Run the convert_to_answer_format.py script to convert raw predictions into the required internal format. You must set the following environment variables:

   • RAW_ANSWER_PATH: Path to your raw model predictions.
   • CONVERTED_ANSWER_PATH: Path where processed JSON files will be stored.
   • MODEL_NAME: Name of your model.
   • METHOD: The method used (e.g., CoT).

Example preprocessing loop:

export RAW_ANSWER_PATH=../../data/reproduction_data/model_predictions/
export CONVERTED_ANSWER_PATH=../../data/reproduction_data/model_predictions_converted/
export MODEL_NAME=chatgpt_cot
export METHOD=CoT
mkdir ${CONVERTED_ANSWER_PATH}/${MODEL_NAME}
for test_set in G1_instruction G1_category G1_tool G2_category G2_instruction G3_instruction
do
    answer_dir=${RAW_ANSWER_PATH}/${MODEL_NAME}/${test_set}
    output_file=${CONVERTED_ANSWER_PATH}/${MODEL_NAME}/${test_set}.json
    python convert_to_answer_format.py\ 
        --answer_dir ${answer_dir} \ 
        --method ${METHOD} \ 
        --output ${output_file}
done

To perform inference with ToolLLaMA, you must first initialize your TOOLBENCH_KEY (obtained via questionnaire). Use toolbench/inference/qa_pipeline.py for standard inference, LoRA-based inference, or open-domain inference.

Standard ToolLLaMA Inference LoRA ToolLLaMA Inference (requires --lora and --lora_path) Open-Domain LoRA Inference (requires --corpus_tsv_path and --retrieval_model_path)

export TOOLBENCH_KEY="your_toolbench_key"
export PYTHONPATH=./

# Standard ToolLLaMA Inference
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path /path/to/your/toolllama \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY

# LoRA Open-Domain Inference
python toolbench/inference/qa_pipeline_open_domain.py \
    --tool_root_dir data/toolenv/tools/ \
    --corpus_tsv_path data/retrieval/G1/corpus.tsv \
    --retrieval_model_path /path/to/your/retrival_model \
    --retrieved_api_nums 5 \
    --backbone_model toolllama \
    --model_path huggyllama/llama-7b \
    --lora \
    --lora_path /path/to/your/toolllama_lora \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_lora_dfs_open_domain_inference_result \
    --toolbench_key $TOOLBENCH_KEY

The ToolBench dataset is available under the Apache License 2.0. You can download the latest version (containing over 120k reasoning paths and full reasoning thoughts) via Google Drive or Tsinghua Cloud Drive.

Note: data_0801.zip refers to the older version of the data.

File Structure:

├── /data/
│  ├── /instruction/
│  ├── /answer/
│  ├── /toolenv/
│  ├── /retrieval/
│  ├── /test_instruction/
│  ├── /test_query_ids/
│  ├── /retrieval_test_query_ids/
│  ├── toolllama_G123_dfs_train.json
│  └── toolllama_G123_dfs_eval.json
├── /reproduction_data/
│  ├── /chatgpt_cot/
│  ├── /chatgpt_dfs/
│  ├── ...
│  └── /toolllama_dfs/

Directory Descriptions:

• instruction and answer: Instruction data and solution path annotations. G1, G2, and G3 refer to single-tool, intra-category multi-tool, and inter-category multi-tool data respectively.
• toolenv: Tool environment data, including API JSON, API code, and API example returns.
• retrieval: Data for tool retrieval.
• test_instruction and test_query_ids: 200 instances sampled from each test set (queries and query IDs).
• retrieval_test_query_ids: Query IDs for retriever test instances.
• toolllama_G123_dfs_train.json and toolllama_G123_dfs_eval.json: Preprocessed data for direct ToolLLaMA training and result reproduction.

You can add custom APIs for close-domain inference by following these steps:

1. Create API Documentation: Create a JSON file (e.g., hello_world.json) following the required schema (including tool_description, tool_name, api_list, etc.).
2. Place Documentation: Put the JSON file in a category folder under data/toolenv/tools/ (e.g., data/toolenv/tools/Customized/hello_world.json).
3. Implement API Code: Create a folder named after the tool inside the category (e.g., data/toolenv/tools/Customized/hello_world/) and write the implementation in api.py.
4. Update Query File: Modify your input query JSON to include the category_name, tool_name, and api_name for the custom tool.
5. Run Inference: Use the --api_customization flag.

Note: Currently, customized API usage is only supported in close-domain settings.

# Example API implementation in api.py
def get_hello_world():
    """
    To get hello world 
    """
    observation = "hello world"
    return observation

# Running inference with customized API
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path ToolBench/ToolLLaMA-7b \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file /path/to/your/query/file \
    --output_answer_file /path/to/your/output/file \
    --api_customization

To implement a custom automatic evaluator:

1. Create a config folder: Under toolbench/tooleval/evaluators, create a folder named after your evaluator. Include a config.yaml (required) and an optional template.txt.
2. Implement the Evaluator Class: Create a class inheriting from BaseEvaluator and implement the fn_completions method. Use the @register_evaluator decorator to register it.
3. Register in Config: Set the registered_cls_name in your config.yaml to the name of your class.
4. Test: Run evaluators_comparison.py to verify performance.

Implementation Example:

from evaluators import register_evaluator,BaseEvaluator
from typing import Dict,List

@register_evaluator
class MyEvaluator(BaseEvaluator):
    def __init__(self,config):
        super().__init__(
            fn_completions=self.fn_completions,
        )
        # set your configures here
    
    def fn_completions(self,query:Dict,answers:List[Dict])->int:
        # implement your evaluator here
        # return the index of the preferred answer
        return 0

You can perform inference using ToolLLaMA (standard, LoRA, or open-domain) or OpenAI models (ChatGPT or Text-Davinci-003) via the qa_pipeline.py script.

For RapidAPI-based inference, you must set the TOOLBENCH_KEY environment variable. If using your own RapidAPI account, provide RAPIDAPI_KEY and use the --use_rapidapi_key flag.

# Setup key
export TOOLBENCH_KEY="your_toolbench_key"

# ToolLLaMA Inference
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path ToolBench/ToolLLaMA-7b \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY

# OpenAI ChatGPT Inference
export OPENAI_KEY="your_openai_key"
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model chatgpt_function \
    --openai_key $OPENAI_KEY \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file chatgpt_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY

# Custom RapidAPI Account
export RAPIDAPI_KEY="your_rapidapi_key"
export OPENAI_KEY="your_openai_key"
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model chatgpt_function \
    --openai_key $OPENAI_KEY \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file chatgpt_dfs_inference_result \
    --rapidapi_key $RAPIDAPI_KEY \
    --use_rapidapi_key