JARVIS: Collaborative AI System

To build a custom tool library for TaskBench, you must create a JSON file containing tool descriptions. TaskBench supports two distinct formats depending on the tool type:

1. Type-specific parameters: Used for tools like image processors. These define input-type and output-type arrays.
2. API with request parameters: Used for functional APIs. These define a parameters array where each object specifies a name, type, and desc.

Use the appropriate format to ensure the correct dependency graph is generated later.

// Tool with type-specific parameters
{
  "id": "Image-to-Image",
  "desc": "Image-to-image is the task of transforming a source image...",
  "input-type": ["image"],
  "output-type": ["image"]
}

// API with request parameters
{
  "id": "send_sms",
  "desc": "Send an sms to a specific phone number",
  "parameters": [
    {
      "name": "phone_number",
      "type": "string",
      "desc": "The phone number to send the sms to"
    },
    {
      "name": "content",
      "type": "string",
      "desc": "The content of the sms"
    }
  ]
}

To run inference on the ToolBench dataset, follow these steps:

1. Download Tool Execution Code: Obtain the tool execution code (./data/toolenv/tools.) from the provided Google Drive or Tsinghua Cloud and save it to ./toolenv/tools.
2. Unzip Embeddings: Unzip the API description embeddings into the data directory:

   unzip data_toolbench/tool_instruction/API_description_embeddings.zip -d data_toolbench/tool_instruction/

3. **Execute Inference**: Use `main.py` with the appropriate `--task` and `--data_type` flags. Ensure `OPENAI_API_KEY` and `RAPIDAPI_KEY` are exported.

**Available Tasks and Data Types:**
- `--task toolbench` with `--data_type G2` or `G3`
- `--task toolbench_retrieve` with `--data_type G2` or `G3`

**Required Flags:**
- `--model_name`: The LLM to use (e.g., `gpt-3.5-turbo`).
- `--task`: The specific ToolBench task.
- `--data_type`: The data type variant (`G2` or `G3`).
- `--tool_root_dir`: Path to the tool environment (e.g., `./toolenv/tools`).

```bash
unzip data_toolbench/tool_instruction/API_description_embeddings.zip -d data_toolbench/tool_instruction/

export OPENAI_API_KEY="your_key"
export RAPIDAPI_KEY="your_key"

python3 main.py \
  --model_name gpt-3.5-turbo \
  --task toolbench \
  --data_type G2 \
  --tool_root_dir ./toolenv/tools

Experimental support for NVIDIA Jetson embedded devices is provided via a Dockerfile. It is recommended to use Jetson AGX Orin family devices (64G RAM preferred).

Setup Requirements

• Set the default docker runtime to nvidia.
• Use inference_mode: local and local_deployment: standard.
• Mount models and configs from the host to the container via volumes.

Deployment Steps

1. Build the image.
2. Run the container (automatically starts the model server).
3. Start the chat interface via docker exec.
4. Start the web application via docker exec.

# Build the docker image
docker build --pull --rm -f "Dockerfile.jetson" -t toolboc/nv-jarvis:r35.2.1 

# Run the container
docker run --name jarvis --net=host --gpus all -v ~/jarvis/configs:/app/server/configs -v ~/src/JARVIS/server/models:/app/server/models toolboc/nv-jarvis:r35.2.1

# Start awesome_chat.py
docker exec jarvis python3 awesome_chat.py --config configs/config.default.yaml --mode server

# Start the web application
docker exec jarvis npm run dev --prefix=/app/web