CogAgent Documentation

To fine-tune the cogagent-9b-20241220 model using multi-round dialogue, organize your data in a JSON format where each entry contains a messages array.

Key requirements:

• Each entry must include an image path.
• The user role's content must include a Task description and a History steps section.
• The History steps section must include all previous operations (e.g., CLICK, SCROLL_DOWN, TYPE) to provide context for the current step.
• For intermediate steps, the assistant response should end with <<General Operation>>.
• For the final step in a process, the assistant response should end with <<END>>.

{
  "messages": [
    {
      "role": "user",
      "content": "Task: ...\n(Platform: Mac)\n(Answer in Action-Operation-Sensitive format.)\nHistory steps: \n0. CLICK(box=[[588,946,616,985]], element_info='System Settings')\t...\n",
      "image": "images/0000000000336.png"
    },
    {
      "role": "assistant",
      "content": "Action: ...\nGrounded Operation:CLICK(box=[[655,842,671,857]], element_info='...')\n<<General Operation>>"
    }
  ]
}

To set up the CogAgent client on macOS, follow these steps to ensure tkinter and the virtual environment are correctly configured.

Note: Do not use conda or virtualenv as they may cause tkinter to malfunction. Use the built-in venv module with the --copies flag.

1. Install python-tk and project dependencies:

brew install python-tk
pip install -r requirements.txt

2. Verify tkinter installation:

/opt/homebrew/bin/python3 -m tkinter

3. Create a virtual environment using --copies:

/opt/homebrew/bin/python3 -m venv --copies /path/to/your/venv

4. Verify the virtual environment's tkinter:

/path/to/your/venv/bin/python3 -m tkinter

5. Create a caches directory for model images:

mkdir caches

brew install python-tk
pip install -r requirements.txt

# Create venv with copies to avoid tkinter issues
/opt/homebrew/bin/python3 -m venv --copies /Users/zr/Code/CogAgent/venv

mkdir caches

CogAgent uses a specific conversation format for multi-turn fine-tuning. The data should be organized in a JSON format where each entry contains a messages list.

Key requirements:

• Role-based loss masking: Different roles are assigned different loss_mask values to calculate loss for multi-turn responses in a single pass.
• Image requirement: Every data entry must include an image path.
• History steps: For the cogagent-9b-20241220 model, the user role's content must include a History steps: section containing all previous operations in the sequence.
• Termination:
  • Intermediate steps should return <<一般操作>> (General Operation).
  • The final step in a complete process should return <<END>>.

Example structure:

{
  "messages": [
    {
      "role": "user",
      "content": "Task: ...\nHistory steps: \n0. CLICK(...) ...",
      "image": "images/0000000000336.png"
    },
    {
      "role": "assistant",
      "content": "Action: ...\nGrounded Operation:CLICK(...)\n<<一般操作>>"
    }
  ]
}

{
  "messages": [
    {
      "role": "user",
      "content": "Task: 在系统设置的桌面与程序坞部分，开启调度中心板块中“显示器具有单独空间”的选项\n(Platform: Mac)\n(Answer in Action-Operation-Sensitive format.)\nHistory steps: \n0. CLICK(box=[[588,946,616,985]], element_info='系统设置')\t点击屏幕下方的“系统设置”菜单，从而打开系统设置\n1. CLICK(box=[[227,561,297,585]], element_info='桌面与程序坞')\t点击左侧菜单中的“桌面与程序坞”选项，进入桌面与程序坞的设置界面.\n2. SCROLL_DOWN(box=[[367,39,691,929]], step_count=75, element_info='滚动')\t在当前屏幕的右侧滚动区域中，向下滚动以查看更多选项.\n",
      "image": "images/0000000000336.png"
    },
    {
      "role": "assistant",
      "content": "Action: 点击“调度中心”板块中的“显示器具有单独空间”选项，从而开启该功能.\nGrounded Operation:CLICK(box=[[655,842,671,857]], element_info='“显示器具有单独空间”的开关按钮')\n<<一般操作>>"
    }
  ]
}

CogAgent (cogagent-9b-20241220) is an Agent-class execution model, not a conversational model. It does not support continuous dialogue natively; instead, you must provide the full execution history in every new request to maintain context.

To achieve optimal GUI Agent performance, you must follow a strict input format. The prompt should be constructed in the following order: query = f'{task}{history}{platform}{format}'

Key Components:

• task: A concise text description of the user's goal.
• history: A list of previous steps. Even if there is no history, you must include the string History steps: .
• platform: The operating system identifier.
• format: The desired output structure.
• [Continue] (Optional): To ask the model to continue its previous response, insert [Continue]\n immediately after the {task} field.

current_platform = identify_os() # "Mac" or "WIN" or "Mobile"
platform_str = f"(Platform: {current_platform})\n"
format_str = "(Answer in Action-Operation-Sensitive format.)\n"

history_str = "\nHistory steps: "
for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
   history_str += f"\n{index}. {grounded_op_func}\t{action}"

query = f"Task: {task}{history_str}\n{platform_str}{format_str}"