ScaleCUA

LibreOffice Calc evaluation requires specific libraries and a method for comparing spreadsheet data.

Required Libraries

pip install openpyxl pandas lxml xmltodict

Generating CSV from XLSX

To compare displayed cell values, you must first convert the XLSX to CSV using the following command structure:

libreoffice --convert-to "csv:Text - txt - csv (StarCalc):44,34,UTF8,,,,false,true,true,false,false,1" --out-dir /home/user /home/user/abc.xlsx

Note: The final 1 in the conversion options specifies the sheet number (starting from 1) to export.

Using compare_table

Evaluation relies on the compare_table function, which accepts two filenames and an options list of rules. Each rule requires a type:

• sheet_data: Compares internal cell values via pandoc.
• sheet_print: Compares shown cell values via CSV (requires the CSV generation step above).

Sheet Indexing (sheet_idx0, sheet_idx1, or sheet_idx):

• Integer i: Extracts the $i$-th sheet from the result (starts at 0).
• String with prefix:
  • R: Extract from result xlsx.
  • E: Extract from expected (golden) xlsx.
  • I: Indicates a sheet number (starting from 0).
  • N: Indicates a sheet name (e.g., "Sheet1").

Rules can also use matching patterns like {"method": "eq", "ref": "abc"}, which are processed by utils._match_value_to_rule.

To run the ScaleCUA playground interactively, follow these steps:

1. Deploy Models: Deploy ScaleCUA models using vLLM. Record the model name and the deployment URL.
2. Configure Models: Update the base_url and model fields in your agent configuration file (e.g., playground/config/agent/scalecua_native_agent.yaml) to point to your deployed model.
3. Configure Environment: Set up the environment parameters (Ubuntu, Web, or Android) in the corresponding YAML config file.
4. Run the Agent: Execute the interactive agent script using the agent_run_interactive.py command.

Supported platforms for the --platform flag include: ubuntu, web, android, and ubuntu_web.

```bash
python agent_run_interactive.py \
  --platform ubuntu \
  --env_config_path playground/config/envs/ubuntu.yaml \
  --agent_config_path playground/config/agent/scalecua_native_agent.yaml
```埋

To develop a new task in WAA, you must create a JSON configuration that defines the task's identity, the natural language instruction, the initial environment setup, the evaluation logic, and the expected result format.

Best Practices:

• Reuse existing machinery: If your task involves a category already supported (e.g., Chrome, Edge, LibreOffice), do not start from scratch. Instead, study and modify the existing getters and evaluators for that application.
• Study existing examples: Review the repository's current task examples to understand common command patterns and configuration setups.

Task Components:

• id: A unique identifier for the task.
• instruction: A natural language string describing what the agent should do.
• config: An array of setup actions to prepare the environment (e.g., launch a program or execute a command).
• evaluator: Defines the success criteria. It includes a function name (func) and the expected outcome to compare against for reward calculation.
• result: (Optional) Specifies the type of evaluation result expected (e.g., checking a specific configuration file).

{
    "id": "8ba5ae7a-5ae5-4eab-9fcc-5dd4fe3abf89-W0S",
    "instruction": "Help me modify the folder used to store my recordings to the Desktop",
    "config": [
        {
            "type": "launch",
            "parameters": {
                "command": "vlc"
            }
        },
        {
            "type": "execute",
            "parameters": {
                "command": [
                    "python",
                    "-c",
                    "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"
                ]
            }
        }
    ],
    "evaluator": {
        "func": "vis_vlc_recordings_folder",
        "expected": {
            "type": "rule",
            "rules": {
                "recording_file_path": "C:\\Users\\Docker\\Desktop"
            }
        }
    },
    "result": {
        "type": "vlc_config",
        "dest": "vlcrc"
    }
}