ROCK (Reinforcement Open Construction Kit)

The evaluation/ directory contains end-to-end demos that combine the sandbox lifecycle, agent installation/execution, and test suites. These are designed to show how individual components of ROCK fit together at the script level.

One primary example is swe_bench/, which demonstrates a single-task SWE-bench Verified workflow:

1. Starts a sandbox.
2. Installs an agent using sandbox.agent.install().
3. Runs the agent on a specific task.
4. Executes the test suite.
5. Parses the results.

Deciding whether to use the evaluation/swe_bench/ scripts or the job/harbor/ framework depends on your goal:

• Use evaluation/swe_bench/ (install-agent) when: You are debugging task setup or test parsing. These scripts provide full visibility into the entire pipeline in a single script form. It uses the Sandbox and sandbox.agent.install() APIs.
• Use job/harbor/ (Job) when: You are performing production benchmark runs. This uses the Job(JobConfig.from_yaml(...)).run() API.

The SDK uses camelCase for all TypeScript interfaces and properties. While the underlying HTTP layer may use snake_case, the SDK automatically converts these to camelCase.

Examples:

• Use status.sandboxId, NOT status.sandbox_id.
• Use result.exitCode, NOT result.exit_code.
• Use result.failureReason, NOT result.failure_reason.

The ROCK Job architecture has transitioned from a monolithic Job class in bench/job.py to a layered, decoupled design. This allows for better separation of concerns between job orchestration, execution, and specific task logic (Trials).

Architectural Layers

• Job (Facade): A thin entry point (rock/sdk/job/api.py) that assembles the JobConfig and Operator.
• Operator: Determines how many Trial instances to distribute (e.g., ScatterOperator).
• Executor: Manages the lifecycle of the sandbox, handles parallel startup, and manages nohup processes (rock/sdk/job/executor.py).
• Trial: Contains the specific task logic, including file uploads, script generation, and result parsing.
  • BashTrial: For simple bash commands.
  • HarborTrial: For Harbor-specific orchestration (rock/sdk/job/trial/harbor.py).

The VNC Proxy is designed to provide a seamless interface for noVNC web clients. It operates with the following routing logic and constraints:

Routing Logic

• Target Port: The proxy is hardcoded to forward all traffic to port 8006 within the sandbox. It ignores any rock_target_port query parameters provided in the URL.
• Path Passthrough: The [/{path}] segment allows the proxy to serve static files (e.g., core/rfb.js) required by the noVNC client.

Error Handling

• If the sandbox does not exist, the proxy returns a 404 error.
• If the sandbox exists but port 8006 is unreachable, the proxy returns a 502 error.

When running in Replay mode using --replay-file, the proxy behaves as follows:

• Sequential Consumption: Each request consumes one record from the file. When the file ends, the proxy returns a 404.
• Streaming Support: Replay mode simulates SSE chunks for streaming requests. It automatically injects the index field into tool_calls chunks to comply with the OpenAI streaming protocol.
• Model Matching: If the model requested in the API call does not match the model in the recorded record, a warning is issued, but the request is not blocked.

rock model-service start \
    --type proxy \
    --replay-file /tmp/rock-traj/LLMTraj.jsonl \
    --port 8081

Use the VNC Proxy (/proxy/vnc) instead of the General Proxy (/proxy) when specifically targeting VNC services. The primary difference is how ports are handled:

| Feature | General Proxy (/proxy) | VNC Proxy (/proxy/vnc) | | :--- | :--- | : | | Port Specification | Via rock_target_port query param | Fixed to port 8006 | | Default Port | Port.SERVER = 8080 | N/A (Always 8006) | | Route Format | /proxy/{path} | /proxy/vnc/{path} | | WebSocket Support | Yes | Yes |

The rock job run command operates under strict mutual exclusivity rules to ensure configuration clarity:

• YAML vs. Flags: You cannot use --job_config at the same time as --script or --script-content.
• Script vs. Content: In flags mode, you must choose either --script (a file path) or --script-content (an inline string). You cannot use both.
• Harbor Requirements: Harbor jobs must be defined via a YAML configuration file (--job_config). They cannot be expressed purely via CLI flags.
• Type Consistency: If you use the --type flag with a YAML file, the explicitly provided type must match the type detected in the YAML (e.g., if the YAML defines a BashJobConfig, --type must be bash).

ROCK enforces a strict hierarchical structure for datasets in OSS to ensure consistency across services. The path follows this pattern:

oss://{bucket}/datasets/{organization}/{dataset_name}/{split}/{task_id}/

Example Path: oss://my-bucket/datasets/qwen/my-bench/train/task-001/task.toml