repository·main·Indexed 19 days ago
https://github.com/allenai/vla-evaluation-harnessA unified framework for evaluating Vision-Language-Action (VLA) models across various robot simulation benchmarks. It utilizes a model-server/benchmark-client architecture to decouple models from environments. Supported benchmarks include BEHAVIOR-1K, CALVIN, DuoBench, Kinetix, LIBERO (including Mem, Plus, and Pro versions), ManiSkill2, Mikasa, MolmoSpaces-Bench, RLBench, and RoboCasa.
Kinetix is a physics-based 2D manipulation benchmark implemented in JAX. It is used within the vla-evaluation-harness to evaluate Vision-Language-Action (VLA) models on 2D manipulation tasks.
To run Kinetix benchmarks, use the following Docker image:
ghcr.io/allenai/vla-evaluation-harness/kinetix:latest
RoboCasa365 is a multi-task kitchen manipulation benchmark built on MuJoCo and robosuite. It evaluates models on 50 target tasks across three categories: atomic_seen, composite_seen, and composite_unseen in pretrain kitchens.
Important Note: This implementation uses a specific Docker image that pins RoboCasa 1.0.1 and robosuite 1.5.2. Because this is a separate integration from the original RoboCasa repository, scores are not directly comparable across the two.
DuoBench is a benchmark for bimanual manipulation tasks using a Franka Research 3 Duo robot in the MuJoCo simulator. It consists of 11 distinct tasks designed to evaluate bimanual control capabilities.
Tasks included:
ball_mazebin_sortblock_balancecarry_pothinge_chestjoin_blockspour_marblesspring_doortransfer_cubetransfer_gatetransfer_reorientRLinf is a distributed reinforcement learning (RL) infrastructure designed to train Vision-Language-Action (VLA) models (such as Pi0, OpenVLA, and GR00T) using algorithms like PPO, GRPO, and SAC. It utilizes a Ray-based scheduler and NCCL collective communications for distributed scaling.
Key technical details:
rlinf/envs/ structure is a key reference point for designing environment wrappers.vla-evaluation-harness provides various configuration files to run different starVLA paradigms across multiple benchmarks like SimplerEnv, CALVIN, and LIBERO.The vla-eval harness provides systematic verification of Vision-Language-Action (VLA) model scores across various codebases and benchmarks. It compares reproduced scores against originally reported scores to ensure consistency.
·: No score or no checkpoint available.Note: CI is binomial at p=0.95 (±1.9pp for 500 episodes).
RoboTwin 2.0 is a dual-arm manipulation benchmark built on SAPIEN. It consists of 50 tasks categorized into Easy and Hard protocols. For running evaluations, use the provided Docker image.
Docker image: ghcr.io/allenai/vla-evaluation-harness/robotwin:latest
client-server model where the environment runs in a container and the agent interacts with it through a client.RoboChallenge is an external benchmark for large-scale real-robot evaluation of embodied policies.
Key Details:
success_rate (unit: %, range: 0-100, higher is better).Important Note for Developers: This benchmark is marked as external_only. The vla-evaluation-harness does not mirror these results locally. The leaderboard.json file in this repository must contain zero rows for this benchmark. A previous API mirror was retired in July 2026.
The VLA Model Leaderboard provides monthly updated performance rankings for Vision-Language-Action (VLA) models. You can view the current standings and results at the official hosted URL.
https://allenai.github.io/vla-evaluation-harness/leaderboard/Traditional VLA benchmarks are synchronous, meaning the simulation environment pauses while the agent is performing inference. This does not reflect real-world robot deployment, where slow inference can lead to failures as the environment continues to change.
vla-evaluation-harness introduces Real-Time Evaluation mode, where the environment time flows independently of the agent's execution. This allows developers to:
The ManiSkill2 benchmark evaluates models using the success_rate metric (0-100%).
To ensure comparability and enable a sortable overall_score, models must be evaluated on the standard 5-task set:
PickCubeStackCubePickSingleYCBPickSingleEGADPickClutterYCBoverall_score: This is the arithmetic mean of the 5 standard tasks listed above. If a model uses a different subset of tasks, the overall_score must be set to null.task_scores: This object contains the per-task success rates. Keys must match the PascalCase task names exactly (e.g., PickCube, OpenCabinetDoor).suite_scores: Not used for ManiSkill2.Task subsets vary significantly across different papers. Because different subsets are used, scores across different task subsets are not directly comparable. Only entries using the standard 5-task set are ranked in the leaderboard.