Codabench

Codabench is a platform for creating flexible benchmarks. The workflow typically involves:

1. Defining Tasks: Specifying datasets and metrics of success.
2. Specifying Submission APIs: Defining how algorithms (code) should be submitted.
3. Adding Documentation: Providing context for participants.
4. Leaderboard Management: Once created, the benchmark accepts submissions, and participant results are automatically appended to a leaderboard.

Codabench also supports inverted benchmarks, where the roles are swapped: you specify reference algorithms, and participants submit datasets.

Codabench is designed as a modern upgrade to CodaLab. While it includes all features from CodaLab Competitions, it introduces several new capabilities:

• Live Logs: View logs in real-time during the submission process.
• Storage Quotas: Management of storage limits.
• Computation Server Management: Tools for all users to manage servers.
• Benchmarking Focus: Enhanced support for dataset submissions and the ability to have multiple leaderboard rows per user.
• Interface: A faster and more intuitive user interface.

Human-in-the-Loop (HITL) is a feature for private Compute Workers that introduces a manual validation step into the submission workflow.

When enabled, the Compute Worker executes the scoring program but pauses after completion. It waits for a human operator to manually approve or reject the results before any scores, detailed results, or output archives are published to the Codabench site. This is useful for competitions requiring manual verification, such as medical AI or security-sensitive evaluations.

The competition management interface provides two primary views for tracking competitions via tab navigation:

• Competitions I'm running tab: (Default view) Displays all competitions you currently manage or run. This view includes administrative actions such as editing, publishing, or deleting.
• Competitions I'm in tab: Displays a list of competitions where you are a participant. This view does not include administrator options. Clicking a competition title in this list navigates you to that competition's detail page.

Codabench uses Minio as its default object storage solution.

• Port Configuration: The Minio port is defined in your .env file.
• Bucket Initialization: A helper container named Create Buckets runs during startup. It checks for the existence of buckets defined in your .env settings and creates them if they are missing. This container typically exits after the initialization is complete.

You can provide large or confidential datasets directly to the worker to avoid downloading them during every job.

1. Place your data in the folder $HOST_DIRECTORY/data on the host machine (e.g., /codabench/data).
2. This folder is automatically mounted inside the competition container at /app/data.
3. Requirement: You must modify your scoring or ingestion programs to point to /app/data to read this information.

Warning: Ensure the file/folder ownership matches the user running the worker:

• Use root for Docker rootfull.
• Use codalab for Podman or Docker rootless (if you created a codalab user).

# Example: If HOST_DIRECTORY=/codabench
# Host path: /codabench/data
# Container path: /app/data

Ranking is determined using a hierarchical sorting process:

1. Primary Column: Ranking is first determined by the column at index 0 in the competition.yaml (though this can be adjusted in the competition editor). Sorting follows the asc or desc preference of that column.
2. Secondary Columns: If scores are identical, the platform sorts by subsequent columns from left to right (based on their index).
3. Tie-breaking: If all column scores are identical, the submitted_at timestamp is used as the final tie-breaker. Earlier submissions are ranked higher.

It is critical to distinguish between the two types of Docker environments in Codabench:

1. Codabench Services: The infrastructure containers (Django, Postgres, Rabbit, etc.) that run the Codabench platform itself.
2. Submission Containers: The isolated Docker environments used to run individual user submissions. These are specific to each benchmark and are managed separately from the core platform services.

For details on the submission environment, see Submission Docker Container Layout.