DOCSEARCHbyAlgolia
Submit Docs

CodaLab Competitions Documentation

repository·develop·Indexed 19 days ago

https://github.com/codalab/codalab-competitions

An open-source platform for organizing and participating in scientific machine learning challenges and data-oriented research competitions. The documentation covers local installation via Docker Compose, administrative tools for stress testing and email approval, and a comprehensive API for creating competitions, managing participant registration, submitting entries, and retrieving submission scores and logs.

Tokens
16.5K
Snippets
56
Records
82
Agent score
68%

What's inside CodaLab Competitions

  1. Overview of CodaLab Competitions

    develop

    CodaLab is an open-source web-based platform designed for researchers, developers, and data scientists to collaborate on machine learning and advanced computation research. It facilitates data-oriented research through an online community where users can share worksheets and participate in scientific competitions.

    Key resources:

  2. Quick installation of CodaLab Competitions (Linux)

    develop

    To host your own instance of CodaLab Competitions, you can use Docker Compose. This requires Docker to be installed and your user to be part of the docker group.

    Follow these steps to set up a local instance:

    1. Install Docker and configure permissions:
      wget -qO- https://get.docker.com/ | sh
sudo usermod -aG docker $USER
    2. Clone the repository and initialize the environment:
      git clone https://github.com/codalab/codalab-competitions
cd codalab-competitions
cp .env_sample .env
pip install docker-compose
docker-compose up -d
    3. Access the platform: Once the containers are running, access the interface at http://localhost/.

    Note: If you only want to participate in or organize competitions on an existing platform, no installation is required; you can simply sign in to an active instance like codalab.lisn.fr.

    $ wget -qO- https://get.docker.com/ | sh
$ sudo usermod -aG docker $USER
$ git clone https://github.com/codalab/codalab-competitions
$ cd codalab-competitions
$ cp .env_sample .env
$ pip install docker-compose
$ docker-compose up -d

  3. Manage Organizer Data Sets

    develop

    The OrganizerDataSet model is used to manage various types of data provided by organizers. These datasets can be bundled together and assigned specific roles within a competition.

    Supported Data Types:

    • Reference Data
    • Scoring Program
    • Input Data
    • Ingestion Program
    • Starting Kit
    • Public Data
    • None

    Key Features:

    • Bundling: Datasets can be grouped using the sub_data_files field (a ManyToMany relationship to other OrganizerDataSet instances).
    • Metadata: You can generate signed URLs for all files in a bundle using write_multidataset_metadata(). This creates a metadata file within the main dataset containing signed links valid for 100 years.
    • Automatic Naming: The full_name field is automatically populated with the format: {name} uploaded by {user}.
    # Example of the available types for OrganizerDataSet
TYPES = (
    ("Reference Data", "Reference Data"),
    ("Scoring Program", "Scoring Program"),
    ("Input Data", "Input Data"),
    ("Ingestion Program", "Ingestion Program"),
    ("Starting Kit", "Starting Kit"),
    ("Public Data", "Public Data"),
    ("None", "None")
)

  4. Manage Competition Content Visibility

    develop

    Competition content (categories and items) can be controlled via ContentVisibility. There are three visibility modes:

    • Hidden
    • Visible
    • Always Visible

    ContentCategory objects (which use an MPTT tree structure) and DefaultContentItem objects (which serve as children to categories) both reference a ContentVisibility instance to determine how they are displayed to users.

  5. Configure Competition Phase Settings

    develop

    A Competition consists of one or more CompetitionPhase objects. Each phase can be customized with the following settings:

    • Execution & Limits: execution_time_limit (in seconds), max_submissions (per user), and max_submissions_per_day (per user).
    • Data & Scoring: scoring_program (the file used for scoring), reference_data (reference files), input_data (input files), and datasets (associated Dataset objects).
    • Docker Configuration: scoring_program_docker_image, default_docker_image, and disable_custom_docker_image.
    • Leaderboard Management: leaderboard_management_mode (controls when results become visible) and force_best_submission_to_leaderboard.
    • Storage Limits: max_submission_size (max MB per submission) and participant_max_storage_use (max MB per participant).

  6. Manage Competition Phase Status

    develop

    Competition phases have several lifecycle states that can be checked via properties:

    • is_active: Returns True if the phase is currently ongoing. A phase is active if the current date is between its start_date and the start_date of the next phase, or if it is the final phase and the competition is active.
    • is_future: Returns True if the phase has not yet started (timezone.now() < self.start_date).
    • is_past: Returns True if the phase has already ended (not active and not in the future).
    • is_blind: Returns True if results are always hidden from participants (based on leaderboard_management_mode).

  7. Manage Job status and transitions

    develop

    Jobs follow a specific lifecycle defined by integer status codes. You can interact with these using friendly names (e.g., 'running', 'finished').

    Available Statuses:

    • PENDING (0)
    • RUNNING (1)
    • FINISHED (2)
    • FAILED (3)

    Transition Rules:

    • Once a job is FINISHED or FAILED, it cannot transition to any other state.
    • A PENDING job can transition to any state except PENDING itself.
    • A RUNNING job can only transition to FINISHED or FAILED.

  8. Configure Leaderboard columns and groups

    develop

    Leaderboards in CodaLab are structured using SubmissionResultGroups and SubmissionScoreDefs. When defining a competition via a bundle, you can organize scores into groups and define how they are displayed.

    Score Definition Options (SubmissionScoreDef):

    • key: Unique identifier for the score.
    • label: Display name.
    • sorting: asc (Ascending) or desc (Descending).
    • numeric_format: String defining decimal precision (e.g., "2").
    • show_rank: Boolean to show/hide rank.
    • ordering: Integer to control display order.
    • computed: Boolean indicating if the score is derived from other scores.

    Computed Scores: Computed scores are defined via SubmissionComputedScore and require:

    • operation: The mathematical operation to perform.
    • weights: A comma-separated string of floats used for weighted averages.
    • fields: A comma-separated list of keys for the score definitions that this computed score depends on.

  9. Configure Submission Computed Scores

    develop

    Computed scores allow for aggregating multiple score definitions using specific mathematical operations. This is managed via the SubmissionComputedScore and SubmissionComputedScoreField models.

    Supported Operations:

    • Max: Returns the maximum value.
    • Avg: Returns the average value.
    • MRR: Returns the Mean Reciprocal Rank.

    Constraints:

    • A SubmissionComputedScoreField cannot be used if the associated SubmissionScoreDef is already marked as computed.
    • You can provide weights as a string to influence the computation.
    # Available operations for SubmissionComputedScore
choices=(('Max', 'Max'), ('Avg', 'Average'), ('MRR', 'MRR'))

  10. Manage Competition Submission constraints

    develop

    When saving a CompetitionSubmission, the system enforces several constraints. If any are violated, a PermissionDenied exception is raised:

    • Submission Limits: The number of submissions must not exceed phase.max_submissions.
    • Daily Limits: The number of successful submissions per day must not exceed phase.max_submissions_per_day.
    • Competition End Date: Submissions are not allowed if the current date is past the competition's end_date.
    • Phase Size Limit: The submission file size must not exceed phase.max_submission_size (converted from MB to bytes).
    • Participant Storage Limit: The new submission plus the participant's existing storage usage must not exceed phase.participant_max_storage_use.

  11. Manage Competition Phase Leaderboards

    develop

    Leaderboards are tied to specific CompetitionPhase instances via the PhaseLeaderBoard model.

    Key Behaviors:

    • Open/Closed Status: A leaderboard's is_open status typically mirrors the is_active status of its associated phase. If the phase is inactive, the leaderboard is considered closed.
    • Submission Entry: The PhaseLeaderBoardEntry links a specific CompetitionSubmission to a leaderboard.
    • Adding Submissions: When adding a submission to a leaderboard via add_submission_to_leaderboard(submission), the system automatically handles replacement logic: if a participant or team already has an entry in that specific leaderboard, the old entry is deleted before the new one is created. This ensures only the most recent submission is reflected in the leaderboard.

  12. Configure Competition Settings

    develop

    The Competition model is the central entity for managing a competition. Key configuration options include:

    • Registration & Teams: has_registration (requires registration), allow_teams (enables team functionality), and require_team_approval (organizers must approve new teams).
    • Visibility & Access: published (makes the competition publicly available), anonymous_leaderboard (hides usernames on the leaderboard), and url_redirect (redirects participants to an external URL).
    • Submission Limits: upper_bound_max_submission_size (sets the maximum allowed size for a submission in MB).
    • Features: enable_forum, enable_teams, enable_detailed_results, and enable_medical_image_viewer.
    • Leaderboard Control: hide_top_three, hide_chart, and force_submission_to_leaderboard.