DOCSEARCHbyAlgolia
Submit Docs

3DTeethSeg22 Challenge

repository·main·Indexed 42 days ago

https://github.com/abenhamadou/3dteethseg_miccai_challenges

Resources and evaluation framework for the 3DTeethSeg22 challenge, focused on the segmentation and labeling of 3D intra-oral teeth scans. Includes dataset format specifications using the FDI numbering system, performance metrics such as Teeth Localization Accuracy (TLA), Teeth Identification Rate (TIR), and Teeth Segmentation Accuracy (TSA), and guidelines for algorithm submission via Docker containers to Grand-Challenge.

Tokens
1.9K
Snippets
3
Records
8
Agent score
3%

What's inside 3DTeethSeg22

  1. Understand the 3DTeethSeg22 Dataset format

    main

    The dataset consists of 3D intra-oral scans (obj files) with ground truth tooth labels and instances provided in JSON format.

    Key details:

    • Labels: Uses the FDI numbering system.
    • Instances: Unique numbers indicate specific tooth instances.
    • Gingiva: The value 0 is reserved for gingiva in both labels and instances arrays.
    • Array Length: The length of the labels and instances arrays matches the total number of vertices in the corresponding .obj scan file.

    JSON Schema Example:

    {
    "id_patient": "6X24ILNE", 
    "jaw": "upper",
    "labels": [0, 0, 44, 33, 34, 0, 0, 45, 0, .. ,41,  0, 0, 37, 0, 34, 45, 0, 31, 36], 
    "instances": [0, 0, 10, 2, 12, 0, 0, 9, 0, 0, .. , 10, 0, 0, 8, 0, 0, 9, 0, 1, 8, 13],
}

  2. Configure a Dockerfile for algorithm submission

    main

    To wrap your algorithm in a Docker container, use the provided reference Dockerfile as a template. You must define three main components:

    1. Base Image: Choose an official image compatible with your requirements (e.g., PyTorch, TensorFlow).
    2. File Copying: Copy all necessary files (model weights, requirements.txt, Python scripts) into the container. Use --chown=algorithm:algorithm to ensure correct permissions.
    3. Dependencies: Install all dependencies specified in your requirements.txt using pip.
    # 1. Base Image
FROM pytorch/pytorch:1.9.0-cuda11.1-cudnn8-runtime

# 2. Copy files
COPY --chown=algorithm:algorithm requirements.txt /opt/algorithm/
COPY --chown=algorithm:algorithm model /opt/algorithm/model
COPY --chown=algorithm:algorithm model.pth /opt/algorithm/checkpoints

# 3. Install dependencies
RUN python -m pip install --user -r requirements.txt

  3. Build, test, and export your Docker container

    main

    Follow these steps to prepare your container for submission:

    1. Build: Run build.bat (Windows) or build.sh (Linux) to build the container and verify dependencies.
    2. Test:
      • Update test/expected_output.json with your algorithm's expected results for the sample test/0EJBIPTC_lower.obj.
      • Run test.bat (Windows) or test.sh (Linux). This runs the container on the sample input and compares the output to your expected JSON.
      • Success is indicated by the message: Tests successfully passed....
    3. Export: Run export.bat (Windows) or export.sh (Linux) to save the Docker image as a .tar.gz file for upload.

  5. Re-submit improvements to an existing algorithm

    main

    If you make improvements to your code, you do not need to create a new algorithm entry on Grand-Challenge.

    1. Go to your existing algorithm's page on grand-challenge.org/algorithms/.
    2. Navigate to Containers and click upload a container to upload a new .tar.gz file. This will overwrite the old container.
    3. Wait for the status to become Active.
    4. Crucial: Re-submitting the container does not automatically re-submit you to the challenge. You must manually visit the 3DTeethSeg22 submission page and create a new submission for your chosen track to trigger a new evaluation.

  6. Submit your algorithm to Grand-Challenge

    main

    Once your container is exported as a .tar.gz and tested locally, follow these steps to submit to the 3DTeethSeg22 challenge:

    1. Create Algorithm Entry: On grand-challenge.org/algorithms/create/, define your algorithm's title, description, and modalities.
      • Inputs: Select 3D Teeth scan (OBJ file).
      • Outputs: Select Dental labels (Anything).
      • Specify GPU and memory requirements.
    2. Attach Container: In your algorithm settings, go to the Containers menu and click upload a Container to upload your .tar.gz file. Wait for the status to change from Ready: False to Active.
    3. Verify (Try-out): Use the Try-out Algorithm tab to upload a sample scan (e.g., 0EJBIPTC_lower.obj) to ensure the live container works.
    4. Final Submission: Go to the 3DTeethSeg22 submission page, select your desired track (Preliminary or Final), choose your algorithm from the dropdown, and click Save.

  7. Evaluate segmentation and labeling performance

    main

    Performance is measured using three primary metrics. Metric calculation scripts are available in evaluation/evaluation.py.

    Metrics Definitions

    • Teeth localization accuracy (TLA): The mean of normalized Euclidean distance between ground truth (GT) teeth centroids and the closest localized teeth centroid. Distances are normalized by the size of the corresponding GT tooth. A nominal penalty of 5 is applied per GT tooth if no centroid is provided (e.g., algorithm crash).
    • Teeth identification rate (TIR): The percentage of true identification cases relative to all GT teeth. A true identification occurs when the closest detected tooth centroid is within half of the GT tooth size and shares the same label as the GT tooth.
    • Teeth segmentation accuracy (TSA): The average F1-score over all instances of teeth point clouds.
      • Formula: F1 = 2 * (precision * recall) / (precision + recall)

  8. Comply with 3DTeethSeg22 Input and Output interfaces

    main

    All algorithms submitted to the 3DTeethSeg22 challenge must follow a specific file-based interface using Docker volumes.

    Input:

    • Read an OBJ file located at /input/3d-teeth-scan.obj.

    Output:

    • Write a JSON file located at /output/dental-labels.json.

    The output JSON must follow the format used by the ground-truth files. It includes patient metadata, the jaw type, tooth labels in FDI format, and instance IDs for each 3D point.

    {
    "id_patient": "", 
    "jaw": "upper", # or "lower", extracted from input
    "labels": [0, 0, 44, 33, 34, 0, 0, 45, 0, .. ,41,  0, 0, 37, 0, 34, 45, 0, 31, 36], 
    # label of each 3D point in FDI format (label 0 corresponds to gingiva)
    "instances": [0, 0, 10, 2, 12, 0, 0, 9, 0, 0, .. , 10, 0, 0, 8, 0, 0, 9, 0, 1, 8, 13],
    # each 3D point with same instance corresponds only to one tooth
    # all points with the same instance should have the same label 
    # preferably tooth instance is in [1, 2, .. , number_of_tooth_detected]
    # by default 0 is attributed to gingiva instance
}