EVA Evaluation Framework

The eva project supports language-based datasets for biomedical tasks.

• PubMedQA: A dataset containing 1,000 samples for Multiple Choice Biomedical Q&A. Automatic download is provided.

The eva project supports multimodal datasets that combine image and text modalities for various tasks. Automatic download is provided for all listed datasets:

• PatchCamelyon: Image + Text for Multiple Choice tasks (Domain: Breast Cancer).
• PathMMU Atlas: Image + Text for Multiple Choice tasks (Domain: Pathology).
• QuiltVQA: Image + Text for Free-form Visual Question Answering (VQA) (Domain: Histopathology).

UniToPatho is a vision dataset consisting of 9,536 hematoxylin and eosin (H&E) stained patches extracted from 292 whole-slide images. It is designed for training deep neural networks for colorectal polyp classification and adenoma grading.

For the current benchmark, only the 800 subset is used, which contains 8,669 images with a resolution of 1812x1812. The dataset is organized into six distinct classes:

• NORM: Normal tissue
• HP: Hyperplastic Polyp
• TA.HG: Tubular Adenoma, High-Grade dysplasia
• TA.LG: Tubular Adenoma, Low-Grade dysplasia
• TVA.HG: Tubulo-Villous Adenoma, High-Grade dysplasia
• TVA.LG: Tubulo-Villous Adenoma, Low-Grade dysplasia

PathMMU Atlas is a multimodal visual question answering (VQA) dataset designed to evaluate vision-language models on pathology knowledge. It combines expert-validated questions from the PathMMU repository with pathology images from the ARCH Book Set educational textbooks.

The BreakHis dataset is used for multiclass classification of breast tumor tissue using microscopic images. For this benchmark, the dataset is filtered to include only 40X magnification samples and classes with at least 7 patients to ensure statistical validity.

Dataset Specifications

• Task: Multiclass classification (4 classes)
• Modality: Vision (WSI patches)
• Image Dimensions: 700 x 460
• File Format: png
• Magnification: 40x (0.25 μm/px)
• Included Classes: TA (Tubular Adenoma), MC (Mucinous Carcinoma), F (Fibroadenoma), and DC (Ductal Carcinoma).

Data Splits

The dataset uses train/validation splits with no patient overlap between them:

• Train: 1132 samples (76.95%)
• Validation: 339 samples (23.04%)

The BACH dataset is used for multiclass classification of breast cancer microscopy images. It contains 408 labeled .tif images across 4 classes: Normal, Benign, Invasive, and InSitu.

Dataset Specifications

• Modality: Vision (microscopy images)
• Image Dimensions: 1536 x 2048 x 3
• Magnification: 20x (0.42 μm/px)
• Classes: 4 (102 images per class)
• Splits: The dataset is split into Train (268 samples) and Validation (132 samples). These splits are designed to prevent data leakage by ensuring images from the same patient are not shared between splits.

Data Organization

When extracted from ICIAR2018_BACH_Challenge.zip, the relevant labeled patches are located in the Photos directory, organized by class name (e.g., Photos/Normal/, Photos/Benign/, etc.).

The eva.models.wrappers module provides several classes to wrap different model formats and functions into a unified interface compatible with the eva ecosystem. Use these wrappers to load models from HuggingFace, ONNX, or custom functions.

from eva.models.wrappers import HuggingFaceModel, ONNXModel, ModelFromFunction, BaseModel

# Wrap a HuggingFace model
hf_model = HuggingFaceModel(model_id="name")

# Wrap an ONNX model
onnx_model = ONNXModel(path="model.onnx")

# Wrap a custom function as a model
custom_model = ModelFromFunction(func=my_function)

The eva.core.data.Dataset class serves as the base class for defining custom datasets within the EVA framework. When implementing a new data loading strategy, you should inherit from this base class to ensure compatibility with the rest of the EVA data pipeline.

from eva.core.data import Dataset

class MyCustomDataset(Dataset):
    # Implement required methods for your specific data type
    pass

The dataset provides two primary splits. Note that because only two splits are provided by the authors, no separate test split is reported in benchmarks.

Splits

• Train: 6,270 samples (72.33%)
• Validation: 2,399 samples (27.67%)

Directory Structure

The 800 subset is organized with CSV files for split definitions and subdirectories for each class:

unitopatho
├── 800
│   ├── test.csv
│   ├── train.csv
│   ├── HP
│   ├── NORM
│   ├── TA.HG
│   ├── ...

unitopatho
├── 800
│   ├── test.csv
│   ├── train.csv
│   ├── HP                    # 1 folder per class 
│   ├── NORM
│   ├── TA.HG
│   ├── ...

The CoNSeP (Colorectal Nuclear Segmentation and Phenotypes) dataset is used for vision-based segmentation tasks. It consists of 41 tiles (1000x1000 pixels) extracted from 16 Whole Slide Images (WSIs).

While the original dataset contains 7 classes, this implementation uses a combined 4-class schema for segmentation:

1. Miscellaneous
2. Inflammatory
3. Epithelial (combines original 'Normal epithelium' and 'Malignant/dysplastic epithelium')
4. Spindle-shaped (combines original 'Fibroblast', 'Muscle', and 'Endothelial')

Technical Specifications:

• Modality: Vision (WSI patches)
• Image Dimensions: 1000 x 1000 x 3
• File Formats: .png for images and .mat for segmentation masks
• Magnification: 40x (0.25 μm/px)

The MoNuSAC dataset uses the splits provided by the original data source. Because a dedicated validation split is not provided by the source, the "test" split is used as the validation split for training workflows.