BioCLIP

TreeOfLife-10M is a large-scale, ML-ready dataset containing over 10 million images covering approximately 454,000 taxa. It is designed for training biology foundation models like BioCLIP. The dataset provides images of biological organisms paired with taxonomic labels, including labels for the most specific taxonomic level possible and higher taxonomic ranks. It incorporates data from sources like the Encyclopedia of Life (eol.org) to ensure high diversity.

Key Specifications:

• License: cc0-1.0
• Languages: English (en), Latin (la)
• Task Categories: Image classification, zero-shot classification, and few-shot classification.
• Tags: biology, images, animals, evolutionary biology, CV, multimodal, clip, species, taxonomy, knowledge-guided, imbalanced.

The dataset is divided into three main splits via the split column in catalog.csv:

1. train: The primary training set.
2. val: The validation set.
3. train_small: A 1M-image, uniformly sampled subset of train used specifically for fine-tuned ablation training.

Important: Because train_small is a subset of train, do not include it when calculating overall dataset statistics to avoid double-counting.

The TreeOfLife-10M dataset is organized into webdataset shards (files named shard-######.tar) located in train, val, and train_small folders. Each shard contains a collection of images and their corresponding metadata files. For every image <treeoflife_id>.jpg, there is a set of text files providing different taxonomic and naming perspectives.

<treeoflife_id>.com.txt
<treeoflife_id>.common_name.txt
<treeoflife_id>.jpg
<treeoflife_id>.sci.txt
<treeoflife_id>.sci_com.txt
<treeoflife_id>.scientific_name.txt
<treeoflife_id>.taxon.txt
<treeoflife_id>.taxonTag.txt
<treeoflife_id>.taxonTag_com.txt
<treeoflife_id>.taxon_com.txt
<treeoflife_id>.taxonomic_name.txt

TreeOfLife-10M is a large-scale biology image dataset designed for training foundation models. It is composed of three primary source components:

1. iNat21: Provides 2.7M images of 10K species (training split used).
2. BIOSCAN-1M: A collection of 1M expert-labeled lab images of insects, covering 494 families. This adds variety through lab-based image distributions compared to in situ images.
3. EOL (Encyclopedia of Life): An image aggregator providing approximately 6.6M images, which were used to expand the dataset to cover an additional 440K taxa.

The resulting dataset contains approximately 454K unique taxonomic labels.

The dataset is organized hierarchically by taxonomic rank. Images are stored in directories following the path <kingdom-phylum-class-order-family-genus-species-1>/. Each directory contains up to 30 images per species. Metadata and visualizations are stored in separate subdirectories.

/dataset/
    <kingdom-phylum-class-order-family-genus-species-1>/
        <eol_content_id_1>_<eol_page_id>_eol_full-size-copy.jpg
        ...
    <kingdom-phylum-class-order-family-genus-species-2>/
        ...
    metadata/
        rarespecies-catalog.csv
        licenses.csv
    visuals/
        phyla_ToL_tree.html
        ...

TreeOfLife-10M is the dataset used to train BioCLIP. It contains images for over 450K distinct taxa using 7-rank Linnaean taxonomy (kingdom through species).

Key details:

• Composition: A combination of iNat21, BIOSCAN-1M, and cleaned data from the Encyclopedia of Life (EOL).
• Format: For model training, the datasets can be combined into the webdataset format.
• Availability: While iNat21 and BIOSCAN-1M cannot be re-released, the cleaned EOL data is available on Hugging Face as TreeOfLife-10M.
• Assembly: To combine the datasets into the required format, follow the instructions in docs/imageomics/treeoflife10m.md.

The dataset uses a standardized 7-rank Linnaean taxonomy. Annotations are derived as follows:

• iNat21: Uses iNaturalist English vernacular names and taxa.
• BIOSCAN-1M: Uses expert-labeled Linnaean taxonomic rankings (excluding ranks like 'tribe').
• EOL: Scientific names (genus-species) are used to look up higher-order taxa via BIOSCAN-1M metadata, EOL aggregate datasets, and the ITIS hierarchy. Unresolved taxa are processed via the Global Names Resolver (GNR) API.

Note on Coverage: Approximately 84% of images have full taxa labels. Some images (primarily from BIOSCAN-1M) may only be labeled to the family level. Some labels may contain indicators for unknown species (e.g., sp. ___(get ex)) or hybrids (e.g., species_a x species_b).

The Rare Species dataset is a biologically meaningful test set designed to evaluate the robustness of the Imageomics BioCLIP model, specifically focusing on species with minimal training samples.

Key characteristics:

• Target Species: Includes 400 species from the IUCN Red List categories: Near Threatened, Vulnerable, Endangered, Critically Endangered, and Extinct in the Wild.
• Data Volume: 30 images per species.
• Taxonomic Scope: Limited to the kingdom Animalia and within 5 phyla.
• Exclusions: Species present in the iNat21 or BIOSCAN-1M datasets (which were used for the TreeOfLife-10M dataset) are excluded to ensure a distinct test set.

To evaluate CLIP-like models for biological applications, the following methodologies are proposed:

• Zero-shot classification accuracy: Test on IID (Independent and Identically Distributed) data, co-mimics, and co-mimics supplemented with textual descriptions of their differences.
• Taxonomic vs. Common Names: Compare the performance of the text encoder when using common names versus taxonomic names.
• Generalization testing: Evaluate how well models generalize from naturalist/citizen science photos to lab/museum photos (e.g., photos with white backgrounds).
• Trait presence: Test the model's ability to identify if specific traits are present in a photo or if an animal possesses certain traits even if they are not visually apparent.

To ensure the generated webdataset shards are valid, run scripts/evobio10m/check_wds.py. You must provide a shard list for both the training and validation splits using the --shardlist argument.

If the script outputs paths to corrupted or 'bad' files, delete those specific .tar files and re-run the dataset generation jobs.

python scripts/evobio10m/check_wds.py --shardlist '/fs/ess/PAS2136/open_clip/data/evobio10m-v2.1/224x224/val/shard-{000003..000031}.tar' --workers 8

Use the src.evaluation.extract_features module to extract model embeddings for the iNat validation set. You can extract features using either OpenAI's pretrained models or BioCLIP models from the Hugging Face Hub.

Replace [path_to_iNat21_val] with the actual path to your iNat21 validation data directory.

# Extract OpenAI embeddings
python -m src.evaluation.extract_features --pretrained openai --exp_type openai --feature_output features --val_root [path_to_iNat21_val]

# Extract BioCLIP embeddings
python -m src.evaluation.extract_features --model hf-hub:imageomics/bioclip --exp_type bioclip --feature_output features --val_root [path_to_iNat21_val]