SpeciesNet Documentation

SpeciesNet v4.0.1a is an ensemble model designed for camera trap analysis. It follows an "Always crop" workflow: a detector is run first to find objects, and the image is then cropped to the top detection bounding box before being passed to the species classifier.

Model Identifier: kaggle:google/speciesnet/pyTorch/v4.0.1a

Classifier Details

• Base Architecture: EfficientNet V2 M
• Inference Framework: PyTorch
• Input Requirements: 480x480px RGB images.
• Preprocessing: The input image must be cropped to the top detection bounding box and resized to 480x480px (note: this may change the original aspect ratio).
• Output: Logits, which are converted to label confidence scores via softmax.
• Label Scope: 2498 total labels, including 2066 species (including human), 430 higher taxa labels, and 2 non-animal labels (blank, vehicle).

Detector Details

• Model: MegaDetector v5a
• Base Architecture: YOLOv5
• Input Requirements: Variable sized RGB images where dimensions are multiples of 64px.
• Preprocessing: Resize RGB image to 1280px on the longest edge, then pad dimensions to multiples of 64px while maintaining the original aspect ratio.
• Output: Labeled bounding boxes with confidence scores for 3 labels: animal, human, and vehicle.

SpeciesNet v4.0.0b is a full-image model that runs a detector and a species classifier independently on the same image. It uses an ensemble approach that allows for label rollup, meaning if classifier confidence is low, predictions can be propagated to higher taxonomic levels (genus, family, order, class, kingdom).

Classifier Details

• Framework: TensorFlow (EfficientNet V2 M)
• Input: 480x480px RGB images
• Preprocessing: Crop either 400px or 30% of the height (whichever is smaller) from the center of the RGB image, then resize to 480x480px (aspect ratio may be lost).
• Output: Logits transformed via softmax into confidence scores.
• Labels: 2498 total (2066 species, 430 higher taxa, 2 non-animal: blank and vehicle).

Detector Details

• Model: MegaDetector v5a (YOLOv5)
• Framework: PyTorch
• Input: Variable sized RGB images where dimensions are multiples of 64px.
• Preprocessing: Resize RGB image to 1280px on the longest edge, then pad to multiples of 64px while maintaining aspect ratio.
• Output: Labeled bounding boxes with confidence scores.
• Labels: 3 (animal, human, vehicle).

Ensemble Output

Because of label rollup, the ensemble can predict more labels than the classifier alone:

• Total Labels: 3493
• Species: 2066
• Higher Taxa Labels: 1425
• Non-animal Labels: 2 (blank, vehicle)

SpeciesNet v4.0.0a is an ensemble model designed for camera trap analysis. It uses a two-stage pipeline: first, a Detector identifies objects, and then a Classifier identifies the species within the cropped bounding boxes.

Pipeline Workflow

1. Detection: The model uses MegaDetector v5a (PyTorch/YOLOv5) to find bounding boxes for animal, human, or vehicle.
2. Cropping: The image is cropped to the top detection bounding box.
3. Classification: The cropped image is fed to the Classifier (TensorFlow/EfficientNet V2 M) to predict species or higher taxa.

Model Specifications

Classifier

• Input: 480x480px RGB images (after cropping and resizing).
• Output: Logits transformed via softmax into confidence scores.
• Labels: 2498 total labels, including 2066 species (including human), 430 higher taxa labels, and 2 non-animal labels (blank, vehicle).

Detector (MegaDetector v5a)

• Input: Variable sized RGB images (dimensions must be multiples of 64px).
• Preprocessing: Resizes the longest edge to 1280px and pads to multiples of 64px while maintaining aspect ratio.
• Labels: 3 (animal, human, vehicle).

Ensemble Output

Due to label rollup (propagating low-confidence species predictions to higher taxa like genus, family, etc.), the ensemble can output more labels than the base classifier. The ensemble total includes 3493 labels, with 1425 higher taxa labels.

SpeciesNet uses an ensemble strategy via run_model to predict a single category for an image by combining an object detector (MegaDetector) and an image classifier. The goal is to minimize human review effort by filtering blanks, providing high-confidence predictions for frequent classes, and using taxonomic 'rollups' when species-level confidence is low.

The Prediction Workflow

1. Input processing: Images are preprocessed. Depending on the model, they are either processed as 'always crop' (detector crops the image before classification) or 'full image' (both models process the whole image).
2. Object detection: MegaDetector identifies objects (animals, humans, vehicles) and provides bounding boxes and confidence scores.
3. Species classification: The classifier provides the top-5 species/taxa classifications with confidence scores.
4. Human/Vehicle override: High-confidence human or vehicle detections from the detector override classifier predictions.
5. Blank decisions: Images are labeled 'blank' if the classifier has very high confidence (>0.99) or if the classifier predicts 'blank' and the detector has low animal confidence.
6. Geofencing: If a location (country/admin1) is provided, species disallowed in that region are 'rolled up' to a higher taxonomic level.
7. Label rollup: If species-level confidence is insufficient, the model propagates predictions to the first matching ancestor in the taxonomy (e.g., genus, family, or order) that has a high confidence score.
8. Animal decisions: If the detector has reasonable confidence in an animal presence, animal is returned.
9. Unknown: If no rules apply, the unknown class is returned.
10. Prediction source: The final result includes a prediction_source field to indicate which part of the ensemble produced the result.

Note: When using run_md_and_speciesnet, the goal is to classify individual detections rather than the whole image; therefore, geofencing and label rollup steps are applied differently.