DOCSEARCHbyAlgolia
Submit Docs

SLEAP (Social LEAP Estimates Animal Poses)

repository·develop·Indexed 20 days ago

https://github.com/talmolab/sleap

An open-source deep learning framework for multi-animal pose tracking. SLEAP provides a complete workflow including a specialized GUI for rapid data labeling, high-speed neural network training and inference, and a developer API. It supports top-down and bottom-up training strategies and is built on two backends: sleap-nn for training/inference and sleap-io for file handling.

Tokens
59.7K
Snippets
161
Records
277
Agent score
69%

What's inside SLEAP

  1. Overview of SLEAP sample and benchmark datasets

    develop

    SLEAP provides several sample and benchmark datasets for experimenting with or extending the software. These datasets are provided as .pkg.slp files, which are self-contained SLEAP labels that include embedded image data.

    Datasets are categorized by animal type and whether they include animal identity:

    • Single Animal: e.g., fly32 (Fruit fly).
    • Multi-Animal (without identity): e.g., mice_of (Mice), mice_hc (Mice), bees (Bumblebees).
    • Multi-Animal (with identity): e.g., flies13 (Fruit flies), gerbils (Gerbils).

    For the full set of labeled datasets and trained models reported in the SLEAP paper, visit the OSF repository.

  2. Overview of SLEAP features

    develop

    SLEAP (Social LEAP Estimates Animal Poses) is an open-source deep-learning framework for multi-animal pose tracking. Key capabilities include:

    • Pose Estimation: Supports both single- and multi-animal estimation using top-down and bottom-up training strategies.
    • Human-in-the-loop: Advanced labeling GUI designed for rapid dataset labeling and active learning.
    • Performance: Fast training (15–60 mins on a single GPU) and high-speed inference (600+ FPS for batch, <10ms latency for realtime).
    • Extensibility: Flexible developer API for building integrated applications.
    • Architecture: Built on two independent backends: sleap-nn (training/inference pipelines) and sleap-io (handling SLEAP files).

  3. Understand the SLEAP end-to-end workflow

    develop

    SLEAP (Social LEAP Estimates Animal Poses) follows a four-phase workflow to transform raw video into quantitative behavioral data:

    Phase 1: Setup

    1. Create a project and import videos: Import experimental footage to build your training dataset.
    2. Define the skeleton: Specify the body parts you want to track and their connectivity.

    Phase 2: Label

    1. Select frames for labeling: Use sampling methods (often based on image features) to pick a diverse set of frames.
    2. Label animal poses: Manually place skeleton body parts on animals using the SLEAP GUI.

    Phase 3: Train

    1. Train the model: Train a neural network using your labeled frames. SLEAP supports various architectures and configurations.
    2. Run inference: Apply the trained model to predict poses on unlabeled frames.
    3. Refine and repeat: Use an Active Learning cycle. Inspect predictions, correct errors, and retrain. Typically, labeling 100-500 frames is sufficient to achieve high accuracy across thousands of frames.

    Phase 4: Deploy

    1. Process additional videos: Apply the model to the rest of your experiment videos.
    2. Track identities: Use tracking algorithms to link detections across frames into continuous tracks.
    3. Proofread tracks: Review and fix identity swaps or errors in the GUI.
    4. Export for analysis: Export pose data and tracks for use in Python, MATLAB, or other tools.

  4. Track corrected vs. uncorrected nodes (Red/Green nodes)

    develop

    SLEAP uses color coding to help you track which nodes you have manually adjusted during the correction process:

    • Red nodes: These are nodes in their original predicted locations that have not yet been moved.
    • Green nodes: These nodes turn green as soon as you click them (indicating they have been interacted with/corrected).

    Usage Tips:

    • If the predicted location is already accurate, you can leave it in the red state; it will still be treated as ground truth.
    • To quickly mark all nodes as green, use the shortcut shift+left-button.

  5. Work with SLEAP core data structures

    develop
    For developers building custom analysis scripts or applications, the Data_structures.ipynb notebook explains the core data structures used by SLEAP to store labels, predictions, and metadata. It demonstrates how to interact with these structures, including generating predictions from a trained model programmatically.

  6. Integrate SLEAP with realtime and utility libraries

    develop

    For real-time applications or data management, consider the following integrations:

    Realtime

    • DeepLabStream: A standalone closed-loop application with SLEAP support.

    Utility Libraries

    • sleap-io: Standalone pose I/O utilities.
    • NWB GUIDE: GUI-based conversion to Neurodata Without Borders (NWB).
    • ndx-pose: An NWB extension specifically for pose data.
    • movement: General-purpose utilities for data downstream of pose tracking.

  7. Manage node visibility and occlusion

    develop

    When a node is occluded (not visible), do not guess its position, as this can train the model to be overconfident. Instead, mark it as not visible.

    • To mark a node as not visible: Right-click the node.
    • Visual indicator: Not visible nodes are denoted by a gray and italicized font with a dark background.
    • Effect: These nodes are interpreted as not present during training. The model is trained to predict a low confidence score, allowing them to be replaced with NaN when filtering low-scoring predictions.

    Note: The exact position of a 'not visible' node does not matter; it will be ignored during training. Always check if nodes were incorrectly predicted as not visible during your correction process.

  8. How tracking works in SLEAP

    develop

    Tracking connects frame-by-frame pose predictions into continuous tracks (identities across frames). The process involves:

    1. Taking predictions from frame N.
    2. Comparing them to candidates from previous frames.
    3. Assigning each instance to an existing track or creating a new one.

    Note that prediction and tracking are distinct; you can run tracking separately after inference to experiment with different methods and parameters using --tracking mode.

  9. Legacy Support for SLEAP <1.5

    develop

    SLEAP maintains backward compatibility for older workflows:

    • Config Files: While SLEAP now uses a YAML-based config structure, you can still upload and work with legacy SLEAP JSON files in the GUI.
    • Model Weights: You can continue to run inference on SLEAP <1.4 TensorFlow model weights, though this is limited to the UNet backbone only.

  10. How the SLEAP package architecture works

    develop

    SLEAP is composed of three interconnected packages that manage different parts of the pose estimation workflow:

    1. sleap: The primary entry point. It provides the Labeling GUI (sleap label), the Unified CLI (sleap), quality control tools, and acts as the integration layer between data handling and neural networks.
    2. sleap-io: Handles all data modeling and I/O. This includes managing labels, videos, skeletons, instances, and tracks. It supports native .slp files and provides import/export capabilities for formats like COCO, NWB, and DeepLabCut, as well as various video backends (MP4, AVI, HDF5, etc.).
    3. sleap-nn: The PyTorch-based neural network backend. It manages model architectures (top-down, bottom-up, etc.), backbone networks (UNet, ConvNeXt, etc.), training (augmentation, loss functions), inference (pose prediction, peak finding), and identity tracking.

  11. Automated quality control tools in SLEAP v1.6+

    develop

    SLEAP v1.6+ includes automated tools to assist in finding annotation and tracking errors:

    • Label Quality Control: Uses statistical anomaly detection to flag potential annotation errors, unusual poses, or labeling mistakes by learning what 'normal' poses look like in your specific data.
    • Instance Size Distribution: Analyzes bounding box sizes of labeled instances. This helps in choosing optimal crop sizes for top-down models. You can click on outliers in the distribution to navigate directly to unusually large or small instances that require review.

  12. Analyze data with Instance Size Distribution and Label QC

    develop

    Use the Analyze menu to perform quality control and data analysis:

    Instance Size Distribution

    Access via Analyze → Instance Size Distribution... or the toolbar. This widget shows the distribution of instance bounding box sizes. Use it to:

    • Choose crop sizes for top-down models (the crop size should encompass most instances).
    • Identify outliers that may indicate labeling errors.
    • Verify consistency across labeled frames.

    Label Quality Control (QC)

    Access via Analyze → Label QC.... This panel uses statistical anomaly detection to find potential labeling errors (e.g., unusual edge lengths, joint angles, or node spacing). Use it to:

    • Catch errors before training.
    • Verify proofreading (check tracking corrections).
    • Identify systematic issues.