repository·master·Indexed 22 days ago
https://github.com/humansignal/label-studio-ml-backendAn SDK for wrapping machine learning models into a web server that integrates with Label Studio for automated labeling, pre-annotation, and interactive model training. It supports deployment via Docker or local Python environments and includes examples for BERT-based text classification, Deepgram Text-to-Speech annotation, and Docling Serve integration.
The Label Studio ML backend is an SDK that allows you to wrap machine learning code into a web server. This server can be connected to a running Label Studio instance to automate labeling tasks through pre-annotations, interactive labeling, or model training.
Note: If you only need to load static pre-annotated data, you should use Label Studio's import pre-annotated data feature instead of running an ML backend.
This ML backend integrates the Deepgram API with Label Studio to perform Text-to-Speech (TTS) tasks. It takes input text from a user in Label Studio, processes it via Deepgram, and returns the generated audio for annotation.
Important: After submitting the text in Label Studio, you must refresh the page to see the generated audio appear.
Because ML model labels often differ from the human-readable labels used in Label Studio, you must define a mapping.
Default Mapping: By default, the YOLO ML backend uses the same name (or a lowercased version) as specified in the value attribute of your Label Studio tag.
<Choice value="Jeep"/> maps to jeep in the model.Precise Mapping: For more control, use the predicted_values attribute to map a single Label Studio label to one or more specific labels from the ML model. This is a comma-separated list.
<Choice value="Car" predicted_values="jeep,cab,limousine"/>Tip: To find the exact names used by your YOLO model, check the ML model logs at the INFO logging level when the backend starts, or refer to YOLO_CLASSES.md in the repository.
The backend supports two distinct ways of annotating time series data:
Range Annotations
start ≠ end, and instant is set to false.Instant Annotations
start = end, and instant is set to true.Note: Instant labels often create highly imbalanced datasets. The backend uses a balanced learning approach (class-weighted loss and balanced accuracy monitoring) to handle this.
The TimelineLabelsModel is the primary interface for the <TimelineLabels> control tag in Label Studio. It extends ControlModel and manages two distinct operational modes for temporal video multi-label classification: Simple mode and Trainable mode.
Simple Mode (self.trainable == False):
Uses pre-trained YOLO classes for prediction without additional training. It relies on cached_yolo_predict to extract probabilities from frames and converts them into timeline labels.
Trainable Mode (self.trainable == True):
Uses a custom-trained LSTM neural network to capture temporal dependencies. It extracts features via cached_feature_extraction, loads a trained classifier, and uses the LSTM to predict probabilities across the video sequence.
timeline_labels.pypredict_regions(video_path): The entry point for prediction requests. It routes the request to either create_timelines_simple or create_timelines_trainable based on the trainable attribute.fit(event, data, **kwargs): The entry point for incremental training. It is triggered by Label Studio events like ANNOTATION_CREATED or ANNOTATION_UPDATED to update the LSTM model with new human-labeled data.Choose the model based on your hardware and accuracy requirements:
Supports mixing different prompt types (rectangles, positive/negative keypoints) and using MobileSAM.
The backend provides multi-tenant support by maintaining separate trained models for each Label Studio project. This prevents cross-project interference and ensures data isolation.
Key Mechanisms:
model_project_{project_id}.pt (e.g., model_project_47.pt).model_project_0.pt).The SAM2 backend multiplexes two modes through the /predict endpoint, determined by the context.event value in the request parameters.
Used to cache frames in a window around a specific frame to reduce latency during navigation.
context.event: prewarmUsed for actual inference based on user prompts (clicks or boxes).
context.event: (Implicitly triggered by presence of result in context)PredictionValue containing the segmentation results (bitmasks, rectangles, or polygons).### Prewarm Request Example
```json
{
"tasks": [{"id": 1, "data": {"video": "..."}}],
"params": {
"context": {
"event": "prewarm",
"frame": 42,
"window": 20,
"direction": "forward"
}
}
}{
"tasks": [{"id": 1, "data": {"video": "..."}}],
"params": {
"context": {
"frame": 42,
"result": [
{"type": "keypointlabels", "value": {"x": 45.2, "y": 30.1, "positive": true}},
{"type": "keypointlabels", "value": {"x": 60.0, "y": 55.0, "positive": false}}
]
}
}
}./dir_with_your_model).To ensure detected keypoints are correctly identified in the Label Studio interface, use the model_index and predicted_values attributes within your Label tags.
For standard YOLO pose models, the mapping follows this index pattern:
0: Nose1: Left Eye2: Right Eye3: Left Ear4: Right Ear5: Left Shoulder6: Right Shoulder7: Left Elbow8: Right Elbow9: Left Wrist10: Right Wrist11: Left Hip12: Right Hip13: Left Knee14: Right Knee15: Left Ankle16: Right Ankle<Label value="left_eye" predicted_values="person" model_index="1" />
<Label value="right_eye" predicted_values="person" model_index="2" />Control models are specialized subclasses of ControlModel that translate YOLO model outputs into Label Studio-compatible annotation formats. Each model is responsible for a specific ControlTag found in the Label Studio configuration.
| Annotation Type | Control Model | Purpose |
|---|---|---|
| Bounding Boxes | RectangleLabelsModel | Handles axis-aligned or oriented (OBB) rectangles for images. |
| Polygons | PolygonLabelsModel | Converts segmentation masks into polygon coordinates. |
| Classification | ChoicesModel | Maps classification outputs to Label Studio Choices format. |
| Keypoints | KeyPointLabelsModel | Converts pose estimation outputs into keypoint annotations. |
| Video Tracking | VideoRectangleModel | Uses YOLO tracking to generate bounding boxes across video frames. |
| Timelines | TimelineLabelsModel | Supports temporal event annotations. |