Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Update label-studio-ml-backend docs #6412

Draft
wants to merge 4 commits into
base: develop
Choose a base branch
from
Draft

docs: Update label-studio-ml-backend docs #6412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b1e0cef
docs: Update label-studio-ml-backend docs
makseq Sep 19, 2024
bb9d32f
Update mmdetection-3.md
caitlinwheeless Sep 19, 2024
ecf8171
Update segment_anything_2_video.md
caitlinwheeless Sep 19, 2024
14f10ce
docs: Update label-studio-ml-backend docs
makseq Sep 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/tutorials/mmdetection-3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ https://mmdetection.readthedocs.io/en/latest/
This example demonstrates how to use the MMDetection model with Label Studio to annotate images with bounding boxes.
The model is based on the YOLOv3 architecture with a MobileNetV2 backbone and trained on the COCO dataset.

![screenshot.png](/tutorials/screenshot.png)
![screenshot.png](screenshot.png)

## Before you begin

Expand Down Expand Up @@ -166,4 +166,4 @@ gunicorn --preload --bind :9090 --workers 1 --threads 1 --timeout 0 _wsgi:app
```

* Use this guide to find out your access token: https://labelstud.io/guide/api.html
* You can use and increased value of `SCORE_THRESHOLD` parameter when you see a lot of unwanted detections or lower its value if you don't see any detections.
* You can use and increased value of `SCORE_THRESHOLD` parameter when you see a lot of unwanted detections or lower its value if you don't see any detections.
4 changes: 2 additions & 2 deletions docs/source/tutorials/segment_anything_2_video.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ This guide describes the simplest way to start using **SegmentAnything 2** with
This repository is specifically for working with object tracking in videos. For working with images,
see the [segment_anything_2_image repository](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/segment_anything_2_image)

![sam2](/tutorials/Sam2Video.gif)
![sam2](./Sam2Video.gif)

## Before you begin

Expand Down Expand Up @@ -83,4 +83,4 @@ If you want to contribute to this repository to help with some of these limitati

## Customization

The ML backend can be customized by adding your own models and logic inside the `./segment_anything_2_video` directory.
The ML backend can be customized by adding your own models and logic inside the `./segment_anything_2_video` directory.
109 changes: 86 additions & 23 deletions docs/source/tutorials/yolo.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,28 @@ segmentation, classification, and video object tracking capabilities directly in
This integration allows you to leverage powerful YOLOv8 models for various machine learning tasks,
making it easier to annotate large datasets and ensure high-quality predictions.

<iframe width="560" height="315" src="https://www.youtube.com/embed/UyaecID1iG8?si=UQQW3mpqu2uDtt7i" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
<div align="left">
<a href="http://www.youtube.com/watch?v=UyaecID1iG8" title="Integrating YOLO V8 with Label Studio">
<img src="http://img.youtube.com/vi/UyaecID1iG8/0.jpg" alt="YOLO Integration Video" style="width:50%;"/>
<br>
Check the video tutorial
</a>
</div>
<br/>

**Supported Features**

| YOLO Task Name | LS Control Tag | Prediction Supported | LS Import Supported | LS Export Supported |
|---------------------------------------|--------------------------------------|----------------------|---------------------|---------------------|
| Object Detection | `<RectangleLabels>` | ✅ | YOLO, COCO | YOLO, COCO |
| Oriented Bounding Boxes (OBB) | `<RectangleLabels model_obb="true">` | ✅ | YOLO | YOLO |
| Image Instance Segmentation: Polygons | `<PolygonLabels>` | ✅ | COCO | YOLO, COCO |
| Image Semantic Segmentation: Masks | `<BrushLabels>` | ❌ | Native | Native |
| Image Classification | `<Choices>` | ✅ | Native | Native |
| Pose Detection | `<KeyPoints>` | ✅ | Native | Native |
| Video Object Tracking | `<VideoRectangle>` | ✅ | Native | Native |
| Video Temporal Classification | `<TimelineLabels>` | Coming soon | Native | Native |
| YOLO Task Name | LS Control Tag | Prediction Supported | LS Import Supported | LS Export Supported |
|--------------------------------------------------------------|--------------------------------------|----------------------|---------------------|---------------------|
| Object Detection | `<RectangleLabels>` | ✅ | YOLO, COCO | YOLO, COCO |
| Oriented Bounding Boxes (OBB) | `<RectangleLabels model_obb="true">` | ✅ | YOLO | YOLO |
| Image Instance Segmentation: Polygons | `<PolygonLabels>` | ✅ | COCO | YOLO, COCO |
| Image Semantic Segmentation: Masks | `<BrushLabels>` | ❌ | Native | Native |
| Image Classification | `<Choices>` | ✅ | Native | Native |
| Pose Detection | `<KeyPoints>` | ✅ | Native | Native |
| Video Object Tracking | `<VideoRectangle>` | ✅ | Native | Native |
| [Video Temporal Classification](./README_TIMELINE_LABELS.md) | `<TimelineLabels>` | ✅ | Native | Native |


* **LS Control Tag**: Label Studio [control tag](https://labelstud.io/tags/) from the labeling configuration.
* **LS Import Supported**: Indicates whether Label Studio supports Import from YOLO format to Label Studio (using the LS converter).
Expand Down Expand Up @@ -73,7 +81,7 @@ This tutorial uses the [YOLO example](https://github.com/HumanSignal/label-studi

4. Then from the **Model** page in the project settings, [connect the model](https://labelstud.io/guide/ml#Connect-the-model-to-Label-Studio). The default URL is `http://localhost:9090`.

5. Add images to Label Studio.
5. Add images or video (depending on tasks you are going to solve) to Label Studio.

6. Open any task in the Data Manager and see the predictions from the YOLO model.

Expand All @@ -88,11 +96,13 @@ This tutorial uses the [YOLO example](https://github.com/HumanSignal/label-studi

**Control tags**

- `<Choices>` - [Classification](https://labelstud.io/tags/choices); image classification task
- `<RectangleLabels>` - [Bounding boxes](https://labelstud.io/tags/rectanglelabels); object detection task
- `<PolygonLabels>` - [Polygons](https://labelstud.io/tags/polygonlables); segmentation task
- `<VideoRectangle>` - [Video bounding boxes](https://labelstud.io/tags/videorectangle); video object tracking task
- `<VideoRectangle>` - [Video bounding boxes](https://labelstud.io/tags/videorectangle); object tracking task for videos
- `<KeyPointLabels>` - [Key points](https://labelstud.io/tags/keypointlabels); pose detection task
- `<Choices>` - [Classification](https://labelstud.io/tags/choices)
- `<TimelineLabels>` - [Temporal labels for videos](https://labelstud.io/tags/timelinelabels); multi-label temporal classification task for videos


**How to skip the control tag?**

Expand Down Expand Up @@ -170,7 +180,7 @@ to specify multiple and different labels from the ML model:
<br/>
Labels are printed in the ML model logs when you start using the ML backend with the INFO logging level.

Or you can find some labels in [YOLO_CLASSES.md](https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/yolo/YOLO_CLASSES.md)
Or you can find some labels in [YOLO_CLASSES.md](YOLO_CLASSES.md)
</details>

<details>
Expand Down Expand Up @@ -222,7 +232,7 @@ The same applies to other YOLO models.
You can load your own YOLO labels using the following steps:

1. Mount your model as `/app/models/<your-model>.pt` inside of your docker.
2. Set `ALLOW_CUSTOM_MODEL_PATH=true` (it is true by default) in your Docker environment parameters ([`docker-compose.yml`](https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/yolo/docker-compose.yml)).
2. Set `ALLOW_CUSTOM_MODEL_PATH=true` (it is true by default) in your Docker environment parameters ([`docker-compose.yml`](docker-compose.yml)).
3. Add `model_path="<your-model>.pt"` to the control tag in the labeling configuration, e.g.:

```xml
Expand All @@ -247,7 +257,7 @@ YOLO provides a classification model and Label Studio supports this with the `Ch

More info: https://docs.ultralytics.com/tasks/classify/

<video src="https://github.com/user-attachments/assets/30c5ce43-2c89-4ddf-a77d-9d1d75ac3419" controls="controls" style="max-width: 800px;" class="gif-border" />
https://github.com/user-attachments/assets/30c5ce43-2c89-4ddf-a77d-9d1d75ac3419


### Labeling config
Expand Down Expand Up @@ -295,7 +305,7 @@ YOLO OBB models are also supported.

More info: https://docs.ultralytics.com/tasks/detect/

<video src="https://github.com/user-attachments/assets/413b4650-422d-43dc-809d-51c08f0ad434" controls="controls" style="max-width: 800px;" class="gif-border" />
https://github.com/user-attachments/assets/413b4650-422d-43dc-809d-51c08f0ad434


### Labeling config
Expand Down Expand Up @@ -352,7 +362,9 @@ Label Studio supports this with the `PolygonLabels` control tag.

More info: https://docs.ultralytics.com/tasks/segment/

<video src="https://github.com/user-attachments/assets/9b2447d3-392d-42be-bc7f-ef2b6c81d54c" controls="controls" style="max-width: 800px;" class="gif-border" />
![Yolo Polygons](./YoloPolygons.gif)

https://github.com/user-attachments/assets/9b2447d3-392d-42be-bc7f-ef2b6c81d54c


### Labeling config
Expand Down Expand Up @@ -567,8 +579,10 @@ Label Studio supports this with the `VideoRectangle` + `Labels` control tags.

More info: https://docs.ultralytics.com/modes/track/

![Video Object Tracking](./YoloVideo.gif)

<video src="https://github.com/user-attachments/assets/7b0d50e6-164a-4d66-87cf-df443b77f638" controls="controls" style="max-width: 800px;" class="gif-border" />

https://github.com/user-attachments/assets/7b0d50e6-164a-4d66-87cf-df443b77f638



Expand Down Expand Up @@ -668,6 +682,55 @@ Small models like `yolov8n.pt` are recommended for real-time tracking, however,

<br>


## Video temporal classification using `TimelineLabels`

This ML backend supports temporal multi-label video classification for the [`<TimelineLabels>` control tag](https://labelstud.io/tags/timelinelabels) in Label Studio.
There are two modes available:
- **Simple:** In the simple mode, the model uses pre-trained YOLO classes to generate predictions without additional training.
- **Trainable:** In the [trainable mode](README_TIMELINE_LABELS.md), the model can be trained on custom labels and annotations submitted in Label Studio using few-shot learning as training is performed on a small number of annotations.

<div align="left">
<a href="https://www.youtube.com/watch?v=tfMn5q1tqKI" title="Video Frame Classification with YOLOv8 and Label Studio">
<img src="https://github.com/user-attachments/assets/6b00fdc1-e19b-4312-86f4-20d8a1de6f20" alt="Video Temporal Classification video" style="width:100%;"/>
<br>
Check the video tutorial
</a>
</div>
<br/>

### Labeling config

```xml
<View>
<Video name="video" value="$video"/>
<TimelineLabels
name="label" toName="video"
model_trainable="false" model_score_threshold="0.25">
<Label value="Ball" predicted_values="soccer_ball" />
<Label value="hamster" />
</TimelineLabels>
</View>
```

### Model training

For more details on using the `TimelineLabels` ML backend, including training the model
and adjusting neural network classifier parameters, please refer to
**[README_TIMELINE_LABELS.md](README_TIMELINE_LABELS.md)**.

### Default model

`yolov8n-cls.pt` is the default classification model for simple mode.


<br>

-------------------

<br>


## Run the YOLO ML backend


Expand Down Expand Up @@ -713,11 +776,11 @@ Then you can start the ML backend:
label-studio-ml start ./dir_with_your_model
```

Also, you can check [Dockerfile]([Dockerfile](https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/yolo/Dockerfile)) for additional dependencies and install them manually.
Also, you can check [Dockerfile](Dockerfile) for additional dependencies and install them manually.

### Parameters

Check the `environment` section in the [`docker-compose.yml`](https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/yolo/docker-compose.yml) file before running the container.
Check the `environment` section in the [`docker-compose.yml`](docker-compose.yml) file before running the container.
All available parameters are listed there.

> Note: You can use lots of YOLO model parameters in labeling configurations directly, e.g. `model_path` or `model_score_threshold`.
Expand Down Expand Up @@ -795,4 +858,4 @@ LOG_LEVEL=DEBUG python cli.py --ls-url http://localhost:8080 --ls-api-key YOUR_A

## For developers

The architecture of the project and development guidelines are described in the [README_DEVELOP.md](https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/yolo/README_DEVELOP.md) file.
The architecture of the project and development guidelines are described in the [README_DEVELOP.md](README_DEVELOP.md) file.