From d0af7c1f20cc7f05d1400731ffb9083d37d2f5da Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Tue, 28 Feb 2023 03:37:47 +0100 Subject: [PATCH 01/12] [WIP] Docs restructure --- .../docs/source/examples/color_example.rst | 2 + .../docs/source/fundamentals/callbacks.rst | 2 + .../{features => fundamentals}/components.rst | 0 .../docs/source/fundamentals/oak-camera.rst | 2 + .../{features => fundamentals}/packets.rst | 0 depthai_sdk/docs/source/index.rst | 68 ++++++------------- depthai_sdk/docs/source/installation.rst | 6 ++ depthai_sdk/docs/source/oak-camera.rst | 2 +- depthai_sdk/docs/source/quickstart.rst | 49 +++++++++++++ 9 files changed, 83 insertions(+), 48 deletions(-) create mode 100644 depthai_sdk/docs/source/examples/color_example.rst create mode 100644 depthai_sdk/docs/source/fundamentals/callbacks.rst rename depthai_sdk/docs/source/{features => fundamentals}/components.rst (100%) create mode 100644 depthai_sdk/docs/source/fundamentals/oak-camera.rst rename depthai_sdk/docs/source/{features => fundamentals}/packets.rst (100%) create mode 100644 depthai_sdk/docs/source/installation.rst create mode 100644 depthai_sdk/docs/source/quickstart.rst diff --git a/depthai_sdk/docs/source/examples/color_example.rst b/depthai_sdk/docs/source/examples/color_example.rst new file mode 100644 index 000000000..199a92a72 --- /dev/null +++ b/depthai_sdk/docs/source/examples/color_example.rst @@ -0,0 +1,2 @@ +Color camera example +===================== \ No newline at end of file diff --git a/depthai_sdk/docs/source/fundamentals/callbacks.rst b/depthai_sdk/docs/source/fundamentals/callbacks.rst new file mode 100644 index 000000000..5e2cbaee5 --- /dev/null +++ b/depthai_sdk/docs/source/fundamentals/callbacks.rst @@ -0,0 +1,2 @@ +Callbacks +========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/features/components.rst b/depthai_sdk/docs/source/fundamentals/components.rst similarity index 100% rename from depthai_sdk/docs/source/features/components.rst rename to depthai_sdk/docs/source/fundamentals/components.rst diff --git a/depthai_sdk/docs/source/fundamentals/oak-camera.rst b/depthai_sdk/docs/source/fundamentals/oak-camera.rst new file mode 100644 index 000000000..3f4b031a6 --- /dev/null +++ b/depthai_sdk/docs/source/fundamentals/oak-camera.rst @@ -0,0 +1,2 @@ +OakCamera +========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/features/packets.rst b/depthai_sdk/docs/source/fundamentals/packets.rst similarity index 100% rename from depthai_sdk/docs/source/features/packets.rst rename to depthai_sdk/docs/source/fundamentals/packets.rst diff --git a/depthai_sdk/docs/source/index.rst b/depthai_sdk/docs/source/index.rst index 8524f500b..8bb3679a0 100644 --- a/depthai_sdk/docs/source/index.rst +++ b/depthai_sdk/docs/source/index.rst @@ -1,5 +1,5 @@ =========== -DepthAI SDK +What is DepthAI SDK? =========== DepthAI SDK is a Python package built on top of the `depthai-python `__ API library that **improves @@ -7,63 +7,37 @@ ease of use when developing apps for OAK devices**. .. image:: https://user-images.githubusercontent.com/18037362/142909472-eae7bed1-695b-48ec-8895-76982989de11.png - -OakCamera class ---------------- - -The :ref:`OakCamera` class abstracts DepthAI API pipeline building, different camera permutations, stream :ref:`recording `/:ref:`replaying `, adds -debugging features, handles :ref:`AI model ` sourcing and decoding, does message syncing & visualization, and much more. - -Script below demonstrates how you can easily accomplish complex tasks, that would -otherwise take 100s of lines of code, and a few hours of assembling code pieces together. - -.. code-block:: python - :linenos: - - from depthai_sdk import OakCamera, ResizeMode - - # Download a public depthai-recording and replay it - with OakCamera(replay='cars-tracking-above-01') as oak: - # Create color camera - color = oak.create_camera('color') - - # Download & run pretrained vehicle detection model and track detections - nn = oak.create_nn('vehicle-detection-0202', color, tracker=True) - nn.config_nn(resize_mode=ResizeMode.STRETCH) - - # Visualize tracklets, show FPS - visualizer = oak.visualize(nn.out.tracker, fps=True) - visualizer.tracking(line_thickness=5).text(auto_scale=True) - - # Start the app in blocking mode - oak.start(blocking=True) - -.. image:: https://user-images.githubusercontent.com/18037362/197657107-1c0a73ae-9022-4a85-abe2-892725edaa5f.gif - -A :ref:`public video ` gets downloaded and frames are sent to the OAK camera. On the OAK camera, we run ``vehicle-detection-0202``, -which is an :ref:`SDK supported model `. Afterwards, we use object tracker for tracking these detections over -time. We visualize tracking results and configure visualizer to best fit our needs. - -.. figure:: https://user-images.githubusercontent.com/18037362/197664572-33d8dc9e-dd35-4e73-8291-30bb8ec641d5.png - - Car tracking pipeline from oak.show_graph() - .. note:: This class will be in **alpha stage** until **depthai-sdk 2.0.0**, so there will likely be some API changes. -Installation ------------- -.. include:: ./includes/install-short.rst .. include:: ./includes/footer-short.rst .. toctree:: :maxdepth: 2 :hidden: - :caption: Home: + :glob: + :caption: Getting started self - oak-camera.rst + installation.rst + quickstart.rst + +.. toctree:: + :maxdepth: 2 + :hidden: + :glob: + :caption: Fundamentals + + fundamentals/* + +.. toctree:: + :maxdepth: 2 + :hidden: + :glob: + :caption: Examples + + examples/* .. toctree:: :maxdepth: 2 diff --git a/depthai_sdk/docs/source/installation.rst b/depthai_sdk/docs/source/installation.rst new file mode 100644 index 000000000..cda281b9e --- /dev/null +++ b/depthai_sdk/docs/source/installation.rst @@ -0,0 +1,6 @@ +Installation +============ + +.. include:: ./includes/install-short.rst + +.. include:: ./includes/footer-short.rst \ No newline at end of file diff --git a/depthai_sdk/docs/source/oak-camera.rst b/depthai_sdk/docs/source/oak-camera.rst index ee737412d..a06a36b00 100644 --- a/depthai_sdk/docs/source/oak-camera.rst +++ b/depthai_sdk/docs/source/oak-camera.rst @@ -62,7 +62,7 @@ Run face-detection-retail-0004 on left camera :language: python Deploy models from Roboflow and Roboflow Universe with Depth SDK -********************************************* +**************************************************************** .. literalinclude:: ../../examples/NNComponent/roboflow_integration.py :language: python diff --git a/depthai_sdk/docs/source/quickstart.rst b/depthai_sdk/docs/source/quickstart.rst new file mode 100644 index 000000000..62938bc68 --- /dev/null +++ b/depthai_sdk/docs/source/quickstart.rst @@ -0,0 +1,49 @@ +Quickstart +========== + +Working with camera +------------------- + +The `OakCamera` class is a high-level interface for accessing the features of the OAK device using the DepthAI SDK. +It provides a simple and intuitive way to create pipelines that capture video from the OAK camera, run neural networks on the video stream, and visualize the results. + +The OakCamera class encapsulates the lower-level details of configuring the OAK device and setting up the pipelines. +It provides a number of methods for creating cameras, neural networks, and other pipeline components, and it takes care of connecting these components together and starting the pipeline. + +Creating color stream +--------------------- + +To create a color stream using the OakCamera class in the DepthAI SDK, we can use the create_camera() method. +Here is an example which creates a color stream with a resolution of 1080p and visualizes the stream: + +.. code-block:: python + + from depthai_sdk import OakCamera + + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p') + oak.visualize(color) + oak.start(blocking=True) + +Creating depth stream +------------ + +To create a depth stream using the OakCamera class in the DepthAI SDK, we can use the create_stereo() method. +Here is an example which creates a depth stream with a resolution of 800p and visualizes the stream: + +.. code-block:: python + + from depthai_sdk import OakCamera + + with OakCamera() as oak: + depth = oak.create_stereo('800p') + oak.visualize(depth) + oak.start(blocking=True) + + + +Recording +--------- + + + From e8da50524d1ec7ae2e17c9e8df0987b8b4d5ff21 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 26 Apr 2023 03:00:56 +0200 Subject: [PATCH 02/12] Remove unused fields in TrackingConfig --- depthai_sdk/src/depthai_sdk/visualize/configs.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/depthai_sdk/src/depthai_sdk/visualize/configs.py b/depthai_sdk/src/depthai_sdk/visualize/configs.py index af9961921..dfb26e7fb 100644 --- a/depthai_sdk/src/depthai_sdk/visualize/configs.py +++ b/depthai_sdk/src/depthai_sdk/visualize/configs.py @@ -95,10 +95,8 @@ class TrackingConfig: deletion_lost_threshold: int = 5 line_thickness: int = 1 fading_tails: bool = False - speed: bool = False line_color: Tuple[int, int, int] = (255, 255, 255) line_type: int = 16 # cv2.LINE_AA - bg_color: Tuple[int, int, int] = (0, 0, 0) @dataclass From 06d8b12395cfd20b99b0c72dc750eb4a3b36b876 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 26 Apr 2023 03:01:05 +0200 Subject: [PATCH 03/12] Update docs requirements.txt --- depthai_sdk/docs/requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/depthai_sdk/docs/requirements.txt b/depthai_sdk/docs/requirements.txt index f491e0ce4..cbcc68b66 100644 --- a/depthai_sdk/docs/requirements.txt +++ b/depthai_sdk/docs/requirements.txt @@ -1,3 +1,4 @@ Sphinx==4.1.2 sphinx-rtd-theme==0.5.0 -e ../ +autodocsumm==0.2.10 From a0cc4577ccb0fd2d74a2606f06b295d14047ffa5 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 26 Apr 2023 03:01:14 +0200 Subject: [PATCH 04/12] Refactor documentation --- depthai_sdk/docs/source/api_reference.rst | 10 + .../source/components/camera_component.rst | 10 +- .../docs/source/components/imu_component.rst | 2 +- .../docs/source/components/nn_component.rst | 24 +- .../source/components/stereo_component.rst | 11 +- depthai_sdk/docs/source/conf.py | 1 + .../docs/source/features/ai_models.rst | 12 +- .../docs/source/features/recording.rst | 1 + .../docs/source/fundamentals/callbacks.rst | 2 - .../docs/source/fundamentals/components.rst | 3 +- .../docs/source/fundamentals/oak-camera.rst | 2 - .../docs/source/fundamentals/packets.rst | 23 +- .../{features => fundamentals}/visualizer.rst | 109 +++++--- .../docs/source/includes/install-short.rst | 15 +- depthai_sdk/docs/source/index.rst | 22 +- depthai_sdk/docs/source/installation.rst | 6 - depthai_sdk/docs/source/quickstart.rst | 115 +++++++-- .../visualizer_formats/detection_format.json | 35 +++ .../source/visualizer_formats/example.json | 74 ++++++ .../source/visualizer_formats/format.json | 244 ++++++++++++++++++ .../visualizer_formats/line_format.json | 25 ++ .../visualizer_formats/text_format.json | 13 + 22 files changed, 646 insertions(+), 113 deletions(-) create mode 100644 depthai_sdk/docs/source/api_reference.rst delete mode 100644 depthai_sdk/docs/source/fundamentals/callbacks.rst delete mode 100644 depthai_sdk/docs/source/fundamentals/oak-camera.rst rename depthai_sdk/docs/source/{features => fundamentals}/visualizer.rst (61%) delete mode 100644 depthai_sdk/docs/source/installation.rst create mode 100644 depthai_sdk/docs/source/visualizer_formats/detection_format.json create mode 100644 depthai_sdk/docs/source/visualizer_formats/example.json create mode 100644 depthai_sdk/docs/source/visualizer_formats/format.json create mode 100644 depthai_sdk/docs/source/visualizer_formats/line_format.json create mode 100644 depthai_sdk/docs/source/visualizer_formats/text_format.json diff --git a/depthai_sdk/docs/source/api_reference.rst b/depthai_sdk/docs/source/api_reference.rst new file mode 100644 index 000000000..88bffd3b8 --- /dev/null +++ b/depthai_sdk/docs/source/api_reference.rst @@ -0,0 +1,10 @@ +API Reference +============= + +.. automodule:: depthai_sdk + :autosummary: + :members: + :special-members: __init__ + :show-inheritance: + :undoc-members: + :imported-members: diff --git a/depthai_sdk/docs/source/components/camera_component.rst b/depthai_sdk/docs/source/components/camera_component.rst index 03507e250..b52a01b5c 100644 --- a/depthai_sdk/docs/source/components/camera_component.rst +++ b/depthai_sdk/docs/source/components/camera_component.rst @@ -17,17 +17,17 @@ Usage color = oak.create_camera('color') # Visualize color camera frame stream - oak.visualize(color, fps=True) + oak.visualize(color.out.main, fps=True) # Start the pipeline, continuously poll oak.start(blocking=True) Component outputs ################# -- ``out.main`` - Uses one of the outputs below. -- ``out.camera`` - Default output. Streams either ColorCamera's video (NV12) or MonoCamera's out (GRAY8) frames. Produces :ref:`FramePacket`. -- ``out.replay`` - If we are using :ref:`Replaying` feature. It doesn't actually stream these frames back to the host, but rather sends read frames to syncing mechanism directly (to reduce bandwidth by avoiding loopback). Produces :ref:`FramePacket`. -- ``out.encoded`` - If we are encoding frames, this will send encoded bitstream to the host. When visualized, it will decode frames (using cv2.imdecode for MJPEG, or pyav for H.26x). Produces :ref:`FramePacket`. +- :attr:`main ` - Uses one of the outputs below. +- :attr:`camera ` - Default output. Streams either ColorCamera's video (NV12) or MonoCamera's out (GRAY8) frames. Produces :ref:`FramePacket`. +- :attr:`replay ` - If we are using :ref:`Replaying` feature. It doesn't actually stream these frames back to the host, but rather sends read frames to syncing mechanism directly (to reduce bandwidth by avoiding loopback). Produces :ref:`FramePacket`. +- :attr:`encoded ` - If we are encoding frames, this will send encoded bitstream to the host. When visualized, it will decode frames (using cv2.imdecode for MJPEG, or pyav for H.26x). Produces :ref:`FramePacket`. Reference ######### diff --git a/depthai_sdk/docs/source/components/imu_component.rst b/depthai_sdk/docs/source/components/imu_component.rst index 7238b1f31..505f5d7e0 100644 --- a/depthai_sdk/docs/source/components/imu_component.rst +++ b/depthai_sdk/docs/source/components/imu_component.rst @@ -24,7 +24,7 @@ Usage Component outputs ################# -- ``out.main`` - Main output, produces :ref:`IMUPacket` +- :attr:`main ` - Main output, produces :ref:`IMUPacket`. Reference ######### diff --git a/depthai_sdk/docs/source/components/nn_component.rst b/depthai_sdk/docs/source/components/nn_component.rst index 0615c9f7d..c9f62eb8e 100644 --- a/depthai_sdk/docs/source/components/nn_component.rst +++ b/depthai_sdk/docs/source/components/nn_component.rst @@ -7,10 +7,10 @@ object tracking, and MultiStage pipelines setup. It also supports :ref:`Roboflow DepthAI API nodes ----------------- -For neural inferencing NNComponent will a DepthAI API node: +For neural inference, NNComponent will use DepthAI API node: -- If we are using MobileNet-SSD based AI model, this component will create `MobileNetDetectionNetwork `__ or `MobileNetSpatialDetectionNetwork `__ if ``spatial`` argument is set. -- If we are using YOLO based AI model, this component will create `YoloDetectionNetwork `__ or `YoloSpatialDetectionNetwork `__ if ``spatial`` argument is set. +- If we are using MobileNet-SSD based AI model, this component will create `MobileNetDetectionNetwork `__ (or `MobileNetSpatialDetectionNetwork `__ if ``spatial`` argument is set). +- If we are using YOLO based AI model, this component will create `YoloDetectionNetwork `__ (or `YoloSpatialDetectionNetwork `__ if ``spatial`` argument is set). - If it's none of the above, component will create `NeuralNetwork `__ node. If ``tracker`` argument is set and we have YOLO/MobileNet-SSD based model, this component will also create `ObjectTracker `__ node, @@ -26,19 +26,20 @@ Usage with OakCamera(recording='cars-tracking-above-01') as oak: color = oak.create_camera('color') nn = oak.create_nn('vehicle-detection-0202', color, tracker=True) - nn.config_nn(ResizeMode=ResizeMode.STRETCH) + nn.config_nn(resize_mode='stretch') + oak.visualize([nn.out.tracker, nn.out.passthrough], fps=True) - # oak.show_graph() oak.start(blocking=True) Component outputs ################# -- ``out.main`` - Default output. Streams NN results and high-res frames that were downscaled and used for inferencing. Produces :ref:`DetectionPacket` or :ref:`TwoStagePacket` (if it's 2. stage NNComponent). -- ``out.passthrough`` - Default output. Streams NN results and passthrough frames (frames used for inferencing). Produces :ref:`DetectionPacket` or :ref:`TwoStagePacket` (if it's 2. stage NNComponent). -- ``out.spatials`` - Streams depth and bounding box mappings (``SpatialDetectionNework.boundingBoxMapping``). Produces :ref:`SpatialBbMappingPacket`. -- ``out.twostage_crops`` - Streams 2. stage cropped frames to the host. Produces :ref:`FramePacket`. -- ``out.tracker`` - Streams `ObjectTracker's `__ tracklets and high-res frames that were downscaled and used for inferencing. Produces :ref:`TrackerPacket`. +- :attr:`main ` - Default output. Streams NN results and high-res frames that were downscaled and used for inferencing. Produces :ref:`DetectionPacket` or :ref:`TwoStagePacket` (if it's 2. stage NNComponent). +- :attr:`passthrough ` - Default output. Streams NN results and passthrough frames (frames used for inferencing). Produces :ref:`DetectionPacket` or :ref:`TwoStagePacket` (if it's 2. stage NNComponent). +- :attr:`spatials ` - Streams depth and bounding box mappings (``SpatialDetectionNework.boundingBoxMapping``). Produces :ref:`SpatialBbMappingPacket`. +- :attr:`twostage_crops ` - Streams 2. stage cropped frames to the host. Produces :ref:`FramePacket`. +- :attr:`tracker ` - Streams `ObjectTracker's `__ tracklets and high-res frames that were downscaled and used for inferencing. Produces :ref:`TrackerPacket`. +- :attr:`nn_data ` - Streams NN raw output. Produces :ref:`NNDataPacket`. Decoding outputs ################# @@ -50,6 +51,9 @@ NNComponent allows user to define their own decoding functions. There is a set o - :class:`ImgLandmarks ` - :class:`InstanceSegmentation ` +.. note:: + This feature is still in development and is not guaranteed to work correctly in all cases. + Example usage: .. code-block:: python diff --git a/depthai_sdk/docs/source/components/stereo_component.rst b/depthai_sdk/docs/source/components/stereo_component.rst index c586a43c5..9e8cddedb 100644 --- a/depthai_sdk/docs/source/components/stereo_component.rst +++ b/depthai_sdk/docs/source/components/stereo_component.rst @@ -1,7 +1,7 @@ StereoComponent =============== -**StereoComponent** abstracts `StereoDepth `__ node, its configuration, +:class:`StereoComponent ` abstracts `StereoDepth `__ node, its configuration, filtering (eg. `WLS filter `__), and disparity/depth viewing. Usage @@ -23,9 +23,12 @@ Usage Component outputs ################# -- ``out.main`` - Default output. Uses ``out.depth``. -- ``out.disparity`` - Streams `StereoDepth's `__ disparity frames to the host. When visualized, these get normalized and colorized. Produces :ref:`FramePacket`. -- ``out.depth`` - Streams `StereoDepth's `__ depth frames to the host. When visualized, depth gets converted to disparity (for nicer visualization), normalized and colorized. Produces :ref:`FramePacket`. +- :attr:`main ` - Default output. Uses :attr:`depth `. +- :attr:`disparity ` - Streams `StereoDepth's `__ disparity frames to the host. When visualized, these get normalized and colorized. Produces :ref:`DepthPacket`. +- :attr:`depth ` - Streams `StereoDepth's `__ depth frames to the host. When visualized, depth gets converted to disparity (for nicer visualization), normalized and colorized. Produces :ref:`DepthPacket`. +- :attr:`rectified_left ` - Streams `StereoDepth's `__ rectified left frames to the host. +- :attr:`rectified_right ` - Streams `StereoDepth's `__ rectified right frames to the host. +- :attr:`encoded ` - Provides an encoded version of :attr:`disparity ` stream. Reference ######### diff --git a/depthai_sdk/docs/source/conf.py b/depthai_sdk/docs/source/conf.py index fa2af3393..eefe52fef 100644 --- a/depthai_sdk/docs/source/conf.py +++ b/depthai_sdk/docs/source/conf.py @@ -36,6 +36,7 @@ "sphinx_rtd_theme", 'sphinx.ext.autodoc', 'sphinx.ext.napoleon', + 'autodocsumm' ] # Add any paths that contain templates here, relative to this directory. diff --git a/depthai_sdk/docs/source/features/ai_models.rst b/depthai_sdk/docs/source/features/ai_models.rst index e9f4d0151..669b2a55b 100644 --- a/depthai_sdk/docs/source/features/ai_models.rst +++ b/depthai_sdk/docs/source/features/ai_models.rst @@ -3,11 +3,11 @@ AI models Through the :ref:`NNComponent`, DepthAI SDK abstracts: -- **AI model sourcing** using `blobconverter `__ from `Open Model Zoo `__ (OMZ) and `DepthAI Model Zoo `__ (DMZ) -- **AI result decoding** - currently SDK supports on-device decoding for YOLO and MobileNet based results using `YoloDetectionNetwork `__ and `MobileNetDetectionNetwork `__ nodes -- **Decoding** of the ``config.json`` which **allows an easy deployment of custom AI models** trained `using our notebooks `__ and converted using https://tools.luxonis.com -- Formatting of the AI model input frame - SDK uses **BGR** color order and **Planar / CHW** channel layout conventions -- Integration with 3rd party tools/services (:ref:`Roboflow`) +- **AI model sourcing** using `blobconverter `__ from `Open Model Zoo `__ (OMZ) and `DepthAI Model Zoo `__ (DMZ). +- **AI result decoding** - currently SDK supports on-device decoding for YOLO and MobileNet based results using `YoloDetectionNetwork `__ and `MobileNetDetectionNetwork `__ nodes. +- **Decoding** of the ``config.json`` which **allows an easy deployment of custom AI models** trained `using our notebooks `__ and converted using https://tools.luxonis.com. +- Formatting of the AI model input frame - SDK uses **BGR** color order and **Planar / CHW** channel layout conventions. +- Integration with 3rd party tools/services (:ref:`Roboflow`). SDK supported models @@ -28,6 +28,8 @@ With :ref:`NNComponent` you can **easily try out a variety of different pre-trai Both of the models above are supported by this SDK, so they will be downloaded and deployed to the OAK device along with the pipeline. +The following table lists all the models supported by the SDK. The model name is the same as the name used in the :ref:`NNComponent` constructor. + .. list-table:: :header-rows: 1 diff --git a/depthai_sdk/docs/source/features/recording.rst b/depthai_sdk/docs/source/features/recording.rst index c4a7a9e5b..ab49628da 100644 --- a/depthai_sdk/docs/source/features/recording.rst +++ b/depthai_sdk/docs/source/features/recording.rst @@ -1,3 +1,4 @@ +.. _Recording: Recording ========= diff --git a/depthai_sdk/docs/source/fundamentals/callbacks.rst b/depthai_sdk/docs/source/fundamentals/callbacks.rst deleted file mode 100644 index 5e2cbaee5..000000000 --- a/depthai_sdk/docs/source/fundamentals/callbacks.rst +++ /dev/null @@ -1,2 +0,0 @@ -Callbacks -========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/fundamentals/components.rst b/depthai_sdk/docs/source/fundamentals/components.rst index e65a281b0..74b1305bc 100644 --- a/depthai_sdk/docs/source/fundamentals/components.rst +++ b/depthai_sdk/docs/source/fundamentals/components.rst @@ -16,12 +16,13 @@ their initialization, configuration, and linking. This improves ease of use when .. autoclass:: depthai_sdk.components.Component :members: :undoc-members: + :noindex: .. toctree:: :maxdepth: 1 :hidden: :glob: - :caption: Components: + :caption: Components ../components/* diff --git a/depthai_sdk/docs/source/fundamentals/oak-camera.rst b/depthai_sdk/docs/source/fundamentals/oak-camera.rst deleted file mode 100644 index 3f4b031a6..000000000 --- a/depthai_sdk/docs/source/fundamentals/oak-camera.rst +++ /dev/null @@ -1,2 +0,0 @@ -OakCamera -========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/fundamentals/packets.rst b/depthai_sdk/docs/source/fundamentals/packets.rst index 325373a31..895def582 100644 --- a/depthai_sdk/docs/source/fundamentals/packets.rst +++ b/depthai_sdk/docs/source/fundamentals/packets.rst @@ -4,17 +4,17 @@ Packets Packets are **synchronized collections** of one or more `DepthAI messages `__. They are used **internally for visualization** and also forwarded to the callback function if the user: -#. Specified a callback for visualizing of an output (``oak.visualize(component, callback=cb)``) -#. Used callback output (``oak.callback(component, callback=cb)``) +#. Specified a callback for visualizing of an output via :meth:`OakCamera.visualize(..., callback=fn) `. +#. Used callback output via :meth:`OakCamera.callback(..., callback=fn, enable_visualizer=True) `. -Example -####### +API Usage +##### -#. **oak.visualize**: In the example below SDK won't show the frame to the user, but instead it will send the packet to the callback function. SDK will draw detections (bounding boxes, labels) on the ``packet.frame``. -#. **oak.callback**: This will also send ``DetectionPacket`` to the callback function, the only difference is that the SDK won't draw on the frame, so you can draw detections on the frame yourself. +#. :meth:`OakCamera.visualize() `: In the example below SDK won't show the frame to the user, but instead it will send the packet to the callback function. SDK will draw detections (bounding boxes, labels) on the ``packet.frame``. +#. :meth:`OakCamera.callback() `: This will also send :class:`DetectionPacket ` to the callback function, the only difference is that the SDK won't draw on the frame, so you can draw detections on the frame yourself. .. note:: - If you specify callback function in **oak.visualize**, you need to trigger drawing of detections yourself via **visualizer.draw** method. + If you specify callback function in :meth:`OakCamera.visualize() `, you need to trigger drawing of detections yourself via :meth:`Visualizer.draw() ` method. .. code-block:: python @@ -35,7 +35,7 @@ Example oak.visualize(nn.out.main, fps=True, callback=cb) # 2. Callback: - oak.callback(nn.out.main, callback=cb) + oak.callback(nn.out.main, callback=cb, enable_visualizer=True) oak.start(blocking=True) @@ -64,6 +64,13 @@ DetectionPacket :members: :undoc-members: +NNDataPacket +------------ + +.. autoclass:: depthai_sdk.classes.packets.NNDataPacket + :members: + :undoc-members: + DepthPacket --------------- diff --git a/depthai_sdk/docs/source/features/visualizer.rst b/depthai_sdk/docs/source/fundamentals/visualizer.rst similarity index 61% rename from depthai_sdk/docs/source/features/visualizer.rst rename to depthai_sdk/docs/source/fundamentals/visualizer.rst index 5c740d21d..25fd070aa 100644 --- a/depthai_sdk/docs/source/features/visualizer.rst +++ b/depthai_sdk/docs/source/fundamentals/visualizer.rst @@ -1,39 +1,32 @@ Visualizer ========== -Visualizer helps user to visualize the output of AI models in a more intuitive and customizable way. +DepthAI SDK visualizer serves as a tool to visualize the output of the DepthAI pipeline. + +It can be used to visualize the output of the camera, neural network, depth and disparity map, the rectified streams, the spatial location of the detected objects, and more. Getting Started ############### -:func:`Visualizer ` is created upon calling -:func:`oak.visualize `, which returns :func:`Visualizer ` instance. -Once it is created, the visualizer configs can be modified using :func:`output `, -:func:`stereo `, -:func:`text `, -:func:`detections `, -:func:`tracking ` methods. +:class:`Visualizer ` is created upon calling +:meth:`OakCamera.visualize() `, which returns :func:`Visualizer ` instance. +Once it is created, the visualizer configs can be modified using :meth:`output() `, +:meth:`stereo() `, +:meth:`text() `, +:meth:`detections() `, +:meth:`tracking() ` methods. -Example how :func:`Visualizer ` can be created: +Example how :class:`Visualizer ` can be created: .. code-block:: python with OakCamera() as oak: - cam = oak.create_camera(...) + cam = oak.create_camera('color') visualizer = oak.visualize(cam.out.main) + oak.start(blocking=True) :func:`Visualizer ` is primarily used alongside with :ref:`Packets` -in ``oak_outputs.xout`` module. - - -Flow -#### - -The flow of the :class:`Visualizer ` is as follows: - -1. Once the stream produces the output, corresponding ``Packet`` is sent to the ``Xout`` class. -2. The ``Xout`` calls ``visualize`` function and parses ``Packet`` and add the required objects to the visualizer. -3. After objects are added, ``Xout`` calls :func:`draw ` method, that draws objects on the frame. +in :mod:`depthai_sdk.oak_outputs` module. Configs ####### @@ -48,11 +41,11 @@ and :class:`TrackingConfig `. Each config's type has its own set of parameters, which effects how the corresponding object will be visualized. There are the following methods for modifying the default configuration: -:func:`output `, -:func:`stereo `, -:func:`text `, -:func:`detections `, -:func:`tracking `. +:meth:`output() `, +:meth:`stereo() `, +:meth:`text() `, +:meth:`detections() `, +:meth:`tracking() `. The arguments should be passed as key-value arguments with the same signature as the corresponding config, e.g., ``text(font_size=2, font_color=(255,123,200))``. @@ -74,9 +67,9 @@ Example how to configure the visualizer: Objects ####### -:func:`Visualizer ` operates with objects. Objects can be seen as a hierarchical structure. +:class:`Visualizer ` operates with objects. Objects can be seen as a hierarchical structure. The root object is ``self``, and the children are the list of the created objects. -:func:`add_child ` should be used to add the object to the children list. +:meth:`add_child ` should be used to add the object to the children list. The parent object shares the config and frame shape with all children. All objects must be derived from :class:`GenericObject `. @@ -91,21 +84,21 @@ Implemented objects: Objects can be added to the visualizer using the following methods: -* :func:`add_text `, -* :func:`add_detections `, -* :func:`add_trail `, -* :func:`add_circle `, -* :func:`add_line `. +* :meth:`add_text() `, +* :meth:`add_detections() `, +* :meth:`add_trail() `, +* :meth:`add_circle() `, +* :meth:`add_line() `. Create your own object ###################### If the provided functionality is not enough, you can create your own object. To do so, you need to create a class derived from :class:`GenericObject ` and implement the -:func:`prepare `, -:func:`serialize `, -and :func:`draw ` methods. -The :func:`draw ` method should draw the object on the passed ``frame`` argument. +:meth:`prepare `, +:meth:`serialize `, +and :meth:`draw ` methods. +The :meth:`draw ` method should draw the object on the passed ``frame`` argument. .. code-block:: python @@ -130,7 +123,7 @@ The :func:`draw ` method shoul Example usage ############# -The following script will visualize the output of face detection model. +The following script will visualize the output of face detection model: .. code-block:: python @@ -157,6 +150,44 @@ The following script will visualize the output of face detection model. oak.start(blocking=True) +Serialization +############# +The visualizer provides a way to serialize the output objects to JSON, which can be used for further processing. + +JSON schemas +++++++++++++ + +General config +~~~~~~~~~~~~~~ + +.. literalinclude:: ../visualizer_formats/format.json + :language: json + +Objects +~~~~~~~ + +- Detection: + +.. literalinclude:: ../visualizer_formats/detection_format.json + :language: json + +- Text: + +.. literalinclude:: ../visualizer_formats/text_format.json + :language: json + +- Line: + +.. literalinclude:: ../visualizer_formats/line_format.json + :language: json + +Example JSON output +~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../visualizer_formats/example.json + :language: json + + References ########## @@ -201,4 +232,4 @@ Configs :members: :undoc-members: -.. include:: ../includes/footer-short.rst \ No newline at end of file +.. include:: ../includes/footer-short.rst diff --git a/depthai_sdk/docs/source/includes/install-short.rst b/depthai_sdk/docs/source/includes/install-short.rst index 18533be9d..a3dde8a10 100644 --- a/depthai_sdk/docs/source/includes/install-short.rst +++ b/depthai_sdk/docs/source/includes/install-short.rst @@ -1,5 +1,16 @@ -To install this package, run the following command in your terminal window +DepthAI SDK is available on PyPI. You can install it with the following command: -.. code-block:: bash +.. code-block:: sh + # Linux and macOS python3 -m pip install depthai-sdk + + # Windows + py -m pip install depthai-sdk + +Additionally, you can install the following extra requirements: + +* ``depthai-sdk[visualize]`` - to install the visualization tools. +* ``depthai-sdk[replay]`` - to install the *mcap* and *rosbag* replay tools. +* ``depthai-sdk[record]`` - to install the recording tools. +* ``depthai-sdk[minimal]`` - to install the minimal set of dependencies (excluding OpenCV). \ No newline at end of file diff --git a/depthai_sdk/docs/source/index.rst b/depthai_sdk/docs/source/index.rst index 8bb3679a0..e25244311 100644 --- a/depthai_sdk/docs/source/index.rst +++ b/depthai_sdk/docs/source/index.rst @@ -1,6 +1,6 @@ -=========== +==================== What is DepthAI SDK? -=========== +==================== DepthAI SDK is a Python package built on top of the `depthai-python `__ API library that **improves ease of use when developing apps for OAK devices**. @@ -8,7 +8,7 @@ ease of use when developing apps for OAK devices**. .. image:: https://user-images.githubusercontent.com/18037362/142909472-eae7bed1-695b-48ec-8895-76982989de11.png .. note:: - This class will be in **alpha stage** until **depthai-sdk 2.0.0**, so there will likely be some API changes. + DepthAI SDK is in **alpha stage** until **depthai-sdk 2.0**, so there will likely be API changes during the development. .. include:: ./includes/footer-short.rst @@ -20,7 +20,6 @@ ease of use when developing apps for OAK devices**. :caption: Getting started self - installation.rst quickstart.rst .. toctree:: @@ -31,13 +30,6 @@ ease of use when developing apps for OAK devices**. fundamentals/* -.. toctree:: - :maxdepth: 2 - :hidden: - :glob: - :caption: Examples - - examples/* .. toctree:: :maxdepth: 2 @@ -46,3 +38,11 @@ ease of use when developing apps for OAK devices**. :caption: Features features/* + +.. toctree:: + :maxdepth: 2 + :hidden: + :glob: + :caption: References + + api_reference.rst diff --git a/depthai_sdk/docs/source/installation.rst b/depthai_sdk/docs/source/installation.rst deleted file mode 100644 index cda281b9e..000000000 --- a/depthai_sdk/docs/source/installation.rst +++ /dev/null @@ -1,6 +0,0 @@ -Installation -============ - -.. include:: ./includes/install-short.rst - -.. include:: ./includes/footer-short.rst \ No newline at end of file diff --git a/depthai_sdk/docs/source/quickstart.rst b/depthai_sdk/docs/source/quickstart.rst index 62938bc68..cc2b0fcf4 100644 --- a/depthai_sdk/docs/source/quickstart.rst +++ b/depthai_sdk/docs/source/quickstart.rst @@ -1,20 +1,29 @@ Quickstart ========== +The DepthAI SDK is a powerful tool for building computer vision applications using Luxonis devices. +This quickstart guide will help you get started with the SDK. + +Installation +------------ + +.. include:: ./includes/install-short.rst + Working with camera ------------------- -The `OakCamera` class is a high-level interface for accessing the features of the OAK device using the DepthAI SDK. -It provides a simple and intuitive way to create pipelines that capture video from the OAK camera, run neural networks on the video stream, and visualize the results. +The :class:`depthai_sdk.OakCamera` class is a fundamental part of the DepthAI SDK, providing a high-level interface for accessing the features of the OAK device. +This class simplifies the creation of pipelines that capture video from the OAK camera, run neural networks on the video stream, and visualize the results. -The OakCamera class encapsulates the lower-level details of configuring the OAK device and setting up the pipelines. -It provides a number of methods for creating cameras, neural networks, and other pipeline components, and it takes care of connecting these components together and starting the pipeline. +With :class:`depthai_sdk.OakCamera`, you can easily create color and depth streams using the :meth:`depthai_sdk.OakCamera.create_camera` and :meth:`depthai_sdk.OakCamera.create_stereo` methods respectively, and add pre-trained neural networks using the :meth:`depthai_sdk.OakCamera.create_nn` method. +Additionally, you can add custom callbacks to the pipeline using the :meth:`depthai_sdk.OakCamera.callback` method and record the outputs using the record() method. -Creating color stream +Creating color and depth streams --------------------- -To create a color stream using the OakCamera class in the DepthAI SDK, we can use the create_camera() method. -Here is an example which creates a color stream with a resolution of 1080p and visualizes the stream: +To create a color stream we can use the :meth:`depthai_sdk.OakCamera.create_camera` method. + +Here is an example which creates color and depth streams and visualizes the stream: .. code-block:: python @@ -22,28 +31,100 @@ Here is an example which creates a color stream with a resolution of 1080p and v with OakCamera() as oak: color = oak.create_camera('color', resolution='1080p') - oak.visualize(color) + stereo = oak.create_stereo(resolution='800p') # works with stereo devices only! + oak.visualize([color, stereo]) oak.start(blocking=True) -Creating depth stream ------------- +Creating YOLO neural network for object detection +-------------------------------------------------- + +DepthAI SDK provides a number of pre-trained neural networks that can be used for object detection, pose estimation, semantic segmentation, and other tasks. +To create a neural network, we can use the :meth:`depthai_sdk.OakCamera.create_nn` method and pass the name of the neural network as an argument. -To create a depth stream using the OakCamera class in the DepthAI SDK, we can use the create_stereo() method. -Here is an example which creates a depth stream with a resolution of 800p and visualizes the stream: +Here is an example which creates a YOLO neural network for object detection and visualizes the results: .. code-block:: python - from depthai_sdk import OakCamera + from depthai_sdk import OakCamera - with OakCamera() as oak: - depth = oak.create_stereo('800p') - oak.visualize(depth) - oak.start(blocking=True) + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p') + # List of models that are supported out-of-the-box by the SDK: + # https://docs.luxonis.com/projects/sdk/en/latest/features/ai_models/#sdk-supported-models + yolo = oak.create_nn('yolov6n_coco_640x640', input=color) + + oak.visualize([color, yolo]) + oak.start(blocking=True) + +Adding custom callbacks +----------------------- + +Callbacks are functions that are called when a new frame is available from the camera or neural network. +:class:`depthai_sdk.OakCamera` provides a mechanism for adding custom callbacks to the pipeline using the :meth:`depthai_sdk.OakCamera.callback` method. + +Here is an example which creates a YOLO neural network for object detection and prints the number of detected objects: + +.. code-block:: python + + from depthai_sdk import OakCamera + def print_num_objects(packet): + print(f'Number of objects detected: {len(packet.detections)}') + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p') + yolo = oak.create_nn('yolov6n_coco_640x640', input=color) + + oak.callback(yolo, callback=print_num_objects) + oak.start(blocking=True) Recording --------- +DepthAI SDK provides a simple API for recording the outputs. The :meth:`depthai_sdk.OakCamera.record` method takes a list of outputs and a path to the output file. +Here is an example which creates a YOLO neural network for object detection and records the results: + +.. code-block:: python + + from depthai_sdk import OakCamera + from depthai_sdk.record import RecordType + + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p') + yolo = oak.create_nn('yolov6n_coco_640x640', input=color) + + oak.record([color, yolo], path='./records', record_type=RecordType.VIDEO) + oak.start(blocking=True) + +There are several formats supported by the SDK for recording the outputs: + +#. :attr:`depthai_sdk.record.RecordType.VIDEO` - record video files. +#. :attr:`depthai_sdk.record.RecordType.MCAP` - record `MCAP `__ files. +#. :attr:`depthai_sdk.record.RecordType.BAG` - record `ROS bag `__ files. + +You can find more information about recording in the :ref:`record` section. + +Output syncing +------- + +There is a special case when one needs to synchronize multiple outputs. +For example, recording color stream and neural network output at the same time. +In this case, one can use the :meth:`depthai_sdk.OakCamera.sync`. +This method takes a list of outputs and returns a synchronized output to the specified callback function. +Here is an example which synchronizes color stream and YOLO neural network output: + +.. code-block:: python + + from depthai_sdk import OakCamera + + def callback(synced_packets): + print(synced_packets) + + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p') + yolo = oak.create_nn('yolov6n_coco_640x640', input=color) + oak.sync([color, yolo], callback=callback) + oak.start(blocking=True) +.. include:: ./includes/footer-short.rst \ No newline at end of file diff --git a/depthai_sdk/docs/source/visualizer_formats/detection_format.json b/depthai_sdk/docs/source/visualizer_formats/detection_format.json new file mode 100644 index 000000000..80c2e90b7 --- /dev/null +++ b/depthai_sdk/docs/source/visualizer_formats/detection_format.json @@ -0,0 +1,35 @@ +{ + "type": "detections", + "detections": { + "type": "array", + "items": { + "type": "object", + "bbox": { + "type": "array", + "items": { + "type": "number" + }, + "description": "bbox absolute coordinates in format [x1, y1, x2, y2]" + }, + "label": { + "type": "string", + "description": "class label" + }, + "color": { + "type": "array", + "items": { + "type": "integer" + }, + "description": "bbox color in RGB format" + } + } + }, + "children": { + "type": "array", + "items": { + "type": "object" + }, + "description": "array of child objects (e.g. detection, text, line)", + "default": [] + } +} \ No newline at end of file diff --git a/depthai_sdk/docs/source/visualizer_formats/example.json b/depthai_sdk/docs/source/visualizer_formats/example.json new file mode 100644 index 000000000..56f270aad --- /dev/null +++ b/depthai_sdk/docs/source/visualizer_formats/example.json @@ -0,0 +1,74 @@ +{ + "platform": "pc", + "frame_shape": [720, 1280], + "config": { + "output": { + "img_scale": 1.0, + "show_fps": false, + "clickable": true + }, + "stereo": { + "colorize": 2, + "colormap": 2, + "wls_filter": false, + "wls_lambda": 8000, + "wls_sigma": 1.5 + }, + "detection": { + "thickness": 1, + "fill_transparency": 0.15, + "box_roundness": 0, + "color": [0, 255, 0], + "bbox_style": 0, + "line_width": 0.5, + "line_height": 0.5, + "hide_label": false, + "label_position": 0, + "label_padding": 10 + }, + "text": { + "font_face": 0, + "font_color": [255, 255, 255], + "font_transparency": 0.5, + "font_scale": 1.0, + "font_thickness": 2, + "font_position": 0, + "bg_transparency": 0.5, + "bg_color": [0, 0, 0], + "line_type": 16, + "auto_scale": true + }, + "tracking": { + "max_length": -1, + "deletion_lost_threshold": 5, + "line_thickness": 1, + "fading_tails": false, + "line_color": [255, 255, 255], + "line_type": 16 + }, + "circle": { + "thickness": 1, + "color": [255, 255, 255], + "line_type": 16 + } + }, + "objects": [ + { + "type": "detections", + "detections": [ + { + "bbox": [101, 437, 661, 712], + "label": "laptop", + "color": [210, 167, 218] + } + ], + "children": [ + { + "type": "text", + "text": "Laptop", + "coords": [111, 469] + } + ] + } + ] +} diff --git a/depthai_sdk/docs/source/visualizer_formats/format.json b/depthai_sdk/docs/source/visualizer_formats/format.json new file mode 100644 index 000000000..10d826ad7 --- /dev/null +++ b/depthai_sdk/docs/source/visualizer_formats/format.json @@ -0,0 +1,244 @@ +{ + "platform": { + "type": "string", + "enum": [ + "pc", + "robothub" + ] + }, + "frame_shape": { + "type": "array", + "items": { + "type": "integer" + }, + "description": "Frame shape in (height, width) format." + }, + "config": { + "type": "object", + "output": { + "img_scale": { + "type": "number", + "minimum": 0.0, + "maximum": 1.0, + "default": 1.0, + "description": "Scale of output image." + }, + "show_fps": { + "type": "boolean", + "default": false, + "description": "Show FPS on output image." + }, + "clickable": { + "type": "boolean", + "default": false, + "description": "Show disparity or depth value on mouse hover." + } + }, + "stereo": { + "type": "object", + "colorize": { + "type": "integer", + "default": 2, + "description": "cv2 colormap." + }, + "colormap": { + "type": "integer", + "default": 2, + "description": "0 - gray, 1 - color, 2 - blended color and depth." + }, + "wls_filter": { + "type": "boolean", + "default": false + }, + "wls_lambda": { + "type": "number", + "default": 8000.0 + }, + "wls_sigma": { + "type": "number", + "default": 1.5 + } + }, + "detection": { + "type": "object", + "thickness": { + "type": "integer", + "default": 1 + }, + "fill_transparency": { + "type": "number", + "default": 0.15, + "minimum": 0.0, + "maximum": 1.0, + "description": "Transparency of bbox fill." + }, + "box_roundness": { + "type": "integer", + "default": 0, + "description": "Roundness of bbox corners, used only when bbox_style is set to BboxStyle.ROUNDED_*." + }, + "color": { + "type": "array", + "items": { + "type": "integer" + }, + "default": [0, 255, 0], + "description": "Default bbox color in RGB format." + }, + "bbox_style": { + "type": "integer", + "default": 0, + "description": "``depthai_sdk.visualize.configs.BBoxStyle`` enum value." + }, + "line_width": { + "type": "number", + "default": 0.5, + "minimum": 0.0, + "maximum": 1.0, + "description": "Horizontal line width of bbox." + }, + "line_height": { + "type": "number", + "default": 0.5, + "minimum": 0.0, + "maximum": 1.0, + "description": "Vertical line height of bbox." + }, + "hide_label": { + "type": "boolean", + "default": false, + "description": "Hide class label on output image." + }, + "label_position": { + "type": "integer", + "default": 0, + "description": "``depthai_sdk.visualize.configs.TextPosition`` enum value." + }, + "label_padding": { + "type": "integer", + "default": 10, + "description": "Padding between label and bbox." + } + }, + "text": { + "font_face": { + "type": "integer", + "default": 0, + "description": "cv2 font face." + }, + "font_color": { + "type": "array", + "items": { + "type": "integer" + }, + "default": [255, 255, 255], + "description": "Font color in RGB format." + }, + "font_transparency": { + "type": "number", + "default": 0.5, + "minimum": 0.0, + "maximum": 1.0 + }, + "font_scale": { + "type": "number", + "default": 1.0 + }, + "font_thickness": { + "type": "integer", + "default": 2 + }, + "font_position": { + "type": "integer", + "default": 0, + "description": "``depthai_sdk.visualize.configs.TextPosition`` enum value." + }, + "bg_transparency": { + "type": "number", + "default": 0.5, + "minimum": 0.0, + "maximum": 1.0, + "description": "Text outline transparency." + }, + "bg_color": { + "type": "array", + "items": { + "type": "integer" + }, + "default": [0, 0, 0], + "description": "Text outline color in RGB format." + }, + "line_type": { + "type": "integer", + "default": 16, + "description": "cv2 line type." + }, + "auto_scale": { + "type": "boolean", + "default": true, + "description": "Automatically scale font size based on bbox size." + } + }, + "tracking": { + "max_length": { + "type": "integer", + "default": -1, + "description": "Maximum length of tracking line, -1 for infinite." + }, + "deletion_lost_threshold": { + "type": "integer", + "default": 5, + "description": "Number of frames after which lost track is deleted." + }, + "line_thickness": { + "type": "integer", + "default": 1 + }, + "fading_tails": { + "type": "boolean", + "default": false, + "description": "Enable fading tails - reduces line thickness over time." + }, + "line_color": { + "type": "array", + "items": { + "type": "integer" + }, + "default": [255, 255, 255], + "description": "Tracking line color in RGB format." + }, + "line_type": { + "type": "integer", + "default": 16, + "description": "cv2 line type." + } + }, + "circle": { + "thickness": { + "type": "integer", + "default": 1 + }, + "color": { + "type": "array", + "items": { + "type": "integer" + }, + "default": [0, 255, 0], + "description": "Circle color in RGB format." + }, + "line_type": { + "type": "integer", + "default": 16, + "description": "cv2 line type." + } + } + }, + "objects": { + "type": "array", + "items": { + "type": "object" + }, + "description": "Array of objects (e.g. detection, text, line).", + "default": [] + } +} \ No newline at end of file diff --git a/depthai_sdk/docs/source/visualizer_formats/line_format.json b/depthai_sdk/docs/source/visualizer_formats/line_format.json new file mode 100644 index 000000000..5f9091455 --- /dev/null +++ b/depthai_sdk/docs/source/visualizer_formats/line_format.json @@ -0,0 +1,25 @@ +{ + "type": "line", + "pt1": { + "type": "array", + "items": { + "type": "number" + }, + "description": "Absolute (x, y) coordinates of the first point." + }, + "pt2": { + "type": "array", + "items": { + "type": "number" + }, + "description": "Absolute (x, y) coordinates of the second point." + }, + "children": { + "type": "array", + "items": { + "type": "object" + }, + "description": "array of child objects (e.g. detection, text, line).", + "default": [] + } +} \ No newline at end of file diff --git a/depthai_sdk/docs/source/visualizer_formats/text_format.json b/depthai_sdk/docs/source/visualizer_formats/text_format.json new file mode 100644 index 000000000..5815916c7 --- /dev/null +++ b/depthai_sdk/docs/source/visualizer_formats/text_format.json @@ -0,0 +1,13 @@ +{ + "type": "text", + "text": { + "type": "plain_text" + }, + "coords": { + "type": "array", + "items": { + "type": "number" + }, + "description": "The absolute coordinates of the text in the format (x1, y1)." + } +} \ No newline at end of file From 2a9d9f1ab9e4204b4b9c5039a01e24e3d9a5546c Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 26 Apr 2023 16:09:40 +0200 Subject: [PATCH 05/12] Update documentation --- .../docs/source/features/replaying.rst | 12 ++-- .../docs/source/fundamentals/visualizer.rst | 6 +- .../docs/source/includes/footer-short.rst | 3 +- depthai_sdk/docs/source/quickstart.rst | 71 ++++++++++++++----- 4 files changed, 67 insertions(+), 25 deletions(-) diff --git a/depthai_sdk/docs/source/features/replaying.rst b/depthai_sdk/docs/source/features/replaying.rst index d5bc001f2..fc3d75b57 100644 --- a/depthai_sdk/docs/source/features/replaying.rst +++ b/depthai_sdk/docs/source/features/replaying.rst @@ -21,9 +21,9 @@ Replaying support Replaying feature is quite extensible, and supports a variety of different inputs: #. Single image. -#. Folder with images. Images are getting rotated every 3 seconds. `Example here `__.. -#. Url to a video/image. -#. Url to a YouTube video. +#. Folder with images. Images are getting rotated every 3 seconds. `Example here `__. +#. URL to a video/image. +#. URL to a YouTube video. #. Path to :ref:`depthai-recording `. #. A name of a :ref:`public depthai-recording `. @@ -49,13 +49,15 @@ Script below will also do depth reconstruction and will display 3D detections co .. figure:: https://user-images.githubusercontent.com/18037362/193642506-76bd2d36-3ae8-4d0b-bbed-083a94463155.png - Live view pipeline uses live camera feeds (MonoCamera, ColorCamera) whereas Replaying pipeline uses XLinkIn nodes to which we send recorded frames + Live view pipeline uses live camera feeds (MonoCamera, ColorCamera) whereas Replaying pipeline uses XLinkIn nodes to which we send recorded frames. Public depthai-recordings ######################### We host several depthai-recordings on our servers that you can easily use in your -application (eg. ``OakCamera(recording='cars-california-01')``). Recording will get downloaded & cached on the computer for future use. +application, e.g., :class:`OakCamera(recording='cars-california-01') `. Recording will get downloaded & cached on the computer for future use. + +The following table lists all available recordings: .. list-table:: :header-rows: 1 diff --git a/depthai_sdk/docs/source/fundamentals/visualizer.rst b/depthai_sdk/docs/source/fundamentals/visualizer.rst index 25fd070aa..aa5ce408b 100644 --- a/depthai_sdk/docs/source/fundamentals/visualizer.rst +++ b/depthai_sdk/docs/source/fundamentals/visualizer.rst @@ -20,6 +20,8 @@ Example how :class:`Visualizer ` ca .. code-block:: python + from depthai_sdk import OakCamera + with OakCamera() as oak: cam = oak.create_camera('color') visualizer = oak.visualize(cam.out.main) @@ -47,10 +49,10 @@ There are the following methods for modifying the default configuration: :meth:`detections() `, :meth:`tracking() `. The arguments should be passed as key-value arguments with the same signature as the corresponding config, -e.g., ``text(font_size=2, font_color=(255,123,200))``. +e.g., :meth:`Visualizer.text(font_size=2, font_color=(255,123,200)) `. The modified configuration will be applied to every created objects. The methods support -fluent interface and can be chained, e.g., ``visualizer.text(font_size=2).detections(color=(255, 0, 0))``. +fluent interface and can be chained, e.g., :meth:`Visualizer.text(font_size=2).detections(color=(255, 0, 0)) `. Example how to configure the visualizer: diff --git a/depthai_sdk/docs/source/includes/footer-short.rst b/depthai_sdk/docs/source/includes/footer-short.rst index 3909829a1..754e010b4 100644 --- a/depthai_sdk/docs/source/includes/footer-short.rst +++ b/depthai_sdk/docs/source/includes/footer-short.rst @@ -2,4 +2,5 @@

Got questions?

- Head over to Discussion Forum for technical support or any other questions you might have. \ No newline at end of file + Head over to Discussion Forum for technical support or any other questions you might have. +| diff --git a/depthai_sdk/docs/source/quickstart.rst b/depthai_sdk/docs/source/quickstart.rst index cc2b0fcf4..4ed932795 100644 --- a/depthai_sdk/docs/source/quickstart.rst +++ b/depthai_sdk/docs/source/quickstart.rst @@ -12,16 +12,24 @@ Installation Working with camera ------------------- -The :class:`depthai_sdk.OakCamera` class is a fundamental part of the DepthAI SDK, providing a high-level interface for accessing the features of the OAK device. +The :class:`OakCamera ` class is a fundamental part of the DepthAI SDK, providing a high-level interface for accessing the features of the OAK device. This class simplifies the creation of pipelines that capture video from the OAK camera, run neural networks on the video stream, and visualize the results. -With :class:`depthai_sdk.OakCamera`, you can easily create color and depth streams using the :meth:`depthai_sdk.OakCamera.create_camera` and :meth:`depthai_sdk.OakCamera.create_stereo` methods respectively, and add pre-trained neural networks using the :meth:`depthai_sdk.OakCamera.create_nn` method. -Additionally, you can add custom callbacks to the pipeline using the :meth:`depthai_sdk.OakCamera.callback` method and record the outputs using the record() method. +With :class:`OakCamera `, you can easily create color and depth streams using the :meth:`OakCamera.create_camera() ` and :meth:`OakCamera.create_stereo() ` methods respectively, and add pre-trained neural networks using the :meth:`OakCamera.create_nn() ` method. +Additionally, you can add custom callbacks to the pipeline using the :meth:`OakCamera.callback() ` method and record the outputs using the :meth:`OakCamera.record() ` method. Creating color and depth streams --------------------- -To create a color stream we can use the :meth:`depthai_sdk.OakCamera.create_camera` method. +To create a color stream we can use the :meth:`OakCamera.create_camera() ` method. +This method takes the name of the sensor as an argument and returns a :class:`CameraComponent ` object. + +The full list of supported sensors: ``color``; ``left``; ``right``; ``cam_{socket},color``, ``cam_{socket},mono``, where ``{socket}`` is a letter from A to H representing the socket on the OAK device. +Custom socket names are usually used for FFC devices. + +To visualize the stream, we can use the :meth:`OakCamera.visualize() ` method. +This method takes a list of outputs and displays them. +Each component has its own outputs, which can be found in the :ref:`components` section. Here is an example which creates color and depth streams and visualizes the stream: @@ -39,7 +47,9 @@ Creating YOLO neural network for object detection -------------------------------------------------- DepthAI SDK provides a number of pre-trained neural networks that can be used for object detection, pose estimation, semantic segmentation, and other tasks. -To create a neural network, we can use the :meth:`depthai_sdk.OakCamera.create_nn` method and pass the name of the neural network as an argument. +To create a neural network, we can use the :meth:`OakCamera.create_nn() ` method and pass the name of the neural network as an argument. + +Similarly to the :meth:`OakCamera.create_camera() ` method, the :meth:`OakCamera.create_nn() ` method returns a :class:`NNComponent ` object. Here is an example which creates a YOLO neural network for object detection and visualizes the results: @@ -60,7 +70,7 @@ Adding custom callbacks ----------------------- Callbacks are functions that are called when a new frame is available from the camera or neural network. -:class:`depthai_sdk.OakCamera` provides a mechanism for adding custom callbacks to the pipeline using the :meth:`depthai_sdk.OakCamera.callback` method. +:class:`OakCamera ` provides a mechanism for adding custom callbacks to the pipeline using the :meth:`OakCamera.callback() ` method. Here is an example which creates a YOLO neural network for object detection and prints the number of detected objects: @@ -81,7 +91,7 @@ Here is an example which creates a YOLO neural network for object detection and Recording --------- -DepthAI SDK provides a simple API for recording the outputs. The :meth:`depthai_sdk.OakCamera.record` method takes a list of outputs and a path to the output file. +DepthAI SDK provides a simple API for recording the outputs. The :meth:`OakCamera.record() ` method takes a list of outputs and a path to the output file. Here is an example which creates a YOLO neural network for object detection and records the results: .. code-block:: python @@ -105,26 +115,53 @@ There are several formats supported by the SDK for recording the outputs: You can find more information about recording in the :ref:`record` section. Output syncing -------- +-------------- There is a special case when one needs to synchronize multiple outputs. For example, recording color stream and neural network output at the same time. -In this case, one can use the :meth:`depthai_sdk.OakCamera.sync`. +In this case, one can use the :meth:`OakCamera.sync() `. This method takes a list of outputs and returns a synchronized output to the specified callback function. Here is an example which synchronizes color stream and YOLO neural network output: .. code-block:: python - from depthai_sdk import OakCamera + from depthai_sdk import OakCamera - def callback(synced_packets): - print(synced_packets) + def callback(synced_packets): + print(synced_packets) - with OakCamera() as oak: - color = oak.create_camera('color', resolution='1080p') - yolo = oak.create_nn('yolov6n_coco_640x640', input=color) + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p') + yolo = oak.create_nn('yolov6n_coco_640x640', input=color) - oak.sync([color, yolo], callback=callback) - oak.start(blocking=True) + oak.sync([color.out.main, yolo.out.main], callback=callback) + oak.start(blocking=True) + +Encoded streams +--------------- + +Luxonis devices support on-device encoding of the outputs to ``H.264``, ``H.265`` and ``MJPEG`` formats. +To enable encoding, we should simply pass the ``encode`` argument to the :meth:`OakCamera.create_camera() ` or :meth:`OakCamera.create_stereo() ` methods. +Possible values for the ``encode`` argument are ``h264``, ``h265`` and ``mjpeg``. + +Each component has its own encoded output: + +- :class:`CameraComponent.Out.encoded ` +- :class:`StereoComponent.Out.encoded ` +- :class:`NNComponent.Out.encoded ` + +Here is an example which visualizes the encoded color, YOLO neural network and disparity streams: + +.. code-block:: python + + from depthai_sdk import OakCamera + + with OakCamera() as oak: + color = oak.create_camera('color', resolution='1080p', fps=20, encode='h264') + stereo = oak.create_stereo('400p', encode='h264') + yolo = oak.create_nn('yolov6nr3_coco_640x352', input=color) + + oak.visualize([color.out.encoded, stereo.out.encoded, yolo.out.encoded]) + oak.start(blocking=True) .. include:: ./includes/footer-short.rst \ No newline at end of file From f01e6c0d851b5c8cd01eb9b9c8c8dafccac2e73e Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Thu, 11 May 2023 13:33:21 +0200 Subject: [PATCH 06/12] Rename .readthedocs.yml to .readthedocs.yaml --- depthai_sdk/docs/{.readthedocs.yml => .readthedocs.yaml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename depthai_sdk/docs/{.readthedocs.yml => .readthedocs.yaml} (100%) diff --git a/depthai_sdk/docs/.readthedocs.yml b/depthai_sdk/docs/.readthedocs.yaml similarity index 100% rename from depthai_sdk/docs/.readthedocs.yml rename to depthai_sdk/docs/.readthedocs.yaml From c7497c7c3fb42168b96393ca2e7cc88d6d883585 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Tue, 28 Feb 2023 03:37:47 +0100 Subject: [PATCH 07/12] [WIP] Docs restructure --- depthai_sdk/docs/source/fundamentals/callbacks.rst | 2 ++ depthai_sdk/docs/source/fundamentals/oak-camera.rst | 2 ++ depthai_sdk/docs/source/installation.rst | 6 ++++++ 3 files changed, 10 insertions(+) create mode 100644 depthai_sdk/docs/source/fundamentals/callbacks.rst create mode 100644 depthai_sdk/docs/source/fundamentals/oak-camera.rst create mode 100644 depthai_sdk/docs/source/installation.rst diff --git a/depthai_sdk/docs/source/fundamentals/callbacks.rst b/depthai_sdk/docs/source/fundamentals/callbacks.rst new file mode 100644 index 000000000..5e2cbaee5 --- /dev/null +++ b/depthai_sdk/docs/source/fundamentals/callbacks.rst @@ -0,0 +1,2 @@ +Callbacks +========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/fundamentals/oak-camera.rst b/depthai_sdk/docs/source/fundamentals/oak-camera.rst new file mode 100644 index 000000000..3f4b031a6 --- /dev/null +++ b/depthai_sdk/docs/source/fundamentals/oak-camera.rst @@ -0,0 +1,2 @@ +OakCamera +========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/installation.rst b/depthai_sdk/docs/source/installation.rst new file mode 100644 index 000000000..cda281b9e --- /dev/null +++ b/depthai_sdk/docs/source/installation.rst @@ -0,0 +1,6 @@ +Installation +============ + +.. include:: ./includes/install-short.rst + +.. include:: ./includes/footer-short.rst \ No newline at end of file From 566f920eab3cc25feba240e3359a8a52a203ab03 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 26 Apr 2023 03:01:14 +0200 Subject: [PATCH 08/12] Refactor documentation --- depthai_sdk/docs/source/fundamentals/callbacks.rst | 2 -- depthai_sdk/docs/source/fundamentals/oak-camera.rst | 2 -- depthai_sdk/docs/source/installation.rst | 6 ------ 3 files changed, 10 deletions(-) delete mode 100644 depthai_sdk/docs/source/fundamentals/callbacks.rst delete mode 100644 depthai_sdk/docs/source/fundamentals/oak-camera.rst delete mode 100644 depthai_sdk/docs/source/installation.rst diff --git a/depthai_sdk/docs/source/fundamentals/callbacks.rst b/depthai_sdk/docs/source/fundamentals/callbacks.rst deleted file mode 100644 index 5e2cbaee5..000000000 --- a/depthai_sdk/docs/source/fundamentals/callbacks.rst +++ /dev/null @@ -1,2 +0,0 @@ -Callbacks -========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/fundamentals/oak-camera.rst b/depthai_sdk/docs/source/fundamentals/oak-camera.rst deleted file mode 100644 index 3f4b031a6..000000000 --- a/depthai_sdk/docs/source/fundamentals/oak-camera.rst +++ /dev/null @@ -1,2 +0,0 @@ -OakCamera -========= \ No newline at end of file diff --git a/depthai_sdk/docs/source/installation.rst b/depthai_sdk/docs/source/installation.rst deleted file mode 100644 index cda281b9e..000000000 --- a/depthai_sdk/docs/source/installation.rst +++ /dev/null @@ -1,6 +0,0 @@ -Installation -============ - -.. include:: ./includes/install-short.rst - -.. include:: ./includes/footer-short.rst \ No newline at end of file From f22515713d318b1f60aea1bc25239fa23d792964 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Thu, 11 May 2023 14:00:04 +0200 Subject: [PATCH 09/12] Update .readthedocs.yaml --- depthai_sdk/docs/.readthedocs.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/depthai_sdk/docs/.readthedocs.yaml b/depthai_sdk/docs/.readthedocs.yaml index a076386c0..02d58b8f6 100644 --- a/depthai_sdk/docs/.readthedocs.yaml +++ b/depthai_sdk/docs/.readthedocs.yaml @@ -8,7 +8,7 @@ version: 2 # Build documentation in the docs/ directory with Sphinx sphinx: builder: dirhtml - configuration: source/conf.py + configuration: depthai_sdk/docs/source/conf.py # Build documentation with MkDocs #mkdocs: @@ -22,4 +22,5 @@ formats: python: version: 3.8 install: - - requirements: requirements.txt \ No newline at end of file + - requirements: requirements.txt + - requirements: depthai_sdk/docs/requirements.txt \ No newline at end of file From 04344a0aab80cbfd885e2c962b310d3e6b982c89 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Thu, 11 May 2023 14:08:19 +0200 Subject: [PATCH 10/12] Move depthai_sdk installation from requirements.txt to .readthedocs.yaml --- depthai_sdk/docs/.readthedocs.yaml | 4 +++- depthai_sdk/docs/requirements.txt | 1 - 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/depthai_sdk/docs/.readthedocs.yaml b/depthai_sdk/docs/.readthedocs.yaml index 02d58b8f6..59b3a8572 100644 --- a/depthai_sdk/docs/.readthedocs.yaml +++ b/depthai_sdk/docs/.readthedocs.yaml @@ -23,4 +23,6 @@ python: version: 3.8 install: - requirements: requirements.txt - - requirements: depthai_sdk/docs/requirements.txt \ No newline at end of file + - requirements: depthai_sdk/docs/requirements.txt + - method: pip + path: depthai_sdk \ No newline at end of file diff --git a/depthai_sdk/docs/requirements.txt b/depthai_sdk/docs/requirements.txt index cbcc68b66..b0abb2e0d 100644 --- a/depthai_sdk/docs/requirements.txt +++ b/depthai_sdk/docs/requirements.txt @@ -1,4 +1,3 @@ Sphinx==4.1.2 sphinx-rtd-theme==0.5.0 --e ../ autodocsumm==0.2.10 From 1eb0ebb09129a1ddb6c00797e80af41fc6130839 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 17 May 2023 13:17:10 +0200 Subject: [PATCH 11/12] Remove info about extra requirements --- depthai_sdk/docs/source/includes/install-short.rst | 7 ------- 1 file changed, 7 deletions(-) diff --git a/depthai_sdk/docs/source/includes/install-short.rst b/depthai_sdk/docs/source/includes/install-short.rst index a3dde8a10..1325dac1d 100644 --- a/depthai_sdk/docs/source/includes/install-short.rst +++ b/depthai_sdk/docs/source/includes/install-short.rst @@ -7,10 +7,3 @@ DepthAI SDK is available on PyPI. You can install it with the following command: # Windows py -m pip install depthai-sdk - -Additionally, you can install the following extra requirements: - -* ``depthai-sdk[visualize]`` - to install the visualization tools. -* ``depthai-sdk[replay]`` - to install the *mcap* and *rosbag* replay tools. -* ``depthai-sdk[record]`` - to install the recording tools. -* ``depthai-sdk[minimal]`` - to install the minimal set of dependencies (excluding OpenCV). \ No newline at end of file From 32367c7973d01be5d259f62473105d4d91771d63 Mon Sep 17 00:00:00 2001 From: Daniil Pastukhov Date: Wed, 17 May 2023 13:17:37 +0200 Subject: [PATCH 12/12] Remove excessive OakCamera occurrences --- depthai_sdk/docs/source/quickstart.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/depthai_sdk/docs/source/quickstart.rst b/depthai_sdk/docs/source/quickstart.rst index 4ed932795..b81e902a5 100644 --- a/depthai_sdk/docs/source/quickstart.rst +++ b/depthai_sdk/docs/source/quickstart.rst @@ -15,8 +15,8 @@ Working with camera The :class:`OakCamera ` class is a fundamental part of the DepthAI SDK, providing a high-level interface for accessing the features of the OAK device. This class simplifies the creation of pipelines that capture video from the OAK camera, run neural networks on the video stream, and visualize the results. -With :class:`OakCamera `, you can easily create color and depth streams using the :meth:`OakCamera.create_camera() ` and :meth:`OakCamera.create_stereo() ` methods respectively, and add pre-trained neural networks using the :meth:`OakCamera.create_nn() ` method. -Additionally, you can add custom callbacks to the pipeline using the :meth:`OakCamera.callback() ` method and record the outputs using the :meth:`OakCamera.record() ` method. +With :class:`OakCamera `, you can easily create color and depth streams using the :meth:`create_camera() ` and :meth:`create_stereo() ` methods respectively, and add pre-trained neural networks using the :meth:`create_nn() ` method. +Additionally, you can add custom callbacks to the pipeline using the :meth:`callback() ` method and record the outputs using the :meth:`record() ` method. Creating color and depth streams ---------------------