Home
In this project, ViSP and Unity are integrated to create Augmented Reality (AR) applications. ViSP (Visual Servoing Platform) is a cross-platform library that provides real-time visual tracking and visual servoing implementation useful in robotics, computer vision, augmented reality and computer animation. Pose estimation or tracking algorithms in ViSP have the potential to benefit AR community and could be used for AR applications in general.
Unity is a cross-platform 3D game engine to build high quality games. Unity engine provides a framework to create/add/manipulate 3D scenes consisting of camera, lighting and 3D objects.
ViSP offers several methods to track and estimate the pose of the camera. Basic methods intend to estimate the camera pose from 2D/3D point correspondences. 2D point coordinates are obtained by detecting and tracking fiducial markers, for instance by tracking blobs or by detecting corners of an AprilTag. Advanced methods rely on the knowledge of the CAD model of the object to track. The model-based tracker allows to track and estimate the camera pose of a markerless object using multiple types of primitives (edges, texture or both). Return to table of content.
This project aims to place arbitrary marked objects thanks to AprilTag markers at specific places within the virtual-camera field of view. Marked objects are detected, localized and tracked using detection and model-based tracking approaches implemented in ViSP. While Unity supports creation of virtual scenes, it does not have libraries to perform computer vision tasks such as pose estimation and tracking. However, Unity allows integration of 3rd party libraries in the form of plugin (DLLs for dynamically linked libraries under Windows, Bundles under MacOS, or shared libraries under Ubuntu) which can be imported into Unity as Assets. So, it is possible to export ViSP library functions as a plugin and import it in Unity. This combination of Unity and ViSP creates a highly capable AR system.
A scene created using Unity game engine consists of cameras, lighting and game objects. Unity allows creation and import of assets that influence the visual and behavioral aspects of game objects. A game object’s behavior can be programmed using a C# script. Unity also allows import of plugins which can be called by C# behavior scripts to extend the functionality of Unity SDK. ViSP exports the pose estimation and tracking functionality to the Unity scripts associated with game objects.
Above figure shows the schematic of interoperability between ViSP and Unity. ViSP is implemented in C++ and is therefore unmanaged while the behaviour scripts are in managed C# environment. Therefore, implementation of exported API must take the following into consideration:
- Data type interoperability: data types of parameters and return values of API functions must be the same in both environments (C# and C++).
- Garbage collection: any object created dynamically in C++ will not free the memory until explicitly freed. Therefore, API functions in C++ must also export suitable destructors which can be called by C# script once the use of required methods is complete.
Return to table of content.
Start by looking at this tutorial which explains how one can generate ViSP wrapper plugin to be used in Unity Project using an example.
Return to table of content.