This is a follow-up on the OpenVino’s inference tutorials:
Version 2019 R1.0
Version 2018 R5.0
Version 2018 R4.0
Caution!
-
As of OpenVINO’s Release 2019 R1 and afterwards, the model’s binaries are not included in the toolkit, as they are part of the model zoo. You are supposed to download them manually as described in the tutorial. Be aware that if you choose to download them in a different path than the default, our
scripts/setupenv.shwill not fully work and you will have to add the path to the models yourself when running the program. -
The API got broken since 2019 R2, if you’re using an older OpenVINO version, run
git checkout OpenVINO\<\=2019R1and work from there.
This project consists on showcasing the advantages of the Intel’s OpenVINO toolkit. We will develop a Driver Behaviour case scenario, where we will detect drowsiness based on blinking and yawning and gaze direction. For that, we will use the OpenVINO toolkit and OpenCV, all written in C++.
As mentioned previously, we will take the Interactive face detection sample as a starting point, as it provides us with the options to run and stack different models synchronously or asynchronously. We will develop the following features based on computer vision:
-
Sleep/Drowsiness Detection:
-
Counting frecuency of blinking.
-
Yawn detection.
-
-
Gaze detection.
Using OpenVino’s model detection we can easily detect faces with great accuracy. We are currently using for testing 2 different face detection models that are included with OpenVino out-of-the-box:
-
face-detection-adas-0001
-
face-detection-retail-0004
Using the image detected inside the face ROI (region of interest), we feed a facial landmarks detector to identify points of iterest. Using 6 points for each eye and 6 points for the mouth it is possible to calculate 'Eye Aspect Ratio (EAR)' that gives 2 values for eye/mouth open or closed (based on this paper).
At the moment of writing this guide, the facial landmarks detection model included with OpenVino (facial-landmarks-35-adas-0001) has not enough points to run this calculations. We are using dlib’s facial landmarks detector instead.
Once we have a positive detection for blink/yawn, we count frames of those events and trigger an alarm when they hit a threshold.
Using the face’s ROI, we feed a head-pose detector model provided by OpenVino (head-pose-estimation-adas-0001). Analizing the output of that model we can easily detect when the face is not centered or not looking to the front.
To run the application in this tutorial, the OpenVINO™ toolkit and its dependencies must already be installed and verified using the included demos. Installation instructions may be found at: https://software.intel.com/en-us/articles/OpenVINO-Install-Linux
If to be used, any optional hardware must also be installed and verified including:
-
USB camera - Standard USB Video Class (UVC) camera.
-
Intel® Core™ CPU with integrated graphics.
-
Intel® ARC dGPU.
-
VPU - USB Intel® Movidius™ Neural Compute Stick and what is being referred to as "Myriad"
A summary of what is needed:
-
Target and development platforms meeting the requirements described in the "System Requirements" section of the OpenVINO™ toolkit documentation which may be found at: https://software.intel.com/en-us/openvino-toolkit
Note: While writing this tutorial, an Intel® i7-12700H with integrated Intel® Iris® Xe Graphics and Intel® Graphics Arc A770 GPU was used as both the development and target platform.
-
Optional:
-
Intel® Movidius™ Neural Compute Stick
-
USB UVC camera
-
Intel® Core™ CPU with integrated graphics.
-
Intel® Graphics Arc dGPU
-
-
OpenVINO™ toolkit supported Linux operating system. This tutorial was run on 64-bit Ubuntu 22.04.1 LTS updated to kernel 5.15.71+ following the OpenVINO™ toolkit installation instructions. Also tested on Ubuntu 20.04
-
The latest OpenVINO™ toolkit(2022.3 LTS) installed and verified. Supported versions 2019 R2 and greater
-
Git(git) for downloading from the GitHub repository.
-
GRPC for getting remote frame as input and do inference
export MY_INSTALL_DIR=$HOME/.local
mkdir -p $MY_INSTALL_DIR
export PATH="$MY_INSTALL_DIR/bin:$PATH"
#add $HOME/.local/bin to the path in /etc/environment to avoid exporting everytime
#echo "export PATH=$MY_INSTALL_DIR/bin:$PATH">>/home/${USERNAME}/.bashrc
git clone --recurse-submodules -b v1.46.2 --depth 1 --shallow-submodules https://github.com/grpc/grpc grpc
cd grpc
mkdir build && cd build
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$MY_INSTALL_DIR ..
make -j 5
make install-
BOOST library. To install on Ubuntu, run:
apt-get install libboost-dev-
LibAO and libsndfile to play some beeping sounds. On Ubuntu, run:
apt-get install libao-dev libsndfile1-dev libx11-dev libopenblas-dev liblapack-dev libgflags-devBy now you should have completed the Linux installation guide for the OpenVINO™ toolkit, however before continuing, please ensure:
-
That after installing the OpenVINO™ toolkit you have run the supplied demo samples
-
If you have and intend to use a GPU: You have installed and tested the GPU drivers
-
If you have and intend to use a USB camera: You have connected and tested the USB camera
-
If you have and intend to use a Myriad: You have connected and tested the USB Intel® Movidius™ Neural Compute Stick
-
That your development platform is connected to a network and has Internet access. To download all the files for this tutorial, you will need to access GitHub on the Internet.
-
You have build and installed grpc > 1.46.2 for cpp
1. Clone the repository to DMS folder at desired location:
git clone https://github.com/intel-sandbox/thirdparty.incluit.OpenVino-Driver-Behaviour.git DMS2. The first step is to configure the build environment for the OpenCV toolkit by sourcing the "setupvars.sh" script.
source <--path to openvino dir-->/setupvars.sh
#source /opt/intel/openvino/bin/setupvars.shFor older versions than 2019 R1, OpenVINO was installed in a different dir, run this instead:
source /opt/intel/computer_vision_sdk/bin/setupvars.sh3. Change directory and clone dlib to third-party dir:
cd DMS
git clone https://github.com/davisking/dlib/ -b v19.17 third-party/dlib4. OpenVINO’s Release compatibility
bash scripts/download_models.shIn case of using the OpenVINO’s 2019 R1.0, before continuing, if you have not manually downloaded all the models before, it is necessary to download the following models.
cd /opt/intel/<openvino_path>/deployment_tools/tools/model_downloader/
sudo ./downloader.py --name face-detection-adas-0001
sudo ./downloader.py --name landmarks-regression-retail-0009
sudo ./downloader.py --name face-detection-retail-0004
sudo ./downloader.py --name head-pose-estimation-adas-00015. Create a directory to build the tutorial in and change to it.
mkdir build
cd build6. Before running each of the following sections, be sure to source the helper script. That will make it easier to use environment variables instead of long names to the models:
source ../scripts/setupenv.sh7. Compile:
cmake -DCMAKE_BUILD_TYPE=Release -DOpenCV_DIR="<--path to opencv install dir-->" ..
#cmake -DCMAKE_BUILD_TYPE=Release -DOpenCV_DIR="/home/wid/ov_cv/build_opencv" ../
make -j88. Move to the executable’s dir:
cd intel64/Release1. First, let us see how face detection works on a single image file using the default synchronous mode.
./driver_behavior -m $face132 -i ../../../data/img_1.jpg2. For video files:
./driver_behavior -m $face132 -i ../../../data/video1.mp43. For remote frames:
./driver_behavior -m $face132 -i remote4. You can also run the command in asynchronous mode using the option "-async":
./driver_behavior -m $face132 -i ../../../data/video1.mp4 -async4. You can also load the models into the GPU or MYRIAD:
Note: In order to run this section, the GPU and/or MYRIAD are required to be present and correctly configured.
./driver_behavior -m $face132 -d GPU -i ../../../data/video1.mp4./driver_behavior -m $face132 -d MYRIAD -i ../../../data/video1.mp4You can also experiment by using different face detection models, being the ones available up to now:
-
face-detection-adas-0001:
-
-m $face1{16,32}
-
-
face-detection-retail-0004:
-
-m $face2{16,32}
-
By default they will be loaded into the CPU, so remember to pass the corresponding argument:
-
-d {CPU, GPU, GPU.0, GPU.1, AUTO, MYRIAD}
In order to enable drowsiness and yawn detection, we add to the pipeline a face landmarks detection.
./driver_behavior -m $face232 -dlib_lm -i ../../../data/video2.mp4
To analize if the driver is paying attention to the road, we enable the head/pose model and work with that information:
./driver_behavior -m $face232 -m_hp $hp32 -i ../../../data/video3.mp4
Removing the '-i' flag, if the computer has a video camera enabled, the programs uses its feed to run the face detection models and the following calculations.
./driver_behavior -m $face232
./driver_behavior -m $face232 -dlib_lm
./driver_behavior -m $face232 -d GPU -dlib_lm -async
./driver_behavior -m $face232 -m_hp $hp32Passing parameter 'remote' to '-i' flag, if the system has some app or a container which host camera enabled app that can feed frame via grpc, the programs uses its feed to run the face detection models and the following calculations.
./driver_behavior -m $face232 -i remote
./driver_behavior -m $face232 -dlib_lm -i remote
./driver_behavior -m $face232 -d GPU -dlib_lm -async -i remote
./driver_behavior -m $face232 -m_hp $hp32 -i remoteWe integrated our program to the Intel® IoT DevCloud platform. This developer tool enabled us to run the inference proccess on different hardware targets. The following is the comparison graph where greater is better:
Driver Assistance has been optimized for having compatibility with OpenVINO’s releases 2018’s (R4, R5) and 2019’s (Lastest version tested 2019 R1.0.1). It is important for the user to be aware that some changes regarding detection models had been introduced between releases from 2018 and 2019. In first instance, 2019 releases do not have the detection model’s binaries included within the toolkit; the user will have to follow the instructions described in the Open Model Zoo link suggested at the “Foreword” section of this installation guide. Be aware that if you choose to download them in a different path than the default, our “scripts/setupenv.sh” will not fully work and you will have to add the path to the models yourself when running the program. In case of using the OpenVINO’s 2019 R1.0 or greater, before continuing, it is necessary to manually download all the models.
cd /opt/intel/<openvino_path>/deployment_tools/tools/model_downloader/
sudo ./downloader.py --name <detectionModelName>If you are using OpenVINO’s Release >= R2 and < R2020.1 you will need to execute the following script:
bash scripts/download_models.shAfter, the user will be able to initiate the building process and to start using Driver Assistance System
Firstly, in order to successfully execute the building process, please make sure that all the declared prerequisites –hardware and software- have been met. In particular, regarding software prerequisites, it is fundamental that the OpenVINO’s toolkit version had been downloaded by following the Intel’s intrucctions described in the following links:
-
Version 2019 R2 (latest) https://docs.openvinotoolkit.org/latest/_docs_install_guides_installing_openvino_linux.html
-
Version 2019 R1.1 https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_linux.html
-
Version 2019 R1.01 https://docs.openvinotoolkit.org/2019_R1.01/_docs_install_guides_installing_openvino_linux.html
-
Version 2019 R1.0 https://docs.openvinotoolkit.org/2019_R1/_docs_install_guides_installing_openvino_linux.html
-
Version 2018 R5.0 https://docs.openvinotoolkit.org/2018_R5/_docs_install_guides_installing_openvino_linux.html
Secondly, make sure that “BOOST” library has been downloaded. If not, execute the following commands:
apt-get install libboost-dev
apt-get install libboost-log-devIn third place, it is fundamental for the building process to configure de build environment for the OpenCV toolkit by executing the following command:
2019 R1.X source /opt/intel/openvino/bin/setupvars.sh
2018 R4-R5 source /opt/intel/computer_vision_sdk/bin/setupvars.shFinally, before executing the compilation process be sure to source the helper script. That will make it easier to use environment variables instead of long names to the models: source ../scripts/setupenv.sh
If you encounter a problem like this:
Building CXX object dlib_build/dlib/CMakeFiles/dlib.dir/external/libjpeg/jdhuff.cpp.o
/home/ieisw/OpenVino-Driver-Behaviour/third-party/dlib/dlib/external/libjpeg/jdhuff.cpp:23:32: error: unknown option after ‘#pragma GCC diagnostic’ kind [-Werror=pragmas]
#pragma GCC diagnostic ignored "-Wshift-negative-value"
^
cc1plus: all warnings being treated as errors
dlib_build/dlib/CMakeFiles/dlib.dir/build.make:1574: recipe for target 'dlib_build/dlib/CMakeFiles/dlib.dir/external/libjpeg/jdhuff.cpp.o' failed
make[2]: *** [dlib_build/dlib/CMakeFiles/dlib.dir/external/libjpeg/jdhuff.cpp.o] Error 1
CMakeFiles/Makefile2:193: recipe for target 'dlib_build/dlib/CMakeFiles/dlib.dir/all' failed
make[1]: *** [dlib_build/dlib/CMakeFiles/dlib.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2It’s DLib.
DLib has its own BLAS library which tries to compile if it can’t find any of them installed (openblas, intel mkl, libblas). When this happens, it needs to compile its own libjpeg and throws the error mentioned above. There are 2 ways to solve this:
Lightweight solution, install another libjpeg (on Ubuntu):
sudo apt-get install libjpeg8-dev OR sudo apt-get install libjpeg9-devRecommended solution, install a full BLAS library as it will boost the program’s performance a bit. We recommend installing Intel’s MKL as it works faster and takes advantage of your Intel’s hardware.
You could also install openblas:
sudo apt-get install libopenblas-devor libblas (untested):
sudo apt-get install libblas-devWith that, DLib shouldn’t compile the file that’s causing the trouble.