This project was created as part of the Machine Learning course at the University of Würzburg.
This project aims to control a slideshow and a tetris game with gestures. The gestures are recognized by a machine learning model that was trained on annotated video gestures. For generating train data, videos of gestures were recorded, annotated and preprocessed before being fed into the model.
As a machine learning library, my self-made Neural Network Framework is used. The framework is programmed in Python and numpy without any further Machine Learning libraries. The mathematics behind backpropagation taught in Machine Learning lecture is put into practice in this framework.
By training the library on the annotated video gestures, the model is able to recognize gestures in real-time. The model is then used to control a slideshow and a tetris game.
This is an optional step for setting up a virtual environment designated for the project.
git clone <link-to-repo>
python -m venv .venv
source .venv/bin/activatepip install -r requirementsTo run the slideshow, run the following command:
python src/gesture_control/slidehow/live_detection.pyThen go to your browser and open 127.0.0.1:8000.
To run tetris, run the following command:
python src/gesture_control/tetris/live_detection.pyThen go to your browser and open 127.0.0.1:8000.
The dataset was created by recording videos of gestures with a webcam. The videos were then annotated with the help of a annotation tool. The tool allows to annotate each gesture by specifying the interval of the gesture and labeling it. The tool then saves the timestamps of the gestures which can later be joined with the video frames to create the dataset.
The preprocessing of the video gestures is done with the help of OpenCV and Mediapipe which return three-dimensional coordinates of detected body landmarks. In training, the landmark coordinates are combined with gesture annotations and saved as a numpy array.
As contemplating only a single frame for gesture classification does not provide enough information about hand movement, a bundle of consecutive frames is considered.
Instead of using the raw landmark coordinates, the relative coordinates are used. The relative coordinates are calculated by subtracting the coordinates of the previous frame from the coordinates of the current frame. This way, the hand movement is captured instead of the absolute position of the hand which is not relevant for gesture classification.