Repository for the Roboarm controller.
This is a controller for a Simulated Robotic Arm. With the help of a Leap Motion hands sensor, we train a Machine Learning model to identify the position of the fingers of a person, and we use that to indicate a movement to the Robotic Arm. The main focus of this exercise is to learn the different technologies used for it: Gazebo, Kafka, Tensorflow, and Java.
Here you can find a couple of youtube videos to show it working:
Controlling
Training and controlling steps
Disclaimer: The arm seems to move slowly in the video, because of issues on the recording of the video itself.
At the architecture folder you can find more info about how the components are connected to each other.
- gazebo http://gazebosim.org
- leap motion sdk (for the hands sensor)
- maven
- java sdk
- cmake
- plantuml (optional, architecture diagrams)
- conda (see readme of the training module)
sudo leapd
cd .Leap\ Motion/
Visualizer
LeapControlPanel
LeapControlPanel --showsettings
https://developer-archive.leapmotion.com/documentation/java/supplements/Leap_JSON.html
Each of the following commands on a different terminal
1- Start the leap motion framework
sudo leapd
2- Start the simulation Note: Do the cd and then start the sh script. There is an open issue regarding it.
cd RoboArmSimulation
./startGazeboSim.sh
3- Start the class roboarmcontroller.Application (in IntelliJ) with the spring profile "controllingML"
1- Start the leap motion framework
sudo leapd
2- Start the class roboarmcontroller.Application (in IntelliJ) with the spring profile "training"
3- Start the training module
cd RoboArmController/src
conda activate roboarmcontroller
jupyter notebook
4- Copy the timestamp printed out on the roboarmcontroller logs
5- Execute the jupyter notebook training/RoboArmControllerTraining.ipynb
6- Copy the path of the exported model to the property training.input.exported.model.path
Each of the following commands on a different terminal
1- Start the leap motion framework
sudo leapd
2- Start the simulation Note: Do the cd and then start the sh script. There is an open issue regarding it.
cd RoboArmSimulation
./startGazeboSim.sh
3- Start the class roboarmcontroller.Application (in IntelliJ) with the spring profile "controllingSimple"
Note: There is a known issue making the simulation loading to halt unless data is provided, so it is important to execute the steps in the order specified here, and input data by placing the hands over the sensor for the simulation to actually load.
Optional to see the hand visualizer
cd ~/.Leap\ Motion/
Visualizer