Antipodal Robotic Grasping

We present a novel generative residual convolutional neural network based model architecture which detects objects in the camera’s field of view and predicts a suitable antipodal grasp configuration for the objects in the image.

This repository contains the implementation of the Generative Residual Convolutional Neural Network (GR-ConvNet) from the paper:

Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network

Sulabh Kumra, Shirin Joshi, Ferat Sahin

arxiv | video

If you use this project in your research or wish to refer to the baseline results published in the paper, please use the following BibTeX entry:

@inproceedings{kumra2020antipodal,
  author={Kumra, Sulabh and Joshi, Shirin and Sahin, Ferat},
  booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  title={Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network}, 
  year={2020},
  pages={9626-9633},
  doi={10.1109/IROS45743.2020.9340777}}
}

Requirements

• numpy
• opencv-python
• matplotlib
• scikit-image
• imageio
• torch
• torchvision
• torchsummary
• tensorboardX
• pyrealsense2
• Pillow

Installation

• Checkout the robotic grasping package

$ git clone https://github.com/skumra/robotic-grasping.git

• Create a virtual environment

$ python3.6 -m venv --system-site-packages venv

• Source the virtual environment

$ source venv/bin/activate

• Install the requirements

$ cd robotic-grasping
$ pip install -r requirements.txt

Datasets

This repository supports both the Cornell Grasping Dataset and Jacquard Dataset.

Cornell Grasping Dataset

1. Download the and extract Cornell Grasping Dataset.
2. Convert the PCD files to depth images by running python -m utils.dataset_processing.generate_cornell_depth <Path To Dataset>

Jacquard Dataset

1. Download and extract the Jacquard Dataset.

Model Training

A model can be trained using the train_network.py script. Run train_network.py --help to see a full list of options.

Example for Cornell dataset:

python train_network.py --dataset cornell --dataset-path <Path To Dataset> --description training_cornell

Example for Jacquard dataset:

python train_network.py --dataset jacquard --dataset-path <Path To Dataset> --description training_jacquard --use-dropout 0 --input-size 300

Model Evaluation

The trained network can be evaluated using the evaluate.py script. Run evaluate.py --help for a full set of options.

Example for Cornell dataset:

Weights training on Cornell https://pan.baidu.com/s/1bie7XJF_c6mWfktwZgExcQ eh9y (train on 2080ti)

python evaluate.py --network <Path to Trained Network> --dataset cornell --dataset-path <Path to Dataset> --iou-eval

Example for Jacquard dataset:

python evaluate.py --network <Path to Trained Network> --dataset jacquard --dataset-path <Path to Dataset> --iou-eval --use-dropout 0 --input-size 300

Run Tasks

A task can be executed using the relevant run script. All task scripts are named as run_<task name>.py. For example, to run the grasp generator run:

python run_grasp_generator.py

Realtime test with Realsense D435i, distance about 1m.

python run_realtime.py

Realtime test with Realsense D405, distance about 1m.

TODO

• 调整 infer的crop 现在是直接中心裁剪224 对于1280x720 损失太多 导致 object 占据画面比重太大
• 接入 yolo 来辅助 crop

Run on a Robot

To run the grasp generator with a robot, please use our ROS implementation for Baxter robot. It is available at: https://github.com/skumra/baxter-pnp