Hardware-FasterRCNN

This is the implementation for a hardware Faster-RCNN neural network architecture used in object detection in autonomous cars we will divide this project into 2 parts The first part is the software implementation of our Faster-RCNN neural network The second part of the code is the hardware implementation of the backbone network of the Faster-RCNN (VGG-16) to fasten the simulation The hardware implementation will be only used in inference mode and we will use the software implementation to get the correct weights we will use in the hardware part

Software implementation

Introduction

• We will be simulating the Faster-RCNN architecture using google colabs as this will fasten the training using google servers
• we will use a unique custom dataset which is a subset of the VOCtrainval_11-May-2012 that contains objects that is present on the road normally
• This data set is found in the dataset folder where it is divided into 2 folders one which contains the images and the annotation files(XML format) and the other folder contains the tfrecord format that will be used in training ( explained later )

About Roboflow for Data Management

Roboflow makes managing, preprocessing, augmenting, and versioning datasets for computer vision seamless. Developers reduce 50% of their code when using Roboflow's workflow, automate annotation quality assurance, save training time, and increase model reproducibility.

This website is used to change from one format to another to the annotation files and this will be used to generate the formats we will need in training in case you need to train on your custom dataset ( note the dataset attached in the dataset folder already has the correct format ) for the full tutorial on how to work with roboflow with your custom dataset please check the following link Roboflow

Folder structure

The software folder contains 2 items and 2 folders:

1. Models contains the actual Faster-RCNN model being implemented
2. tensorflow-object-detection-faster-rcnn contains some .ipynb files that we can use to train the dataset on google colab
3. Faster-RCNN.ipynb this is the modified file that we will use for inference or training the Faster-RCNN model
4. frozen_inference_graph.pb this is the file that will be outputed from the Faster-RCNN.ipynb after training the model using the dataset in the folder, This file will be used in inference mode

Inference mode

1. Open Faster-RCNN.ipynb
2. Run this code snippet

3. Run this code snippet

• Number of evalutation steps isn't important when running the inference mode
• If you run this code snippet and doing the training mode i suggest doing a small number of steps as a start ie. num_steps = 100 for example
• The extracted frozen_graph.pb attached was done on 1000 steps because the number of the attached images isn't big enough so we don't run on the problem of overfitting
• Make sure that the selected_model = 'faster_rcnn_inception_v2' to choose the Faster-RCNN model

4. Run this code snippet

5. This should be your folder after the last step

6. Run this code snippet

7. This should be your folder after the last step ( Model folder should be added)

8. Run this code snippet (may remove it )

9. Create 2 directories named Test and Train as shown in the following screenshot

10. check the Dataset\Dataset1\VOC-test.v2.tfrecord folder where you can see the test and train directories

11. Add the content in the directories to the directories created in step 9 after this step it should look something like this

12. we will work with the following code snippet in this step

• In the first variable we should copy the path of the tfrecord file in the test directory we created in step 9
• In the second variable we should copy the path of the tfrecord file in the train directory we created in step 9
• In the third variable we should copy the path of the pbtxt file in the test directory we created in step 9
• Note: in order to copy the path simply right click on the file you want and then paste in the code snippet
• After the following steps this should be the shape of the code snippet

13. Run the code snippet

14. Run this code snippet

15. Run these code snippets

16. Create a new sub directory to the data directory named test after this step the folders should look like this

17. Go to Dataset/Dataset2/subest_udacity_dataset where you can see the images we will use for testing these images are a subset of the udacity dataset

18. upload the images you want to test in the inference mode to the test directory created in step 16 from the directory stated in step 17 your folders may look like this after uploading

19. Upload the frozen_inference_graph.pb file found in software directory to the tensorflow-object-detection-faster-rcnn folder in google colabs your folders should look something like this( note the added pb file)

20. Now check this code snippet

• Delete the frozen_inference_graph.pb in the following directory /content/models/research/pretrained_model/frozen_inference_graph.pb
• Upload the new frozen_inference_graph.pb found in the software directory in github

21. Run the code snippet

22. add the following 2 lines at the start of this code snippet then run it !pip install tf-slim num_classes = get_num_classes(label_map_pbtxt_fname)

23.Run the remaining code snippets and enjoy the results!

Results