Add GPU training support via CuPy, Jupyter notebooks, ROC plots and 4…

… more * Change NumPy to CuPy so that GPU training is only possible * Add notebooks for local Jupyter and Google Colab (there are differencies i.e. in plotting). Google Colab training with GPU is possible (and recommended!) * Add plotting ROC curves for every class and misclassified examples after training process * Update requirements * Update command line parameters * Update logger setup * Add plots generated after best models training
VictorAtPL · Mar 26, 2019 · 9d46d45 · 9d46d45
1 parent 316d86f
commit 9d46d45
Show file tree

Hide file tree

Showing 34 changed files with 532 additions and 75 deletions.
diff --git a/Final_Report_PL.pdf b/Final_Report_PL.pdf
diff --git a/README.md b/README.md
@@ -1,16 +1,17 @@
 Fully Connected Deep Network for MNIST dataset 
 ==
 * project implemented as part of the *Deep Learning Methods* course attended on MSc degree of *Data Science* at the *Faculty of Mathematics and Information Sciences* in *Warsaw University of Technology*,
-* written purely in NumPy library.
+* written purely in NumPy (`numpy` branch) and CuPy (`master` branch) libraries.
 
 Prerequisites
 ---
 * Python 3.6
 * Python's `pip`
 * `wget`, `gunzip`
-* `python3-tkinter` (if you want to plot metrics) 
+* `python3-tkinter` (if you want to plot metrics)
+* `cuda` (if version with `cupy` will be used)
 
-How to run
+How to run via CLI
 ---
 1. Download datasets to `data/` directory:
     ```bash
@@ -27,32 +28,42 @@ How to run
 
 3. Install python requirements:
     ```bash
-    pip3 install -r requirements.txt
+    pip3 install -r requirements.txt 
+    ```
+
+4. Install system requirements:
+    ```bash
+    sudo apt install python3-tk
     ```
 
-4. Start training:
+5. Start training:
     ```bash
-    python3 train.py
+    python3 run.py
     ```
 
+How to run via Google Colab
+---
+1. Open `google_colab.ipynb` in Google Colab by clicking [this](http://colab.research.google.com/github/VictorAtPL/MNIST_FCDN_NumPy/blob/master/google_colab.ipynb) link,
+2. run all cells.
+
 Results
 ---
-Results obtained so far with *Weight decay* and *Adam optimizer*:
+Results obtained so far with *Weight decay*, *Adam optimizer* and *Dropout*:
 ```
-2019-03-18 02:21:44,833 - MNIST_FCDN_NumPy - INFO - epoch 43 - training loss 0.000000 - training accuracy 0.995783
-2019-03-18 02:21:46,699 - MNIST_FCDN_NumPy - INFO - epoch 43 - validation loss 0.005396 - validation accuracy 0.978100
+2019-03-26 00:18:41,382 - MNIST_FCDN_NumPy - INFO - epoch 14 - training loss 0.381925 - training accuracy 0.977683
+2019-03-26 00:18:42,486 - MNIST_FCDN_NumPy - INFO - epoch 14 - validation loss 11.725633 - validation accuracy 0.978400
+2019-03-26 00:18:42,488 - MNIST_FCDN_NumPy - INFO - epoch 14 - time taken 1.79
 ```
 
-Sample plot drawn by this application looks like this:
-![Sample plot](sample_plot.png)
+Sample plots drawn by this application looks like this:
+![Sample plot](experiments_images/best_model.png)
 
+![Sample plot 2](experiments_images/best_model_misclassified.png)
 
-Remarks
----
-* as stated on [Yann Lecun's MNIST page](http://yann.lecun.com/exdb/mnist/), the best result for _Neural Nets_ is *0.35* error rate (in %). So far, the model presented in this repository achieves *2.2* error rate. Possibly, increasing model capacity 
-with enabled _dropout_ mechanism might result in decreasing error rate,
-* `train.py` script accepts a lot of arguments. Read more by executing `train.py -h` in console.
+![Sample plot 3](experiments_images/best_model_rocs.png)
 
-TODO
+
+Remarks
 ---
-* implement *Dropout* mechanism
+* as stated on [Yann Lecun's MNIST page](http://yann.lecun.com/exdb/mnist/), the best result for _Neural Nets_ is *0.35* error rate (in %). The model presented in this repository achieves *1.5* error rate. Possibly, adding data augmentation and/or using conv nets might result in further decreasing error rate,
+* `run.py` script accepts a lot of arguments. Read more by executing `train.py -h` in console.
diff --git a/dataset.py b/dataset.py
@@ -2,7 +2,7 @@
 import struct
 from typing import Generator, List, Dict, Tuple
 
-import numpy as np
+import cupy as np
 
 from dataset_type import DatasetType
 from utils import get_logger

diff --git a/experiments_images/best_model.png b/experiments_images/best_model.png
diff --git a/experiments_images/best_model_colab.png b/experiments_images/best_model_colab.png
diff --git a/experiments_images/best_model_misclassified.png b/experiments_images/best_model_misclassified.png
diff --git a/experiments_images/best_model_misclassified_colab.png b/experiments_images/best_model_misclassified_colab.png
diff --git a/experiments_images/best_model_rocs.png b/experiments_images/best_model_rocs.png
diff --git a/experiments_images/exp1_1.png b/experiments_images/exp1_1.png
diff --git a/experiments_images/exp1_2.png b/experiments_images/exp1_2.png
diff --git a/experiments_images/exp2_2.png b/experiments_images/exp2_2.png
diff --git a/experiments_images/exp3_1.png b/experiments_images/exp3_1.png
diff --git a/experiments_images/exp3_2.png b/experiments_images/exp3_2.png
diff --git a/experiments_images/exp4_1.png b/experiments_images/exp4_1.png
diff --git a/google_colab.ipynb b/google_colab.ipynb
@@ -0,0 +1,183 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Thanks to this notebook you can run training of the model on `Google Colab` - it means that you can train model without downloading files to your local computer and without having GPU!\n",
+    "\n",
+    "After the training, plots with losses, accuracies, missclassified examples and ROC are plotted."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 136
+    },
+    "colab_type": "code",
+    "id": "uTVcwQTIsYa0",
+    "outputId": "45e4a32d-82f9-4c19-bab3-59cf1c9f9c95"
+   },
+   "outputs": [],
+   "source": [
+    "!git clone https://github.com/VictorAtPL/MNIST_FCDN_NumPy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 34
+    },
+    "colab_type": "code",
+    "id": "7fYVH7b5sgTJ",
+    "outputId": "7d8e79af-f9d2-4d30-ec87-5afabeac18c8"
+   },
+   "outputs": [],
+   "source": [
+    "%cd MNIST_FCDN_NumPy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 85
+    },
+    "colab_type": "code",
+    "id": "g-LevFszsiIf",
+    "outputId": "ca044407-1df4-40a3-ee09-6d664be4afb2"
+   },
+   "outputs": [],
+   "source": [
+    "!sed -i 's/80/100/g' requirements.txt\n",
+    "!pip install -r requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 799
+    },
+    "colab_type": "code",
+    "id": "d-s47af_tA_l",
+    "outputId": "106bd3fa-b9c9-4f87-c788-864dee4ef0f5"
+   },
+   "outputs": [],
+   "source": [
+    "%cd data/\n",
+    "!wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
+    "!wget http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
+    "!wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
+    "!wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
+    "!gunzip -f *\n",
+    "%cd .."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {},
+    "colab_type": "code",
+    "id": "mwlbdk80wvph"
+   },
+   "outputs": [],
+   "source": [
+    "!export PYTHONPATH=`pwd`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 316
+    },
+    "colab_type": "code",
+    "id": "tsSDqx4XskDp",
+    "outputId": "7bfb2d6c-d81f-4324-c686-7590d5743b8f",
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "\n",
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "from utils import setup_logger\n",
+    "from run import train\n",
+    "\n",
+    "plt.style.use('seaborn-pastel')\n",
+    "\n",
+    "setup_logger()\n",
+    "\n",
+    "# Run following line if you don't want information from logger being printed\n",
+    "#setup_logger(logging.NOTSET)\n",
+    "\n",
+    "# Run following line if you want the training process to be most verbose\n",
+    "#setup_logger(logging.DEBUG)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "args = {\n",
+    "    \"data_dir\": \"data/\",\n",
+    "    \"batch_size\": 1024,\n",
+    "    \"neurons_in_hidden_blocks\": [512, 512],\n",
+    "    \"weight_decay_lambda\": 0.2,\n",
+    "    \"learning_rate\": 0.001,\n",
+    "    \"epochs\": 15,\n",
+    "    \"dropout_keep_prob\": 0.6,\n",
+    "    \"train_with_validation\": True,\n",
+    "    \"plot_graphs\": True\n",
+    "}\n",
+    "\n",
+    "train(args)"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "name": "Google_Colab.ipynb",
+   "provenance": [],
+   "version": "0.3.2"
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/jupyter.ipynb b/jupyter.ipynb
@@ -0,0 +1,109 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Thanks to this notebook you can run training of the model quite interactively - it means that you can see plots updated live with every step of training.\n",
+    "\n",
+    "After the training, plot with missclassified examples and ROC are also plotted."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 316
+    },
+    "colab_type": "code",
+    "id": "tsSDqx4XskDp",
+    "outputId": "7bfb2d6c-d81f-4324-c686-7590d5743b8f",
+    "scrolled": false
+   },
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "\n",
+    "%matplotlib notebook\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "from utils import setup_logger\n",
+    "from run import train\n",
+    "\n",
+    "plt.style.use('seaborn-pastel')\n",
+    "\n",
+    "setup_logger()\n",
+    "\n",
+    "# Run following line if you don't want information from logger being printed\n",
+    "#setup_logger(logging.NOTSET)\n",
+    "\n",
+    "# Run following line if you want the training process to be most verbose\n",
+    "#setup_logger(logging.DEBUG)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [],
+   "source": [
+    "args = {\n",
+    "    \"data_dir\": \"data/\",\n",
+    "    \"batch_size\": 1024,\n",
+    "    \"neurons_in_hidden_blocks\": [512, 512],\n",
+    "    \"weight_decay_lambda\": 0.2,\n",
+    "    \"learning_rate\": 0.001,\n",
+    "    \"epochs\": 15,\n",
+    "    \"dropout_keep_prob\": 0.6,\n",
+    "    \"train_with_validation\": True,\n",
+    "    \"plot_graphs\": True\n",
+    "}\n",
+    "\n",
+    "train(args)"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "name": "Google_Colab.ipynb",
+   "provenance": [],
+   "version": "0.3.2"
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}