050_introduction_to_convnets.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    },
    "colab": {
      "name": "5.1-introduction-to-convnets.ipynb",
      "provenance": []
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g6kfSHPn5lKs"
      },
      "source": [
        "# Introduction to Convnets\n",
        "\n",
        "**Author:** [FarrokhKarimi](https://github.com/farrokhkarimi)  \n",
        "**Note:** For this notebook the best runtime type is GPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bPL366WV45UT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "6f53b42d-656d-4242-9360-6f128f425cbe"
      },
      "source": [
        "import keras\n",
        "keras.__version__"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'2.8.0'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "tU1-GU3y45Ub"
      },
      "source": [
        "# 5.1 - Introduction to convnets\n",
        "\n",
        "First, let's take a practical look at a very simple convnet example. We will use our convnet to classify MNIST digits, a task that you've already been \n",
        "through in section 1, using a densely-connected network (our test accuracy then was 97.8%). Even though our convnet will be very basic, its \n",
        "accuracy will still blow out of the water that of the densely-connected model from section 1.\n",
        "\n",
        "The 6 lines of code below show you what a basic convnet looks like. It's a stack of `Conv2D` and `MaxPooling2D` layers. We'll see in a \n",
        "minute what they do concretely.\n",
        "Importantly, a convnet takes as input tensors of shape `(image_height, image_width, image_channels)` (not including the batch dimension). \n",
        "In our case, we will configure our convnet to process inputs of size `(28, 28, 1)`, which is the format of MNIST images. We do this via \n",
        "passing the argument `input_shape=(28, 28, 1)` to our first layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sMzMGrye45Uc"
      },
      "source": [
        "from keras import layers\n",
        "from keras import models\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
        "model.add(layers.MaxPooling2D((2, 2)))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(layers.MaxPooling2D((2, 2)))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eNhkluep45Uh"
      },
      "source": [
        "Let's display the architecture of our convnet so far:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DOkms4kS45Ui",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c979416-f0ed-4282-d122-4fb9b11b07cf"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d (Conv2D)             (None, 26, 26, 32)        320       \n",
            "                                                                 \n",
            " max_pooling2d (MaxPooling2D  (None, 13, 13, 32)       0         \n",
            " )                                                               \n",
            "                                                                 \n",
            " conv2d_1 (Conv2D)           (None, 11, 11, 64)        18496     \n",
            "                                                                 \n",
            " max_pooling2d_1 (MaxPooling  (None, 5, 5, 64)         0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_2 (Conv2D)           (None, 3, 3, 64)          36928     \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 55,744\n",
            "Trainable params: 55,744\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "nPwol7_Y45Un"
      },
      "source": [
        "You can see above that the output of every `Conv2D` and `MaxPooling2D` layer is a 3D tensor of shape `(height, width, channels)`. The width \n",
        "and height dimensions tend to shrink as we go deeper in the network. The number of channels is controlled by the first argument passed to \n",
        "the `Conv2D` layers (e.g. 32 or 64).\n",
        "\n",
        "The next step would be to feed our last output tensor (of shape `(3, 3, 64)`) into a densely-connected classifier network like those you are \n",
        "already familiar with: a stack of `Dense` layers. These classifiers process vectors, which are 1D, whereas our current output is a 3D tensor. \n",
        "So first, we will have to flatten our 3D outputs to 1D, and then add a few `Dense` layers on top:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FCvMc8Us45Uo"
      },
      "source": [
        "model.add(layers.Flatten())\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(10, activation='softmax'))"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bl5HpHvj45Ut"
      },
      "source": [
        "We are going to do 10-way classification, so we use a final layer with 10 outputs and a softmax activation. Now here's what our network \n",
        "looks like:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SMI01uk945Uu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c0b868b0-01d5-4e81-8087-81e86046f448"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d (Conv2D)             (None, 26, 26, 32)        320       \n",
            "                                                                 \n",
            " max_pooling2d (MaxPooling2D  (None, 13, 13, 32)       0         \n",
            " )                                                               \n",
            "                                                                 \n",
            " conv2d_1 (Conv2D)           (None, 11, 11, 64)        18496     \n",
            "                                                                 \n",
            " max_pooling2d_1 (MaxPooling  (None, 5, 5, 64)         0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_2 (Conv2D)           (None, 3, 3, 64)          36928     \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 576)               0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 64)                36928     \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 10)                650       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 93,322\n",
            "Trainable params: 93,322\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2FF4UV4U45Uz"
      },
      "source": [
        "As you can see, our `(3, 3, 64)` outputs were flattened into vectors of shape `(576,)`, before going through two `Dense` layers.\n",
        "\n",
        "Now, let's train our convnet on the MNIST digits. We will reuse a lot of the code we have already covered in the MNIST example from Section 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5ns_-RV945U0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1434278e-512c-40b4-b96b-23c8a2f8b77c"
      },
      "source": [
        "from keras.datasets import mnist\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
        "\n",
        "train_images = train_images.reshape((60000, 28, 28, 1))\n",
        "train_images = train_images.astype('float32') / 255\n",
        "\n",
        "test_images = test_images.reshape((10000, 28, 28, 1))\n",
        "test_images = test_images.astype('float32') / 255\n",
        "\n",
        "train_labels = to_categorical(train_labels)\n",
        "test_labels = to_categorical(test_labels)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n",
            "11501568/11490434 [==============================] - 0s 0us/step\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ehmRFUng45U5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "30bea5b0-47d6-4c4c-e0c8-93f1de5ff1e6"
      },
      "source": [
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(train_images, train_labels, epochs=5, batch_size=64)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/5\n",
            "938/938 [==============================] - 17s 5ms/step - loss: 0.1730 - accuracy: 0.9463\n",
            "Epoch 2/5\n",
            "938/938 [==============================] - 4s 5ms/step - loss: 0.0475 - accuracy: 0.9847\n",
            "Epoch 3/5\n",
            "938/938 [==============================] - 4s 5ms/step - loss: 0.0336 - accuracy: 0.9898\n",
            "Epoch 4/5\n",
            "938/938 [==============================] - 4s 5ms/step - loss: 0.0261 - accuracy: 0.9919\n",
            "Epoch 5/5\n",
            "938/938 [==============================] - 4s 5ms/step - loss: 0.0196 - accuracy: 0.9943\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f1986178d50>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n0U60Njb45U-"
      },
      "source": [
        "Let's evaluate the model on the test data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IexJjrnB45U_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "896a2953-f607-4159-d160-4dd037a97497"
      },
      "source": [
        "test_loss, test_acc = model.evaluate(test_images, test_labels)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "313/313 [==============================] - 1s 3ms/step - loss: 0.0286 - accuracy: 0.9919\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DHCodlp45VE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "11f1835c-869f-4e19-db7a-54f8240ce08f"
      },
      "source": [
        "test_acc"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9919000267982483"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fSW_axDa45VI"
      },
      "source": [
        "While our densely-connected network from Section 1 had a test accuracy of 97.8%, our basic convnet has a test accuracy of 99.3%: we \n",
        "decreased our error rate by 68% (relative). Not bad! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iw6oXXDF6ec9"
      },
      "source": [
        "**Contact** : [FarrokhKarimi](https://zil.ink/farrokhkarimi)  \n",
        "**Reference** : [tensorflow.com](https://www.tensorflow.org/)  \n",
        "Special thanks to [François Chollet](https://fchollet.com/)"
      ]
    }
  ]
}