Custom_layer_activation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Custom_layer_activation.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNQckiin8lSC3iAa1tRoHdF",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/armeh429/A-journey-for-learning-advanced-tensorflow/blob/main/Custom_layer_activation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0xPbAf_8_93u"
      },
      "source": [
        "###Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aRIR6bJJ6-SH"
      },
      "source": [
        "try:\n",
        "  %tensorflow_version 2.x\n",
        "except Exection:\n",
        "  pass\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers import Layer\n"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E_0PTOW_Ac--"
      },
      "source": [
        "###Adding an activation layer\n",
        "\n",
        "To use the built-in activations in Keras, we can specify an activation parameter in the __init__() method of our custom layer class. From there, we can initialize it by using the tf.keras.activations.get() method. This takes in a string identifier that corresponds to one of the available activations in Keras. Next, you can now pass in the forward computation to this activation in the call() method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MOUlcu3jALhT"
      },
      "source": [
        "class SimpleDense(Layer):\n",
        "\n",
        "    # add an activation parameter\n",
        "    def __init__(self, units=32, activation=None):\n",
        "        super(SimpleDense, self).__init__()\n",
        "        self.units = units\n",
        "        \n",
        "        # define the activation to get from the built-in activation layers in Keras\n",
        "        self.activation = tf.keras.activations.get(activation)\n",
        "\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        w_init = tf.random_normal_initializer()\n",
        "        self.w = tf.Variable(name=\"kernel\",\n",
        "            initial_value=w_init(shape=(input_shape[-1], self.units),\n",
        "                                 dtype='float32'),\n",
        "            trainable=True)\n",
        "        b_init = tf.zeros_initializer()\n",
        "        self.b = tf.Variable(name=\"bias\",\n",
        "            initial_value=b_init(shape=(self.units,), dtype='float32'),\n",
        "            trainable=True)\n",
        "        super().build(input_shape)\n",
        "\n",
        "\n",
        "    def call(self, inputs):\n",
        "        \n",
        "        # pass the computation to the activation layer\n",
        "        return self.activation(tf.matmul(inputs, self.w) + self.b)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kn7o5kG7CWZO"
      },
      "source": [
        "We can now pass in an activation parameter to our custom layer. The string identifier is mostly the same as the function name so 'relu' below will get tf.keras.activations.relu"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wBm310hWB9d9",
        "outputId": "bdf69604-5473-47b3-889c-ba037bdc009f"
      },
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "\n",
        "(x_train, y_train),(x_test, y_test) = mnist.load_data()\n",
        "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
        "\n",
        "model = tf.keras.models.Sequential([\n",
        "    tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
        "    SimpleDense(128, activation='relu'),\n",
        "    tf.keras.layers.Dropout(0.2),\n",
        "    tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(x_train, y_train, epochs=5)\n",
        "model.evaluate(x_test, y_test)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n",
            "11501568/11490434 [==============================] - 0s 0us/step\n",
            "Epoch 1/5\n",
            "1875/1875 [==============================] - 4s 2ms/step - loss: 0.2958 - accuracy: 0.9144\n",
            "Epoch 2/5\n",
            "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1415 - accuracy: 0.9582\n",
            "Epoch 3/5\n",
            "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1071 - accuracy: 0.9675\n",
            "Epoch 4/5\n",
            "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0871 - accuracy: 0.9739\n",
            "Epoch 5/5\n",
            "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0740 - accuracy: 0.9765\n",
            "313/313 [==============================] - 1s 1ms/step - loss: 0.0708 - accuracy: 0.9788\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.07083190232515335, 0.9787999987602234]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EgMfKsVKCbcq"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}