01_building_neural_nets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "## Building and Training Neural Networks\n",
    "\n",
    "Before we dig into how to build neural networks from scratch, let's load a few popular libraries that we'll need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from mpl_toolkits import mplot3d\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "We'll also need our custom `draw_nn_graph` function for visualizing these networks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from visualization import draw_nn_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "## Constructing a Simple Neural Network\n",
    "\n",
    "Let's take another look at the simple linear-regression-based neural network that we examined in the previous lesson:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"247pt\" height=\"111pt\"\n",
       " viewBox=\"0.00 0.00 246.83 111.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 107.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-107.2376 242.8304,-107.2376 242.8304,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-51.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-47.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"187.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.2116\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"187.2116\" y=\"-40.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 5.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.8306,-51.6188C94.5633,-51.6188 110.2316,-51.6188 125.2435,-51.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"125.4996,-55.1189 135.4996,-51.6188 125.4996,-48.1189 125.4996,-55.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"108.0928\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">3.0</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad2d5aa90>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load simple model for comparison\n",
    "nn_linear_regression = tf.keras.models.load_model('models/linear-regression.h5')\n",
    "draw_nn_graph(nn_linear_regression)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "To build a simple model like this, we'll use the high-level [Keras](https://www.tensorflow.org/guide/keras) API provided by tensorflow, which we import as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "We can create the skeleton of a Keras neural network using the following code: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build an empty model\n",
    "model = keras.Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Now, we'll add the output neuron pictured in the diagram above (note that it isn't hooked up to our network yet): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build output layer\n",
    "output_layer = keras.layers.Dense(\n",
    "    name=\"output\",\n",
    "    units=1, \n",
    "    input_shape=(1,)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "We can hook this output neuron up to our network using the `add` method of our `model`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hook up output layer\n",
    "model.add(output_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Let's visualize the result of these operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"247pt\" height=\"111pt\"\n",
       " viewBox=\"0.00 0.00 246.83 111.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 107.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-107.2376 242.8304,-107.2376 242.8304,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-51.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-47.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"187.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.2116\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"187.2116\" y=\"-40.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.8306,-51.6188C94.5633,-51.6188 110.2316,-51.6188 125.2435,-51.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"125.4996,-55.1189 135.4996,-51.6188 125.4996,-48.1189 125.4996,-55.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"108.0928\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">1.0</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad04ef208>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# visualize model\n",
    "draw_nn_graph(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Notice that the weight and bias values shown above do not match the respective values from our original diagram. Since we did not tell it to do otherwise, Keras has initialized the biases in our model to zero and the weights to randomly chosen values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Let's now repeat the steps above, but also take care to set the weight and bias values of our model to the correct values. First, we'll start with a clean `Sequential` model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start with a clean model\n",
    "model_correct = keras.Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "This time, we'll pass the `kernel_initializer` and `bias_initializer` arguments when we create our output neuron in order to properly set the weights and biases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build the layer with the right parameter values\n",
    "output_correct = keras.layers.Dense(\n",
    "    name=\"output\",\n",
    "    units=1,\n",
    "    input_shape=(1,),\n",
    "    kernel_initializer=keras.initializers.constant(3.0),\n",
    "    bias_initializer=keras.initializers.constant(5.0)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Now, let's hook this new output neuron up to our new network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the correct output layer to our model\n",
    "model_correct.add(output_correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "When we visualize our new network, we can see that the weight and bias values are correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"247pt\" height=\"111pt\"\n",
       " viewBox=\"0.00 0.00 246.83 111.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 107.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-107.2376 242.8304,-107.2376 242.8304,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-51.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-47.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"187.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.2116\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"187.2116\" y=\"-40.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 5.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.8306,-51.6188C94.5633,-51.6188 110.2316,-51.6188 125.2435,-51.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"125.4996,-55.1189 135.4996,-51.6188 125.4996,-48.1189 125.4996,-55.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"108.0928\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">3.0</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad04f81d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now can see that values are correct\n",
    "draw_nn_graph(model_correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "## Building a Multi-Layer Neural Network\n",
    "\n",
    "There is no limit to how many layers we can add to our neural network. Let's create an example network containing one hidden layer and one output layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"429pt\" height=\"353pt\"\n",
       " viewBox=\"0.00 0.00 429.07 353.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 349.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-349.2376 425.068,-349.2376 425.068,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-242.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-238.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- hidden_0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>hidden_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_0</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-161.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.444,-235.9599C96.1946,-232.5248 114.1573,-227.5599 129.5928,-220.6188 136.3127,-217.597 143.0734,-213.9152 149.6008,-209.9483\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"151.7704,-212.7164 158.3313,-204.3974 148.0147,-206.8093 151.7704,-212.7164\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-234.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.92</text>\n",
       "</g>\n",
       "<!-- hidden_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>hidden_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-293.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-297.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_1</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-282.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M78.8541,-255.2285C85.4371,-257.3691 92.2093,-259.5635 98.5928,-261.6188 112.0017,-265.9361 126.4271,-270.5449 140.0506,-274.8813\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"139.2124,-278.2874 149.8028,-277.9826 141.3338,-271.6166 139.2124,-278.2874\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-274.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.13</text>\n",
       "</g>\n",
       "<!-- hidden_2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>hidden_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_2</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-40.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M61.5296,-208.0029C71.6101,-193.2606 84.5267,-176.5261 98.5928,-163.6188 110.4994,-152.6932 118.4584,-156.3303 129.5928,-144.6188 141.1042,-132.5109 138.0921,-125.361 147.5928,-111.6188 150.7674,-107.027 154.2343,-102.3816 157.8208,-97.8152\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"160.8072,-99.6881 164.3685,-89.7097 155.3619,-95.2894 160.8072,-99.6881\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-167.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.14</text>\n",
       "</g>\n",
       "<!-- input_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>input_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-128.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-124.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_1</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.5303,-124.5198C96.1264,-124.1019 113.958,-125.0849 129.5928,-129.6188 136.0098,-131.4796 142.4188,-134.1081 148.606,-137.1593\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"147.0163,-140.278 157.4875,-141.8919 150.3082,-134.1003 147.0163,-140.278\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-133.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.45</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M68.9631,-157.2707C78.1034,-165.8226 88.4774,-174.9528 98.5928,-182.6188 111.4719,-192.3792 118.3823,-189.9801 129.5928,-201.6188 140.9131,-213.3713 138.2259,-220.2572 147.5928,-233.6188 150.748,-238.1195 154.1773,-242.6785 157.7182,-247.1678\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"155.1644,-249.5758 164.1771,-255.1456 160.6049,-245.1711 155.1644,-249.5758\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-205.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.88</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M76.7548,-110.9534C96.4793,-101.3962 121.2626,-89.3878 143.365,-78.6785\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"145.1777,-81.6894 152.6508,-74.1792 142.1253,-75.3899 145.1777,-81.6894\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-102.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.04</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"369.4492\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"369.4492\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"369.4492\" y=\"-161.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- hidden_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>hidden_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M251.2136,-172.6188C269.0341,-172.6188 289.149,-172.6188 307.667,-172.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"307.7658,-176.1189 317.7658,-172.6188 307.7657,-169.1189 307.7658,-176.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.26</text>\n",
       "</g>\n",
       "<!-- hidden_1&#45;&gt;output_0 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>hidden_1&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M241.2929,-263.7087C264.8236,-246.9837 294.3091,-226.0262 319.0638,-208.4313\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"321.1389,-211.2504 327.2621,-202.6042 317.0835,-205.5448 321.1389,-211.2504\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-247.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.97</text>\n",
       "</g>\n",
       "<!-- hidden_2&#45;&gt;output_0 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>hidden_2&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M241.2929,-81.5289C264.8236,-98.2539 294.3091,-119.2114 319.0638,-136.8063\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"317.0835,-139.6928 327.2621,-142.6334 321.1389,-133.9872 317.0835,-139.6928\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-126.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.04</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad04f1f28>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import this way to make code more readable\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "# can add as many layers as we wish\n",
    "model_3_layers = keras.Sequential()\n",
    "model_3_layers_hidden = layers.Dense(\n",
    "    name=\"hidden\",\n",
    "    units=3,\n",
    "    input_shape=(2,)\n",
    ")\n",
    "model_3_layers_output = layers.Dense(\n",
    "    name=\"output\",\n",
    "    units=1\n",
    ")\n",
    "model_3_layers.add(model_3_layers_hidden)\n",
    "model_3_layers.add(model_3_layers_output)\n",
    "draw_nn_graph(model_3_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "In order to shorten the code above, we can instead pass our layers as arguments to the `Sequential` function, which allows us to trim away the calls to `add`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"429pt\" height=\"353pt\"\n",
       " viewBox=\"0.00 0.00 429.07 353.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 349.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-349.2376 425.068,-349.2376 425.068,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-242.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-238.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- hidden_0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>hidden_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_0</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-161.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.444,-235.9599C96.1946,-232.5248 114.1573,-227.5599 129.5928,-220.6188 136.3127,-217.597 143.0734,-213.9152 149.6008,-209.9483\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"151.7704,-212.7164 158.3313,-204.3974 148.0147,-206.8093 151.7704,-212.7164\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-234.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.92</text>\n",
       "</g>\n",
       "<!-- hidden_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>hidden_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-293.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-297.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_1</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-282.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M78.8541,-255.2285C85.4371,-257.3691 92.2093,-259.5635 98.5928,-261.6188 112.0017,-265.9361 126.4271,-270.5449 140.0506,-274.8813\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"139.2124,-278.2874 149.8028,-277.9826 141.3338,-271.6166 139.2124,-278.2874\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-274.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.13</text>\n",
       "</g>\n",
       "<!-- hidden_2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>hidden_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_2</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-40.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M61.5296,-208.0029C71.6101,-193.2606 84.5267,-176.5261 98.5928,-163.6188 110.4994,-152.6932 118.4584,-156.3303 129.5928,-144.6188 141.1042,-132.5109 138.0921,-125.361 147.5928,-111.6188 150.7674,-107.027 154.2343,-102.3816 157.8208,-97.8152\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"160.8072,-99.6881 164.3685,-89.7097 155.3619,-95.2894 160.8072,-99.6881\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-167.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.14</text>\n",
       "</g>\n",
       "<!-- input_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>input_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-128.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-124.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_1</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.5303,-124.5198C96.1264,-124.1019 113.958,-125.0849 129.5928,-129.6188 136.0098,-131.4796 142.4188,-134.1081 148.606,-137.1593\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"147.0163,-140.278 157.4875,-141.8919 150.3082,-134.1003 147.0163,-140.278\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-133.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.45</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M68.9631,-157.2707C78.1034,-165.8226 88.4774,-174.9528 98.5928,-182.6188 111.4719,-192.3792 118.3823,-189.9801 129.5928,-201.6188 140.9131,-213.3713 138.2259,-220.2572 147.5928,-233.6188 150.748,-238.1195 154.1773,-242.6785 157.7182,-247.1678\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"155.1644,-249.5758 164.1771,-255.1456 160.6049,-245.1711 155.1644,-249.5758\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-205.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.88</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M76.7548,-110.9534C96.4793,-101.3962 121.2626,-89.3878 143.365,-78.6785\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"145.1777,-81.6894 152.6508,-74.1792 142.1253,-75.3899 145.1777,-81.6894\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-102.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.04</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"369.4492\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"369.4492\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"369.4492\" y=\"-161.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- hidden_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>hidden_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M251.2136,-172.6188C269.0341,-172.6188 289.149,-172.6188 307.667,-172.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"307.7658,-176.1189 317.7658,-172.6188 307.7657,-169.1189 307.7658,-176.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.26</text>\n",
       "</g>\n",
       "<!-- hidden_1&#45;&gt;output_0 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>hidden_1&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M241.2929,-263.7087C264.8236,-246.9837 294.3091,-226.0262 319.0638,-208.4313\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"321.1389,-211.2504 327.2621,-202.6042 317.0835,-205.5448 321.1389,-211.2504\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-247.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.97</text>\n",
       "</g>\n",
       "<!-- hidden_2&#45;&gt;output_0 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>hidden_2&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M241.2929,-81.5289C264.8236,-98.2539 294.3091,-119.2114 319.0638,-136.8063\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"317.0835,-139.6928 327.2621,-142.6334 321.1389,-133.9872 317.0835,-139.6928\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-126.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.04</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad04f8dd8>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# More convenient way of writing the above\n",
    "model_3_layers_more_convenient = tf.keras.Sequential(\n",
    "    [\n",
    "        model_3_layers_hidden,\n",
    "        model_3_layers_output\n",
    "    ]\n",
    ")\n",
    "draw_nn_graph(model_3_layers_more_convenient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "To make our code even more compact, we can create the layers inline in order to fully specify the model in a single step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"425pt\" height=\"353pt\"\n",
       " viewBox=\"0.00 0.00 425.07 353.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 349.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-349.2376 421.068,-349.2376 421.068,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-242.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-238.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- hidden_0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>hidden_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_0</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-161.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.444,-235.9599C96.1946,-232.5248 114.1573,-227.5599 129.5928,-220.6188 136.3127,-217.597 143.0734,-213.9152 149.6008,-209.9483\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"151.7704,-212.7164 158.3313,-204.3974 148.0147,-206.8093 151.7704,-212.7164\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-234.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.59</text>\n",
       "</g>\n",
       "<!-- hidden_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>hidden_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-293.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-297.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_1</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-282.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M78.8541,-255.2285C85.4371,-257.3691 92.2093,-259.5635 98.5928,-261.6188 112.0017,-265.9361 126.4271,-270.5449 140.0506,-274.8813\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"139.2124,-278.2874 149.8028,-277.9826 141.3338,-271.6166 139.2124,-278.2874\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-274.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.61</text>\n",
       "</g>\n",
       "<!-- hidden_2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>hidden_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-55.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_2</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-40.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M61.5296,-208.0029C71.6101,-193.2606 84.5267,-176.5261 98.5928,-163.6188 110.4994,-152.6932 118.4584,-156.3303 129.5928,-144.6188 141.1042,-132.5109 138.0921,-125.361 147.5928,-111.6188 150.7674,-107.027 154.2343,-102.3816 157.8208,-97.8152\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"160.8072,-99.6881 164.3685,-89.7097 155.3619,-95.2894 160.8072,-99.6881\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-167.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.2</text>\n",
       "</g>\n",
       "<!-- input_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>input_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-128.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-124.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_1</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.5303,-124.5198C96.1264,-124.1019 113.958,-125.0849 129.5928,-129.6188 136.0098,-131.4796 142.4188,-134.1081 148.606,-137.1593\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"147.0163,-140.278 157.4875,-141.8919 150.3082,-134.1003 147.0163,-140.278\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-133.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.3</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M68.9631,-157.2707C78.1034,-165.8226 88.4774,-174.9528 98.5928,-182.6188 111.4719,-192.3792 118.3823,-189.9801 129.5928,-201.6188 140.9131,-213.3713 138.2259,-220.2572 147.5928,-233.6188 150.748,-238.1195 154.1773,-242.6785 157.7182,-247.1678\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"155.1644,-249.5758 164.1771,-255.1456 160.6049,-245.1711 155.1644,-249.5758\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-205.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.58</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M76.7548,-110.9534C96.4793,-101.3962 121.2626,-89.3878 143.365,-78.6785\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"145.1777,-81.6894 152.6508,-74.1792 142.1253,-75.3899 145.1777,-81.6894\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-102.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.97</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"365.4492\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"365.4492\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"365.4492\" y=\"-161.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "</g>\n",
       "<!-- hidden_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>hidden_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M250.899,-172.6188C267.6306,-172.6188 286.355,-172.6188 303.7387,-172.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"303.7939,-176.1189 313.7938,-172.6188 303.7938,-169.1189 303.7939,-176.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"282.3304\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.22</text>\n",
       "</g>\n",
       "<!-- hidden_1&#45;&gt;output_0 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>hidden_1&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M241.1604,-263.0854C263.7029,-246.6773 291.6505,-226.3349 315.3315,-209.0982\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"317.4886,-211.8571 323.5139,-203.1424 313.3691,-206.1976 317.4886,-211.8571\"/>\n",
       "<text text-anchor=\"middle\" x=\"282.3304\" y=\"-243.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.8</text>\n",
       "</g>\n",
       "<!-- hidden_2&#45;&gt;output_0 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>hidden_2&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M241.1604,-82.1522C263.7029,-98.5603 291.6505,-118.9026 315.3315,-136.1394\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"313.3691,-139.04 323.5139,-142.0952 317.4886,-133.3804 313.3691,-139.04\"/>\n",
       "<text text-anchor=\"middle\" x=\"282.3304\" y=\"-122.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.18</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad04b12e8>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# even more convenient - create everything at once\n",
    "model_3_layers_even_more_convenient = tf.keras.Sequential(\n",
    "    [\n",
    "        layers.Dense(\n",
    "            name=\"hidden\",\n",
    "            units=3,\n",
    "            input_shape=(2,)\n",
    "        ),\n",
    "        layers.Dense(\n",
    "            name=\"output\",\n",
    "            units=1\n",
    "        )\n",
    "    ]\n",
    ")\n",
    "draw_nn_graph(model_3_layers_even_more_convenient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "## Adding an Activation Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "In order to add activation functions to our neurons so that we can model functions with more interesting shapes, all we need to do is pass an additional `activation` argument to each neuron that we create:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"447pt\" height=\"353pt\"\n",
       " viewBox=\"0.00 0.00 447.45 353.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 349.2376)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-349.2376 443.4528,-349.2376 443.4528,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-242.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-238.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- hidden_0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>hidden_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-172.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-183.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_0</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-168.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-153.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: relu</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.444,-235.9599C96.1946,-232.5248 114.1573,-227.5599 129.5928,-220.6188 136.3127,-217.597 143.0734,-213.9152 149.6008,-209.9483\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"151.7704,-212.7164 158.3313,-204.3974 148.0147,-206.8093 151.7704,-212.7164\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-234.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;1.06</text>\n",
       "</g>\n",
       "<!-- hidden_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>hidden_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-293.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-304.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_1</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-289.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-274.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: relu</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M78.8541,-255.2285C85.4371,-257.3691 92.2093,-259.5635 98.5928,-261.6188 112.0017,-265.9361 126.4271,-270.5449 140.0506,-274.8813\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"139.2124,-278.2874 149.8028,-277.9826 141.3338,-271.6166 139.2124,-278.2874\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-274.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.88</text>\n",
       "</g>\n",
       "<!-- hidden_2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>hidden_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"199.2116\" cy=\"-51.6188\" rx=\"51.7379\" ry=\"51.7379\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-62.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden_2</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-47.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"199.2116\" y=\"-32.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: relu</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M61.5296,-208.0029C71.6101,-193.2606 84.5267,-176.5261 98.5928,-163.6188 110.4994,-152.6932 118.4584,-156.3303 129.5928,-144.6188 141.1042,-132.5109 138.0921,-125.361 147.5928,-111.6188 150.7674,-107.027 154.2343,-102.3816 157.8208,-97.8152\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"160.8072,-99.6881 164.3685,-89.7097 155.3619,-95.2894 160.8072,-99.6881\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-167.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.03</text>\n",
       "</g>\n",
       "<!-- input_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>input_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-128.6188\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-124.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_1</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.5303,-124.5198C96.1264,-124.1019 113.958,-125.0849 129.5928,-129.6188 136.0098,-131.4796 142.4188,-134.1081 148.606,-137.1593\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"147.0163,-140.278 157.4875,-141.8919 150.3082,-134.1003 147.0163,-140.278\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-133.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.94</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_1 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M68.9631,-157.2707C78.1034,-165.8226 88.4774,-174.9528 98.5928,-182.6188 111.4719,-192.3792 118.3823,-189.9801 129.5928,-201.6188 140.9131,-213.3713 138.2259,-220.2572 147.5928,-233.6188 150.748,-238.1195 154.1773,-242.6785 157.7182,-247.1678\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"155.1644,-249.5758 164.1771,-255.1456 160.6049,-245.1711 155.1644,-249.5758\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-205.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.76</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden_2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M76.7548,-110.9534C96.4793,-101.3962 121.2626,-89.3878 143.365,-78.6785\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"145.1777,-81.6894 152.6508,-74.1792 142.1253,-75.3899 145.1777,-81.6894\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-102.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.47</text>\n",
       "</g>\n",
       "<!-- output_0 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>output_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"378.6416\" cy=\"-172.6188\" rx=\"60.623\" ry=\"60.623\"/>\n",
       "<text text-anchor=\"middle\" x=\"378.6416\" y=\"-183.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output_0</text>\n",
       "<text text-anchor=\"middle\" x=\"378.6416\" y=\"-168.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"378.6416\" y=\"-153.9188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: sigmoid</text>\n",
       "</g>\n",
       "<!-- hidden_0&#45;&gt;output_0 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>hidden_0&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M251.1102,-172.6188C268.7446,-172.6188 288.765,-172.6188 307.6306,-172.6188\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"307.6699,-176.1189 317.6699,-172.6188 307.6698,-169.1189 307.6699,-176.1189\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-176.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.93</text>\n",
       "</g>\n",
       "<!-- hidden_1&#45;&gt;output_0 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>hidden_1&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M242.1896,-264.6363C265.3833,-248.9954 294.3592,-229.4553 319.6363,-212.4095\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"321.7189,-215.2266 328.053,-206.7336 317.8052,-209.4229 321.7189,-215.2266\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-247.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.73</text>\n",
       "</g>\n",
       "<!-- hidden_2&#45;&gt;output_0 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>hidden_2&#45;&gt;output_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M242.1896,-80.6013C265.3833,-96.2422 294.3592,-115.7823 319.6363,-132.8281\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"317.8052,-135.8147 328.053,-138.504 321.7189,-130.011 317.8052,-135.8147\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.3304\" y=\"-122.4188\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;1.07</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faad043a518>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# activation function\n",
    "model_w_activation = tf.keras.Sequential(\n",
    "    [\n",
    "        layers.Dense(\n",
    "            name=\"hidden\",\n",
    "            units=3,\n",
    "            input_shape=(2,),\n",
    "            activation=\"relu\"\n",
    "        ),\n",
    "        layers.Dense(\n",
    "            name=\"output\",\n",
    "            units=1,\n",
    "            activation=\"sigmoid\"\n",
    "        )\n",
    "    ]\n",
    ")\n",
    "draw_nn_graph(model_w_activation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "## Looking \"Under the Hood\" of Our Model: The Weights\n",
    "\n",
    "All of the numbers that define our neural network (the weights and the biases) are stored in a data structure that we'll call \"the model weights\" - we can look at this data structure as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[-1.0570867 ,  0.88223386, -0.02645457],\n",
       "        [-0.9439961 ,  0.75666344, -0.472427  ]], dtype=float32),\n",
       " array([0., 0., 0.], dtype=float32),\n",
       " array([[-0.9254637],\n",
       "        [ 0.734987 ],\n",
       "        [-1.0732952]], dtype=float32),\n",
       " array([0.], dtype=float32)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = model_w_activation.get_weights()\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "This looks quite complicated - let's step back and figure out what we're dealing with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "So, `weights` is a list - it turns out there are two entries per layer of our network (excluding the input layer) - since our network has a hidden layer and an output layer, we can see that there are four entries in `weights`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Each of these entries is a `numpy` array with a particular shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "hidden (Dense)               (None, 3)                 9         \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 1)                 4         \n",
      "=================================================================\n",
      "Total params: 13\n",
      "Trainable params: 13\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_w_activation.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Let's look at the weights associated with our hidden layer - you should compare the values below against the numbers printed next to the arrows in the network visualization above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weights for first neuron: [-1.0570867 -0.9439961]\n",
      "weights for second neuron: [0.88223386 0.75666344]\n",
      "weights for third neuron: [-0.02645457 -0.472427  ]\n"
     ]
    }
   ],
   "source": [
    "hidden_layer_weights = weights[0]\n",
    "print(\"weights for first neuron:\", hidden_layer_weights[:,0])\n",
    "print(\"weights for second neuron:\", hidden_layer_weights[:,1])\n",
    "print(\"weights for third neuron:\", hidden_layer_weights[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "As we can see from the output above, the columns of this array correspond to different neurons and the rows correspond to individual inbound edges. The next entry in the `weights` list is the array of biases for this layer, one per neuron: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_layer_biases = weights[1]\n",
    "hidden_layer_biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "If we continue moving through the `weights` list, you can see that we arrive at the output layer weights and biases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weight shape: (3, 1)\n",
      "weight values: [[-0.9254637]\n",
      " [ 0.734987 ]\n",
      " [-1.0732952]]\n"
     ]
    }
   ],
   "source": [
    "output_layer_weights = weights[2]\n",
    "print(\"weight shape:\", output_layer_weights.shape)\n",
    "print(\"weight values:\", output_layer_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bias shape: (1,)\n",
      "bias values: [0.]\n"
     ]
    }
   ],
   "source": [
    "output_layer_biases = weights[3]\n",
    "print(\"bias shape:\", output_layer_biases.shape)\n",
    "print(\"bias values:\", output_layer_biases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "<span style=\"color:blue;font-weight:bold\">Exercise</span>: Create a neural network with:\n",
    "\n",
    "* One hidden layer named `\"hidden-layer-one\"`consisting of five neurons with the activation function `\"tanh\"`, with an `input_shape` of `(2,)` indicating two input neurons\n",
    "* An output layer named `\"output-layer-one\"` consisting of three neurons with the `\"softmax\"` activation function\n",
    "\n",
    "Store your neural network in the variable `my_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true,
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# delete this entire line and replace it with your code\n",
    "\n",
    "my_model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(name=\"hidden-layer-one\", units=5, input_shape=(2,), activation=\"tanh\"),\n",
    "    tf.keras.layers.Dense(name=\"output-layer-one\", activation=\"softmax\", units=3)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "deletable": false,
    "editable": false,
    "strategicml": {
     "execution_time": 1584778624471,
     "exercise_id": "c17ab3a5-ecfe-4c3f-89c5-6e53c6600a1d",
     "points": 4
    },
    "tags": [
     "test"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color:#e8fcef\n",
       ";color:#333333;padding:10px;border-style:solid;border-width:2px;border-color:#038c33\">\n",
       "<div style=\"font-weight:bold\">Correct!</div>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "check_variable_definition(\"my_model\")\n",
    "assert str(type(my_model)) ==  \"<class 'tensorflow.python.keras.engine.sequential.Sequential'>\", \"Your variable <code>my_model</code> does not contain an object of the right type - did you create it using <code>keras.Seqential()</code>?\"\n",
    "assert len(my_model.layers) == 2, \"Your model does not contain the correct number of layers.\"\n",
    "assert my_model.layers[0].name == 'hidden-layer-one', \"Your hidden layer does not have the correct name value.\"\n",
    "assert my_model.layers[1].name == 'output-layer-one', \"Your output layer does not have the correct name value.\"\n",
    "assert my_model.layers[0].units == 5, \"Your hidden layer does not contain the correct number of neurons.\"\n",
    "assert my_model.layers[1].units == 3, \"Your output layer does not contain the correct number of neurons.\"\n",
    "assert my_model.layers[0].activation.__name__ == \"tanh\",  \"Your hidden layer does not use the correct activation function.\"\n",
    "assert my_model.layers[1].activation.__name__ == \"softmax\",  \"Your output layer does not use the correct activation function.\"\n",
    "success()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_8\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "hidden-layer-one (Dense)     (None, 5)                 15        \n",
      "_________________________________________________________________\n",
      "output-layer-one (Dense)     (None, 3)                 18        \n",
      "=================================================================\n",
      "Total params: 33\n",
      "Trainable params: 33\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "my_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"585pt\" height=\"990pt\"\n",
       " viewBox=\"0.00 0.00 584.63 990.43\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 986.4335)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-986.4335 580.6315,-986.4335 580.6315,4 -4,4\"/>\n",
       "<!-- input_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>input_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-506.2168\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-502.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_0</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>hidden&#45;layer&#45;one_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"238.8096\" cy=\"-491.2168\" rx=\"91.4341\" ry=\"91.4341\"/>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-502.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden&#45;layer&#45;one_0</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-487.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-472.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: tanh</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.9476,-503.1451C97.5411,-501.8913 117.5393,-500.3802 137.642,-498.8612\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"138.1162,-502.3354 147.8241,-498.0918 137.5888,-495.3553 138.1162,-502.3354\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-505.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.15</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>hidden&#45;layer&#45;one_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"238.8096\" cy=\"-91.2168\" rx=\"91.4341\" ry=\"91.4341\"/>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-102.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden&#45;layer&#45;one_1</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-87.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-72.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: tanh</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M63.538,-473.2369C69.7644,-463.2925 76.0262,-452.1155 80.5928,-441.2168 93.6735,-409.9986 83.9019,-397.7103 98.5928,-367.2168 108.3549,-346.954 120.9285,-347.9726 129.5928,-327.2168 153.0805,-270.9508 121.8688,-246.4961 147.5928,-191.2168 152.3854,-180.918 158.6204,-170.943 165.5822,-161.5367\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"168.5107,-163.4716 171.8693,-153.4232 162.9775,-159.1839 168.5107,-163.4716\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-371.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.86</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>hidden&#45;layer&#45;one_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"238.8096\" cy=\"-291.2168\" rx=\"91.4341\" ry=\"91.4341\"/>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-302.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden&#45;layer&#45;one_2</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-287.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-272.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: tanh</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M63.1735,-472.8602C73.264,-459.2466 85.7403,-443.7844 98.5928,-431.2168 110.859,-419.2225 116.798,-419.6454 129.5928,-408.2168 138.5688,-400.1993 153.9756,-384.2111 170.4333,-366.5646\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"173.2941,-368.6275 177.5353,-358.9186 168.1652,-363.8636 173.2941,-368.6275\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-435.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.89</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_3 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>hidden&#45;layer&#45;one_3</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"238.8096\" cy=\"-691.2168\" rx=\"91.4341\" ry=\"91.4341\"/>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-702.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden&#45;layer&#45;one_3</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-687.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-672.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: tanh</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden&#45;layer&#45;one_3 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden&#45;layer&#45;one_3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M51.8164,-545.1179C61.017,-570.5789 76.1335,-603.1048 98.5928,-625.2168 111.3725,-637.7987 127.0032,-648.3031 143.1339,-656.9589\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"141.6373,-660.1248 152.1319,-661.5789 144.8346,-653.8977 141.6373,-660.1248\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-652.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.49</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_4 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>hidden&#45;layer&#45;one_4</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"238.8096\" cy=\"-891.2168\" rx=\"91.4341\" ry=\"91.4341\"/>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-902.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">hidden&#45;layer&#45;one_4</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-887.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"238.8096\" y=\"-872.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: tanh</text>\n",
       "</g>\n",
       "<!-- input_0&#45;&gt;hidden&#45;layer&#45;one_4 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>input_0&#45;&gt;hidden&#45;layer&#45;one_4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M49.8546,-545.7882C64.814,-602.9814 97.3508,-710.7066 147.5928,-791.2168 153.4623,-800.6223 160.2665,-809.9575 167.4616,-818.9289\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"164.8124,-821.2177 173.8714,-826.7118 170.2158,-816.7676 164.8124,-821.2177\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-759.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.53</text>\n",
       "</g>\n",
       "<!-- input_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>input_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"40.2964\" cy=\"-392.2168\" rx=\"40.0939\" ry=\"40.0939\"/>\n",
       "<text text-anchor=\"middle\" x=\"40.2964\" y=\"-388.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">input_1</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.5688,-386.7036C96.3366,-386.0585 114.2784,-387.2547 129.5928,-393.2168 143.9063,-398.7892 157.5358,-407.1728 170.0095,-416.7343\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"168.044,-419.6445 178.0351,-423.1699 172.4232,-414.1835 168.044,-419.6445\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-397.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.73</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M56.5821,-354.9521C75.5994,-313.2709 109.3659,-244.6062 147.5928,-191.2168 154.0471,-182.2025 161.2205,-173.1143 168.6288,-164.2857\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"171.357,-166.4805 175.1875,-156.6023 166.033,-161.9358 171.357,-166.4805\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-269.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.21</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M58.2356,-355.8786C67.9767,-339.9435 81.5445,-322.5491 98.5928,-312.2168 110.3655,-305.0819 123.6862,-300.0621 137.3168,-296.5739\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"138.3107,-299.937 147.2791,-294.2962 136.7505,-293.1131 138.3107,-299.937\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-316.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.74</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden&#45;layer&#45;one_3 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden&#45;layer&#45;one_3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M70.1751,-419.8977C94.6226,-442.8314 125.7725,-472.8197 129.5928,-480.2168 152.5267,-524.6219 125.8484,-546.2173 147.5928,-591.2168 152.5351,-601.4446 158.8633,-611.3754 165.8778,-620.7567\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"163.2874,-623.1262 172.2011,-628.853 168.8042,-618.8175 163.2874,-623.1262\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-484.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.12</text>\n",
       "</g>\n",
       "<!-- input_1&#45;&gt;hidden&#45;layer&#45;one_4 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>input_1&#45;&gt;hidden&#45;layer&#45;one_4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M63.4728,-425.6508C69.6254,-435.5126 75.8646,-446.5322 80.5928,-457.2168 92.3773,-483.8462 89.6503,-492.5034 98.5928,-520.2168 111.0698,-558.8832 120.6676,-566.5796 129.5928,-606.2168 147.7402,-686.8094 114.04,-715.727 147.5928,-791.2168 152.2065,-801.5969 158.3302,-811.6226 165.2292,-821.0574\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"162.6073,-823.3898 171.473,-829.1907 168.1598,-819.1272 162.6073,-823.3898\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.0928\" y=\"-610.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.11</text>\n",
       "</g>\n",
       "<!-- output&#45;layer&#45;one_0 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>output&#45;layer&#45;one_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"486.829\" cy=\"-291.2168\" rx=\"89.6056\" ry=\"89.6056\"/>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-302.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output&#45;layer&#45;one_0</text>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-287.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-272.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: softmax</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_0&#45;&gt;output&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_0&#45;&gt;output&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M304.1565,-427.3774C313.7005,-415.9586 322.7468,-403.7012 330.0264,-391.2168 342.982,-368.9978 329.8142,-354.3785 348.0264,-336.2168 359.3901,-324.8845 373.6298,-316.4052 388.6262,-310.0608\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"390.0029,-313.2806 398.0423,-306.3799 387.4544,-306.761 390.0029,-313.2806\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-340.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.62</text>\n",
       "</g>\n",
       "<!-- output&#45;layer&#45;one_1 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>output&#45;layer&#45;one_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"486.829\" cy=\"-687.2168\" rx=\"89.6056\" ry=\"89.6056\"/>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-698.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output&#45;layer&#45;one_1</text>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-683.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-668.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: softmax</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_0&#45;&gt;output&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_0&#45;&gt;output&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M304.6585,-554.7712C314.141,-566.2249 323.0367,-578.5677 330.0264,-591.2168 344.5245,-617.4536 326.3025,-635.5615 348.0264,-656.2168 354.8205,-662.6767 369.7972,-668.0319 387.5042,-672.3693\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"387.0591,-675.8578 397.5876,-674.6748 388.6194,-669.0339 387.0591,-675.8578\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-673.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.37</text>\n",
       "</g>\n",
       "<!-- output&#45;layer&#45;one_2 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>output&#45;layer&#45;one_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"486.829\" cy=\"-489.2168\" rx=\"89.6056\" ry=\"89.6056\"/>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-500.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">output&#45;layer&#45;one_2</text>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-485.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bias: 0.00</text>\n",
       "<text text-anchor=\"middle\" x=\"486.829\" y=\"-470.5168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">act: softmax</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_0&#45;&gt;output&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_0&#45;&gt;output&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M330.2815,-489.4184C336.2865,-489.3385 342.2425,-489.2697 348.0264,-489.2168 360.4804,-489.1028 373.5633,-489.0348 386.5134,-488.9989\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"386.8637,-492.4982 396.8561,-488.9767 386.8486,-485.4983 386.8637,-492.4982\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-493.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.79</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_1&#45;&gt;output&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_1&#45;&gt;output&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M309.882,-148.5287C340.7607,-173.4289 377.0302,-202.6763 408.6665,-228.1874\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"406.6223,-231.0352 416.6037,-234.5879 411.0163,-225.5861 406.6223,-231.0352\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-207.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.1</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_1&#45;&gt;output&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_1&#45;&gt;output&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M306.5085,-152.709C315.9563,-164.5391 324.3587,-177.5404 330.0264,-191.2168 363.1562,-271.1609 304.0112,-505.7097 348.0264,-580.2168 356.2457,-594.13 365.8484,-589.8639 379.0264,-599.2168 389.3883,-606.571 400.0427,-614.6177 410.4321,-622.762\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"408.3018,-625.5395 418.316,-628.9991 412.645,-620.0497 408.3018,-625.5395\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-603.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.58</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_1&#45;&gt;output&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_1&#45;&gt;output&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M305.9982,-152.927C315.5051,-164.732 324.0595,-177.6683 330.0264,-191.2168 365.8199,-272.4895 301.1875,-314.7678 348.0264,-390.2168 356.5495,-403.946 365.636,-400.1707 379.0264,-409.2168 388.2125,-415.4226 397.7068,-422.0842 407.0775,-428.818\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"405.116,-431.7188 415.2696,-434.7444 409.2189,-426.0473 405.116,-431.7188\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-413.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.73</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_2&#45;&gt;output&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_2&#45;&gt;output&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M330.1062,-291.2168C348.6049,-291.2168 368.1494,-291.2168 386.927,-291.2168\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"386.9664,-294.7169 396.9664,-291.2168 386.9664,-287.7169 386.9664,-294.7169\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-295.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.71</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_2&#45;&gt;output&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_2&#45;&gt;output&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M306.1263,-352.8711C315.6184,-364.6824 324.1346,-377.6354 330.0264,-391.2168 350.1653,-437.6395 319.6926,-576.2902 348.0264,-618.2168 349.8882,-620.9717 371.0057,-631.8861 396.2315,-644.2482\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"394.7782,-647.4335 405.3006,-648.6702 397.8461,-641.1416 394.7782,-647.4335\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-638.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.1</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_2&#45;&gt;output&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_2&#45;&gt;output&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M302.9485,-356.2421C312.5867,-367.4683 321.9784,-379.355 330.0264,-391.2168 340.2937,-406.3496 334.8476,-415.5385 348.0264,-428.2168 360.5535,-440.2681 375.8543,-450.1611 391.6641,-458.2073\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"390.4884,-461.5267 401.0143,-462.7338 393.5386,-455.2262 390.4884,-461.5267\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-454.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.48</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_3&#45;&gt;output&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_3&#45;&gt;output&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M306.0973,-629.5499C315.5927,-617.74 324.1176,-604.7907 330.0264,-591.2168 349.6046,-546.241 321.7718,-412.6513 348.0264,-371.2168 349.9406,-368.1958 372.728,-354.6856 399.3047,-339.6035\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"401.062,-342.6308 408.045,-334.6629 397.6173,-336.537 401.062,-342.6308\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-375.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.82</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_3&#45;&gt;output&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_3&#45;&gt;output&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M330.082,-694.5915C346.3829,-694.7889 363.208,-694.7298 379.0264,-694.2168 381.702,-694.13 384.4144,-694.0311 387.1511,-693.9218\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"387.3076,-697.4184 397.1444,-693.4831 387.0005,-690.4251 387.3076,-697.4184\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-698.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.62</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_3&#45;&gt;output&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_3&#45;&gt;output&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M304.6145,-627.6381C314.1025,-616.1873 323.0114,-603.8518 330.0264,-591.2168 344.3693,-565.3833 327.0604,-548.0377 348.0264,-527.2168 359.0786,-516.241 372.9797,-508.3575 387.6866,-502.711\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"389.1524,-505.9068 397.4457,-499.3134 386.8509,-499.2959 389.1524,-505.9068\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-531.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">&#45;0.56</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_4&#45;&gt;output&#45;layer&#45;one_0 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_4&#45;&gt;output&#45;layer&#45;one_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M306.1382,-829.5677C315.6289,-817.7557 324.1416,-804.8011 330.0264,-791.2168 350.4056,-744.1739 321.2505,-604.9363 348.0264,-561.2168 356.4663,-547.4362 370.3142,-555.8268 379.0264,-542.2168 415.7023,-484.9225 368.8349,-452.128 397.0264,-390.2168 401.7122,-379.9263 407.8676,-369.9666 414.7635,-360.5844\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"417.6657,-362.5523 420.9958,-352.4944 412.1203,-358.2803 417.6657,-362.5523\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-565.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.52</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_4&#45;&gt;output&#45;layer&#45;one_1 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_4&#45;&gt;output&#45;layer&#45;one_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M323.6192,-856.9474C342.8238,-847.2095 362.4274,-835.5611 379.0264,-822.2168 397.6396,-807.2533 415.1032,-788.5379 430.3316,-769.8521\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"433.2116,-771.8545 436.7181,-761.8567 427.7423,-767.4857 433.2116,-771.8545\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-845.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.86</text>\n",
       "</g>\n",
       "<!-- hidden&#45;layer&#45;one_4&#45;&gt;output&#45;layer&#45;one_2 -->\n",
       "<g id=\"edge25\" class=\"edge\">\n",
       "<title>hidden&#45;layer&#45;one_4&#45;&gt;output&#45;layer&#45;one_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M302.0937,-825.1306C311.7874,-814.0854 321.4203,-802.5328 330.0264,-791.2168 354.809,-758.6304 364.913,-751.6467 379.0264,-713.2168 398.376,-660.5289 373.1256,-639.0023 397.0264,-588.2168 401.8412,-577.986 408.0765,-568.0633 415.0172,-558.7019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"417.9176,-560.6732 421.2802,-550.6261 412.3861,-556.3833 417.9176,-560.6732\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5264\" y=\"-771.0168\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0.72</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7faaabfdb748>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_nn_graph(my_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": null,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "strategicml": {
   "lesson_id": "01ae638e-1e68-49bd-abc7-9d3c0de80c32",
   "release": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}