Linear_Regression_Prob_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear Regression\n",
    "========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 2\n",
    "\n",
    "* [Problem Description](#problem_description)\n",
    "* [Solution 1: Applying Gradient Descent (Keras)](#gradient_descent_keras)\n",
    "* [Solution 2: Applying Gradient Descent (Tensorflow)](#gradient_descent_tensorflow)\n",
    "* [Solution 3: Analytical solution with Least Squares Error minimization](#least-squares-error-minimization)\n",
    "* [Comparing all 3 solutions](#comparing_solutions)\n",
    "\n",
    "In this problem we will find the best line that fits a given model. For theory details, please see [Linear Regression Theory](Linear_Regression_Theory.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='problem_description'></a>\n",
    "### Problem Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this problem we will be using a function to create a cloud of points and based on these generated points we will try to find the best line to fit the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "'''This function creates a cloud of points in a line specified between two points: initialPoint and finalPoint\n",
    "Input parameters: \n",
    "    initialPoint  : initial point of the line that represents the cloud\n",
    "    finalPoint    : final point of the line that represents the cloud\n",
    "    scatter       : range in y-axis indicating how spread the cloud of points is\n",
    "    multiplicity  : number of points in the cloud for every x point '''\n",
    "def PointsCloud(initialPoint, finalPoint, scatter, multiplicity):\n",
    "    # Angular coefficient\n",
    "    ang_coef = (initialPoint[1]-finalPoint[1])/(initialPoint[0]-finalPoint[0]) \n",
    "    # Linear coefficient\n",
    "    lin_coef = -ang_coef*initialPoint[0]+initialPoint[1]\n",
    "    i = [val for val in np.arange(initialPoint[0], finalPoint[0], 1) for _ in np.arange(0, multiplicity)]\n",
    "    y = [(ang_coef*ii)+lin_coef for ii in i]\n",
    "    random.seed(26051984) # to enable reproducibility\n",
    "    r = [random.uniform(0, 1)*scatter+(yy-(0.5*scatter)) for yy in y]\n",
    "    points = [[i[idx], r[idx]] for idx in range(len(i))]\n",
    "    return points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define and visualize our cloud of points using the function above.<p>\n",
    "This function takes a line as input, which will be the base of the generated points. Our line is represented by the connection between the points (10,12) and (50,53). For each point in x-axis it will have 4 points (multiplicity) and 30 will be our scatter value, representing the range in y-axis indicating how vertically spaced the cloud is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2UXVWZ5/HvQ4wSUIiMmUwsSMcXGkZkIG0ENNjLgG+t\nCCXajEzTi25Zk9WrHYXWSRtnaAEXa8wMji0z9jgT8SWKjbxpCGL7wpuOdItUDBARWPYgCGUgaSEK\nmoYQnvnjnkpuinvPOffuu+8+L7/PWrVS91Tde3edVJ3n7Gc/e29zd0REpL32Sd0AERFJS4FARKTl\nFAhERFpOgUBEpOUUCEREWk6BQESk5RQIRERaToFARKTlFAhERFruOakbUMaLXvQiX7JkSepmiIjU\nysaNG//J3RcUfV8tAsGSJUuYmppK3QwRkVoxswfKfJ9SQyIiLadAICLScgoEIiItp0AgItJyCgQi\nIi1Xi6ohEZE2Wb9pmou+dS+/2L6DF8+fx6o3H8bk0olo76dAICJSIes3TfPhr25mx85dAExv38GH\nv7oZIFowUGpIRKRCLvrWvbuDwIwdO3dx0bfujfaeCgQiIhXyi+07Bjo+CgoEIiIV8uL58wY6PgoK\nBCIiFbLqzYcxb+6cvY7NmzuHVW8+LNp7arBYRCSCYSt/Zr5HVUMiIjUWWvkzuXQi6oV/NqWGRERG\nLEXlTwj1CEREhpCX+pnuU+HT73hqCgQiIgMqSv3MMWOX+7OeN8dsrO0sS6khEZEBFaV+egWBvOOp\nKRCIiAyoaNLXRJ+a/37HU1MgEBEZUNGkrxRzAUIoEIiIDKjoQj+5dIKPnXokE/PnYXR6Ah879cjS\nJaHrN02zfM2NvGT1dSxfcyPrN02P+kfYiwaLRaSxYi3nXGbS17BzAVKsPho1EJjZfOAS4JWAA+8B\n7gUuB5YA9wOnuftjMdshIu0T+4Iaa9JX3kB0XZehvhj4prsfDhwF3A2sBm5w90OBG7LHIiIjVbdJ\nXTMatfqomR0I/D7wWQB3f8rdtwOnAOuyb1sHTMZqg4i0V4oL6ig0bfXRlwDbgM+b2SYzu8TM9gcW\nuvuW7HseBhZGbIOItFSKC+oopKg4ihkIngP8HvBpd18K/IZZaSB3dzpjB89iZivNbMrMprZt2xax\nmSLSRHUr4ZwRWnE0DPNIM93M7F8BP3D3Jdnj19EJBC8HXu/uW8xsEXCzu+f+zyxbtsynpqaitFNE\nqiu06mfcm8BXjZltdPdlRd8XrWrI3R82swfN7DB3vxc4EfhJ9nEmsCb795pYbRCR+hpF1U9eZU/b\ng0S32PMI3gd82cyeC9wH/CmddNQVZnYW8ABwWuQ2iEgNxSyjTFGrX2VRA4G73w706pacGPN9RaT+\nYlb9pKjVrzLNLBaRSnrx/Hk91+8fRdXPOEpL65R60lpDIlJJMat+YpeWzqSeprfvwNmTeoq9ZtCw\nFAhEpJJillGuOHxB4fGQhd/qNqtZqSERqaxY6/ncdE/vuUkzx0MHk+s2q1k9AhFpnaILdegd/YHz\n5g50PDUFAhFpnaIxgtA7+n5bE1d0y2IFAhFpn6KB6Pn79b5z73d8tu2/3TnQ8dQ0RiAi0VS1hLJo\nY5l+K++UXZEnZulrDAoEIhJF1Wfv5g1E/2pH7zv3fsdnW/Xmw/b62aHaC94pNSQiUdSthLJb6DyD\nFCuIhlCPQESiqFsJZbcyd/RFaa9Ypa8xKBCISBR1y5N3KxpDqHraa1AKBCISRd3y5LPl3dE3bdE6\nBQIRiaLornocYlUt1Tnt1YsCgYhEkzJPHjN9U+e0Vy+qGhKRRopZtVTX/ZD7UY9ARBopZvqmCmmv\nUVIgEJFGip2+qVN5aBGlhkSkkZqWvolJPQIRaaSmpW9iUiAQkcZqUvomJgUCEZEeqrpyagxRA4GZ\n3Q88DuwCnnb3ZWZ2EHA5sAS4HzjN3R+L2Q4RGV6bLogzmraERJFxDBavcPej3X1Z9ng1cIO7Hwrc\nkD0WkQqauSBOb9+Bs+eCOMhG7nWUeuXU9ZumWb7mRl6y+jqWr7kx+vlOUTV0CrAu+3wdMJmgDSJS\nQuoLYiopl5BIEXxjBwIHrjezjWa2Mju20N23ZJ8/DCzs9UQzW2lmU2Y2tW3btsjNFJFe6r6mzrB3\n1qH7EYRIEXxjB4Lj3f1o4A+A95rZ73d/0d2dTrB4Fndf6+7L3H3ZggULIjdTRHqJfUGMmQIJubNO\nOQchRfCNGgjcfTr7dyvwNeAY4BEzWwSQ/bs1ZhtEZHgxL4ixUyAhd9YpdxhL0RuJVjVkZvsD+7j7\n49nnbwI+CmwAzgTWZP9eE6sNIhIm5qSs2Gv6h95Zp5qDkGIfh5jlowuBr5nZzPv8rbt/08xuA64w\ns7OAB4DTIrZBRALFuiDGToHUdanoFDOiowUCd78POKrH8V8CJ8Z6XxEZr2HnGcS+UFd9h7S88zbu\n3ohmFotUQF0nbYVMvIp9oS6z73Cqc75+0zSrrryDnc90amWmt+9g1ZV37NXucbJO4U61LVu2zKem\nplI3QySK2RdT6FwQxzU4GWL5mht73tVPzJ/HLatPKHx+qotx6nN+9AXfZvuOnc86Pn/eXG4/700j\nex8z29g1mbcv9QhEEqvzRuipB2SHDSSpz3mvIJB3PDbtRyCSWJ0nbaWceBVSftqrF5N3vOkUCEQS\nS3kxDZVy4lXIPIE5nWrG0sdH7YX7zR3oeGwKBCKJ1XknrdgTr/JmHof0pHb1GRvtd3zUznv7Ecyd\ns3fQmTvHOO/tR4zl/WfTGIFIYnXfSStWqWNRRVJI+elEn+dOjKkXVrWKJlUNiYxBXctDUyqqSAqp\n/EldNZRnlG0rWzWk1JBIZG1d0z9UUeonJC2Vci2hIilWH1VqSCSy1KWKdVUm9ROSlqrqfsaNW31U\nROpdHppSnQfRQzRq9VGRJjl3/WYuu/VBdrkzx4zTjz2ECyePLPXcui5+llrdB9GH1bTVR0Ua4dz1\nm7n0Bz/f/XiX++7HZYJB1Rc/q7Kqpm9iShEAVTUkjRGrMudlH/5Gz/ryOWb8v4+9dSxtU9WRDENr\nDUmrhKyCWWQUk49C7mxj/mwioMFiaYiYJXeplyNIUU4o8cXcr3lQ6hFII8SszDn92EP2GiPoPj4O\noT+b0krVU7VennoE0ggxS+4unDySM45bvLsHMMeMM45bXLpqKFTIzzaKyWxVunNtiqr18tQjkEaI\nXZlz4eSRY7vwzxbys4VOZqvaTlpNUbW5JbmBwMz2BU4CXge8GNgB/Bi4zt3vit88kXLqXnNetH8t\nDPezhV5wzt9w1+4gMGPnM875G+6qxLmta9qranNL+gYCM7uAThC4GbgV2ArsC/wusCYLEh909zvH\n0E6RQnWtOS+TLx72Zwu94FRtJ61uVcuzD2LF4Qt6jjutOHxBgtbk9wh+6O7n9fnaJ8zsXwKLi97A\nzOYAU8C0u59kZgcBlwNLgPuB09z9sYFaLdJDyOzfUCF3pjHXIqr7ZLa881rnNZxuumfbQMdj6xsI\n3P26vCe6+1Y6vYQiZwN3Awdkj1cDN7j7GjNbnT3+ULnmivQWOvs3ROidacx8cWjK7IX7zeWx3z77\n7r97J61Y6Zmi85o6zx7yc6du+2yFVUNmdq2ZbZj18SUzOztLD+U992DgbcAlXYdPAdZln68DJodt\nvMiMy259cKDjoxRaARJ7kbHJpRPcsvoEfrbmbdyy+oSBLtJFO2nFXGK76LzWdb9kqN72pGXKR+8D\nngA+k338GniczljBZwqe+0ngL4Fnuo4tdPct2ecPAwsHabBIL7G3HswroQzdCL3MKpupSjgnl05w\n0buO2mvd/ovedVSp9Eyoorvmuu6XDNVbWbVM+ehr3f3VXY+vNbPb3P3VZta3csjMTgK2uvtGM3t9\nr+9xdzeznn+pZrYSWAmweHHhUIS0nAG9fpFGMfc39qBkmW0LU5Zw5g1Ux0xxFA10p6wUC/25q1bl\nViYQPN/MFrv7zwHMbDHw/OxrT+U8bzlwspm9lU610QFmdinwiJktcvctZraIPuMM7r4WWAudRefK\n/TjSVvs9dw6/eWpXz+OhxjEomXexrXIJZ8wyyDID3akqxUbxc1epyq1MauiDwPfN7CYzuxn4v8B/\nNLP92ZPrfxZ3/7C7H+zuS4B3Aze6+xnABuDM7NvOBK4JaL8IAL/tEQTyjg+i6O4v9lpEVS7hjJni\nqPJ2klVL7YQq0yP4O+BQ4PDs8b10sjpP0hkDGNQa4AozOwt4ADhtiNcQ2cso7tD6VYEUvXbqtYhS\nCk1xFFXeVOmuuVvVUjuhCvcjMLPPuft7uh7vD2xw9xNjN26G9iOQIrPz+NC5Qyt7B5n3fKDwtWPO\nYVj60W/3LeHc9JE3jeQ9Ugj9P5t5jaZcjGMY5X4E02b2v9z9z83shcB1FFcLifQU6w839A4tbxzg\nltUnFL52zLWIznv7Eay66g527tpz09ZdwllXo1gHqa4zi6umMBC4+1+Z2X8zs/8NvApY4+5Xx2+a\nNM04qm+GfZ2icYCUKYqmpSFmhFbexB7EL7ppaVJvJG+toVO7Ht4K/BXwQ8DN7FR3/2rsxkmzVHlJ\ngKotAjZbVXPlIULPeczS1aKblqb1RvKqht7e9XESsAmY2/VYZCChE69ialoVSB2EnvOYs3OLJoxV\nbT+BUHlrDf3pOBsi45GyOzvHrO8m8Kk1Nf1SZaHnPOaCekW9jaqtFRSqb4/AzM7NBof7ff2EbPaw\n1ETMdWHKiL0MhLRLzHkGRb2Nqq0VFCpvsHgz8HUz+2fgR8A2OjOEDwWOBq4H/kv0FsrIpM7RT/TJ\nCU9U4I+naTnfOhjFOY81dlLU26j78t6z9e0RuPs17r4c+DPgLmAOnQXnLgWOcfe/cPc0i2fLUFJ3\nZ6uch29azncQqRa0q/I5L+ptVHnW8zDKlI/+FPjpGNoikaWujEmdh88bH0kdJFNJ2RMaxzkPGRMr\n6m00qZJLm9e3SBW6s6n+eIoueLGDZFVrzlOmC8dxzpXuK6fMonPSEE3rzg6iKA0RM22VepA+T8qe\nUOxUYZVTT1VT2CMws+XufkvRMamHJnVnB1Fm5jDESVulngGbJ2W6MHaqsK3pvmGUSQ39T+D3ShwT\nqawyF7xYQTLmRLrQ9MeKwxf0XDl1xeELgttWRug5zwuCqcfE6iRviYnXAK8FFpjZB7q+dACdCiKJ\npKr5ZKh22/JUYXwkhtDexk339C7863d8UDF/X4qCYFP/z2PI6xE8l85OZM8BXtB1/NfAu2I2qs2q\nPMBV5bYVib1ufiqh6Y+U6/WUfY1+570oCKauUquTvCUmvgt818y+4O4PjLFNrZZ60lfIH17VDZuG\nCL2gxVxaIzT9ETN9EnuZ6TJBrK1jYoMqUzX0PDNba2bfNrMbZz6it6ylUg5wFVW3tHXwLbT6pN9O\nZaPYwSy08iZm5U7MZaahecs8pFQmEFxJZ+XRc4FVXR8SQcpfbv3h9RZ6Qbtw8kjOOG7x7h7AHDPO\nOG7xSDayCS0JTrleT5Gi817lmep1U6Zq6Gl3/3T0lgiQdlCzzB9eGwffRpE+ibmDWWj6I9V6PUWK\nzrvGAEYnr2rooOzTa83sz4GvAU/OfN3dH43ctlZK+cutP7ze2hoAQ41jmWmNAYxG383rzexngAO9\nRrTc3V8as2HdtHn96OQNBo9iM/EqC6n8qWrVUNPpvIcpu3l930AwggbsC3wPeB6dnsdV7n5e1tO4\nHFgC3A+c5u6P5b1W2wJBrF/+Mhf6pv7hNT3IifQyskAwa+/iGb8CNrv71pznGbC/uz9hZnOB7wNn\nA6cCj7r7GjNbDbzQ3T+U14Y2BYKYF6zla27sux/ALatPCHrtqmvzz16kqcFfygeCMlVDZwGXAH+U\nfXwG+BBwi5n9cb8neccT2cO52YcDpwDrsuPrgMkSbWiNmAtltbX8E9r9s+ep8oJ4Mj5lqoaeA/xr\nd38EwMwWAl8EjqWT+vlSvyea2RxgI/By4G/c/VYzW+juW7JveRhYGND+Wkq1Lv78/eby2G939jze\ndFp3prdRTBLU2Ev9lekRHDITBDJbs2OPAs++qnRx913ufjRwMHCMmb1y1tedTi/hWcxspZlNmdnU\ntm3N2Qit6A4sZq1+vyxgG7YMbnPNed4OZKE3HiE9CvVGqqNMILjZzL5uZmea2ZnANdmx/YHtZd7E\n3bcDNwFvAR4xs0UA2b89xxncfa27L3P3ZQsWjGclxHFIuS7+r3b0jtv9jjdJW/diiH3jUfT7nBeE\ntF9AdZRJDb0XeCewPHv8ReDq7G5+Rb8nmdkCYKe7bzezecAbgf8KbADOBNZk/14zfPPrJ+W6+ONI\nj1Q5TdDGmvOi1E/oHIm83+f1m6ZZddUd7NzV6XJOb9/BqqvuAMqvFSTjUWbPYgeuyj4GsQhYl40T\n7ANc4e5fN7N/AK4ws7OAB4DTBnzdWku5Ln7siVEhi7PVeWXTKot945E37nTBtXftDgIzdu5yLrj2\nrrFsDyrl5c0s/r67H29mj7N3Ht/oxIcD8l7Y3e8ElvY4/kvgxCHbW3spZ6nGnhlcZuCx311/3Vc2\nTSl0c5aQG4+8caftfVKOM4FDM7arI28Z6uOzf1/Q73tkcKmXaYiZHinaiSvvrl9pgnz9LvapN2cJ\nGXdK/bcge5QZI8DMjgcOdffPm9mLgBe4+8/iNq25YueqY+8K1e+1i9bdz7vrV5qgv7yL/Tg2Zxm2\nx/GbJ5/u2SuYP29PuXIbx22qqMzm9ecBy4DDgM/T2bnsUvYMHldSW+uTY+bai167VxAAdh/Pu+v/\n6397dK3TBKG/b8NuCFRmP+SQi21oj2PVlXew85k9vxdz9zHOP/mIodoi8ZTpEbyDTq7/RwDu/gsz\nq3S6qOkDj6l2ESt67Yk+d4cT2V193t1jFdIEw17MQ3/fQnbiirn7GYymx9HGG7K6KRMInnJ3NzMH\nyOYPVFqTBx5HsX3fsEL3Kyj6eso0QcjFPPT3rej5eQG0X4+gX++sl9CZ7nn/b0r91EOZCWVXmNn/\nAeab2b8Hrqez3lBlNXngMeUuYkWvXTRpq8qTukImN8XeQD5vkuFEn/+T7uN5k7pSznSX6igzj+Dj\nZvZG4Nd0xgk+4u7fid6yAE0eeEy5i9goNgqp6h1iyMU89gbyRemXvP+Top5O7AlnUg958wjOAf4e\n+FF24a/0xb9bk395U+4iVoU8fiwhF/PQ37eQAFv0f1J0oU85012qI69HcDDwSeBwM9sM3EInMPx9\n1bepbPIvb+rt+6p6Rx8q5GIe+vs2iuf3+96iC33Kme5SHWU2pnkunfLR1wKvyT62u/sr4jevo24b\n08QuXW1raWxsTTyvRRvyaOe2Ziu7MU2ZqqF5wAHAgdnHL4DNYc1Lb1zbQcYoXdUdWhxNPK9lKrWg\nmb1nKS9vjGAtcATwOHArnbTQJ4r2F66DmBfrJpeuptbEO/bYJpdOMPXAo1x264PscmeOGe981d4B\nr4kBUAaTVz66mM7G8w8D08BDlNx/oOq0HWT9NH0Tk7wSz9DXvXrj9O55BbvcuXrjdGPOm4xG30Dg\n7m8BXg18PDv0QeA2M/u2mV0wjsbFEvNirbrrOJq8iUnMINfk8yajkzuhLNuA/sfAN4C/o1M59DLg\n7DG0LZqYF+u6b4kY6840VJN7WqEX65hbUUo79A0EZvZ+M/uKmf0c+C5wEnAPcCpw0JjaF0XMi3UV\nZs8OezGvcvqlyT2tkIu1ZgbLKORVDS0BrgT+wt23jKc541FmAC309Zu6Zk6qAds2TxLMo5nBMgp5\nYwQfcPermxYEoNkDaDHXzEnZY6hCTyuWkB5qmZnBTT1vMjqlNqZpmiaXeMZcMyf1eWtqmWNILb9m\nBssotDIQjGIArao17THXzCmzCYoMZ9iLtVI/MgpllqFunNABtCoPqoakGYrSCP02OxnVJigyOKV+\nZBRa2SMIvYuKnSIJ6W3EXMCsaCtKSUOpHwkVLRCY2SHAF4GFgANr3f1iMzsIuJxOVdL9wGnjXrYi\n9GIZsza7yttsFm1FWXVVTeeJpBazR/A08EF3/1G2x/FGM/sO8CfADe6+xsxWA6uBD0VsR08hd1Ex\nN74J7W3EDCR1zkdXOcCKpBZtjMDdt7j7zIb3jwN3AxPAKcC67NvWAZOx2hBLzAlpob2NmEsK1Dkf\nraUWRPobyxiBmS0BltJZxXRh19yEh+mkjno9ZyWwEmDx4sXxGzmAmEv3hvY2Yi8pUNd8tJZaEOkv\neiAws+cDVwPnuPuvravCxN3dzHqONLr7WmAtdDamid3OQcW6IIamX+q8X3PMHH6dz4tIbFHLR81s\nLp0g8GV3/2p2+BEzW5R9fRGwNWYb6iY0/VLXRe9il+SuOHzBQMdF2iRm1ZABnwXudvdPdH1pA3Am\nsCb795pYbairkN5GXXecil2Se9M92wY6LtImMVNDy4E/Bjab2e3Zsf9EJwBcYWZnAQ8Ap0VsQyvV\nMY8fO4evMQKR/qIFAnf/PtBvyumJsd63LNWUxzHseY2dw9cYgUh/rVxiospLRNRZyHktM7YRsmlO\nXcdORMahlYFANeVxhJzXyaUTvPNVE7vXLZq9R0Ro8K7zHAiR2Fq51pDyxXGE7rTVa4+IZb9zEJNL\nJ0YymFzHsRORcWhlIDhw3ly279jZ8/gotHX8IeZOW+MI3m39fxNpZWqo36rJo1hNuUwKo6obxIeK\nudNW7L13NW4kbdbYQJB3sd3+22f3BvKOD6IoT97kC05IHr7oQh97sFfjRtJmjUwNFa00GbOUsOjO\nNvV2j7EV5eH7pV+KltaIPVFO40bSZo0MBEUX2zLr+cSqh2/zBafMUtB55zzmYK/mGUibNTIQFF1s\niy46ZS5Yw97ZtvmCUxSgU1b11HmvBZFQjQwEZS62eRedogtWyJ1tmy84Ve4N1XWNJpFRaGQgWHH4\nAi79wc97Hi8jNM+fF2Qml04w9cCjXHbrg+xyf9bEqSarem9I8wykrRpZNRS60mRRBUuMiVPjqhpK\nWbqqZR5EqqmRgSA0BVF0wQqpaU9Zppi6dDV0mYemzr8QSa2RqaHQFETMPH/KPHkVSleHTb+s3zTN\nqqvuYOeuTk9qevsOVl11x+7XFJHhNTIQjGJAtijPD8MNLKbMk9d5mYYLrr1rdxCYsXOXc8G1dykQ\niARqZCAYRwXIsHe2owhS567fvNdg8+nHHsKFk0cWPi92ECpTTTWsx/rM+u53XETKa2QggOpWgIQG\nqXPXb96rImqX++7HRcEgdulqFVJPIjK4xgaCKgsJUpfd+mDf40WBoM7LNMzvs2Ls/BGtGCvSZgoE\nNTNTdlr2+GxVXqYhb3zh/JOPYNWVd7DzmT0/59x9jPNPPmI0jRdpsUaWj45CVUsV5/RZK7vf8XEK\nmSdQVNo6uXSCi/7wqL1KTy/6w6OUchIZAfUIeog56Bnq9GMP6Tlr+vRjD0nQmr2FpJ7KjC9UddxH\npO6iBQIz+xxwErDV3V+ZHTsIuBxYAtwPnObuj8Vqw7CqPOg5Mw4wTNVQGaHln8NerKu8DpFI08Xs\nEXwB+BTwxa5jq4Eb3H2Nma3OHn8oYhuGUvWL0oWTR47swt8tZU+o6usQiTRZtDECd/8e8Oisw6cA\n67LP1wGTsd4/ROxtEasq5fIXWodIJJ1xDxYvdPct2ecPAwvH/P6lVP2iFGsgO2VPKHQdIhEZXrLB\nYnd3M+tb82hmK4GVAIsXLx5bu6Daa9PHTN8c2KdW/8Ax1eprMFgkjXEHgkfMbJG7bzGzRcDWft/o\n7muBtQDLli0rVyQ/QlW9KJUZyB52wLdfBWoFKlNFJKJxB4INwJnAmuzfa8b8/rWQdyEvSt+E9Bi2\n91m3p99xEWmGaGMEZnYZ8A/AYWb2kJmdRScAvNHMfgq8IXssXYomVhUNZIcM+LZ1kFyk7WJWDZ3u\n7ovcfa67H+zun3X3X7r7ie5+qLu/wd1nVxU1QshgbtGFvGggO2TAt+qD5CISh2YWj1iZ1ExI6qdo\nIDukHr/Kg+QiEo8CwYgVDeYWBYoyF/K8gezQpaarOkguIvFo0bkh9Uv/FN3Rh6Z+iqgeX0QGpR7B\nEPLu6ovu6ENTP2Xorl5EBqFA0EdeHj/vrr4oNROa+hERGTWlhnooKuHMu6svSs2oMkdEqkY9gh6K\nBnyL7urz7uhVmSMiVaNA0ENRHl+VOSLSJEoN9VA0w1aVOSLSJAoEPSiPLyJtotRQD0V5/CrvaSwi\nMigFgj7y8vhV3tNYRGRQSg0Noep7GouIDEI9gj7yJpRpo3URaRL1CHoomlCmwWQRaRL1CHooGgMo\nM5isCWMiUhcKBD2UGQPoN5isiiIRqRulhnoI2bIxZKtIEZEUFAh6CBkDUEWRiNSNAkEPIUtIaAN4\nEakbjRH0MezCcKEL0omIjFuSQGBmbwEuBuYAl7j7mlG/R6rKHS0zLSJ1M/ZAYGZzgL8B3gg8BNxm\nZhvc/Sejeo/UlTtaZlpE6iTFGMExwD+6+33u/hTwFeCUUb6BKndERMpLEQgmgAe7Hj+UHRsZVe6I\niJRX2aohM1tpZlNmNrVt27aBnqvKHRGR8lIEgmngkK7HB2fH9uLua919mbsvW7BgwUBvoLWARETK\nS1E1dBtwqJm9hE4AeDfw70b5BqrcEREpb+yBwN2fNrP/AHyLTvno59z9rlG/jyp3RETKSTKPwN2/\nAXwjxXuLiMjeKjtYLCIi46FAICLScgoEIiItp0AgItJy5u6p21DIzLYBDwz59BcB/zTC5oyS2jYc\ntW04attw6ty233H3wolYtQgEIcxsyt2XpW5HL2rbcNS24ahtw2lD25QaEhFpOQUCEZGWa0MgWJu6\nATnUtuGobcNR24bT+LY1foxARETytaFHICIiORoVCMzsc2a21cx+3HXsIDP7jpn9NPv3hRVq2/lm\nNm1mt2cfb03UtkPM7CYz+4mZ3WVmZ2fHk5+7nLYlP3dmtq+Z/dDM7sjadkF2vArnrV/bkp+3rB1z\nzGyTmX09e5z8nOW0rRLnLGvL/Wa2OWvHVHYs+Nw1KhAAXwDeMuvYauAGdz8UuCF7nMIXeHbbAP7a\n3Y/OPlJEOedjAAAFJklEQVQtxPc08EF3fwVwHPBeM3sF1Th3/doG6c/dk8AJ7n4UcDTwFjM7jmqc\nt35tg/TnDeBs4O6ux1U4ZzNmtw2qcc5mrMjaMVM2GnzuGhUI3P17wKOzDp8CrMs+XwdMjrVRmT5t\nqwR33+LuP8o+f5zOH8EEFTh3OW1LzjueyB7OzT6capy3fm1LzswOBt4GXNJ1OPk5g75tq7rgc9eo\nQNDHQnffkn3+MLAwZWN6eJ+Z3ZmljpJ1h2eY2RJgKXArFTt3s9oGFTh3WRrhdmAr8B13r8x569M2\nSH/ePgn8JfBM17FKnDN6tw3Sn7MZDlxvZhvNbGV2LPjctSEQ7OadEqlK3BVlPg28lE7XfQvw31M2\nxsyeD1wNnOPuv+7+Wupz16NtlTh37r7L3Y+ms+XqMWb2yllfT3be+rQt6Xkzs5OAre6+sd/3pDpn\nOW2rxO9a5vjs//QP6KRJf7/7i8OeuzYEgkfMbBFA9u/WxO3Zzd0fyf5YnwE+AxyTqi1mNpfOhfbL\n7v7V7HAlzl2vtlXp3GXt2Q7cRGccqBLnrVfbKnDelgMnm9n9wFeAE8zsUqpxznq2rQLnbDd3n87+\n3Qp8LWtL8LlrQyDYAJyZfX4mcE3Ctuxl5j8v8w7gx/2+N3I7DPgscLe7f6LrS8nPXb+2VeHcmdkC\nM5uffT4PeCNwD9U4bz3blvq8ufuH3f1gd19CZ7/yG939DCpwzvq1LfU5m2Fm+5vZC2Y+B96UtSX8\n3Ll7Yz6Ay+h03XYCDwFnAf+Czkj6T4HrgYMq1LYvAZuBO7P/zEWJ2nY8ne7kncDt2cdbq3DuctqW\n/NwB/wbYlLXhx8BHsuNVOG/92pb8vHW18fXA16tyznLaVolzRic9dUf2cRfwn0d17jSzWESk5dqQ\nGhIRkRwKBCIiLadAICLScgoEIiItp0AgItJyCgTSWGb2xKzHf2Jmnyp4zslmlrtol5m9fmZlyh5f\nO8fM9st57lVm9tK815/1/Uea2RfKfr/IMBQIRLq4+wZ3XxPwEucAPQOBmR0BzHH3+wZoz2bgYDNb\nHNAmkVwKBNJK2czbq83stuxjeXZ8d6/BzF5mZj/I1n+/cFYP4/nZ3f09ZvZl63g/8GLgJjO7qcfb\n/hFdsz7N7Akzu8g6+wVcb2bHmNnNZnafmZ3c9bxr6cx0FYlCgUCabF7XZiK3Ax/t+trFdNaYfzXw\nTnovO3wxcLG7H0lnNni3pXTu/l9BZ8bncnf/H8Av6KwXv6LH6y0Huhc025/OMgZHAI8DF9JZCuId\ns9o6BbyuzA8sMoznpG6ASEQ7vLNSI9C52wdmNvN4A/CKzlJGAByQrXDa7TXsWdv9b4GPd33th+7+\nUPa6twNLgO8XtGcRsK3r8VPAN7PPNwNPuvtOM9ucvd6MrXR6GiJRKBBIW+0DHOfu/9x9sCswFHmy\n6/NdlPtb2gHs2/V4p+9Z4+WZmdd092fMrPv19s2eKxKFUkPSVt8G3jfzwMyO7vE9P6CTNoLyOfrH\ngRf0+drdwMvLNrDL75JoxUtpBwUCaav3A8uyXad+AvxZj+85B/iAmd1J5wL+qxKvuxb4Zp/B4uvo\nrGo5qBXZc0Wi0OqjIn1k8wF2uLub2buB0939lIDXm0dng5jl7r6r5HOeB3yXzs5UTw/73iJ5FAhE\n+jCz1wGfAgzYDrzH3f8x8DXfTGeTnZ+X/P5DgQl3vznkfUXyKBCIiLScxghERFpOgUBEpOUUCERE\nWk6BQESk5RQIRERaToFARKTl/j/BOa99IfbD5QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x19904c22e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Defining initial and final points\n",
    "initialPoint = (10, 12)\n",
    "finalPoint = (50,53)\n",
    "points = PointsCloud(initialPoint, finalPoint, 30, 4)\n",
    "# Defining x and y\n",
    "x = np.array([i[0] for i in points], dtype=np.float)\n",
    "y = np.array([i[1] for i in points], dtype=np.float)\n",
    "# Plotting points\n",
    "plt.scatter(x, y)\n",
    "plt.xlabel('Height (m)')\n",
    "plt.ylabel('Weight (kg)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this Problem is to **find the best line that fits the points above**. For that, we will use different approaches and compare the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='gradient_descent_keras'> </a>\n",
    "### Solution 1: Applying Gradient Descent (Keras)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could see linear regression as a 1-layer neural network with a **linear activation function ($\\sigma$)**.\n",
    "<img src='figures/lr_with_activation.png'>\n",
    "\n",
    "In this solution we will be using [Tensorflow](https://www.tensorflow.org) with [Keras](https://keras.io/).\n",
    "\n",
    "So, let's create our single-neuron model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras \n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.optimizers import SGD\n",
    "# Cleaning sessions previously started\n",
    "keras.backend.clear_session()\n",
    "# Creating our model\n",
    "model = Sequential()\n",
    "model.add(Dense(units=1, input_dim=1, activation='linear', kernel_initializer='zero'))\n",
    "# Using Gradient Descent as optimizer\n",
    "# Obs: When the batch size is equal to the amount of samples, Stochastic Gradient Descent becomes Batch Gradient Descent\n",
    "sgd = SGD(lr=0.2)\n",
    "# Adding the Mean Squared Error loss function and Gradient Descent optimizer to our model\n",
    "model.compile(optimizer=sgd, loss='mean_squared_error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once our model is created, let's train using the model.fit() function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# As we have only one sample and the gradient is updated after every batch passing through the neuron, \n",
    "# let's make our batch longer to avoid having the weights (betas) updated very often.\n",
    "batch_size = 50\n",
    "epochs = 1000\n",
    "# Train our model silently (verbose=0)\n",
    "result = model.fit(x, y, batch_size=batch_size, epochs=epochs, verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting our loss to see how it dropped throughout the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEDCAYAAAAyZm/jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEJlJREFUeJzt3H+s3XV9x/Hnay26rNQ4bUXWHyv+yIjbQNi1LoOoZJMA\nMytGnThXY4Q0JtNBhpkEE//xLzRhZokGGyHTDCQq7UKUHyuGjBGEcduVlraAtWJoU9eCLkCmg473\n/jjfurPrvb3n3Htu7718no/k5H7P5/P+nvP+5CSv873f8z0nVYUkqR2/Nt8NSJJOLoNfkhpj8EtS\nYwx+SWqMwS9JjTH4JakxCzb4k9yU5EiSRweofUeSHUmOJXl/3/gFSXb23X6R5NK57VySFrYs1Ov4\nk7wDeB74elX93jS164BXAZ8Cbq+qb09S8xpgP7C6qv5r5A1L0iKxYI/4q+o+4Kf9Y0nemOSuJNuT\n/GuSM7vaJ6tqF/DSCR7y/cCdhr6k1i3Y4J/CZuCTVfUH9I7uvzzEvpcB35iTriRpEVk63w0MKsmp\nwB8B30pyfPiVA+57OvD7wN1z050kLR6LJvjp/Xfyn1X11hns++fA1qp6ccQ9SdKis2hO9VTVs8CP\nknwAID1nD7j7h/A0jyQBAwR/kjVJ7k2yN8meJFeeoPZtk1xSeVGSx5PsT3LNoI0l+QbwfeB3khxM\ncjnwYeDyJI8Ae4ANfc97EPgA8JUke/oeZx2wBviXQZ9bkl7Opr2cszs/fnpV7UiyHNgOXFpVeyfU\nLQG2Ab8Abqqqb3djTwDvBg4CDwMfmrivJOnkmfaIv6oOV9WObvs5YB+wapLSTwK3AUf6xtYD+6vq\nQFW9ANxKd5QuSZofQ3242502OQd4aML4KuC9wAXA2/qmVgFP9d0/CLx9uudZsWJFrVu3bpjWJKlp\n27dvf7qqVg5SO3Dwd5dT3gZc1X3Q2u+LwKer6qW+Sy2HkmQTsAlg7dq1jI+Pz+hxJKlFSX48aO1A\nwZ/kFHqhf3NVbZmkZAy4tQv9FcAlSY4Bh+h9sHrc6m7sV1TVZnpf0GJsbGxh/o6EJL0MTBv86aX5\njcC+qrp+spqqOqOv/h+A71TVPyVZCrw5yRn0Av8y4C9G0bgkaWYGOeI/D9gI7E6ysxu7FlgLUFU3\nTLVjVR1L8gl635hdQu9qnz1T1UuS5t60wV9V9wMDn7ivqo9OuH8HcMfQnUmS5sSi+eauJGk0DH5J\naozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TG\nGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozB\nL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjZk2+JOsSXJvkr1J9iS5cpKa\nDUl2JdmZZDzJ+X1zTybZfXxu1AuQJA1n6QA1x4Crq2pHkuXA9iTbqmpvX833gNurqpKcBXwTOLNv\n/oKqenp0bUuSZmraI/6qOlxVO7rt54B9wKoJNc9XVXV3lwGFJGlBGuocf5J1wDnAQ5PMvTfJY8B3\ngY/1TRVwT5LtSTad4LE3daeJxo8ePTpMW5KkIQwc/ElOBW4DrqqqZyfOV9XWqjoTuBT4XN/U+VX1\nVuBi4K+SvGOyx6+qzVU1VlVjK1euHGoRkqTBDRT8SU6hF/o3V9WWE9VW1X3AG5Ks6O4f6v4eAbYC\n62fVsSRpVga5qifAjcC+qrp+ipo3dXUkORd4JfBMkmXdB8IkWQZcCDw6quYlScMb5Kqe84CNwO4k\nO7uxa4G1AFV1A/A+4CNJXgR+Dnywu8LnNGBr956wFLilqu4a8RokSUOYNvir6n4g09RcB1w3yfgB\n4OwZdydJGjm/uStJjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/\nJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtS\nYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMdMGf5I1\nSe5NsjfJniRXTlKzIcmuJDuTjCc5v2/uoiSPJ9mf5JpRL0CSNJylA9QcA66uqh1JlgPbk2yrqr19\nNd8Dbq+qSnIW8E3gzCRLgC8B7wYOAg8nuX3CvpKkk2jaI/6qOlxVO7rt54B9wKoJNc9XVXV3lwHH\nt9cD+6vqQFW9ANwKbBhV85Kk4Q11jj/JOuAc4KFJ5t6b5DHgu8DHuuFVwFN9ZQeZ8KbRt/+m7jTR\n+NGjR4dpS5I0hIGDP8mpwG3AVVX17MT5qtpaVWcClwKfG7aRqtpcVWNVNbZy5cphd5ckDWig4E9y\nCr3Qv7mqtpyotqruA96QZAVwCFjTN726G5MkzZNBruoJcCOwr6qun6LmTV0dSc4FXgk8AzwMvDnJ\nGUleAVwG3D6q5iVJwxvkqp7zgI3A7iQ7u7FrgbUAVXUD8D7gI0leBH4OfLD7sPdYkk8AdwNLgJuq\nas+I1yBJGkL+72KchWNsbKzGx8fnuw1JWjSSbK+qsUFq/eauJDXG4Jekxhj8ktQYg1+SGmPwS1Jj\nDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbg\nl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5J\naozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUmGmDP8maJPcm2ZtkT5IrJ6n5cJJdSXYneSDJ2X1zT3bj\nO5OMj3oBkqThLB2g5hhwdVXtSLIc2J5kW1Xt7av5EfDOqvpZkouBzcDb++YvqKqnR9e2JGmmpg3+\nqjoMHO62n0uyD1gF7O2reaBvlweB1SPuU5I0IkOd40+yDjgHeOgEZZcDd/bdL+CeJNuTbDrBY29K\nMp5k/OjRo8O0JUkawiCnegBIcipwG3BVVT07Rc0F9IL//L7h86vqUJLXAduSPFZV903ct6o20ztF\nxNjYWA2xBknSEAY64k9yCr3Qv7mqtkxRcxbwVWBDVT1zfLyqDnV/jwBbgfWzbVqSNHODXNUT4EZg\nX1VdP0XNWmALsLGqnugbX9Z9IEySZcCFwKOjaFySNDODnOo5D9gI7E6ysxu7FlgLUFU3AJ8FXgt8\nufc+wbGqGgNOA7Z2Y0uBW6rqrpGuQJI0lEGu6rkfyDQ1VwBXTDJ+ADj7V/eQJM0Xv7krSY0x+CWp\nMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj\n8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/\nJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTHTBn+SNUnuTbI3yZ4kV05S8+Eku5Ls\nTvJAkrP75i5K8niS/UmuGfUCJEnDWTpAzTHg6qrakWQ5sD3Jtqra21fzI+CdVfWzJBcDm4G3J1kC\nfAl4N3AQeDjJ7RP2lSSdRNMe8VfV4ara0W0/B+wDVk2oeaCqftbdfRBY3W2vB/ZX1YGqegG4Fdgw\nquYlScMb6hx/knXAOcBDJyi7HLiz214FPNU3d5AJbxp9j70pyXiS8aNHjw7TliRpCAMHf5JTgduA\nq6rq2SlqLqAX/J8etpGq2lxVY1U1tnLlymF3lyQNaJBz/CQ5hV7o31xVW6aoOQv4KnBxVT3TDR8C\n1vSVre7GJEnzZJCregLcCOyrquunqFkLbAE2VtUTfVMPA29OckaSVwCXAbfPvm1J0kwNcsR/HrAR\n2J1kZzd2LbAWoKpuAD4LvBb4cu99gmPdaZtjST4B3A0sAW6qqj0jXoMkaQjTBn9V3Q9kmporgCum\nmLsDuGNG3UmSRs5v7kpSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklq\njMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY\n/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEv\nSY2ZNviTrElyb5K9SfYkuXKSmjOTfD/Jfyf51IS5J5PsTrIzyfgom5ckDW/pADXHgKurakeS5cD2\nJNuqam9fzU+BvwYuneIxLqiqp2fZqyRpBKY94q+qw1W1o9t+DtgHrJpQc6SqHgZenJMuJUkjM8gR\n/y8lWQecAzw0xG4F3JPkf4CvVNXmKR57E7Cpu/t8kseH6W0BWAG09l+Na26Da14cfnvQwoGDP8mp\nwG3AVVX17BDNnF9Vh5K8DtiW5LGqum9iUfeGMOmbwmKQZLyqxua7j5PJNbfBNb/8DHRVT5JT6IX+\nzVW1ZZgnqKpD3d8jwFZg/bBNSpJGZ5CregLcCOyrquuHefAky7oPhEmyDLgQeHQmjUqSRmOQUz3n\nARuB3Ul2dmPXAmsBquqGJK8HxoFXAS8luQp4C73zZFt77x0sBW6pqrtGu4QFY9GeppoF19wG1/wy\nk6qa7x4kSSeR39yVpMYY/JLUGIN/CElek2Rbkh90f39zirqLkjyeZH+SayaZvzpJJVkx913PzmzX\nnOQLSR5LsivJ1iSvPnndD26A1yxJ/r6b35Xk3EH3XahmuuZBfsZloZrN69zNL0ny70m+c/K6ngNV\n5W3AG/B54Jpu+xrguklqlgA/BN4AvAJ4BHhL3/wa4G7gx8CK+V7TXK+Z3pVcS7vt6ybbf75v071m\nXc0lwJ1AgD8EHhp034V4m+WaTwfO7baXA0+83NfcN/83wC3Ad+Z7PbO5ecQ/nA3A17rtrzH5bxOt\nB/ZX1YGqegG4tdvvuL8D/pbeN5oXg1mtuar+uaqOdXUPAqvnuN+ZmO41o7v/9ep5EHh1ktMH3Hch\nmvGaa4CfcVmgZvM6k2Q18KfAV09m03PB4B/OaVV1uNv+CXDaJDWrgKf67h/sxkiyAThUVY/MaZej\nNas1T/AxekdTC80g/U9VM+jaF5rZrPmXZvgzLvNltmv+Ir2DtpfmqsGTZajf6mlBknuA108y9Zn+\nO1VVSQY+ak/yG/S+/3Dh7Docvbla84Tn+Ay9X3q9eSb7a+GZxc+4LDpJ3gMcqartSd413/3MlsE/\nQVX9yVRzSf7j+L+63b9/RyYpO0TvPP5xq7uxNwJnAI90X2hbDexIsr6qfjKyBczAHK75+GN8FHgP\n8MfVnShdYE7Y/zQ1pwyw70I0mzXP6mdc5tFs1vw+4M+SXAL8OvCqJP9YVX85h/3Onfn+kGEx3YAv\n8P8/6Pz8JDVLgQP0Qv74B0i/O0ndkyyOD3dntWbgImAvsHK+13KCNU77mtE7t9v/od+/DfN6L7Tb\nLNcc4OvAF+d7HSdrzRNq3sUi/3B33htYTDfgtcD3gB8A9wCv6cZ/C7ijr+4Selc6/BD4zBSPtViC\nf1ZrBvbTO2e6s7vdMN9rmmKdv9I/8HHg4912gC9187uBsWFe74V4m+magfPpXZywq+91vWS+1zPX\nr3PfYyz64PcnGySpMV7VI0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSY/4XhWj3Nci0GLYA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1990d0980b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(result.history[\"loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What? Something went very wrong here!** Our results show no error. Let's visualize the first 5 losses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2.2750866550544314e+17, inf, inf, inf, nan]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.history[\"loss\"][0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen above, keras only recorded the first epoch error. The other values are either 'Inf' (infinite) or 'Nan' (not a number), meaning an overflow occurred.<br>\n",
    "We did not normalize our data and sometimes the gradient explodes when we don't normalize our data. Notice that the error in our first epoch was very high (e+17) and then it went to infinite.<br>\n",
    "In order to overcome this problem let's normalize our data. Below we will repeat the whole process with our normalized data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHORJREFUeJzt3Xt8nVW95/HPr7lfmqSX9Ja0JGApBqVQaq2IgiAvKTLU\nOSPnVeagR0eHwwyMetTjKTpnzpyZcc6NcZSRQ+Ul6EFUBGS0g9XKXeS0pSmlpVeaXpPecmmT5tZc\nf/PHfqi7abqfnTYlZe3v+/XKq/tZz3r2Xms3+WZlred5trk7IiKSOcaNdQNEROTtpeAXEckwCn4R\nkQyj4BcRyTAKfhGRDKPgFxHJMAp+EZEMo+AXEckwCn4RkQyTPdYNGM7kyZO9qqpqrJshIvKOsW7d\numZ3L0+n7nkZ/FVVVdTW1o51M0RE3jHMbG+6dTXVIyKSYRT8IiIZRsEvIpJhFPwiIhlGwS8ikmEU\n/CIiGUbBLyKSYYIK/v/z3A5eerNprJshInJeCyr4/+nFnbxS1zzWzRAROa8FFfwA+vB4EZHUggp+\ns7FugYjI+S+o4BcRkXjBBb9mekREUgsq+DXTIyISL6jgB9CAX0QktaCC37S6KyISK6jgB83xi4jE\nCSr4Nd4XEYkXVPADuGb5RURSCiv4NeQXEYkVVvCLiEis4IJfi7siIqkFFfya6RERiRdU8IuISLy0\ngt/MbjSz7WZWZ2ZLh9l/iZmtMrMeM/tqUvlMM3vBzLaY2WYz++JoNn6YdpzLpxcRCUJ2XAUzywLu\nB24AGoC1Zrbc3bckVTsCfAH4xJDD+4GvuPtrZjYeWGdmzww5dlTpfvwiIqmlM+JfANS5+y537wUe\nAxYnV3D3RndfC/QNKT/o7q9Fj9uBrUDFqLR8GBrwi4jESyf4K4D6pO0GziC8zawKuAJYM9JjRURk\n9Lwti7tmVgz8HPiSux87TZ07zKzWzGqbms78A9M10SMiklo6wb8fmJm0XRmVpcXMckiE/o/d/anT\n1XP3B919vrvPLy8vT/fpT36tMzpKRCSzpBP8a4HZZlZtZrnAEmB5Ok9uidNsHgK2uvu3zryZ6dPa\nrohIarFn9bh7v5ndDawEsoCH3X2zmd0Z7V9mZtOAWqAEGDSzLwE1wGXAp4A3zOz16Cm/7u4rzkFf\ndDqniEgaYoMfIArqFUPKliU9PkRiCmio3/M2z8Do7pwiIqkFdeWuxvsiIvGCCn7QHL+ISJyggl9T\n/CIi8YIKfhERiRdc8GumR0QktcCCX3M9IiJxAgt+Le6KiMQJKvi1uCsiEi+o4E/QkF9EJJWggl8D\nfhGReEEFv4iIxAsu+LW4KyKSWlDBr8VdEZF4QQU/aMQvIhInqOA3Le+KiMQKKvhB9+MXEYkTVPBr\njl9EJF5QwQ+a4xcRiRNU8GvALyISL6jgFxGReMEFv2Z6RERSCyr4Tau7IiKxggp+0OKuiEic4IJf\nRERSCy74dQGXiEhqQQW/pvhFROKlFfxmdqOZbTezOjNbOsz+S8xslZn1mNlXR3KsiIi8vWKD38yy\ngPuBRUANcJuZ1QypdgT4AnDvGRw7ujTTIyKSUjoj/gVAnbvvcvde4DFgcXIFd29097VA30iPHU2a\n6hERiZdO8FcA9UnbDVFZOs7m2DOiAb+ISGrnzeKumd1hZrVmVtvU1HRmz6G79YiIxEon+PcDM5O2\nK6OydKR9rLs/6O7z3X1+eXl5mk8/7POc8bEiIpkgneBfC8w2s2ozywWWAMvTfP6zOXbENMcvIhIv\nO66Cu/eb2d3ASiALeNjdN5vZndH+ZWY2DagFSoBBM/sSUOPux4Y79lx1BjTHLyISJzb4Adx9BbBi\nSNmypMeHSEzjpHXsuaIBv4hIvPNmcVdERN4ewQW/1nZFRFILKvh1P34RkXhBBT9ocVdEJE5Qwa/x\nvohIvKCCH3QBl4hInLCCX0N+EZFYYQW/iIjECi74NdEjIpJaUMGvmR4RkXhBBT+gIb+ISIyggl8X\ncImIxAsq+AFcQ34RkZSCCn6N90VE4gUV/KCbtImIxAkq+DXFLyISL6jgFxGReMEFv6Z6RERSCyr4\nTcu7IiKxggp+0OmcIiJxggp+Le6KiMQLKvhBc/wiInGCC34REUlNwS8ikmGCC37N9IiIpBZU8Ovu\nnCIi8dIKfjO70cy2m1mdmS0dZr+Z2X3R/o1mNi9p35+b2WYz22RmPzWz/NHswFBa3BURSS02+M0s\nC7gfWATUALeZWc2QaouA2dHXHcAD0bEVwBeA+e7+HiALWDJqrR/a1nP1xCIiAUlnxL8AqHP3Xe7e\nCzwGLB5SZzHwiCesBsrMbHq0LxsoMLNsoBA4MEptPw0N+UVEUkkn+CuA+qTthqgsto677wfuBfYB\nB4E2d//tmTc3NU3xi4jEO6eLu2Y2gcRfA9XADKDIzG4/Td07zKzWzGqbmprO+DU1xy8iklo6wb8f\nmJm0XRmVpVPno8Bud29y9z7gKeCq4V7E3R909/nuPr+8vDzd9p9EI34RkXjpBP9aYLaZVZtZLonF\n2eVD6iwHPh2d3bOQxJTOQRJTPAvNrNAS51peD2wdxfaLiMgIZcdVcPd+M7sbWEnirJyH3X2zmd0Z\n7V8GrABuAuqALuCz0b41ZvYk8BrQD6wHHjwXHTnR3nP55CIiAYgNfgB3X0Ei3JPLliU9duCu0xz7\n18Bfn0Ub06b78YuIxAvqyl0A1+quiEhKQQW/FndFROIFFfygOX4RkThBBb8G/CIi8YIKfhERiRdc\n8GttV0QktbCCX6u7IiKxwgp+tLgrIhInqODXeF9EJF5QwQ+6gEtEJE5Qwa8pfhGReEEFv4iIxAsq\n+DXgFxGJF1Twi4hIvOCCX2u7IiKpBRX8ptVdEZFYQQU/gOsSLhGRlIIKfo33RUTiBRX8oDl+EZE4\nQQW/pvhFROIFFfwiIhIvuODXVI+ISGpBBb9peVdEJFZQwQ86nVNEJE5Ywa8Bv4hIrLCCH83xi4jE\nCSr4NeAXEYmXVvCb2Y1mtt3M6sxs6TD7zczui/ZvNLN5SfvKzOxJM9tmZlvN7AOj2YGhNOAXEUkt\nNvjNLAu4H1gE1AC3mVnNkGqLgNnR1x3AA0n7vgP8xt0vAeYCW0eh3adp67l6ZhGRcKQz4l8A1Ln7\nLnfvBR4DFg+psxh4xBNWA2VmNt3MSoEPAw8BuHuvu7eOYvtFRGSE0gn+CqA+abshKkunTjXQBPzA\nzNab2ffNrGi4FzGzO8ys1sxqm5qa0u7AKTTXIyKS0rle3M0G5gEPuPsVQCdwyhoBgLs/6O7z3X1+\neXn5Gb2YLuASEYmXTvDvB2YmbVdGZenUaQAa3H1NVP4kiV8E54wu4BIRSS2d4F8LzDazajPLBZYA\ny4fUWQ58Ojq7ZyHQ5u4H3f0QUG9mc6J61wNbRqvxQ2lxV0QkXnZcBXfvN7O7gZVAFvCwu282szuj\n/cuAFcBNQB3QBXw26Sn+E/Dj6JfGriH7Rp0u4BIRSS02+AHcfQWJcE8uW5b02IG7TnPs68D8s2hj\n2jTiFxGJF9SVuyIiEi+44NdMj4hIakEFv07nFBGJF1TwA7hWd0VEUgoq+LW4KyISL6jgB83xi4jE\nCS74RUQkteCCX1P8IiKpBRX8pkl+EZFYQQW/iIjECy74NdMjIpJaUMGviR4RkXhBBT+g1V0RkRhB\nBb/WdkVE4gUV/KA5fhGROEEFvwb8IiLxggp+ERGJF1zwa21XRCS1oIJfV+6KiMQLKvgBXMu7IiIp\nBRX8Gu+LiMQLKvhBc/wiInGCCn5N8YuIxAsq+EEjfhGROIEFv4b8IiJxAgt+ERGJk1bwm9mNZrbd\nzOrMbOkw+83M7ov2bzSzeUP2Z5nZejN7erQafjqa6RERSS02+M0sC7gfWATUALeZWc2QaouA2dHX\nHcADQ/Z/Edh61q2NocVdEZF46Yz4FwB17r7L3XuBx4DFQ+osBh7xhNVAmZlNBzCzSuDjwPdHsd2n\n5VrdFRFJKZ3grwDqk7YborJ063wb+BoweIZtTJsG/CIi8c7p4q6Z3Qw0uvu6NOreYWa1Zlbb1NR0\nLpslIpLR0gn+/cDMpO3KqCydOh8EbjGzPSSmiK4zs0eHexF3f9Dd57v7/PLy8jSbfzLN8YuIxEsn\n+NcCs82s2sxygSXA8iF1lgOfjs7uWQi0uftBd7/H3SvdvSo67nl3v300OyAiIiOTHVfB3fvN7G5g\nJZAFPOzum83szmj/MmAFcBNQB3QBnz13TY5r71i9sojIO0Ns8AO4+woS4Z5ctizpsQN3xTzHi8CL\nI27hCJiWd0VEYgV35a7uxy8iklpQwa/FXRGReEEFP2iOX0QkTlDBrxG/iEi8oIIfdJM2EZE4QQW/\nzuoREYkXVPCLiEi84IJfd+cUEUktrODXTI+ISKywgh8t7oqIxAkq+DXgFxGJF1Twgy7gEhGJE1Tw\nF+Vm09XbP9bNEBE5rwUV/KWFObR19411M0REzmtBBX9JfjbH+wbp6R8Y66aIiJy3ggr+0oIcAI51\na7pHROR0wgr+wlwA2rp7x7glIiLnr6CCf8r4PAAOH+sZ45aIiJy/ggr+aSX5ABxsOz7GLREROX+F\nFfyl+eTnjOPZLYfHuikiIuetoII/PyeLq99Vzp6WzrFuiojIeSuo4AcoH5/LkU4t7oqInE5wwT+h\nMJejXb26PbOIyGkEF/wTi3LpG3Bau3QFr4jIcIIL/poZJQCsrz86xi0RETk/BRf8V8ycQE6W8epu\nBb9IpnN3nt92WFO/Q6QV/GZ2o5ltN7M6M1s6zH4zs/ui/RvNbF5UPtPMXjCzLWa22cy+ONodGKog\nN4vLKst4dXfLuX4pETnPPbGugX/3w1qeWNcw1k05r8QGv5llAfcDi4Aa4DYzqxlSbREwO/q6A3gg\nKu8HvuLuNcBC4K5hjh11C6onsrGhje5e3axNJJM1HO0+6V9JSGfEvwCoc/dd7t4LPAYsHlJnMfCI\nJ6wGysxsursfdPfXANy9HdgKVIxi+4dvcPVE+gedR1fvPdcvFaQfrdrDrqaOsW7GqDreN3DKn/uZ\n/ud//8Bg5tzGPMP/r4dKJ/grgPqk7QZODe/YOmZWBVwBrBlpI0fqgxdNZpzBN1ds5Wdr9/HC9kb+\n7Ee1NLYPfyuH7zy7g4/f9zIvvdlE/8AgKzcfYnDQOdDazSOr9nC8b4B9LV08UZvoorvz1GsNHBpy\na4jBQad/YDDtdh7p7GXl5kPsaupgT/PZX3T24O92smZXC+v3HT3xfH0Dgxw+dpxXdx+hs6efz//z\nWl7Y1sgTtfVULf0VrV0nX/PQ0dPPX/1yM3/8vdUAbDt0jF++vp/+gUEer63nyz97HYD1+44yMOj0\n9g+yob6V3v70+r1+39FT3qMjnb3sb/3DiGxg8A8/pD9atYetB48BUNfYwSt1zXT0xN99dcuBY9z4\n7d9xtLOXHYfbufy//ZZ7nnqDlZsPAfD42nqq71lxov8Dg85g9Lr7W7u5+yevsfs0/ydHOxOnC9c1\nttPZ08+KNw5y24OrORD1YXdzJ3/1i0184/++cdItwhuOdvGztfti2z7UvpYuqpb+it9GbU9luF9m\n2w+18/y2wyfVae7oYelTbzD3b37LwKCzt6WTVTtPnR5tONp1ynO2H+9j84G2E9s9/QM8UVtPX9L/\na2//IC0dPWyob02rjwC7mjpo6ejh+W2HWbWzhav+9jmeeq2B3v5B/vczb9Lcceo9uP5x5Tb+x9Nb\n2HLg2LDP+dbHsb7Vg8b243zl8Q0cO95HXWMHjcdO/hk+0NrNc1sP05Z0VuBb/X9hW+OJ/+PvPr+D\nx2vr+YffbKOusf2U96i3f5B1e48AiV+w59sgw+IaZGafBG50989H258C3u/udyfVeRr4O3f/fbT9\nHPCX7l4bbRcDLwHfdPenTvM6d5CYJmLWrFlX7t17dqP1tXuOcOuyVaeUTy7Opbmjl9KC4T+05YJJ\nhext6eIrN1zM/3rmTQBueu80Vrwx/A/df198KZUTC9nU0Hai/icun8F7KkppbO/h9fpWXt2d+Ab4\nzpLL6eodYP2+o6zbe5SdTScHy599+EJe3tHMkgUzeWbLYV7e0cztC2cxsSiP7z6/g6sumsyC6olM\nKs7lhW2NjM/P4bLKUi6rLOXfPHBqX4f60OzJvLyj+aSym947jfF5OXz44nK6evspzM3mrp+8BsBL\nf3Et1/zjiwBcMm082w61AzBn6ni2H24nN2scvUk/7PNmlXGw7ThP/cerONLZyw9f2cP00nxKCnI4\nfOw4j9c20NbdR2FuFl3DTMPNrSzlvZWlPLp6H++rmkBX7wCbox/oX9z1QT5x/ysn6l5/yRTeW1nK\nt5/dwTUXl3Pr/Eq++3wdV100mesumcLtDyXGFx9991RmlOXzyKrhv58qJxTwhetn87O19RTmZvG5\nq6v5zA/Wntj/hevexdGuPnr6B1i/r5UdjYm/hKaW5A17M8CbL5vO0xsPnlT2wJ/MY9lLO9nQ0HZK\n/f9ycw2/r2tmcnEuF5YX83e/3sY1F5ez8MJJdPb088L2xhPvAcC9t87lN5sOkpeTxaqdLWSNM773\nqSvp7h3gJ6/u41cbDzJ7SjEXlhexcvPJty65Ze4Mlm84MOz78JZbr6zkPRWlXP/uKXzzV1v59aZD\nTC7O5Y/mVXLtnHJeerOJlZsOsaeli+xxxqUzSk7065qLy/mjeRVMLs7jPzy6jmPHE7+gb1swixtq\nptDRM0BxXhav7W2lenIR3X0DzJ5SzA//ZQ+rd7Vw9DSnYBfkZNHdl/h+eV/VBG6omcq7phQzvbSA\nRd95+US9aSX5LL5iBruaOrl8ZhkXlRfz4zV7T3zPr/n69XzzV1tZvuEA11yc6MtbvrPkcrYcOMb3\nfrfrRFn15CI+MmcKD7+ym7LCnBOniF9UXnTKz+5bPxMXlRfxn2+u4e9/vY1th9r58g0X860oF+69\ndS4XlRcxpSSfrz25ga7eAS6ZNp7Pf+hCHvr9brp7B/j3H7rwxJmJI2Vm69x9flp10wj+DwD/1d0/\nFm3fA+Duf5tU53vAi+7+02h7O3Ctux80sxzgaWClu38rnUbNnz/fa2tr06ma0vINB/jqExvSHo2K\niIylCYU5vPyX11Gclz3iY0cS/Ok8+1pgtplVA/uBJcC/HVJnOXC3mT0GvB9oi0LfgIeAremG/mi6\nZe4Mbpk7A3enp3+QbYfayR5nvLyjmV1NHRzvH2TO1GIGHSYX51HX2MHEohz+5P0XsOylnZQV5vLe\nilKe23aYG949lYUXTqK1u4/G9uNkjzMeXb2PvOxxHO8bYObEQuoaO5hUnMuOwx1cMWsC00vzuXjq\neB6vraejp59JRbm0dPZSOaGAS2eUMmtiIa/UNdPdN0BLRy+XVZbyxv42qiYXUVGWz5pdR7hoSvGJ\nhalj3X0nRuVTS/I51NZNXk4WNdNLyM/Joq27l6sumkxdYwdHoymMwtws/mVnC1WTijjcfpxLpo2n\n/kg3HT399PQNsO1QO3taOll44SSK8rKpmlSIYcydWca6vUfp6u3n5stm8OzWw6zf18rV75rELzcc\n4JNXVlI9uYjWrj5aOnowM7LGGUc6e2k8dpy8nCxe39dKSUEOn/9QNY3tidGxu7N+XyvXzinn0dX7\nGGdQlJf4rOQ500oYHHQunVHCz19roLQgl0tnlFCYm8WzWxspyc/mmjnl5GSNo6Wjh3V7j/LRmqkU\n52XzP1ds5ZNXVrK7uYvNB9r4yJwpVEwoYP2+VsqLc7lmTjlbDhyjrrGD5o5eKiYUsKBqImt2t2Bm\nTCvJZ2pJPgfbuplWms/lM8t4cXsTZtDdO8CRzl7mzizj9fpWivOyaWrvYVJRLn0Dg9Qf7WZjQysV\nEwr5V5dNp3bPUd49fTyN7T3MmljItkPt7G7uJC97HO+eXsLUknyqJhdy/wt1FOVm0z/o9A0MUlqQ\nwy1zZ7DvSBfj87PJz8li+6F2LqssY3dzJ7uaOnhf1USOHU+MPJs6ethY38ZlM0t5aXsTd1/3Lqom\nFXGks5ff1zXT0tHL5OJc/vh9M/l/Gw6Qk/WHmd3sccaAOx+ZM4UDrd00d/RwvG+Qg23HyckyBgad\n1u4+rpw1gWPH+9h84BhHOxPvW1lBDpdWlPK7N5tYUD2R+iNd9A86bx5upyAnm8b248ycWMhnrqpi\n26F2NkTv2epdLZQW5DDozodml1OQm+hfcV42184pZ8UbB7movJhppfm8UtdMWWEun7u6mmUv7WTm\nhELW7G6hckIh+TnjyMkah3tianJGWQEXlhfx4vYmSvKz2bS/jaNdfRTnZfOJKypYUD2Rv/rFJmZN\nLOSCSYVMKMzln16s495b57KjsYOe/kEOtx2nrDCHBdUTyc/J4onaeo509rFu31HmVpYyuTiPzQfa\n+NdXVJA9bhzPbDnMgbZurrxgAn0Dg/QNOBsbWpk1sZCKskIunlrM+vpWVu1s4S8+Noe+gUFe3XOE\n3v5BJhfnMXNiIWt3H6Gtu4+KCQVMKsrlygsm0Bq1+1yLHfEDmNlNwLeBLOBhd/+mmd0J4O7LooD/\nLnAj0AV81t1rzexq4GXgDeCtYffX3X1FqtcbrRG/iEimGNWpnrGg4BcRGZmRBH9wV+6KiEhqCn4R\nkQyj4BcRyTAKfhGRDKPgFxHJMAp+EZEMo+AXEckw5+V5/GbWBJzpzXomA82xtcKiPmcG9Tl8Z9Pf\nC9y9PJ2K52Xwnw0zq033IoZQqM+ZQX0O39vVX031iIhkGAW/iEiGCTH4HxzrBowB9TkzqM/he1v6\nG9wcv4iIpBbiiF9ERFIIJvjN7EYz225mdWa2dKzbM1rMbKaZvWBmW8xss5l9MSqfaGbPmNmO6N8J\nScfcE70P283sY2PX+rNjZllmtj76aM/g+2xmZWb2pJltM7OtZvaBDOjzn0ff15vM7Kdmlh9an83s\nYTNrNLNNSWUj7qOZXWlmb0T77os+B+XMuPs7/ovEB8TsBC4EcoENQM1Yt2uU+jYdmBc9Hg+8CdQA\n/wAsjcqXAn8fPa6J+p8HVEfvS9ZY9+MM+/5l4CfA09F20H0G/hn4fPQ4FygLuc9ABbAbKIi2Hwc+\nE1qfgQ8D84BNSWUj7iPwKrCQxGfI/xpYdKZtCmXEvwCoc/dd7t4LPAYsHuM2jQp3P+jur0WP24Gt\nJH5gFpMICqJ/PxE9Xgw85u497r4bqCPx/ryjmFkl8HHg+0nFwfbZzEpJBMRDAO7e6+6tBNznSDZQ\nYGbZQCFwgMD67O6/A44MKR5RH81sOlDi7qs98VvgkaRjRiyU4K8A6pO2G6KyoJhZFXAFsAaY6u4H\no12HgKnR41Dei28DX+MPH9kJYfe5GmgCfhBNb33fzIoIuM/uvh+4F9gHHCTxWd2/JeA+JxlpHyui\nx0PLz0gowR88MysGfg58yd2PJe+LRgDBnJ5lZjcDje6+7nR1QusziZHvPOABd78C6CQxBXBCaH2O\n5rUXk/ilNwMoMrPbk+uE1ufhjEUfQwn+/cDMpO3KqCwIZpZDIvR/7O5PRcWHoz//iP5tjMpDeC8+\nCNxiZntITNtdZ2aPEnafG4AGd18TbT9J4hdByH3+KLDb3ZvcvQ94CriKsPv8lpH2cX/0eGj5GQkl\n+NcCs82s2sxygSXA8jFu06iIVu4fAra6+7eSdi0H/jR6/KfAL5PKl5hZnplVA7NJLAq9Y7j7Pe5e\n6e5VJP4vn3f32wm7z4eAejObExVdD2wh4D6TmOJZaGaF0ff59STWsELu81tG1MdoWuiYmS2M3qtP\nJx0zcmO94j2KK+c3kTjjZSfwjbFuzyj262oSfwZuBF6Pvm4CJgHPATuAZ4GJScd8I3oftnMWK//n\nwxdwLX84qyfoPgOXA7XR//UvgAkZ0Oe/AbYBm4AfkTibJag+Az8lsYbRR+Ivu8+dSR+B+dH7tBP4\nLtEFuGfypSt3RUQyTChTPSIikiYFv4hIhlHwi4hkGAW/iEiGUfCLiGQYBb+ISIZR8IuIZBgFv4hI\nhvn/trJ9TE6qG2wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x19973831e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Normalizing the data\n",
    "x_norm = x\n",
    "y_norm = y\n",
    "x_norm -= x_norm.min()\n",
    "x_norm /= x_norm.max()   \n",
    "y_norm -= y_norm.min()\n",
    "y_norm /= y_norm.max()\n",
    "# Cleaning sessions previously started\n",
    "keras.backend.clear_session()\n",
    "# Creating our model\n",
    "model = Sequential()\n",
    "model.add(Dense(units=1, input_dim=1, activation='linear', kernel_initializer='zero'))\n",
    "# Using Gradient Descent as optimizer\n",
    "sgd = SGD(lr=0.2)\n",
    "# Adding the Mean Squared Error loss function and Gradient Descent optimizer to our model\n",
    "model.compile(optimizer=sgd, loss='mean_squared_error')\n",
    "# Train\n",
    "batch_size = 50\n",
    "epochs = 1000\n",
    "# Train with our normalized data\n",
    "result = model.fit(x_norm, y_norm, batch_size=batch_size, epochs=epochs, verbose=0)\n",
    "# Plot our MSE\n",
    "plt.plot(result.history[\"loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have our model trained, we'll measure our erros, see the parameters we found (beta_0 and beta_1) and plot the line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "beta_0: 0.163225\n",
      "beta_1: 0.606449 \n",
      "Least Squares Eror: 3.02362877583\n",
      "Mean Squared Eror: 0.0188976809382\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNXV+PHvYQQdFUGFGBxA0KBGJIqOoqIRNAZBo6Oo\nQU2MivE10ezhJ74xLtE3kpjXaOIWMq4xwUQwiHHBBZfXBQVE1ohBjMKAEWURZR/O74/b09MM01U1\nXX27ejmf5+HRrq7uudXTc0/d9YiqYowxxgC0S7oAxhhjiocFBWOMMWkWFIwxxqRZUDDGGJNmQcEY\nY0yaBQVjjDFpFhSMMcakWVAwxhiTZkHBGGNM2nZJF6CtunTpor169Uq6GMYYU1JmzJjxkap2DTuv\n5IJCr169mD59etLFMMaYkiIi70U5z7qPjDHGpFlQMMYYk2ZBwRhjTJoFBWOMMWkWFIwxxqRZUDDG\nGJNmQcEYY0yat6AgIneLyIciMjfL8yIivxORhSIyW0QO8VUWY4wx0fhcvHYvcCtwf5bnhwJ9Uv8G\nAHek/muMMRVv4swGbpy8gKWr1rFn52pGDdmPuv413n+ut5aCqr4IrAg45VTgfnWmAp1FpJuv8hhj\nTKmYOLOBKx6eQ8OqdSjQsGodVzw8h4kzG7z/7CTHFGqAxRmPl6SOGWNMRbtx8gLWbWrc6ti6TY3c\nOHmB959dEgPNInKxiEwXkenLly9PujjGGOPV0lXr2nQ8n5IMCg1Aj4zH3VPHtqGqY1W1VlVru3YN\n3eTPGGNK2p6dq9t0PJ+SDAqTgPNSs5COAFar6rIEy2OMMXk1cWYDA8dMoffoxxg4ZkrkMYFRQ/aj\nun3VVseq21cxash+Poq5FW+zj0RkHDAI6CIiS4CrgfYAqnon8DgwDFgIrAUu8FUWY4wptKbB4qax\ngabBYiB0FlHT80nMPhJV9f5D8qm2tlYtn4IxptgNHDOFhlbGAGo6V/Py6OMKXh4RmaGqtWHnlcRA\nszHGlJrWAkLQ8WJRcpnXjDGmWAQtMKsSobGVnpgqkUIXs00sKBhjTA7CxgxaCwhA1uPFwrqPjDEm\nB2ELzGqyTB/NdrxYWFAwxpgchC0wS3JaaRwWFIwxJgdhC8zq+tdww+n9qOlcjeBaCDec3q8g00rj\nsDEFY0xZ87Xb6Kgh+201pgDbtgTq+tfk/LOS2iXVgoIxpmzFWUAWxucCM5/lDmNBwRhTtoIGg/NR\nucZpCQTxXe4gNqZgjClbSe42Gkel7pJqjDFeJbnbaByVukuqMcZ4VarTQstyl1RjjElakruNxmG7\npLaB7ZJqTOXxOT0zqamfhRZ1l1RrKRhjiprP6ZlJTv0sVjamYIwpaj6T2Pt871JlLQVjTFHzOT3T\n99TPUuyaspaCMaao+Zye6fO9m7qmGlatQ2numoqapzkpFhSMMUXN5/TMwft3DTw+cWYDA8dMoffo\nxxg4ZkqbKvRS7Zqy7iNjTFHzOT3zubeWZz0edxC6VFdTW1AwxhQ9X3sMBVXccfcf6lTdnlXrNrV6\nvJhZ95ExpmIFjSnEvdPPloq5yFM0W1AwxlSuoPGKzju2fkef7XhLq9Zu20oIOl4sLCgYY7yLM2Dr\nU1B2tGybPUTdBKJUN+OzMQVjjFfFvmo423jF6lbGA4KOtxQlM1sxspaCMcarUp2aGfdO33I0G2NM\nK0p1amaUO/2wFcu+Zk35ZEHBGOPVnp2raWglABR733rY+ohi7xbLlQUFY4xXSfetx9l/KOhOP8k8\nyj5ZUDDGeJVkwhifd/Ol2i0WxmtQEJETgVuAKqBeVce0eL4T8ADQM1WW36jqPT7LZIwpvKT61n3e\nzZdqt1gYb7OPRKQKuA0YChwAnC0iB7Q47VJgvqoeBAwC/ldEOvgqkzGmsvi8my/V/M9hfE5JPRxY\nqKqLVHUj8CBwaotzFOgoIgLsDKwANnsskzGmgvhcQFaqU07D+Ow+qgEWZzxeAgxocc6twCRgKdAR\n+Lqqbmn5RiJyMXAxQM+ePb0U1hhTfnwPcpfilNMwSS9eGwK8CewJHAzcKiK7tDxJVceqaq2q1nbt\n2vr+58YY01K53s375LOl0AD0yHjcPXUs0wXAGFVVYKGIvAvsD7zusVzGmApSjnfzPvkMCtOAPiLS\nGxcMRgDntDjnfeB44P9EZA9gP2CRxzIZY0xkpZhjOS5vQUFVN4vIZcBk3JTUu1V1nohcknr+TuA6\n4F4RmQMIcLmqfuSrTMaY3FVaBVmuK5bDiEbdB7ZI1NbW6vTp05MuhjEVpWUFCW7Atpz75weOmdLq\nOoSaztW8PPo4/wVYuxbGj4dly+Dyy2O/nYjMUNXasPOSHmg2xpSAUt3pNI7EViy/8QZ897ts2uPz\n8K1vMf+mP3DML58uWA4K2+bCGBOqlLd0yLXbq6ArlleuhL/8Berr4c03adx+ex7f92j+3O8EXu/e\nFz7ZWLCuKwsKxphQpbqlQ5xxAe8b+anCiy+6QDB+PKxfD4ccArffzrAPaliwYevV0oXabM+6j4wx\noXxv6eArXWecbi9vaxw++AB+9SvYbz8YNAgefRQuvBBmzHD/vvMd3m4REJoUomVmLQVjTCifO50W\n806meVvjsHkzPPmkaxX84x/Q2Ahf/jL8/OcwfDjsuONWpyfZMrOgYIyJxNcisLLeyXTRIrj7brjn\nHli6FPbYA376U9cy2HffrC9LMgeFBQVjTF7kOqDreyfTgleu69fDxImuVfDss9CuHQwdCrfdBied\nBO3bh75FkjkoLCgYU2RKcZFYnC4gn3fzUSrXvH3ec+a4QPDAA7BiBfTqBdddB+efD927b3N6seZ3\ntqBgTBEp1VW0cbqAktzJNPbnvWYNPPigCwavvw4dOsDpp8NFF8Hgwa6VkOXnjnpoFpu2aPrnjnpo\nVvSf65HNPjKmiJTqIrE4XUBxZ/nEmbmU0+etCq++CiNHQrducPHF8NlncPPNbtxg3Dg4/visAQHg\nmknz0gGhyaYtyjWT5kUuuy/WUjCmiJTqIrG4XUC5dpXEvdNvrcxZjy9fDn/6E9x1F8yfDzvtBGef\n7VoFhx8OIpHLvWrdpjYdLyRrKRhTRHxmCvMpqdSUcVtWVVkq8vTxLVvgqafgrLOgpgZ+8hPYZRfX\nXbRsGfzxjzBgQJsCQrGzloIxRSTJqYhx+J4tk21QNm7LqjHLhqCfW/0h/OIXbjrpe+/B7rvDZZe5\nLqO+fXO+jia77tielWu3bRXsumP4zCTfLCgYU0SSnIoYl6/ZMkFdRHG7rWoyXt++cRPHLZzGiNmT\nOfbdN9zYwQknwK9/DaeeCttvn6crgqu/1pdR42exqbE5KLWvEq7+WnPASWoWmm2dbUyBleKU0yQF\nbWGdrWUVdaB64swGxo59nFPfeJLhc5+ly9rVLOvYhdUjzmX/K34AvXvn9Vpa/uxs3wMfW5VH3Trb\nWgrGFFCpTjlNUlAXUc4tq88+g/Hjqauvp+6ll9jcroqnvzCAZ446mWMuPZe62p75voxtBLWsfK7y\nDmNBwZgCSvKPvVSFdRFF7rZSdbkK6uvdNtWffOK2mvj1r9nuvPMYusceDM134XOU5Cw0m31kTAGV\n6pTTJMWe2bRyJdx6K/TvD7W1cN99UFfntq1+6y0YNcrtSVREkpyFZi0FY3Jw5cQ5jHttMY2qVIlw\n9oAeXF/XL/R1iW/QVoJy6iJShRdeaM5VsGEDHHoo3H67W1vQuXOBSp8b2xDPmBJy5cQ5PDD1/fTj\nRtX047DAUKpTTpMWuYto2TLXErjrLli4EDp1covLRo50LYUSkeQsNJt9ZMqWr1k++1zxeKvz26tE\neOeGYV7LZTOXWrF5MzzxhGsVPPaYy1Vw7LEuGAwfDtXWCgObfWQqnM9ZPtkWPGU73lJSWzqUnXfe\nac5VsGyZGxcYNcrlKujTJ+nSlSwLCqYs+ZzlUyWStaXgk81cwuUq+PvfXatgyhS36dywYa5VMGxY\npFwFxaJYW30WFExZ8jnL5+wBPbYaU8g87lM+rqlYK6JQs2c35ypYudItKrv+eperoKYEyt9CMbf6\nLCiYsuRzlk/TYHIus4/iiHtNcSuiggeUTz5pzlUwbVrkXAWloJhbfYFBQUR2AE4GjgH2BNYBc4HH\nVDX5jb+NycL3LJ/r6/p5CwLZKt+41xSnIipYUpimXAX19fDXv8LatXDggXDLLXDuuW5julbKVmqt\nn2Jer5I1KIjItbiA8DzwGvAhsAOwLzAmFTB+oqqzC1BOY9qkVDeWi3I3n+s1xamIgpLC5OUzbcpV\nUF8P//wn7LyzCwIXXQSHHZZ1a+pi7oYJUszrVYJaCq+r6tVZnrtJRD4H+N8gxJgcTX9vBR+sXo8C\nH6xez/T3VhR1RQHhd/NxdiKNUxF5SQrT2AjPPOMCwSOPwKZNcOSRbo3BWWe5wJCSrTVQzN0wQQbv\n37XVcanB+3dNoDRbyxoUVPWxoBeq6oe41oMxRSfOArO44nRn+OxWKJqFc++/76aR3n23+//dd4fv\nfc8tMDvggG1OD2oNJN0Nk+vv+rm3lrfpeCGFDjSLyKNAy/l3q4HpwB9UdX3Aa08EbgGqgHpVHdPK\nOYOAm4H2wEeqemzk0huTxbjXFmc97jMoTJzZsNU++Q2r1jFqfPS+d5/dCnG6n6IkhQmsIDduhEcf\nda2CyZPdsRNOgN/8Bk45JTBXQVBrIMlumDhdV0kHsyBRhu8XAZ8Cf0z9+wRYgxtb+GO2F4lIFXAb\nMBQ4ADhbRA5ocU5n4HbgFFXtC5yZwzUYs424C8xyde2j87ZKnAKwqVG59tFo8zJ8p7Ws61/Dy6OP\n490xJ/Hy6OMit2Cu/lpf2ldt3a+fmRSmqYJsWLUOpbmCfGbC8/DTn0L37nDGGTB3Lvz857BokQsO\nZ54ZmrwmqAJNKg0oxEsFWsxpV6NMST1KVQ/LePyoiExT1cNEJOibfjiwUFUXAYjIg8CpwPyMc84B\nHlbV9yHdJWVMbL4XmGW7K27tbhrIerylsLv5XDfiiyusXJkVZPXG9Zy04CW+PuspDmuYD9tt5zKX\njRwJX/0qVFVl/TmtCWoNJDmhIM7dftF05bUiSlDYWUR6NlXcItITaBoB2hjwuhogsw2/BBjQ4px9\ngfYi8jzQEbhFVe+PUnBjghyx9668/M6KVo/HFdRtkA/ZBpOTHCcJKhfA0pVr6ffBQkbMnswp81+g\n48Z1vLNbd3456EL++8FfxtqaOqwC9ZUGNEycrqtinh0XJSj8BHhJRN4BBOgNfFdEdgLuy8PPPxQ4\nHqgGXhWRqar6duZJInIxcDFAz5424cmE+/fHrd+tZTveFnG6DeJIapwk0IoV8Oc/89T9N9Png0Ws\n2257Htv/aB486KtMrzmAml135L9j5ioo1go07t1+UsEsTJSg8ATQB9g/9XgBoKq6ATdAnE0DkLnu\nv3vqWKYlwMeq+hnwmYi8CBwEbBUUVHUsMBbcLqkRymwqnM+BvKD3jjIgm6ukxkm2sWVLc66CCRNg\nwwa6fPFLXNP/Mibsdwxrtt8JyP9YSLFVoMUarOKKEhTuUtULgVkAqRbCJNzdfZBpQB8R6Y0LBiNw\nYwiZHgFuFZHtgA647qXfRi++Ma3Lx5YQ2f7Yg9571JD9tpp9BFsPyMaR1EZ8aUuXNucqeOcdl6jm\n29+GkSPZ9eCDOXhmA09PXsCnBd4SPMkVzcUYrOKKEhQaROR2Vf2uiOwKPEbArKMmqrpZRC4DJuOm\npN6tqvNE5JLU83eq6j9F5ElgNrAFN211bs5XY0qOrz/oOE37sKmGQe/t8+4xkY34WstVMGgQXHut\n24coI1dBEluCl+qK5mIWKcmOiPwa2AXX/z9GVSf4Llg2lmSnfLT8gwZXud5wer+8/EHnGnAGjpnS\nakugpnM1L48+LtZ7x1Ww2UcLF7rFZffe63IVfP7zbkdSD7kKonzePl4bRdjvuZT2XYqdZEdETs94\n+Brwc+B1QEXkdFV9OH4xTSXzvUVBrneuUcYjkuo28LkRH+vXw8MPu1bBc8+5XUhPOsntPzR0qLdc\nBXHGf3yOHYW1Qsq1lRK0eO1rGf9OBmbiVh03PTYmlmJd1VnMC4u8mDULvv992HNPtwnde+/B//yP\n24Ji0iS34thj8po4n7fP31XYLLOkZqH5FrT30QWFLIhJTlJN4E7V7VvdUK1TdbLZs4p5YVHefPIJ\njBvnWgXTp7tcBcOHu1bBoEEFzVUQ5/P2+bsKu2kp1puauLL+5kXkytTAcrbnjxMRazGUuGzbE0yc\n2XL2cP5lmzRTqMk02dT1r+GG0/tR07kawfVP52ucI1Gq8PLLcMEF0K0bXHIJbNjgchUsXQp/+Qsc\nd1zBk9fU9a9h+KE16VlUVSIMPzRa95zP31VYK6RcW5RBs4/mAP8QkfXAG8ByXD6FPsDBwDPAL72X\n0HiV5NbDq7Js/ZDtuMnRhx825yp46y23JfU3vuFaBbW1iUfhiTMbmDCjIT3dtlGVCTMaqN1rt8iB\nwcd3NawVUq4tyqDuo0eAR0SkDzAQ6IbbDO8B4GJVLe02kgGSbQIXa6KRshhAbGyEp592gWDSJJer\n4Kij3IyiM8/cKldBk6S6EYs1J0LY9OKKXbymqv8C/lWAspgEJFkxJ3mnFVQBFmslFcl77zXnKli8\nGLp0cYPII0fCF7+Y9WVJBkLfNyZxgl1YK6RSF6+ZMpZkxZzUnVZYBVjMlVSrNm50rYH6enjqKXfs\nq1+Fm25yM4c6dAh9iyQDoc8bk7Jo9RWYBYUKl3QTOIk7rbAKsGQqqfnz3ZYT998PH30EPXrAVVe5\ngeS99mrTWyXZjejzxqSkW30JiZJ5baCqvhx2zJSucmwCBwmrAIu6kvrsM/jb31yr4JVXmnMVXHQR\nnHACE2d/wI3jFrB01dw2BfgkuxF93piU67RRn6K0FH4PHBLhmDElIawC9FlJtfZzg44Dbirp9Oku\nEIwbB2vWwP77u1SW3/wmfO5zQLxWSNKJ5OPcmOS6eaFpXdA2F0cCRwFdReTHGU/tgtvgzhRQse6x\nUqzlChKlJVAUradUrgLq62H2bLf53Ne/7loFRx21zVTSOK0Q34nkfX1P4mxeaFoX1FLogMuwth0u\nK1qTT4AzfBbKbK1YB8uKtVxh8tES8BYMt2yB5593geDhh93istpauPNOGDECOnXK+tJS3UMo6nu0\n9nmHBcKkx8xKUdA6hReAF0TkXlV9r4BlMi0kPViW6x9kMYvbXZFrJZctJ0K3T1fAL3/pBo4XLdoq\nVwEHHxypXHG6Snx2s8T9ngR93sW8eWGpijKmsL2IjAV6ZZ6vqvH3pTWRJDlYFvcPshzFqeQycyJs\n17iZwYumc9bspzh+0XTXShg8GK67Dk47batcBVGU6h5CYYI+bxszyL8oQeEh4E6gHmgMOdd4kOQX\n3/4gtxWnkru+rh+7LXuf6vvvY/icZ/jcZytZs2sX2l1+uctV8IUv5FyuOF0lPrtZ4n5Pgj7v3379\nYBszyLMoQWGzqt7hvSQmqyQHy+wPcls5VXLr1qVzFfz4+eehqgqGDYOLLqLjsGFuamkexOkqSWoP\noTBBn7eNGeRf0Oyj3VL/+6iIfBf4O7Ch6XlVXeG5bCYlyS++/UFuq02V3KxZbtD4gQdg1SrYe283\ndvCtb7n8BRUg7vck7PO2MYP8Cro9mQEo0DTvbVTGcwrs7atQprCCZtKU6x9k3P1wIKCSW73arSe4\n6y63vmD77ZtzFRx7bMG3pi4GcVswUHk3H0mJlKO5mFRijmafuYyjvHcprkUI4uXzbMpVUF/vVhyv\nWwf9+rkZROeeC7vtFv4exngUNUdzaFBokau5yWpgjqp+mGP5claJQcFncnLfic+LUV6v+cMP3d5D\n9fWwYAF07AjnnOOmkhZBroK2KLfgb7YWNShEGd0aCRwJPJd6PAjXtdRbRH6hqn/KuZQmEp9TP3Pa\ndqHExf48GxvdbqRNuQo2b4aBA2H0aJerYKed8ljawvC5wMz3a01+RQkK2wFfVNX/AIjIHsD9wADg\nRcCCQh4ktX9LtsVUVSV0h9tWOX+e//53c66CJUtcroIf/CA0V0Ex8bUQMU5QKdWV8eUqyohXj6aA\nkPJh6tgKwPIm5kFYnuRRQ/ajuv3W203la+pnawEh6Hg5aNPnuWEDPPQQDBniZg5ddx0ceCCMHw8N\nDW5TuhIKCNm+Zz4XmPl8rcm/KC2F50XkH7hFbADDU8d2AlZ5K1kFSXL/lposd801eVqAVoxdCpE+\nz5a5Cnr2hKuvdrkKevaMXYYk+FyIGBZUgn6XlboyvlhFCQqX4gLBwNTj+4EJ6kaoB/sqWCVJcv8W\nnwvjirlLodXP89NPm3MVvPoqtG/fnKvgK19xC85KmM+FiEFBZeLMBkaNn8WmRtf6bFi1jlHjZwF4\nT2pk2i60+0id8ar6o9S/8Vpq81iLXLYvf6ESnNxwej9qOlcjuBZCPqa6QrRugYkzGxg4Zgq9Rz/G\nwDFT0l1mBetSUIXXX4eLL4Zu3dz4wMqVrltoyZLmrqMSDwgQ/D2L+z3Ilndh8P5dufbReemA0GRT\no3Lto/MAv92jpu2CVjS/pKpHi8ga3GK19FO4WLGL99JViKT3fPfVCgmb2ZToZnsff+xWGd91F8yZ\nAzvu2Jyr4MgjS2oqaaakFiIG5WNYubb1ocem47Y4rbgEbZ19dOq/HbOdY/LD9x9FUtP9wmY2FXyz\nvS1b4LnnmnMVbNwIhx0Gf/iDy1WwS/Hf5wT9LsO6aeJ+z3yOC5TqyvhyFGkXLhE5GuijqveISBeg\no6q+67dolcXXH4XvvvmgiiJsZlPBNttraIB773WtgnffhV13hf/6L9dVdNBBbX8/z7J9pmG/y6Bu\nmqbfSa7fs7CfHRTEP9uwmVXrtm0tdK5u3+ZyGP9CxxRE5GrgcuCK1KEOwANR3lxEThSRBSKyUERG\nB5x3mIhsFhHvGd2y9WGXuiT65sOm0mabwdR03GcfN5s2wSOPwNe+5mYLXXkl9O4Nf/kLLF0Kv/ud\nt4AQ5zsW9JmG/S7DumniCPvZQeMC15zSl/bttu6Oa99OuOaUvrHLZfIvSkvhNKA/8AaAqi4VkdAu\nJRGpAm4DTgCWANNEZJKqzm/lvF8BT7Wx7G1WrotkkuqbD5tKG9aH7aWP+1//ci2C++6DDz5wg8ej\nR7tcBfvsE/ltcu1yi/sdC/pMk5y6Gfazo3RN2ZhBaYgSFDaqqoqIAqTWJ0RxOLBQVRelXvcgcCow\nv8V53wMmAIdFfN+clXL6yCBJJcKJW1HkbSxl3TqYMMGNFbzwgpspdNJJbtB46NA25yqIU7HH/Y4F\nfaZhv8vO1e1Du2nCxiTirKoPCuI2ZlA6ovy1/E1E/gB0FpFvAxcCf4zwuhpgccbjJbitMdJEpAbX\nEhlMQFAQkYuBiwF6xlg4VK6LZJJKhBO3oojyfKCZM12r4IEH3HbV++yTl1wFcSr2uN+xoM80rGV1\nzSl9GfXQLDZtaR5XyOymCQp2QGAgTHqGnCmcKOsUfgOMx93N7wdcpaq/z9PPvxm4XFW3hJRhrKrW\nqmpt166tz4eOIsn1AD557ZsPkMj88tWr4Y474NBD4ZBDXOvg5JPdrKK334YrroidvCZOxR73Oxb0\nmYb9Luv613DjmQdt9fyNZx60VYssW7ALGzPw+T0yxSVoncIPgVeAN1T1aeDpNr53A9Aj43H31LFM\ntcCD4qYodgGGichmVZ3Yxp8VSbne7SSVCKdg88tV4aWXXAB46CHXXfSlL8Hvf+9yFey6a15/XJwu\nt7jfsShdbrm2vHIJdoVYVW+KS1D3UXfcnfz+IjIHeBkXJF6JmIpzGtBHRHrjgsEI4JzME1S1d9P/\ni8i9wD98BQRIfpFMonv5eOK1ovjPf5pzFbz9tstVcN55bqzg0EO9LTCLU7Hn43fh6zMNC3a21YSB\naEl2OuDu6I/C5VU4ElilqgeEvrnIMFxgqQLuVtX/EZFLAFT1zhbn3osLCuOD3tN3kh1fFbfP7Gll\npbERJk92YwVNuQqOPtoFgjPOKFiugnLc3z/oOwjY97PM5TPJTjWwC9Ap9W8pMCfwFSmq+jjweItj\nd2Y59/wo7+mTzymr5TrzKW/+/W+Xp+Cee9yeQ127wg9/6BaY7b9/qy/xWXGXY1dJXf8apr+3gnGv\nLaZRlSoRhh+69XWWWyA0bRc0pjAW6AusAV7DdR3dpKorC1S2gvNZcZfrzKdYNmxwC8zq6+GZZ9yx\nIUPg5pvdorMOHbK+tFzXnIDf1uqEGQ3pFeWNqkyY0UDtXrulg2Cpf3YmvqDZRz2B7YEPcGMCSyjz\n/Ak+K+5ynfmUk3nz4Ec/gpoatwndggVwzTWutfDEEzB8eGBAgPJNzBK2SjyOcv3MTH4FbYh3orhp\nQX1x4wk/AQ4UkRXAq6p6dYHKWDA+F3qV8synvNy5fvop/PWvrlUwdarLVVBX58YKjj++zVtTl2vL\nKx9pMS2ZjYkjcEwhlTdhroisAlan/p2MW61cdkHBZ8Wd5AyhuNnPcu6macpVUF8PDz7oAsMXvwj/\n+7/wzW+6cYMclWtiljgVd5xN64xpEjSm8H1cC+EoXC7mV1L/7ibiQHOpiTIQF/f9C91n63Mvnqad\nO7cJOD13cKuM6+th7lwvuQpKueUVJE7FHXcvKmMguKXQC5eX+UequqwwxUlW2EBcKfK5F09mwBHd\nQq83X6XDn66l8V9Tqdq0EQ4/HMaOdQEhz7kKkl5z4kucijsfm9YZEzSm8ONCFqQYlOO0UZ978dw4\neQGdPv4PI+c+w9dnP02P1f9h1Q47M+GwkzjrjmvcqmOPynG2TJyKOx97URnTtu0jy1zcCrQYFzzF\n7Udu7c61YzvlpvaL+PSOsQxaNIMq3cJLex3EjV8+j8n7HsmG7TpwlueAUM5yrbite8jkgwWFDHEq\n0GKdN5/PvXi2X7SQCxdM4Yx5U9jh4+V8sPPu3H7EmfztSyewuPPn06+pKtH8xqXOuodMPlhQyBCn\nAvXd9ZRrKyR2RbF2LXVzp1D3RD28+KKbOnryyXDRRQx8cQuN7badSpotDafxz7qHTFwWFDLEqUB9\nzgFPpBXKQZcHAAAReUlEQVQyc6abPfTnPzfnKrjhBperoFs3AD4/d0qrLatsaTiLSTF29RlTDCwo\ntJDrnZbPOeBxWiFtCiirVrkcxnfdBW+8ATvs4DahGzkSjj12m6mkpdqHXaxdfcYUg9AkOyYanwln\n4rRCQrc2UHXdQued51oAl17qdiq99VaX4P5Pf4JBg1pdW1CqiVdsuwdjsrOWQp74HOSL0wrJFjg2\nLlkKv/61axW8/bZbR3D++W6B2SGHRF5gVop92LbdgzHZWVDII18VZJxumsyAUrWlkS+/+wYjZk3m\n+HemwZZGOOYY+NnPXDfRjjvmvexx+Or375QlwX2njAT3xlQqCwolIE4rZNSQ/bj1rqc55Y0nOXPO\nM3T79GM+3rET737j2/S54gdZcxUkzWe/f7ZGkM2kNcaCQslocytkwwaYOJG6+nrqnnmGLSK80PsQ\nfn/q9xjw/fM49fDe4e+RIJ9TfFet3baVEHTcmEpiQaHczJ3rxgnuvx9WrIC99oJrr6XdBRcwuEcP\nBiddvoh857aw3UKNaZ0FhRZKcv76mjXNuQpee83lKjjttOZcBe2Sm2SW6+cZpeLO9b1LdSqtMYVg\nQSFDSc1fV3UBoClXwWefwQEHwE03wTe+EStXQb7E+TxHDdmPUQ/NYtOW5tXR7dtJuuKO8962HYQx\n2VlQyFASu6R+9JFbO3DXXS6t5U47NecqOOKIohotjf15tryUjMdx37sUp9IaUwgWFDK01l0RdLyt\ncu6a2rIFnn3WtQomToSNG2HAAPjjH11A6NgxL+XLt7iL7jY1br2H0qZGTVf6vtcalGQ3ojF5YEEh\nQ5VIq5u55WPXz7DujlYroa4K99wDd9/tktrvtht85ztu24l+/WKXyTcfi+6ajvscLC6pbkRj8qzi\ntrmYOLOBgWOm0Hv0YwwcM4WJMxvSz2Xb3TMfu34GdXc0VUINq9ZR1biZA19/ll3POg3day+46iq3\nGd24cdDQADffXBIBAeJt/ZGtcm867nNbEdsGw1SyimophN0B1mS5+8zHrp9Bd743Tl5Atw/e46zZ\nTzF87hS6rl3FBzvvxn3HnsP59dfC3nvH/vlJiDKgm62bJmyGkM/BYtsGw1SyigoKcRObx+lnbq27\nY4dN6zl38TSGvPoPDl8yj83Sjme/cDgPHjSEF3sfwpZ2VZxfogGhSdCAbpRumqDP29dgsa1jMJWs\nooJCnMTmUSqwoKCRGXD6frCQEbOfom7+83TcsJbFu9cw5tjzmXDg8Szfedd0uUohL0EcYUE6qRlC\nto7BVLKKCgpxEpuHVWBhQaOu14703jSD6vvvZd9lC9mwXQc+/OrJdPx/32dGx3247+9zK64SKtZu\nGlvHYCpZRQWFwft35YGp77d6PExYBdZq0Ni4mafv+Bt1696A8eM5aP16OPhguPI2tj/nHHp07gxA\nHTD9/ZWMe20xjapUiTD80PKfR1/M3TS2jsFUKq9BQUROBG4BqoB6VR3T4vlzgctxy5LWAN9R1Vm+\nyvPcW8vbdDxTWAWWGTS6frqS4XOf5azZT7H3yqUuV8EFFzTnKmhh4swGJsxoSM9yalRlwowGavfa\nrSAVU1Jz8q2bxpji4y0oiEgVcBtwArAEmCYik1R1fsZp7wLHqupKERkKjAUG+CpTnO6KsAqsR8cO\n7PPmyy5XwcLX2U638Fr3vvz5K9/k5/deFZirIMmV1EnOybduGmOKj8+WwuHAQlVdBCAiDwKnAumg\noKqvZJw/FejusTyxuiuyVmCdNsCVVzJ5bD3Vy//D8h07U3/4afyt3wks+/xe3HB6v9DkNUn2rSe9\ntUecbporJ87Zqsvt7AE9uL6uNNZwGFOsfAaFGmBxxuMlBLcCRgJPeCxP7O6KdAW2fr3bbmLUL932\nE+3aUX3iibx23GmMWt+TxWs2sWfnam7I446gvhTrYG+YKyfO2Wp8qFE1/dgCgzG5K4qBZhEZjAsK\nR2d5/mLgYoCePXvm/HNid1fMmeP2H3rggeZcBb/4hctt3KMHA4AXcyhXPvrWfW5RHYev8Ypxry3O\netyCgjG58xkUGoAeGY+7p45tRUS+BNQDQ1X149beSFXH4sYbqK2tjbXnRJu7K9ascVtT19fD669D\nhw5QV5fXXAVxg9XEmQ2MGj8rvYFcw6p1jBo/a6v3zsbnYK/P8QqfW5IYU8l8BoVpQB8R6Y0LBiOA\nczJPEJGewMPAN1X1bY9laRtVmDrVBYK//rU5V8Fvf+tyFXTpkvcfGadv/dpH57W6o+i1j85LNLeA\nz/EKn5sXGlPJvAUFVd0sIpcBk3FTUu9W1Xkicknq+TuBq4DdgdvF/TFvVtVaX2UK1ZSroL4e5s93\nuQpGjHCtggEDiipXQaaVWXILZzvekq85+XHHK4K6ns4e0KPVNSdnD+ixzTFjTHRexxRU9XHg8RbH\n7sz4/4uAi3yWIdSWLfDMM825CjZtcslq6uvhrLO2ylVge+y3TZzxirCup6ZxA5t9ZEx+FcVAcyIW\nL27OVfDeey5XwaWXulwFBx64zenFvMd+5+r2rFq3baugc3X7BErTLM54RZSup+vr+lkQMCbPKi6f\nAuAq/732gquvhj593EDy0qVuzKCVgADFvcf+Naf0pX27rbu22rcTrjmlb17ePygHRZC6/jXccHo/\najpXI7gN/m44vV+kIFqqU2WNKXWV2VLo3x9+9jO48ELo3TvSS4q5kvI5WBy3hZTreEUx74tkTDmr\nzKBwUduHMYq9kvI1WJzUimfbF8mYZFRm91EOfKZ/LGZJtZDidD0ZY3JXmS2FHBT75m2+ZkZ1yjKI\n3akAg9i2fbUxhWdBoQ2KtZIK6/ePEzCyLc0o0iUbxpiYLCiUiKCKPWxmVJyB4lVZFsBlO26MKW02\nplACmloCDavWoTRX7E1TQ4P6/eNOpc02kF4sA+zGmPyyoFBAQfP9g54Lq9iDKu64A8WVOsBuTKWy\noFAgQXf7cVoCEFxxx73Tt1lAxlQWG1MokLC7/aC1AGFrJMJmRsWd71+sA+zGmPyzoJBHQYPBuXTj\nZLYEwir2bBV3sU+lNcYUFwsKeRI2LTTsbj9OSyCM3ekbY6KyoNAGuU4LretfE3q3n2tLwBhj8smC\nQkRhLYGw7qEod/vWxWOMSZoFhYjCWgJRNswLutu3loAxphjYlNSI4kwLNcaYUmFBIaKw+f51/WsY\nfmhNOnF8lQjDD7W7f2NMabGgEFFYS2DizAYmzGigURWARlUmzGiInKXMGGOKgQWFiMJW9hZzuk5j\njInKBprbIGgwuJjTdRpjTFQWFNogaJ1CsafrNMaYKKz7KKKwTets9pExphxYSyGisHUKURan+UqZ\naYwx+WJBIaIoYwZBYw5hK6KNMaYYWPdRRHHzEtjsJGNMKbCgEFHcMQObnWSMKQUWFCKKm4HMch0b\nY0qBjSm0QZxN66IkyjHGmKR5bSmIyIkiskBEForI6FaeFxH5Xer52SJyiM/ygBvwHThmCr1HP8bA\nMVMKtg2F5To2xpQCby0FEakCbgNOAJYA00RkkqrOzzhtKNAn9W8AcEfqv14kPQPItsc2xhQ7ny2F\nw4GFqrpIVTcCDwKntjjnVOB+daYCnUWkm68C2QwgY4wJ5jMo1ACLMx4vSR1r6zmIyMUiMl1Epi9f\nvjznAtkMIGOMCVYSs49Udayq1qpqbdeuXXN+H5sBZIwxwXwGhQagR8bj7qljbT0nb2x/ImOMCeYz\nKEwD+ohIbxHpAIwAJrU4ZxJwXmoW0hHAalVd5qtANgPIGGOCeZt9pKqbReQyYDJQBdytqvNE5JLU\n83cCjwPDgIXAWuACX+VpYjOAjDEmO6+L11T1cVzFn3nszoz/V+BSn2UwxhgTXUkMNBtjjCkMCwrG\nGGPSLCgYY4xJs6BgjDEmzYKCMcaYNAsKxhhj0iwoGGOMSRO3VKB0iMhy4L08vFUX4KM8vE+psOst\nb5V0vZV0rZC/691LVUM3jyu5oJAvIjJdVWuTLkeh2PWWt0q63kq6Vij89Vr3kTHGmDQLCsYYY9Iq\nOSiMTboABWbXW94q6Xor6VqhwNdbsWMKxhhjtlXJLQVjjDEtlH1QEJETRWSBiCwUkdGtPC8i8rvU\n87NF5JAkypkvEa733NR1zhGRV0TkoCTKmQ9h15px3mEisllEzihk+fItyvWKyCAReVNE5onIC4Uu\nYz5F+C53EpFHRWRW6nq952PxRUTuFpEPRWRulucLV0+patn+wyX3eQfYG+gAzAIOaHHOMOAJQIAj\ngNeSLrfn6z0K2DX1/0NL9XqjXGvGeVNweT3OSLrcnn+3nYH5QM/U488lXW7P1/vfwK9S/98VWAF0\nSLrsOV7vl4FDgLlZni9YPVXuLYXDgYWqukhVNwIPAqe2OOdU4H51pgKdRaRboQuaJ6HXq6qvqOrK\n1MOpuLzYpSjK7xbge8AE4MNCFs6DKNd7DvCwqr4PoKqlfM1RrleBjiIiwM64oLC5sMXMD1V9EVf+\nbApWT5V7UKgBFmc8XpI61tZzSkVbr2Uk7u6jFIVeq4jUAKcBdxSwXL5E+d3uC+wqIs+LyAwROa9g\npcu/KNd7K/BFYCkwB/iBqm4pTPEKrmD1lNd0nKZ4ichgXFA4OumyeHQzcLmqbnE3k2VvO+BQ4Hig\nGnhVRKaq6tvJFsubIcCbwHHAPsDTIvJ/qvpJssUqbeUeFBqAHhmPu6eOtfWcUhHpWkTkS0A9MFRV\nPy5Q2fItyrXWAg+mAkIXYJiIbFbViYUpYl5Fud4lwMeq+hnwmYi8CBwElGJQiHK9FwBj1HW6LxSR\nd4H9gdcLU8SCKlg9Ve7dR9OAPiLSW0Q6ACOASS3OmQSclxrdPwJYrarLCl3QPAm9XhHpCTwMfLPE\n7yBDr1VVe6tqL1XtBYwHvluiAQGifZcfAY4Wke1EZEdgAPDPApczX6Jc7/u4VhEisgewH7CooKUs\nnILVU2XdUlDVzSJyGTAZN5vhblWdJyKXpJ6/EzcrZRiwEFiLu/soSRGv9ypgd+D21B30Zi3BzcUi\nXmvZiHK9qvpPEXkSmA1sAepVtdUpjsUu4u/3OuBeEZmDm5VzuaqW5O6pIjIOGAR0EZElwNVAeyh8\nPWUrmo0xxqSVe/eRMcaYNrCgYIwxJs2CgjHGmDQLCsYYY9IsKBhjjEmzoGAqgoh82uLx+SJya8hr\nTgnafTV1ziAR+UeW536YWi+Q7bXjRWTvoPdvcX4/Ebk36vnG5MKCgjFZqOokVR0T4y1+CLQaFESk\nL1ClqpEXW6nqHKB7agGiMV5YUDAVT0S6isgEEZmW+jcwdTzdmhCRfURkaioPxfUtWh47p+763xKR\nP6dWnX4f2BN4TkSea+XHnotbgdxUhk9F5MZUXoBnROTw1MZ2i0TklIzXPYpb3WuMFxYUTKWoTiWf\neVNE3gR+kfHcLcBvVfUwYDhuX6iWbgFuUdV+uD2GMvXHtQoOwO3/P1BVf4fbvXOwqg5u5f0GAjMy\nHu8ETFHVvsAa4HrgBNwur5llnQ4cE+WCjclFWW9zYUyGdap6cNMDETkft2EewFeAAzJ2Ut1FRHZu\n8fojgbrU//8F+E3Gc6+r6pLU+74J9AJeCilPN2B5xuONwJOp/58DbFDVTaktHHplnPchrgVijBcW\nFIxxLeYjVHV95sE2bLe9IeP/G4n2d7UO2CHj8SZt3nNmS9N7prb9zny/HVKvNcYL6z4yBp7CZWgD\nQEQObuWcqbiuJYjep78G6JjluX8CX4hawAz7AiW5yZ0pDRYUjIHvA7WphOjzgUtaOeeHwI9FZDau\nMl8d4X3HAk9mGWh+DLcrZlsNTr3WGC9sl1RjIkitN1inqioiI4CzVbW1nNBR368aeA43KN0Y8TXb\nAy8AR6tqSeYiNsXPgoIxEYjIMbicwAKsAi5U1YUx33MI8E9VfT/i+X2AGlV9Ps7PNSaIBQVjjDFp\nNqZgjDEmzYKCMcaYNAsKxhhj0iwoGGOMSbOgYIwxJs2CgjHGmLT/D+KI8gyLI8I+AAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x199770881d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get beta_0 and beta_1\n",
    "[beta_1, beta_0] = model.get_weights()\n",
    "beta_1 = beta_1[0] #Keras returns our beta_1 as a vector\n",
    "print(\"beta_0: %s\" % beta_0[0])\n",
    "print(\"beta_1: %s \" % beta_1[0])\n",
    "# Measure errors\n",
    "mse = model.evaluate(x_norm, y_norm, verbose = 0)\n",
    "# mse = np.mean(np.square(y_norm - (beta_0 + np.multiply(beta_1,x_norm)))) could also be calculated directly with our equation\n",
    "lse = np.sum(np.square(y_norm - (beta_0 + np.multiply(beta_1,x_norm))))\n",
    "print('Least Squares Eror: %s' % lse)\n",
    "print('Mean Squared Eror: %s' % mse)\n",
    "# Plotting\n",
    "y_min = beta_0 + (beta_1*np.min(x_norm))\n",
    "y_max = beta_0 + (beta_1*np.max(x_norm))\n",
    "plt.scatter(x_norm, y_norm)\n",
    "plt.xlabel('Height (m)')\n",
    "plt.ylabel('Weight (kg)')\n",
    "plt.plot([np.min(x_norm),np.max(x_norm)], [y_min,y_max],c='r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I encourage you to train with different epochs. Notice the more we train, the less our errors get.<p>\n",
    "If you train with 10000 epochs, you will obtain the following results: <br>\n",
    "beta_0: 0.16777468<br>\n",
    "beta_1: 0.61720169<br>\n",
    "Least Squares Eror: 2.99910858853<br>\n",
    "Mean Squared Eror: 0.0187444284558"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='gradient_descent_tensorflow'> </a>\n",
    "### Solution 2: Applying Gradient Descent (Tensorflow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find our solution directly with Tensorflow. First let's import the packages we will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "# This is a .py file to with useful functions\n",
    "from utils import show_graph\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import numpy.random as rand\n",
    "# To enable reproducibility\n",
    "rand.seed(26051984)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining our graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "# Defining variables to be updated during training\n",
    "betas = tf.Variable(np.zeros((2,1), np.float64)) # initializing beta_0 as 0 and beta_1 as 0\n",
    "Xb = np.hstack([np.ones(shape=(len(x_norm),1)), np.reshape(x_norm, (len(x_norm),1))])\n",
    "Yb = np.reshape(y_norm,(len(y_norm),1))\n",
    "# Input is constant (it will never be changed)\n",
    "x_const = tf.constant(Xb)\n",
    "# Output is also a constant\n",
    "y_const = tf.constant(Yb)\n",
    "# Defining our output\n",
    "output = tf.matmul(x_const,betas)\n",
    "error = output-y_const\n",
    "mse = tf.reduce_mean(tf.square(error))\n",
    "lse = tf.reduce_sum(tf.square(error))\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run our graph and plot our results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 MSE = 0.170857409193 LSE = 27.3371854708\n",
      "beta_0: 0.173145229628\n",
      "beta_1: 0.611950453073 \n"
     ]
    }
   ],
   "source": [
    "num_epochs = 1000\n",
    "learning_rate = 0.2\n",
    "\n",
    "# Creating our session\n",
    "with tf.Session() as session:\n",
    "    # Initializing our variables\n",
    "    session.run(tf.global_variables_initializer())\n",
    "    for epoch in range(num_epochs):\n",
    "        # Run our graph\n",
    "        out = session.run(optimizer)\n",
    "        best_betas = betas.eval()\n",
    "        # Calculate error every 1000 epochs\n",
    "        if epoch % 1000 == 0:\n",
    "            print(\"Epoch %s MSE = %s LSE = %s\" %(epoch, mse.eval(), lse.eval()))\n",
    "\n",
    "# Getting beta 0 and beta 1\n",
    "beta_0 = best_betas[0][0]\n",
    "beta_1 = best_betas[1][0]\n",
    "# Printing results\n",
    "print(\"beta_0: %s\" % beta_0)\n",
    "print(\"beta_1: %s \" % beta_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we make our training with 10000 epochs, we will notice our errors will drop significantly. With 10000 epochs, these are the results we will get:<br>\n",
    "Epoch 9000 MSE = 0.0187344786198 LSE = 2.99751657917<br>\n",
    "beta_0: 0.173145032501<br>\n",
    "beta_1: 0.6119508201"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='least-squares-error-minimization'></a>\n",
    "### Solution 3: Least Squares Error minimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now it is time to compare our results above with our analytic solution. You can check the theory at [Linear Regression Theory](Linear_Regression_Theory.ipynb).<br>\n",
    "The line that best fits our model is given by:\n",
    "\n",
    "$$\\begin{align*} \n",
    "\\boxed{y = \\beta_0 + \\beta_1 x \\text{ , where }\n",
    "            \\left\\{\n",
    "                \\begin{array}{ll}\n",
    "                  \\beta_0 = \\bar{y} - \\beta_1\\bar{x}\\\\\n",
    "                  \\beta_1 = \\frac{\\sum_{i=1}^N \\left(x_i - \\bar{x}\\right)\\left(y_i - \\bar{y}\\right)}{\\sum_{i=1}^N \\left(x_i - \\bar{x}\\right)^2}\n",
    "                \\end{array}\\right.}\n",
    "\\end{align*} $$\n",
    "\n",
    "Using NumPy, let's find $\\beta_0$ and $\\beta_1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "beta_0: 0.173145032501\n",
      "beta_1: 0.6119508201 \n"
     ]
    }
   ],
   "source": [
    "# Finding y and x means\n",
    "y_mean = np.mean(y_norm)\n",
    "x_mean = np.mean(x_norm)\n",
    "# Finding beta_1 and beta_0\n",
    "beta_1 = np.sum((x_norm-x_mean)*(y_norm-y_mean))/np.sum(np.square(x_norm-x_mean))\n",
    "beta_0 = y_mean - (beta_1*x_mean)\n",
    "print(\"beta_0: %s\" % beta_0)\n",
    "print(\"beta_1: %s \" % beta_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The equation of our line is: $$y = 0.1731 + 0.6119 x$$\n",
    "Let's plot our line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNXV+PHvcRgUN9BAXAYQoqBCUNFRUNQISlBUQNxQ\nE+OevL5mD4/4i4kak0h+JnElGkcRlyiJYBACSjRoVAwKiIAoRFxhcEFZVBhhGM77x+1pe4bpqpqu\nrq7q7vN5Hh7t6pruWz0999TdzhVVxRhjjAHYLu4CGGOMSQ4LCsYYY9IsKBhjjEmzoGCMMSbNgoIx\nxpg0CwrGGGPSLCgYY4xJs6BgjDEmzYKCMcaYtDZxF6C1OnbsqN26dYu7GMYYU1Tmz5//sap28juv\n6IJCt27dmDdvXtzFMMaYoiIi7wY5z7qPjDHGpFlQMMYYk2ZBwRhjTJoFBWOMMWkWFIwxxqRZUDDG\nGJNmQcEYY0xaZEFBRMaLyEci8mqW50VEbhWR5SKySEQOjaosxhhjgoly8doE4Hbg/izPnwT0SP3r\nB9yR+q8xxpS9KQtquXHmMlatq2PvDu0YPWR/RvStivx9I2spqOqzwBqPU4YD96szB+ggIntFVR5j\njCkWUxbUctWji6ldV4cCtevquOrRxUxZUBv5e8c5plAFrMh4vDJ1zBhjytqNM5dRV9/Q5FhdfQM3\nzlwW+XsXxUCziFwmIvNEZN7q1avjLo4xxkRq1bq6Vh3PpziDQi3QJeNx59SxbajqXapararVnTr5\nJvkzxpiitneHdq06nk9xBoWpwPmpWUj9gfWq+n6M5THGmLyasqCWAWNn0X3MdAaMnRV4TGD0kP1p\nV1nR5Fi7ygpGD9k/imI2EdnsIxF5GDgO6CgiK4FrgEoAVb0TmAEMBZYDG4ELoyqLMcYUWuNgcePY\nQONgMeA7i6jx+ThmH4mqRv4m+VRdXa22n4IxJukGjJ1FbQtjAFUd2jF7zKCCl0dE5qtqtd95RTHQ\nbIwxxaalgOB1PCmKbuc1Y4xJCq8FZhUiNLTQE1MhUuhitooFBWOMyYHfmEFLAQHIejwprPvIGGNy\n4LfArCrL9NFsx5PCgoIxxuTAb4FZnNNKw7CgYIwxOfBbYDaibxU3jOxDVYd2CK6FcMPIPgWZVhqG\njSkYY0paVNlGRw/Zv8mYAmzbEhjRtyrn94orS6oFBWNMyQqzgMxPlAvMoiy3HwsKxpiS5TUYnI/K\nNUxLwEvU5fZiYwrGmJIVZ7bRMMo1S6oxxkQqzmyjYZRrllRjjIlUsU4LLcksqcYYE7c4s42GYVlS\nW8GypBpTfqKcnhnX1M9CC5ol1VoKxphEi3J6ZpxTP5PKxhSMMYkW5Sb2Ub52sbKWgjEm0aKcnhn1\n1M9i7JqyloIxJtGinJ4Z5Ws3dk3VrqtD+bJrKug+zXGxoGCMSbQop2cOPKCT5/EpC2oZMHYW3cdM\nZ8DYWa2q0Iu1a8q6j4wxiRbl9Mynl67OejzsIHSxrqa2oGCMSbyocgx5Vdxh8w+1b1fJurr6Fo8n\nmXUfGWPKlteYQtg7/WxbMSd8i2YLCsaY8uU1XtFhx5bv6LMdb27dxm1bCV7Hk8KCgjEmcmEGbKPk\ntTtatmQPQZNAFGsyPhtTMMZEKumrhrONV6xvYTzA63hzQXZmSyJrKRhjIlWsUzPD3unbHs3GGNOC\nYp2aGeRO32/FclSzpqJkQcEYE6m9O7SjtoUAkPS+db/1EUnvFsuVBQVjTKTi7lsPk3/I604/zn2U\no2RBwRgTqTg3jInybr5Yu8X8RBoURORE4BagArhbVcc2e7498CDQNVWW36vqvVGWyRhTeHH1rUd5\nN1+s3WJ+Ipt9JCIVwDjgJKAXcI6I9Gp22v8Cr6nqwcBxwB9EpG1UZTLGlJco7+aLdf9nP1FOST0C\nWK6qb6nqZmAiMLzZOQrsIiIC7AysAbZEWCZjTBmJcgFZsU459RNl91EVsCLj8UqgX7NzbgemAquA\nXYCzVXVr8xcSkcuAywC6du0aSWGNMaUn6kHuYpxy6ifuxWtDgFeAvYFDgNtFZNfmJ6nqXapararV\nnTq1nP/cGGOaK9W7+ShF2VKoBbpkPO6cOpbpQmCsqiqwXETeBg4AXoqwXMaYMlK0d/ObN8Njj0Ft\nLfzoRwV72yiDwlygh4h0xwWDUcC5zc55DzgeeE5E9gD2B96KsEzGGBNYLHssL10Kd98N990HH38M\nBx4I3/8+VFT4/2weRBYUVHWLiFwBzMRNSR2vqktE5Hup5+8ErgcmiMhiQIArVfXjqMpkjMldMW5C\nH0ZBVyxv3AiTJrlg8Nxz0KYNDB8Ol14KJ5xQsIAAIBo0D2xCVFdX67x58+IuhjFlpXkFCW7AtpT7\n5weMndXiOoSqDu2YPWZQft5k4UKoqYEHH4T166FHD7jkEvjOd2CPPfLzHikiMl9Vq/3OsxXNxhhf\npZrSwUtkaxw++wweftgFg3nzYPvt4YwzXKvg2GPTW7PF1TKzoGCM8VXMKR1yrVzzumJZFV56yQWC\niRNhwwb4+tfhllvgW9+C3XffpsxxJduLe0qqMaYIFOsuYo2Va+26OpQvK9cgO7/lZcXymjVw661w\n8MHQv78LCKNGwZw5sGgR/OAH2wQEiHcPCgsKxhhfUad0iGq7zjCVa85rHFThmWdcC2DvveGHP4Qd\ndoA//xnef98NJvfrl+4makmcLTPrPjLG+Ioy02mSM5m2ao3Dhx+6aaR33w1vvAHt27tB40svdS2F\nVogz2Z4FBWNMIFEtAivqTKYNDfDUU26s4LHHYMsWOOYYuPpqN3i84445vWyce1BYUDDG5EWuA7pR\nZzKNpHJduRLGj3f/3n0XOnZ03USXXAIHHBCy1PHuQWFBwZiEKcZFYmG6gKK8mw9SuQb+vOvrYfp0\n1yp44gnYuhUGD4Ybb4Rhw9zU0lZI6v7OFhSMSZBi3fc3TBdQnJlMA33eb74J99wD994LH3zgBo+v\nugouvhi6d8+pTFMW1DL6kYXUb9X0+45+ZGHT942JBQVjEqRYF4mF6QIK21USpmWV7fO+efpiRix7\nzg0a/+tfsN12cPLJrnto6FCXhiKEa6cuSQeERvVblWunLon992xBwZgEKdZFYmG7gHLtKgnbsmpe\n5v0+fo9RC2cycsnTUPcpdOsG118PF14IVfmrrNfV1bfqeCFZUDAmQYp139+4ZsuEbVlViFC5uY5T\nlj7PqIUzqa59nc3bteHJnv05+dZfwvHHu1ZCGbGgYEyCxDkVMYyoZ8tk6yIK1bJasIBrZ45j+JJn\n2HXzRt7cvTO/Oe4iHv36ID7ZqQMnDx6cl7K3ZLcdK1m7cdtWwW47Vkb2nkFZUDAmQeKcihhWVLNl\nvLqIWt2y+vRTeOghN4Po5Zc5q01b/rH/ACYePIS5nXunVxlXRdwyu+bU3oyetJD6hi/HFSorhGtO\n7Z1+bAnxjCkTSZ2KmFReXUSBWlaqLtdQTQ389a9u74KDDoLbbuOpPgO5+qkVBW+Z+QX/OGehWVAw\npoCKdcppnLy6iDwr108+gQcecDOIliyBnXeG885zaSeqq0GEk4H6XTvEckfuFfzjnIVmQcGYAirW\nKadx8usialK5bt3qktGdOxomT3b7HPfr51oJo0a5wNBMEltmcc5CK69hdWNiVqxTTuMUKEPrBx/A\n2LHQs6ebMfT44/Dd77qdzebMcesLWggISRVnqnJrKRiTg6unLObhF1fQoEqFCOf068KvR/Tx/bli\nnXIap6xdRAftCTNmuO6hadNcMrpjj4Vrr4XTT4d2xfuZWkI8Y4rI1VMW8+Cc99KPG1TTj/0CQ7FO\nOY1bky6e996D8TUwfDysWAGdOsGPf+xaAz17xlvQPIlzFpqoqv9ZCVJdXa3z5s2LuximCEQ1pW/f\nq2bQ0MLfTYUIb94wNNJyFWOyvLyor3etgbvvdsnoAL75TRcIhg2Dtm3jLV8REJH5qlrtd561FExJ\ninKWT0sBwet4c3GldChKy5e7QDBhgtvEpqrK7VVw0UUuBYXJOwsKpiRFOcunQiRrSyFKZTNz6Ysv\n4NFHXTB4+mmoqIBTTnGtghNPDJ2MLimS2uorjU/XmGainOVzTr8uTcYUMo9HKR/XlNSKCHBrCWpq\n3NqCNWtcWurf/AYuuMClqy4hSW71WVAwJSnKWT6Ng8m5zD4KI+w1ha2IIgkoGza4VcY1NW7qaNu2\ncNppboHZwIElm4wuya0+z6AgIjsApwDHAHsDdcCrwHRVXRJ98YzJTdSzfH49ok9kQSBb5Rv2msJU\nRHnfFGb+fBcIHnoIPvvMbWH5hz/A+ee7rS1bIdGtnyySvF4la1AQketwAeEZ4EXgI2AHoCcwNhUw\nfqqqiwpQTmNapVgTywW5m8/1msJURHnZFGb9evjLX9xYwYIFbh3BWWe5VsFRR6WT0bVGkrthvCR5\nvYpXS+ElVb0my3N/FJGvAl0jKJMxeTHv3TV8sP4LFPhg/RfMe3dNoisK8L+bD5OSIUxFlPOmMKow\ne7YLBH/7G9TVwSGHwLhxcO650KFDoLJnaw0kuRvGy8ADOrU4LjXwgE4xlKaprEFBVad7/aCqfoRr\nPRiTOGEWmIUVpjsjym6Fgi6c+/hjuP9+Fwxefx122cV1DV16KRx6aKtaBV6tgbi7YXL9XT+9dHWr\njheS70CziEwDms+/Ww/MA/6sql94/OyJwC1ABXC3qo5t4ZzjgJuBSuBjVf1G4NIbk8XDL67IejzK\noDBlQW2TPPm16+oYPSl433uU3Qphup+CbAozZf4KZt3xVwbPnsqQN+bQtqEejjzSbXp/1lk55x7y\nag3E2Q0Tpusq7mDmJcjQ/lvA50BN6t+nwGe4sYWabD8kIhXAOOAkoBdwjoj0anZOB+BPwDBV7Q2c\nmcM1GLONsAvMcnXdtCVNNk4BqG9QrpsWbF5GoORvIYzoW8XsMYN4e+zJzB4zKHAL5ppTe1NZ0fTu\nPr0pzPvv89oVYzj0hCO49Z7RHPPOAh485CSGXXYHU8Y94haahUhG51WBRv15efEKVn7iTHjnJ8iU\n1KNU9fCMx9NEZK6qHi4iXt/0I4DlqvoWgIhMBIYDr2Wccy7wqKq+B+kuKWNCi3qBWbZug5bupoGs\nx5vzu5vPNRFfWM3L1WWXtvxu51Ucee3lMH06vRoa+E/XPvz+mG8xs+dRbGrTNn1+2L59r9ZAnBMK\nwtztJzkHVpCgsLOIdG2suEWkK9AY9jd7/FwVkNmGXwn0a3ZOT6BSRJ4BdgFuUdX7gxTcGC/9v7Yb\ns99c0+LxsLy6DfIh22BynOMk6XLtvsV1B90xHmprYY894Gc/Y+Car/H27tuWuRBjIXHthxCm6yrJ\ns+OCBIWfAs+LyJuAAN2By0VkJ+C+PLz/YcDxQDvgPyIyR1X/m3mSiFwGXAbQtatNeDL+3vmk5coo\n2/HWCNNtEEZc4yRs3gxTp7pB43/+0x078US49VY49VSorGTz2FmQwLGQKIW920/i5j4QLCg8DvQA\nDkg9Xgaoqm7CDRBnUwtkrvvvnDqWaSXwiapuADaIyLPAwUCToKCqdwF3gcuSGqDMpsxFOZDn9dpB\nBmRzVfBxkv/+98tkdKtXQ5cu8MtfujGCZjdnUXeHJLECTWqwCitIULhHVS8CFgKkWghTcXf3XuYC\nPUSkOy4YjMKNIWR6DLhdRNoAbXHdSzcFL74xLctHSohsf+xerz16yP5NZh9BxoBsSAVJxFdX57ax\nrKmBZ591yeiGDXPJ6IYMcY9bELaCLNZ04kkMVmEFCQq1IvInVb1cRHYDpuMx66iRqm4RkSuAmbgp\nqeNVdYmIfC/1/J2q+rqIPAEsArbipq2+mvPVmKIT1R90mDtXv6mGXq8d5d1jpIn4Fi/+MhndunWw\n775www0uGd2eewZ6iThSghfriuYkC7TJjoj8f2BXXP//WFWdHHXBsrFNdkpH8z9ocJXrDSP75OUP\nOteAM2DsrBZbAlUd2jF7zKBQrx1WXmcfff45TJzogsFLL7lkdKef7loFxx1XsGR0QT7vKH42CL/f\nczHlXQq9yY6IjMx4+CLwC+AlQEVkpKo+Gr6YppxFnaIg1zvXIOMRcXUbhE7Epwpz57pAMHGiCwy9\nesFNN8G3vw1f+Ur+ChtQmPGfKMeO/FohpdpK8eo+OrXZ4wW4Vcen4lY4W1AwoSR1VWeSk5XlbO1a\nl4yupgYWLYIdd4Szz3ZpJ/r3zykZXb6E+byj/F353bQUa94lP165jy4sZEFMfOJqArdvV9liQrX2\n7cLP1AkjyQuLWkUVnnvOzSB65BG3o9mhh8Idd8A550D79nGXEAj3eUf5u/K7aUnqTU1YXt1HVwPj\nVHVtlucHATuq6j+iKpyJXpxN4Gw3pzHetAIlMNVw9Wq47z4XDJYtg113hQsvdGMFhx4ad+m2MaJv\nFfPeXdNkrOT0w4J1z0X5u/JrhZRkixLv7qPFwD9E5AvgZWA1bj+FHsAhwFPAbyMvoYlUnE3gdVlS\nP2Q7bjxs3QpPPeW6hx57DOrrYcAAGDMGzjwTdtop7hJmNWVBLZPn16an2zaoMnl+LdX77B44METx\nXfVrhZRMi7IZr+6jx4DHRKQHMADYC5cM70HgMlUt7jaSAeJtAif1TquoBhBra+Hee13qiXfecQPF\nV1zhWgW9evn+eKa4uhGT2jfv1wop+hZlFr7rFFT1DeCNApTFxCDOijnOOy2vCjCplVTali0wY4Zr\nFcyY4VoJgwa5dQWnnQbbb9/ql4wzEEZ9YxIm2Pm1Qsp18ZopYXFWzHHdaflVgImtpN5+27UI7r0X\nVq1yi8quvBIuvtgtNgshzkAY5Y1JUbX6EsKCQpmLuwkcx52WXwWYqEpq0yY3RlBT48YMttsOTjrJ\nbWd58slQmZ+ZWnF2I0Z5Y5L4Vl8CBdl5bYCqzvY7ZopXKTaBvfhVgImopJYudbOH7rvPbW3ZtStc\nd52bRdQle1qLXFshcXYjRnljUqrTRqMUpKVwG9B8HltLx4wpCn4VYJSVVEvvmz6+cSNMmuRaBc8/\nD23awPDhbtB48OCsyegahekqiXsj+TA3JrkmLzQt81qncCRwFNBJRH6S8dSuuAR3poCSmmMlqeXy\nEqQlUMjWU68P3+LsRTPhzvNg/Xro0QN+9zv4znfcJjYBhekqiXoj+ai+J2GSF5qWebUU2uJ2WGuD\n2xWt0afAGVEWyjSV1MGypJbLTz5aAmEruZ02bWTY688yauFMDv7gDTZVVMLZZ7q0E9/4Rk4r+Io1\nh1DQ12jp8/YLhHGPmRUjr3UK/wb+LSITVPXdApbJNBP3YFmuf5BJFra7IqdKTpXDVi3jzIUzOfX1\nZ9mp/guWdtyHa4+/jKlfH8TLN5+dU3kaFWsOIT9en3eSkxcWqyBjCtuLyF1At8zzVTV8XloTSJyD\nZWH/IEtRqyu5NWvgwQehpobJr77KhsodmHbgsUw8eAiv7NUTRPhW//DbzBZrDiE/Xp+3jRnkX5Cg\n8AhwJ3A30OBzrolAnF98+4PcVqBKThX+/W83g2jSJDe1tLoa/vxn/rhzHyYsXpufPREyhOkqiTOH\nkB+vz/umsw+xMYM8CxIUtqjqHZGXxGQV52CZ/UFuy7OS+/DDL5PRvfGGy0R6ySXu3yGHAG5jkl9E\nVLYwXSVx5RDy4/V525hB/nnNPto99b/TRORy4O/ApsbnVXVNxGUzKXF+8e0PclvNK7nttjZw/MpF\nXDf3BfjFky4NxTHHwNVXwxlnuL0LyljY74lfULExg/zyainMx22m0zgNYnTGcwp8LapCmcLymklT\nqn+QYfPhAEz42/Mc+/w0zn31SfZc9xF07Ag//KFrFRxwQJTFLzphWzBQfjcfcQm0R3OSlOMezVHu\nZRzktYtxLYKXUJ9nfT1Mn+4WmD3xhEtGd8IJbirp8OE5JaMzphCC7tHsGxSa7dXcaD2wWFU/yrF8\nOSvHoBDl5uRRb3yeRDld85tvfpmM7oMPYK+94KKLXDK67t0jLnFhlFrwN00FDQpBBpovBo4Enk49\nPg7XtdRdRH6lqg/kXEoTSJRTPz3TLpSowJ/npk3w97+7VsGsWS4Z3cknu+6hoUNdGooSEeUCs6h/\n1uRXkG91G+BAVf0QQET2AO4H+gHPAhYU8iCu/C0VIukdr5ofL1W+n+drr7nZQ/ffD598At26wfXX\nu2R0VcVdUUW1EDFMUCnWlfGlarsA53RpDAgpH6WOrQFs38Q8aPyjqF1Xh/LlH8WUBbWAG+xtV9k0\n3VS+pn62FBC8jpeClj7P3bWe2zYtdFtY9u4Nt9/uNq6ZOdN1HV19dUkEhGzfsygXmEX5syb/grQU\nnhGRf+AWsQGcnjq2E7AuspKVkTjzt1RluWuuytMCtCR2KWR+nrstXczFS//FKUueofLzz6BnT7jx\nRjj/fPjqV0O/V5JEuRDRL6h4/S7LdWV8UgUJCv+LCwQDUo/vByarG6EeGFXBykmc+VuiXBiX2C6F\nTz9lxIvTGPFIDbz8Muywg1tPcOmlbn1BiXadRbkQ0SuoTFlQy+hJC6lvcK3P2nV1jJ60ECDyTY1M\n6/l2H6kzSVV/nPo3SYttHmvCZfvyF2qDkxtG9qGqQzsE10LIx1RXCNYtMGVBLQPGzqL7mOkMGDsr\n3WWW9y4FVXjhBTdjaK+94H/+xy0yu+02t7XlAw/AsceWbEAA7+9Z2O9Btn0XBh7QieumLUkHhEb1\nDcp105YA0XaPmtbzWtH8vKoeLSKf4RarpZ/CxYpdIy9dmYg753tUrRC/mU0FSbb3ySeuwr/7bliy\nBHbeGc47z7UKqqtLLgjEtRDRaz+GtRtbHnpsPG6L05LFK3X20an/7pLtHJMfUf9RxDXdz29mU2R9\n3Fu3wjPPuKmkjz4KmzfDEUe4x2efDbsU71fa63fp100T9nsW5bhAsa6ML0WBJlqLyNFAD1W9V0Q6\nAruo6tvRFq28RPVHEfV0P6+Kwm9mU977uD/4ACZMcK2CN9+EDh3gu9916woOOii3C4xBts/U73fp\n1U3T+DvJ9Xvm995eQXzDpi2sq9u2tdChXWWry2Gi5zumICLXAFcCV6UOtQUeDPLiInKiiCwTkeUi\nMsbjvMNFZIuIRL6jW7Y+7GJXsL75Zu/pNZU22wymxuN56eNuaIAZM+C006BzZ7jqKjd19IEH3FjB\nrbcWPCCE+Y55faZ+v0u/bpow/N7ba1zg2mG9qdyuaTdd5XbCtcN6hy6Xyb8gLYXTgL7AywCqukpE\nfNvfIlIBjAMGAyuBuSIyVVVfa+G83wH/bGXZW61UF8nEtRGO31Ravz7sUH3c773n0k6MHw8rV0Kn\nTvCTn7hWQc+eoa8t1y63sN8xr880zqmbfu8dpGvKxgyKQ5CgsFlVVUQUILU+IYgjgOWq+lbq5yYC\nw4HXmp33fWAycHjA181ZMW8f6SWujXDCVhSt7uOur4dp09zYwMyZ7tjgwXDTTTBsGLRtG/qaIFzF\nHvY75vWZ+v0uO7Sr9O2m8RuTCLOq3iuI25hB8QgSFP4mIn8GOojIpcBFQE2An6sCVmQ8XolLjZEm\nIlW4lshAPIKCiFwGXAbQtWvu2xaW6iKZuDbCCVtRBHkecJvV3HOPGy/48EPXPXT11W56abduOZY+\nuzAVe9jvmNdn6teyunZYb0Y/spD6rV+OK2R203gFO8AzEMY9Q84UTpB1Cr8HJuHu5vcHfqmqt+Xp\n/W8GrlTVrT5luEtVq1W1ulOnludDBxHneoAoRTn/3Euk88u/+AIeeggGDnTdQb//PfTr51oK77wD\nv/pVJAEBwlXsYb9jXp+p3+9yRN8qbjzz4CbP33jmwU1aZNmCnd+YQZTfI5MsXusUfgS8ALysqk8C\nT7bytWuBLhmPO6eOZaoGJoqbotgRGCoiW1R1SivfK5BSvduJayOcSKbSvvqq6x564AFYu9alpf7N\nb+CCC2DvvfNTcB9hutzCfseCdLnl2vLKJdgVYlW9SRav7qPOuDv5A0RkMTAbFyReCLgV51ygh4h0\nxwWDUcC5mSeoajoRvYhMAP4RVUCA+BfJFCKXT6GvKy8VxYYN8Ne/umAwZw5UVsLIkW7QeNAgl7K6\ngMJU7Pn4XURV+foFO0s1YSDYJjttcXf0R+H2VTgSWKeqvXxfXGQoLrBUAONV9Tci8j0AVb2z2bkT\ncEFhktdrRr3JTlQVd5S7pxUlVZg/3wWChx+Gzz5zW1heeqlLRtexY6zFK8X8/l7fQcC+nyUun5vs\ntAN2Bdqn/q0CFnv+RIqqzgBmNDt2Z5ZzLwjymlGKcspqqc58arV169xYQU0NvPIKtGsHZ57pgsGA\nAa1KOxFlxV2KXSUj+lYx7901PPziChpUqRDh9MOaXmepBULTel5jCncBvYHPgBdxXUd/VNW1BSpb\nwUVZcZfqzKdAVGH2bBcIHnkE6urgkENg3Dg491y38riVSnXNCUTbWp08vza9orxBlcnza6neZ/d0\nECz2z86E59VZ2xXYHvgANyawkhLfPyHKirtUZz55+vhj+OMf3aY1xxzjtrY8/3yYO9elrL788pwC\nApTuxix+q8TDKNXPzOSXV0K8E8VNC+qNG0/4KfB1EVkD/EdVrylQGQsmyoVexTzzqVV3rlu3uv2M\na2pcEKivh/793TqDs85yWUrzoFRbXvnYFtM2szFheI4ppPZNeFVE1gHrU/9Owa1WLrmgEGXFHecM\nobC7nwXqplm1yi0uu+ceeOst2G031xK45BL4+tfzfUkluzFLmIo7TNI6Yxpl7T4SkR+IyEQReQ/4\nNy4YLAVGArsXqHwFNaJvFacfVpVO7dzSQFzY1589ZhBvjz2Z2WMGFSwghOmO8Oxy2LKF/9xyH8/1\nGsCWzl3g5z9n9e57wF/+4oLEzTdHEhCgdDdmCdPNGCZpnTGNvFoK3XD7Mv9YVd8vTHHi5TcQV4yi\nyMXTef2HnPXck9TddB5HfvQBq3fqQM0RI/nrQYP5cI+u3HBgH0bssEPerqElca85iUqY1mo+ktYZ\n4zWm8JNCFiQJSnHaaL5y8VQ21HPCGy9yzsKZHP3OKyDwYo9qJhx1Ef/a7wi2VKS+SgX8vEpxtkyY\nijsfuagr7zp2AAASOElEQVSMCbTJTrkIW4EmccFT2H7k6/Zvw7s3TmD4wifpuHE9q3bpyJ+OOZce\nY37Ad59teQvGbNtwmmByrbiLeTKDSQ4LChnCVKBJnTefU0VRVweTJ0NNDSc8+yxbKyp4bv/+TOh1\nAsv7DuCnJ/ViSN8qKp6b4bndpiks6x4y+WBBIUOYO62ou55ybYW0qqJYtMhNJX3wQbfyeN994YYb\n2O6CC/jGnnvyjWan+223aQrPuodMWBYUMoS504pyDnikrZDPP4eJE10weOklt1HNyJEu7cRxx3km\no6vK0rLKtg1nkiSxq8+YJLCg0Eyud1pRzgEP0wppMaBMXkSHVxdw3LOPuYDw+efQq5fbwezb34av\nfCVQuYq1DzupXX3GJIEFhTyJsoIM0wrJDCi7fvE5I5Y8zTkLZ3Lg6ndcMrqzz3atgiOPbFUyOije\nPuxSnGVmTL5YUMiTKCvIMK2QVWs3csTKJYxaOJOhy2azw5bNLN5jX37+zcv5zd9+C+3bhypbMfZh\nW7oHY7KzoJBHUVWQObVCVq+G++7jmXtvZZ/VK/i07Y480ucEJh70TZbsuZ/r9w8ZEKIWVb9/+ywb\n3LfP2ODemHJlQaEIBG6FbN0KTz3lBo0fewzq69n54Gqu6n8mU/Y7irq2bpVxuff7Z+sls5m0xlhQ\nKBqerZDaWhg/3iWje/ddN1B8xRVw8cV8pXdv+i2o5Vnr909bt3HbVoLXcWPKiQWFYrVlC8yY4VoF\nM2a4VsKgQTB2LJx2Gmy/ffpU6/dvyrKFGpOdBYVmEj9//e23XYvg3ntdJtI994Qrr4SLLoL99ou7\ndNvI9fMMUnHn+trFOpXWmEKwoJAhsfPXN21yYwQ1NW7MYLvt4MQT3XaWJ58MlckcIA3zeY4esj+j\nH1lI/dYvV0dXbifpijvMaxfrVFpjCsGCQobEzV9futQFgvvvd1tbdu0K110HF14IXboUvjytFPrz\nbD7wm/E47GsXY5eaMYVgQSFDtuye+cr6Gai7Y+NGmDTJBYPnn4c2bWDYMLfAbPBgqKho+cUTKOyi\nu/qGpjmU6hs0XelHvdYg8d2IxkTEgkKGCpHIsn76dXc8PfGfrLllHIMX/ItdN23g867d2fl3v4Pv\nfAf22CP0+8ch1KI7n0o/ysHixHYjGlMA2bOdlagpC2oZMHYW3cdMZ8DYWU22pYwy62dL3R0Vn3/G\nsuv/wNrehzDwnCGcMvdxZu1bzahzfsvh3x7HlMHnFW1AgHDbP/ptSxnl1pJ+21oaU8rKqqXgdwcY\nZdbP9J2vKn1XLWPUwpmcsvQ5dqr/gjf36M4tx1/G33sPZH27Xdx5W7YWfS6eIAO62bpp/GYIRTlY\nbGkwTDkrq6DgNzjpVxGF6Wfev209/V94nFELZ3LAx++ysXJ7ph1wLP86ehhP7tINbaGLqhQqIa8B\n3SDdNF6fd1SDxbaOwZSzsgoKYTY2D1KBbRM0vtmTEZ8uh5oapk+aTMXmTSzcswdXDbmCaQceS8PO\nu3DDyD4smbmsLCshvyAd1wwhW8dgyllZBYUwG5v7VWCZQaPjhrWc8uIkDrnxSVhTC+3bU3HpJTx9\n9DCufqdNi3e+5VgJJbWbxtYxmHJWVkFh4AGdeHDOey0e9+NXgf3h8dc4YtlLjFo4kxOWv0jl1gZe\n6tyLB47/Fr+Y8EvYcUcGArNbeI0RfauY9+4aHn5xBQ2qVIhw+mGlP48+yd00to7BlKtIg4KInAjc\nAlQAd6vq2GbPnwdciVuW9BnwP6q6MKryPL10dauOZ8pWgR3CZ3DddUz84ziqPl3Nmna7MuGwU5l4\n0BDe7NgFAX6x446erz1lQS2T59emZzk1qDJ5fi3V++xekIoprjn51k1jTPJEFhREpAIYBwwGVgJz\nRWSqqr6WcdrbwDdUda2InATcBfSLqkxhuisyK7A2DVsY9OZczlv8T459az5s3cqq/Q7lt8ddxJM9\n+rO5zZdpJ4Lc9ca5kjrOOfnWTWNM8kTZUjgCWK6qbwGIyERgOJAOCqr6Qsb5c4DOEZYnVHfFiL5V\n7LjiHT784ziGzHuCr25YS13HPZCrroKLLqJ2/fbMenQxm3O4642zbz3u1B5hummunrK4SZfbOf26\n8OsRffJcQmPKS5RBoQpYkfF4Jd6tgIuBxyMsT27dFZs2wd//DjU1fHPWLJeMbuhQuPRS2g0d6tJQ\nACNSp0eVETQqSR3s9XP1lMVNxocaVNOPLTAYk7tEDDSLyEBcUDg6y/OXAZcBdO3aNef3aVV3xWuv\nfZmMbs0a2GcfuP56uOAC6NxygybXu9589K1HmaI6jKjGKx5+cUXW4xYUjMldlEGhFshM5dk5dawJ\nETkIuBs4SVU/aemFVPUu3HgD1dXVoXJOeFbcGzbAI4+4YPDCCy4l9fDhLhndCSe4VkIEwvatT1lQ\ny+hJC9MJ5GrX1TF60sImr51NlIO9UY5XRJmSxJhyFmVQmAv0EJHuuGAwCjg38wQR6Qo8CnxbVf8b\nYVm8vfyyCwQPPQSffgo9e8KNN8L558NXv1qQIoTpW79u2pIWM4peN21JrHsLRDleEWXyQmPKWWRB\nQVW3iMgVwEzclNTxqrpERL6Xev5O4JfAV4A/iftj3qKq1VGVqYn16+Hhh10wePll2GEHOOMM1yo4\n5pii2sV9bZa9hbMdby6qOflhxyu8up7O6delxTUn5/RL/j4TxiRZpGMKqjoDmNHs2J0Z/38JcEmU\nZWjRz38ON9/s9i446CC47TY47zzYbTfPH7Mc+60TZrzCr+upcdzAZh8Zk1+JGGguuN12g3PPda2C\nww8P1CpIco79Du0qWVe3baugQ7t4t+kMM14RpOvp1yP6WBAwJs/KMyj87Get/pG45/N7uXZY7xb3\nM752WO+8vH6uLaQw4xXFOlXWmGJXnkEhB0mupKIcLA7bQsp1vCLJeZGMKWUWFAJKeiUV1WBxXC0k\ny4tkTDzKbjvOXEW5/WOSxdVCGtG3ihtG9qGqQzsEt/vdDSP7xN5VZ0yps5ZCQElP3hbVzKj2WQax\n2xdgENvSVxtTeBYUWiGplZRfv3+YgJFtYlYRLeMwxrSCBYUi4VWxe/X7A6EGitdlWQCX7bgxprjZ\nmEIRaGwJ1K6rQ/myYp+ywKWS8ur39wsYfrINpCdlgN0Yk18WFApoyoJaBoydRfcx0xkwdla6Uvd7\nzq9i96q4ww4Ul+sAuzHlyoJCgXjd7YdpCYB3xR32Tt9mARlTXmxMoUD87va91gL4rZHwmxkVdr5/\nUgfYjTH5Z0Ehj7wGg3PpxslsCfhV7Nkq7qRPpTXGJIsFhTzxmxbqd7cfpiXgx+70jTFBWVBohVyn\nhY7oW+V7t59rS8AYY/LJgkJAfi0Bv+6hIHf71sVjjImbBYWA/FoCQRLmed3tW0vAGJMENiU1oDDT\nQo0xplhYUAjIb77/iL5VnH5YVXrj+AoRTj/M7v6NMcXFgkJAfi2BKQtqmTy/lgZ1u581qDJ5fm2T\nlcnGGJN0FhQC8lvZGzbHkDHGJIENNLeC12BwkrfrNMaYoCwotILXOoWkb9dpjDFBWPdRQH5J62z2\nkTGmFFhLISC/dQpBFqdFtWWmMcbkiwWFgIKMGXiNOfitiDbGmCSw7qOAwu5LYLOTjDHFwIJCQGHH\nDGx2kjGmGFhQCCjsDmS217ExphjYmEIrhElaF2SjHGOMiVukLQUROVFElonIchEZ08LzIiK3pp5f\nJCKHRlkecAO+A8bOovuY6QwYO6tgaShsr2NjTDGIrKUgIhXAOGAwsBKYKyJTVfW1jNNOAnqk/vUD\n7kj9NxJxzwCy9NjGmKSLsqVwBLBcVd9S1c3ARGB4s3OGA/erMwfoICJ7RVUgmwFkjDHeogwKVcCK\njMcrU8daew4icpmIzBOReatXr865QDYDyBhjvBXF7CNVvUtVq1W1ulOnTjm/js0AMsYYb1EGhVqg\nS8bjzqljrT0nbyw/kTHGeIsyKMwFeohIdxFpC4wCpjY7ZypwfmoWUn9gvaq+H1WBbAaQMcZ4i2z2\nkapuEZErgJlABTBeVZeIyPdSz98JzACGAsuBjcCFUZWnkc0AMsaY7CJdvKaqM3AVf+axOzP+X4H/\njbIMxhhjgiuKgWZjjDGFYUHBGGNMmgUFY4wxaRYUjDHGpFlQMMYYk2ZBwRhjTJoFBWOMMWnilgoU\nDxFZDbybh5fqCHych9cpFna9pa2crrecrhXyd737qKpv8riiCwr5IiLzVLU67nIUil1vaSun6y2n\na4XCX691HxljjEmzoGCMMSatnIPCXXEXoMDsektbOV1vOV0rFPh6y3ZMwRhjzLbKuaVgjDGmmZIP\nCiJyoogsE5HlIjKmhedFRG5NPb9IRA6No5z5EuB6z0td52IReUFEDo6jnPngd60Z5x0uIltE5IxC\nli/fglyviBwnIq+IyBIR+Xehy5hPAb7L7UVkmogsTF1v5PuxREVExovIRyLyapbnC1dPqWrJ/sNt\n7vMm8DWgLbAQ6NXsnKHA44AA/YEX4y53xNd7FLBb6v9PKtbrDXKtGefNwu3rcUbc5Y74d9sBeA3o\nmnr81bjLHfH1/j/gd6n/7wSsAdrGXfYcr/dY4FDg1SzPF6yeKvWWwhHAclV9S1U3AxOB4c3OGQ7c\nr84coIOI7FXoguaJ7/Wq6guqujb1cA5uX+xiFOR3C/B9YDLwUSELF4Eg13su8KiqvgegqsV8zUGu\nV4FdRESAnXFBYUthi5kfqvosrvzZFKyeKvWgUAWsyHi8MnWstecUi9Zey8W4u49i5HutIlIFnAbc\nUcByRSXI77YnsJuIPCMi80Xk/IKVLv+CXO/twIHAKmAx8ENV3VqY4hVcweqpSLfjNMklIgNxQeHo\nuMsSoZuBK1V1q7uZLHltgMOA44F2wH9EZI6q/jfeYkVmCPAKMAjYF3hSRJ5T1U/jLVZxK/WgUAt0\nyXjcOXWstecUi0DXIiIHAXcDJ6nqJwUqW74FudZqYGIqIHQEhorIFlWdUpgi5lWQ610JfKKqG4AN\nIvIscDBQjEEhyPVeCIxV1+m+XETeBg4AXipMEQuqYPVUqXcfzQV6iEh3EWkLjAKmNjtnKnB+anS/\nP7BeVd8vdEHzxPd6RaQr8Cjw7SK/g/S9VlXtrqrdVLUbMAm4vEgDAgT7Lj8GHC0ibURkR6Af8HqB\ny5kvQa73PVyrCBHZA9gfeKugpSycgtVTJd1SUNUtInIFMBM3m2G8qi4Rke+lnr8TNytlKLAc2Ii7\n+yhKAa/3l8BXgD+l7qC3aBEmFwt4rSUjyPWq6usi8gSwCNgK3K2qLU5xTLqAv9/rgQkishg3K+dK\nVS3K7Kki8jBwHNBRRFYC1wCVUPh6ylY0G2OMSSv17iNjjDGtYEHBGGNMmgUFY4wxaRYUjDHGpFlQ\nMMYYk2ZBwZQFEfm82eMLROR2n58Z5pV9NXXOcSLyjyzP/Si1XiDbz04Ska95vX6z8/uIyISg5xuT\nCwsKxmShqlNVdWyIl/gR0GJQEJHeQIWqBl5spaqLgc6pBYjGRMKCgil7ItJJRCaLyNzUvwGp4+nW\nhIjsKyJzUvtQ/LpZy2Pn1F3/UhH5S2rV6Q+AvYGnReTpFt72PNwK5MYyfC4iN6b2BXhKRI5IJbZ7\nS0SGZfzcNNzqXmMiYUHBlIt2qc1nXhGRV4BfZTx3C3CTqh4OnI7LC9XcLcAtqtoHl2MoU19cq6AX\nLv//AFW9FZe9c6CqDmzh9QYA8zMe7wTMUtXewGfAr4HBuCyvmWWdBxwT5IKNyUVJp7kwJkOdqh7S\n+EBELsAlzAM4AeiVkUl1VxHZudnPHwmMSP3/Q8DvM557SVVXpl73FaAb8LxPefYCVmc83gw8kfr/\nxcAmVa1PpXDolnHeR7gWiDGRsKBgjGsx91fVLzIPtiLd9qaM/28g2N9VHbBDxuN6/TLnzNbG10yl\n/c58vR1SP2tMJKz7yBj4J26HNgBE5JAWzpmD61qC4H36nwG7ZHnudWC/oAXM0BMoyiR3pjhYUDAG\nfgBUpzZEfw34Xgvn/Aj4iYgswlXm6wO87l3AE1kGmqfjsmK21sDUzxoTCcuSakwAqfUGdaqqIjIK\nOEdVW9oTOujrtQOexg1KNwT8me2BfwNHq2pR7kVsks+CgjEBiMgxuD2BBVgHXKSqy0O+5hDgdVV9\nL+D5PYAqVX0mzPsa48WCgjHGmDQbUzDGGJNmQcEYY0yaBQVjjDFpFhSMMcakWVAwxhiTZkHBGGNM\n2v8Bskf02O72o4YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1997247c630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Finding the extreme y coordinates for our line\n",
    "y_min = beta_0 + (beta_1*np.min(x_norm))\n",
    "y_max = beta_0 + (beta_1*np.max(x_norm))\n",
    "# Plotting:\n",
    "plt.scatter(x_norm, y_norm)\n",
    "plt.xlabel('Height (m)')\n",
    "plt.ylabel('Weight (kg)')\n",
    "plt.plot([np.min(x_norm),np.max(x_norm)], [y_min,y_max],c='r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculating **Least Squares Error** and **Mean Squared Error**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Least Squares Error: 2.99751657917\n",
      "Mean Squared Error: 0.0187344786198\n"
     ]
    }
   ],
   "source": [
    "lse = np.sum(np.square(y_norm - (beta_0 + np.multiply(beta_1,x_norm))))\n",
    "mse = np.mean(np.square(y_norm - (beta_0 + np.multiply(beta_1,x_norm))))\n",
    "\n",
    "print('Least Squares Error: %s' % lse)\n",
    "print('Mean Squared Error: %s' % mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the least possible errors. There is no other line that can achieve lower errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='comparing_solutions'></a>\n",
    "### Comparing Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table below shows a comparisson among the 3 approaches used to solve this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col0 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col1 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col2 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col3 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col0 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col1 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col2 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col3 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col0 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col1 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col2 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col3 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col0 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col1 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col2 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col3 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col0 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col1 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col2 {\n",
       "            text-align:  center;\n",
       "        }    #T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col3 {\n",
       "            text-align:  center;\n",
       "        }</style>  \n",
       "<table id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"blank level0\" ></th> \n",
       "        <th class=\"col_heading level0 col0\" >Least Squared Error</th> \n",
       "        <th class=\"col_heading level0 col1\" >Mean Squared Error</th> \n",
       "        <th class=\"col_heading level0 col2\" >beta_0</th> \n",
       "        <th class=\"col_heading level0 col3\" >beta_1</th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3\" class=\"row_heading level0 row0\" >Analytic Solution (reference)</th> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col0\" class=\"data row0 col0\" >2.99752</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col1\" class=\"data row0 col1\" >0.0187345</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col2\" class=\"data row0 col2\" >0.173145</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row0_col3\" class=\"data row0 col3\" >0.611951</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3\" class=\"row_heading level0 row1\" >Keras (1000 epochs)</th> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col0\" class=\"data row1 col0\" >2.99943</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col1\" class=\"data row1 col1\" >0.0187464</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col2\" class=\"data row1 col2\" >0.171156</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row1_col3\" class=\"data row1 col3\" >0.609212</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3\" class=\"row_heading level0 row2\" >Keras (10000 epochs)</th> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col0\" class=\"data row2 col0\" >2.99911</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col1\" class=\"data row2 col1\" >0.0187444</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col2\" class=\"data row2 col2\" >0.167775</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row2_col3\" class=\"data row2 col3\" >0.617202</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3\" class=\"row_heading level0 row3\" >Tensorflow (1000 epochs)</th> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col0\" class=\"data row3 col0\" >27.3372</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col1\" class=\"data row3 col1\" >0.170857</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col2\" class=\"data row3 col2\" >0.173145</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row3_col3\" class=\"data row3 col3\" >0.61195</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3\" class=\"row_heading level0 row4\" >Tensorflow (10000 epochs)</th> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col0\" class=\"data row4 col0\" >2.99752</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col1\" class=\"data row4 col1\" >0.0187345</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col2\" class=\"data row4 col2\" >0.173145</td> \n",
       "        <td id=\"T_fff5fc92_8e8c_11e7_809e_5cc9d35574d3row4_col3\" class=\"data row4 col3\" >0.611951</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1997f0cc908>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "# The annotated results\n",
    "analytic_solution = {'Mean Squared Error': 0.0187344786198, 'Least Squared Error': 2.99751657917, 'beta_0': 0.173145032501, 'beta_1': 0.6119508201 }\n",
    "keras_1k = {'Mean Squared Error': 0.0187464144081, 'Least Squared Error': 2.99942620921, 'beta_0': 0.17115624, 'beta_1': 0.60921186 }\n",
    "keras_10k = {'Mean Squared Error': 0.0187444284558, 'Least Squared Error': 2.99910858853, 'beta_0': 0.16777468, 'beta_1': 0.61720169 }\n",
    "tensorflow_1k = {'Mean Squared Error': 0.170857409193, 'Least Squared Error': 27.3371854708, 'beta_0': 0.173145229628, 'beta_1': 0.611950453073 }\n",
    "tensorflow_10k = {'Mean Squared Error': 0.0187344786198, 'Least Squared Error': 2.99751657917, 'beta_0': 0.173145032501, 'beta_1': 0.6119508201 }\n",
    "# Columns\n",
    "index=['Analytic Solution (reference)', 'Keras (1000 epochs)', 'Keras (10000 epochs)', 'Tensorflow (1000 epochs)', 'Tensorflow (10000 epochs)']\n",
    "# Showing with panda\n",
    "data = [analytic_solution, keras_1k, keras_10k, tensorflow_1k, tensorflow_10k]\n",
    "df = pd.DataFrame(data, index)\n",
    "df.style.set_properties(**{'text-align': 'center'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the analytic solution is the one that has the least errors. And the more we train, the better our **Linear Regression** model gets."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}