F0B-05-CS1Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FLIP (00): Data Science \n",
    "**(Module 01: Data Science)**\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- You are free to use,but NOT allowed to change and distribute this package.\n",
    "\n",
    "Prepared by and for \n",
    "**Student Members** |\n",
    "2006-2018 [TULIP Lab](http://www.tulip.org.au), Australia\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 6 Case Study (I) - Classification\n",
    "\n",
    "\n",
    "The purpose of this session is to demonstrate different coefficient and linear regression.\n",
    "\n",
    "\n",
    "## Content\n",
    "\n",
    "### Part 1 Linear Regression\n",
    "\n",
    "1.1 [Linear Regression Package](#lrp)\n",
    "\n",
    "1.2 [Evaluation](#eva)\n",
    "\n",
    "\n",
    "### Part 2 Classificiation\n",
    "\n",
    "2.1 [Skulls Dataset](#data)\n",
    "\n",
    "2.2 [Data Preprocessing](#datapre)\n",
    "\n",
    "2.3 [KNN](#knn)\n",
    "\n",
    "2.4 [Decision Tree](#dt)\n",
    "\n",
    "2.5 [Random Forest](#rf)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">1. Linear Regression</span>\n",
    "\n",
    "<a id = \"lrp\"></a>\n",
    "\n",
    "\n",
    "### <span style=\"color:#0b486b\">1.1 Linear Regression Package</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will learn how to use sklearn package to do linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_diabetes \n",
    "from sklearn.linear_model import LinearRegression \n",
    "import matplotlib.pyplot as plt \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create an instance of the diabetes data set by using the <b>load_diabetes</b> function as a variable called <b>diabetes</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes = load_diabetes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will work with one feature only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_X = diabetes.data[:, None, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create an instance of the LinearRegression called LinReg."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "LinReg = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to perform <b>train/test split</b> we have to split the <b>X</b> and <b>y</b> into two different sets: The <b>training</b> and <b>testing</b> set. Luckily there is a sklearn function for just that!\n",
    "\n",
    "Import the <b>train_test_split</b> from <b>sklearn.cross_validation</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from sklearn.cross_validation import train_test_split  \n",
    "#Cross has been abandoned in sklearn_ Validation, which integrates the contents into the model_ In selection\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now <b>train_test_split</b> will return <b>4</b> different parameters. We will name this <b>X_trainset</b>, <b>X_testset</b>, <b>y_trainset</b>, <b>y_testset</b>. \n",
    "\n",
    "Now let's use <b>diabetes_X</b> as the <b>Feature Matrix</b> and <b>diabetes.target</b> as the <b>response vector</b> and split it up using <b>train_test_split</b> function we imported earlier (<i>If you haven't, please import it</i>). The <b>train_test_split</b> function should have <b>test_size = 0.3</b> and a <b>random state = 7</b>. \n",
    "\n",
    "\n",
    "The <b>train_test_split</b> will need the parameters <b>X</b>, <b>y</b>, <b>test_size=0.3</b>, and <b>random_state=7</b>. The <b>X</b> and <b>y</b> are the arrays required before the split, the <b>test_size</b> represents the ratio of the testing dataset, and the <b>random_state</b> ensures we obtain the same splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_trainset, X_testset, y_trainset, y_testset = train_test_split(diabetes_X, diabetes.target, test_size=0.3, random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the <b>LinReg</b> model using <b>X_trainset</b> and <b>y_trainset</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LinReg.fit(X_trainset, y_trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's <i>plot</i> the graph.\n",
    "<p> Use plt's <b>scatter</b> function to plot all the datapoints of <b>X_testset</b> and <b>y_testset</b> and color it <b>black</b> </p>\n",
    "<p> Use plt's <b>plot</b> function to plot the line of best fit with <b>X_testset</b> and <b>LinReg.predict(X_testset)</b>. Color it <b>blue</b> with a <b>linewidth</b> of <b>3</b>. </p> <br>\n",
    "<b>Note</b>: Please ignore the FutureWarning. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1ae03f34748>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmsUlEQVR4nO2dfZQdVZmvn52mO9B8SHII3EjIaUBYDuBVsUWchZlZMmBEUS5wFVerKHp7GcY1eBm8hpWZketMZgmMg3pBxuZDProdVEYlMihGg6JcBTveEMNHgEg3kBVMCAQIgUDS7/2jquF096lTp86p7/N71qrV1bt27b1r16lfvfW+u3Y5M0MIIUS5mJV1A4QQQsSPxF0IIUqIxF0IIUqIxF0IIUqIxF0IIUrIHlk3AOCAAw6wvr6+rJshhBCFYvXq1U+Z2bx620LF3Tm3J3AnMNvPf7OZfdE5dx3wF8CzftZPmNka55wDvgacAuzw03/fqI6+vj5GR0ebPR4hhBCAc248aFszlvtO4N1mtt051w382jn3Y3/b583s5mn53wsc4S/vAK70/wohhEiJUJ+7eWz3/+32l0ZvPn0QuMHf77fA/s65+e03VQghRLM0FVB1znU559YAm4GVZna3v2m5c26tc+4y59xsP+1g4PGa3Z/w06aXOeicG3XOjW7ZsqX1IxBCCDGDpsTdzHab2VuABcBxzrljgAuBNwJvB+YCX4hSsZkNmVm/mfXPm1c3HiCEEKJFIg2FNLNtwB3AYjPb5LtedgLfAo7zs20EDqnZbYGfJoQQIiVCxd05N885t7+/vhdwEvDgpB/dHx1zGrDO32UF8HHncTzwrJltSqDtQghRCEZGRujr62PWrFn09fUxMjKSeJ3NjJaZD1zvnOvCuxl818xudc6tcs7NAxywBviMn/82vGGQj+ANhfxk7K0WQoiCMDIywuDgIDt27ABgfHycwcFBAAYGBhKr1+Vhyt/+/n7TOHchRBnp6+tjfHzmcPRqtcrY2FhbZTvnVptZf71tmn5ACCGmEacb5bHHHouUHhcSdyGEqGHSjTI+Po6ZvepGaVXgFy5cGCk9LiTuQghRw7Jly171j0+yY8cOli1b1lJ5y5cvp7e3d0pab28vy5cvb7mNzSBxF0KIGuJ2owwMDDA0NES1WsU5R7VaZWhoKNFgKiigKoQQU0gyABo3CqgKIUSTZOVGiRuJuxBC1JCVGyVu5JYRQoiCIreMEEJ0GBJ3IYQoIRJ3IYQoIRJ3IYQoIRJ3IYQoIRJ3IXJOFnOBi+LTzHzuQoiMyGoucFF8ZLkLkWPinsRKdA4SdyFyTFZzgYviI3EXhaETfc9ZzQUuio/EXRSCuD+gUBTKMomVSB+JuygEnep7LsskViJ9NHGYKASzZs2i3m/VOcfExEQGLRIiezRxmCg88j0LEQ2JuygE8j0LEQ2JuygE8j0LEY1Qn7tzbk/gTmA23hutN5vZF51zhwI3ARVgNfAxM3vZOTcbuAF4G7AV+LCZjTWqQz53IYSITrs+953Au83szcBbgMXOueOBi4HLzOwNwDPAp/z8nwKe8dMv8/MJIYRIkVBxN4/t/r/d/mLAu4Gb/fTrgdP89Q/6/+NvP9E55+JqsBBCiHCa8rk757qcc2uAzcBKYAOwzcx2+VmeAA721w8GHgfwtz+L57qZXuagc27UOTe6ZcuWtg5CCJE8nfiGcJFpStzNbLeZvQVYABwHvLHdis1syMz6zax/3rx57RYnhEiQTn1DuMhEGi1jZtuAO4B3Avs75yanDF4AbPTXNwKHAPjbX4cXWBVCFJROfUO4yISKu3NunnNuf399L+Ak4AE8kT/Tz3Y2cIu/vsL/H3/7KsvDa7BCiJbR7JTFoxnLfT5wh3NuLfA7YKWZ3Qp8ATjfOfcInk/9Gj//NUDFTz8fWBp/s4UQaaI3hItH6JeYzGwt8NY66X/E879PT38J+O+xtE4IkQuWL18+5YtQoDeE847eUBVChKI3hIuHZoUUQoiColkhhRCiw5C4C9Fh6GWkdMi6n0MDqkKI8jD5MtJkYHTyZSRA/vMYyUM/y+cuRAfR19fH+Pj4jPRqtcrY2Fj6DSopafWzfO5CCEAvI6VFHvpZ4i5EB1HGl5Gy9m3XIw/9LHEXooMo2+cK8zqhWS762cwyX972treZEMJjeHjYqtWqOeesWq3a8PBwocpPk2q1anjfl5iyVKvVrJsW2s8PPGB2xRVm997beh3AqAXoqgKqQuSI6aMswLP49DZofWbNmkU9DXPOMTExkUGLwrniCvjsZ1/7v7cXHnoIDj44eJ8gFFAVoiBoat1o5MG33QxPPw1/8Rfg3FRhB9ixA3btqr9fO0jcRceQx8DbdPIwyqJI5MK33YCVKz1Br1Tgzjvr57n8cqhWE6g8yF+T5iKfu0ia4eFh6+3tneKX7e3tzZ2/Oc8+5LyStxjCyy+bnXuuGTRerrii/bpo4HPPXNhN4i5SIEg0J4Uza0GYpCg3oaTIm1BHYf16swMPbCzoBxxg9uCD8dUpcRcdj3MuUNzzJqBFFrh2KOqN7RvfCLfSlyzxLPq4aSTuGi0jOoKg18Fr6erq4vrrr9eolIwo0tQI27bBGWfAqlWN891+O5x8cnLt0GgZ0fHUC7xNZ/fu3bl4AaZTKUIwedUqL0A6Z06wsC9aBFu3ejZ7ksIehsRd5IbJ0SzOOfbYYw+cc7GNaqn9klAjNOwwO/IwrLHeiKpdu+C88zxRP/HE4H2//nWYmIBf/hLmzm2trlgJ8tekucjnLur5W0nI79qoLsCcc7HV1elEiR9k7XOfWf/hBpsa+tLnzjW7//446mrtWFFAVeSdRqNZSGAo4PDwsHV1dWnYYYK0ImBZBpNf+w3+j9AA6eCg2c6dcdTV3m+vkbgroCpyQdBr5JMk8Tq5XvVPliIFSJ99FvbffyVwUsN8P/4xLF7cfn1xTZuggKqYQh7f1Azzqybhd631wzvnqFarEvYYKUKA9Je/9Hzp++8PQcI+e/YoW7Z4Nnscwg4pxReCTPrJBTgEuAO4H7gPOM9PvwjYCKzxl1Nq9rkQeARYD7wnrA65ZdIja79mlHblqX0iOnl923bXLrPPfa6x28VbzrO99krmt5cLnzswHzjWX98XeAg4yhf3C+rkPwq4F5gNHApsALoa1SFxT4+8XnBmr/lbgVf94Z30Ek/ZyJshsWGD2YIFjQV9r7122vz5f5WKzz+O+EJb4j5jB7gF7/klSNwvBC6s+f924J2NypS4p0fQm5oaISKSIA9v2159dbiV/qlPtRcgzYpG4h7J5+6c6wPeCtztJ33WObfWOXetc26On3Yw8HjNbk/4adPLGnTOjTrnRrds2RKlGaIN8jCWWHQOAwMDjI2NMTExwdjYWGrxjOeeg/e+1/Onf/rTwfluvdWT96uvhp6eVJqWGk2Lu3NuH+A/gM+Z2XPAlcDhwFuATcBXolRsZkNm1m9m/fPmzYuyq2iDvE+RKkQ7/OpXnqC/7nXwk5/Uz/OOd8DmzZ6ov+996bYvTZoSd+dcN56wj5jZ9wHM7E9mttvMJoCrgOP87BvxgrCTLPDTRA7QCBFRNnbvhs9/3hP1RYuC8/3Lv3hvkP72t9AJ9mSouDvnHHAN8ICZ/WtN+vyabP8NWOevrwDOcs7Nds4dChwB3BNfk0W7hD0q53GopIiPspzfRx+Fvj7YYw9PuOux996wdq1npf/t33o3gI4hyBk/uQAn4AXd1lIz7BG4EfiDn74CmF+zzzK8UTLrgfeG1aGAan7I2wgHES9lOL/f+lZ4gPQTnzB76aWsW5o86A1V0SxFeqtQRKeo5/f55+Gss+C22xrnu+UW+MAH0mlTHtAbqqJpivBWoWidop3fu+7yXCn77Rcs7P398Kc/eTZ7UsJeRFeWxF1MQUMly00Rzu/EBCxd6on6CScE5/vyl728v/sdHHhgcu2ZnINofHwcM2N8fLwQ8/5L3MUUNFSy3KRxflu1csfH4fDDoasLLr64fp4994Q1azwr/QtfSCdAumzZsimTy0FB5v0PcsanuSigmi/y8FahSI4kz28rAdsbbwwPkH70o2YvvhhbMyOR57e6UUBVCJEGzQZst2+HgQFYsaJxeT/4AZx2WrxtjEqeg9AKqIpCUcTglfAIC9j+9reeK2XffYOF/S1vgU2bPJs9a2GHArsqg0z6NBe5ZcQkZRiH3cnUn3XU2X77/Z9Q18s//7PZxETWR1CfvLoqkVtGFIU8PwKLcKZ+3WoB8HPgyMD8e+wB99wDb31rWi0sF3LLiMKQ5Djsdt09cheFMzAwwCc+8VM8o/1xgoT9Ix+BHTvglVck7IkRZNKnucgtIyZJ6mMi7bp75C5qzPbtZqefHj7q5eabs25puSDOj3UksUjcxSRJiWi7N428fMEqb77fu+8OF/Q3vcls48ZMm1laJO6iUCQhYO2OVY5rrHM7x5aXp4fdu83+4R/CRf1LX8pvgLQsSNxF5mRtcebBcm9XnLN+enjiCbM3vrGxoM+aZTY6mkpzhEncRcbkweLMg8+9XXHO6k3Jm24Kt9I/9CGzF15ItBmiDhJ3kSlZW5yTtPv00O7+7Ypzmv34wgtmZ54ZLurf+U7sVYsISNxFS8TlSsnz3Bxp0q44p/EE9LvfhQv6UUd5LhqRPRJ3EZk4hSQvlnvWxNGnScQuJibMLrooXNS/+EUvmCryg8RdRCZOQc6Dzz0vZB1YrmXjRrNjjgkX9bvvzqyJIgSJu4gsKnG7UvIkap3O974XLuhnnOG9mCTyjcS9w2nFcpYrpVzs2GF21lnhov7tb2fdUhGFRuKuuWUKQjvzmrTyJZmo05xq3pV88vvfe5Nz9fbCTTfVz3PkkfDYY568f+Qj6bZPJEiQ6qe5yHJvTLs+61ZdLM26UuRTzxcTE2b/9E/hVvrf/Z0CpEUHTflbbNqdBjfpaXQ1TW8+ePJJeM97YO3axvl+8xs4/vh02iSSRVP+Fpx2p8FN+ksySU7TGwdJuYzyUu4PfuB93Wj+/GBhP+00eP55z2aXsHcIQSb95AIcAtwB3A/cB5znp88FVgIP+3/n+OkO+DrwCLAWODasDrllGhPXvCZJjVZJOviax8m2si73xRfNBgbCXS833thWc0TOoZ3RMsD8SYEG9gUeAo4CLgGW+ulLgYv99VOAH/sifzxwd1gdEvfGxC0kcQt91PZFqT+vk20FlVupVNrq27D2rllj1tPTWNDf8Aaz8fG2Dk8UhLbEfcYOcAtwErAemG+v3QDW++vfBD5Sk//VfEGLxD2cuAQ5SYtzsn2VSsUqlUrdtkatP6+TbQWVO32J2rfB5S4NtdKXLlWAtNOITdyBPuAxYD9gW026m/wfuBU4oWbbz4H+OmUNAqPA6MKFC9Pqi44nDRdKI/Fupv7aG0WQaGY92VZQua3WNXnMU/c90GA0VNTvuqutQxEFJhZxB/YBVgOn+/9vm7b9GYsg7rWLLPf0SHoSrzAxDau/3s2hHXFO0zfe6o1oZlmnhgr6qaeaPf98W4cgSkDb4g50A7cD59ekyS1TQJK23MPEO6z+ZiziPEy2Va/cSqXSUt96x9xjcF2oqF93XSxNFyWhLXHHc7ncAHx1WvqlTA2oXuKvv4+pAdV7wuqQuKdH0i8chYl3WP1hrpg8z0vTSt+uXWsGLzQU9EMPNXv00fSOQxSHdsX9BP+HuhZY4y+nABU8l8vDwM+AufbazeAKYAPwB0JcMtYB4p63SbOSbE8zAteo/qLPadNM305MmF1ySWMLHcz22+9K27Urg4MQhaFtt0zSS5nFvRNfzc/juPQ8sHmz2dvfHi7qcEJuXE8i30jcM6TolmgWlE2ofvSjcEF/85ufsEMOOTrRG2LZ+lVI3DNFn5hrTFkF56WXzD75yXBRv+aa9utqdnhpWZ+IOhmJe4bIcg+mjIKzbp3ZPvs0FvRDDjHbsCG+OpsxIPQ7LCeNxF0ThyXE5ORP4+PjOOembItz0q4i08o883nEDC67zJu865hjYPv2+vnOPx927fLmTj/ssPjqX7hwYWh63id3E/EjcU+AkZERBgcHX50G18xeFfhqtcrQ0BADAwNZNjEXFF1wnnoK3vlOmDXLE+4gfvEL7wbwla9AV1f87Whm1s9mbgCiZASZ9GkuZXPLdOojcFT/eVH76bbbwn3pJ59stm1bem0K6/slS5bMcN8U3QUm5HNPnU4MorbiPy+Sz33nTrNPfzpc1K+6KuuWzqRePzvnbMmSJVk3TbSJxD1limqRtkOrx5z30TL332+2//6NBf31rzd7+OH46oy7Tzrx99gpSNxTpkgWaVyU6WllYsLsq18Nt9LPO8/slVfirbvd3069G0OZzo2YisQ9A/JukcZNGazDp54yO+GEcFFftcrLn8Q5bqcfg24MrU5oJvKPxF1MIQlRKvLTyk9+Ei7oJ55o9swzr+2T1PG2Y2UH3RgqlUphz41ojMRdvEqSIlykp5WXXzb7zGfCRf3KK+vvn9STSjvlNroxFOnciOaRuHcAzV68ZXCftMMDD5hVKo0F/aCDzB56qHE5US3sZs9POzffTj+3nYjEveREEYRODa5dfnm4lf7ZzzYfII0ipEl+QLydekTxkbiXnChC00nW3dNPmy1aFC7qK1dGLzuKkKbZ53K/dBYS95ITxRoP+vZnpVIpjRCsXBku6H/5l574t0OzQtqpT0sieSTuJSeqZTg8PFx3eFyRH+FfecXs3HPDRf3yy9NvWyc9LYl0aSTumjisBDQzcVQtAwMD7LPPPjPSizgj48MPw0EHQXc3fOMb9fMccAA8+KAn73/91+m2D6KfHyFiIUj101yKaLnnzbcZtT1FdxVceWW4lb5kiTfkMQ/k7fciygENLHfnbc+W/v5+Gx0dzboZTTM5pW/tXOS9vb2Fmsp3cq756VSrVcbGxtJvUBNs2wZnnAGrVjXOd/vtcPLJqTRJiExxzq02s/562+SWaYEyfGSinqvAOcf4+Dh9fX2MjIzEUs/kR0tmzZrVcrl33OF9CGPOnGBhf9e7YOtWz2aXsAuB3DKtUHSXxiSTroLJttceSxzB1XbGXb/yitnf/E246+VrX/Mm+hKiE0FumXgpokujEUkdTyvlPvIILFoEmzYFlztnDtx1F/zZn7XcNCFKQVtuGefctc65zc65dTVpFznnNjrn1vjLKTXbLnTOPeKcW++ce088h5Avyjb6odnP3UV1sUT5jN5VV3mulyOOCBb2wUHYuROefnqqsMfh+hGidASZ9JMLsAg4FlhXk3YRcEGdvEcB9wKzgUOBDUBXWB1Fc8uYlWv0QzPjsFtxsYSVu22b2UknhbtebrstuO165V50MrT7EhPQ16S4XwhcWPP/7cA7w8ovoriXiWYEspUXcYLKXbbsp6GC/ud/brZlS3jbi/iCUJkMA5EtSYn7GLAWuBaY46dfDny0Jt81wJkBZQ4Co8DowoULU+uMMpDUfOyNymw1iPxa0LbL9t33qlBRv+yyaAHSogW39aQh4iQJcT8I6MLz2S8HrrWI4l67yHJvnqzEoVULecMGswULGgv6fvuZrVuXbruyomjtFfmmkbi3NM7dzP5kZrvNbAK4CjjO37QROKQm6wI/rSNII7CX1Rj7qEHka6/1AqSHHw5PPFG/zHPO8QKkzz4LRx8dX7uSGK8fF1GCzEK0RZDq1y7MtNzn16z/T+Amf/1opgZU/0hJA6rTScuiztINEea6efZZs8WLG1vpYPajH8XrWkp6vH6chFnu8seLKNCOWwb4d2AT8ArwBPAp4EbgD3g+9xXTxH4Z3iiZ9cB7w8q3koh7Wo/beXysv/POcEE/7jizzZu9/EndCPPYN9NpdOzyx4uotCXuaSxlEPe0LOq8CMCuXWYXXBAu6pdeOjNAmpQIFyW4GmSdF+HmJPJFR4h71o+zaV6YWR7ro4+a9fU1FvS99zZbuza4jKREuOjiWJSbk8gPpRf3PFizeWhDklx3XbiVfvbZZi+9FF5WUiJc9HNQ9JuTSJ/Si3teLoqsnx7i5rnnzN7//nBRv+WWaOUmKcJFPgdFvzmJ9Cm9uOtxNl7uuitc0Ht67rUrrri55TqKLMJJon4RUSi9uOfFci8yu3ebLV0aLurwv5q2KiVU5UTnNT+UXtw76XE27gvr3nvDBb27e5fNn7840g20k85JnORdOHVe80Xpxd0s/xdFHMR5YX3+881Y6TcYzH61X6O4vtJ6mirTeS+CcGb5lFymcx0XHSHunUC7F9bTTzcj6GZw2gwBj1p3GnGQIohhFIrgXswqvlW2cx0XEveS0OqFdfPNzYn6ggX9geJS7+ICrFKp1L3A0hCqIohhFIowMCCrPi/buY4LiXtJiPID37XL7PjjwwV98eLX3iANs46Gh4etUqnMqL+eBZWGpVUEMYxCEQQsKwu6bOc6LiTuJaGZC2vduuas9P/8z+A6Gvk1owhQ0j7SIohhFIriesjC9122cx0XEvcSEXRhXXhhuKD39HgvJrVDniyooohhFBQ0rE8Zz3UcSNxLyjPPNGelf+lLU/drR0DyZkFJDDsHneuZSNxLxve/35yor18/c992LSBZUELkh0bi3tKXmET6TEzAokXe141OPz0434knwu7dnrwfeeTM7e1+yWlgYIChoSGq1SrOOarVKkNDQwwMDITu2+hLVWl8xUqITsJ54p8t/f39Njo6mnUzcskDD8BRR4XnW7ECTj01PN+sWbOod86dc0xMTLTQwuYYGRlhcHBwyo2lt7eXoaEhAM455xxefvnlV7f19PRw7bXXNnXTEKJTcc6tNrP+utsk7vnku9+FD384LNcEe+31eq666itNi2BfXx/j4+Mz0qvVKmNjY5Hb2SyN6t2+fTtbt26dsa1SqfDUU08l1iYhik4jcZdbJmUauR927IAPfchzvTQW9v8NOKCLF1/8U6SPY9f7oDTA9u3bE3WFNPowdD1hBwLTg5BrR4gagpzxaS5FD6g2G8UPCkb+4z/e1lSAFN4YyzDEKC8jxUWjUTb10ieXKMekQG/raCRKMUGjZZIjiqjMFLK/DxX0j3/cm463/v6NhyE2umCDyqpUKolc5I36qd6NZrItzZK3IZpFQjfG4iJxT5AoouK9ADTfYG2oqN9998y6olyEYXmDXkZK0poPutkMDw9bd3f3lHq7u7sj1Zunl6uKhm6MxUXiniDNikozk3edfrrZ9u2N62v28Tnsgg1zh6R9kbfrFpBAtY5ujMVF4p4gjURlxw6zs84KF/Vzz/3VjHLbFbuwCzZolseiXuRyLbSObozFpZTinpcAUD1R2XPPd1pX1+4QUX/QDj74eBseHp5xLEuWLAmdnTHs2Ju5YKeXE+T7LspFnpffRNHQjbG4tCXuwLXAZmBdTdpcYCXwsP93jp/ugK8DjwBrgWPDyrcWxD1vP8bh4WFbuLBqsCzUSl+27LUAadCxBFndk4LVzLG30kd561eRHroxFpN2xX0RcOw0cb8EWOqvLwUu9tdPAX7si/zxwN1h5VsL4p6nx8hNm8ze/OZw18tvflN//yi+76hfRGoUwAy6kHWRC1Ec2nbLAH3TxH09MN9fnw+s99e/CXykXr5GS1Rxz0MAqJnJuz74QbPnn29cTrOjViYFvN1jl3UuRHloJO6tvqF6kJlt8tefBA7y1w8GHq/J94SfNgPn3KBzbtQ5N7ply5ZIlS9cuDBSely89BJ87GPhk3fdcIMn7z/8IeyzT+Myg9rsnJvyf29vL8uXL2/72NudOKyI6M1V0ZEEqX7twkzLfdu07c/4f28FTqhJ/znQH1Z+3n3ua9Z4H7poZKUffrjZ2Fj0soOOZcmSJYEulSj5p5OHp5400ZOKKDOUzS1jlrxveGLC7MtfDne9LF06NUDaCs0cS22eSqVilUql6dE1tWQZr9Dn2YSIlyTE/VKmBlQv8dffx9SA6j3NlJ+nce5PPml27LHhov7rX6fXpjDrM2qQNU5Ltt15dfRhZSFapy1xB/4d2AS8gudD/xRQ8V0uDwM/A+b6eR1wBbAB+EMzLhnLibjfcku4oMMKO+SQo1J/pA8T76gCFpcF3d68OulY0LLcRZlp23JPeslK3F96yZuYK0zUe3oGM/XZhol3EYQzKwtaPndRZiTu01i71qy3t7Gg9/WZPfpoPiy/sDYUweXRab5+IdJA4m5egPTSS8Ot9AsuMNu167X98uCzbUa88x6sjHoDkiALEU5Hi/vmzWbHHRcu6nfeWX//PFjuZvkUu6QEW64UIZqjI8X91lvDBX3xYrNnn21cjoSmMfUEW9P3CpEOHSPuO3eanXNOuKhffXW0cvNoNeeVOG6GeXCFCVEESi/u991ntt9+jQV9wQKzDRvaqiYzinRzicPqluUuRHM0EvdW55bJBatWefO8HH00PPdc/Tznnw+7dsHjj8Nhh6XbvjgYGRlhcHCQ8fFxzIzx8XEGBwdzOz/KY489Fim9HsuXL6e3t3dK2uTcOkKI5ii0uJ94YvC2O+7wbPavfAW6utJrU9wUbaKvOCZ1GxgYYGhoiGq1inOOarXK0NAQAwMDcTVTiNLjPMs+W/r7+210dDTyftMmTuSkk+B734PXvS6mhuWAWbNmUe8cOeeYmJjIoEWNmXzSqL0h9fb2SpyFSADn3Goz66+3rdCW+49+BG96E3zzm56V/tOfNi/sWU8D22z9WU1v3CpxW91ZnychCkuQMz7NJe03VLMe3hjlU3n1vmvaKUMxsz5PQuQdyj5aJipZj8Zopv56wgZYpVJpWtyKNMqmHlmfJyHyjsR9GlmPo26m/naFLc9Wb7M3nazPUxEp+g1dREPiPo2sLcJm6m9H2IaHh62rq6upp4O0haAI0wQXlTzf0EUySNynkfVF0Ez9QcLW1dXVsJ1B7pzpN4es+iDJycY6Hd0MOw+Jex3atVqT3r+RSDcSuKALfPqFnpUQZPVhkU5AbqzOQ+IeM2lZlM26V2oJusCntzErIZB1mRzq286jkbgXepx7VqT11ujAwEDgi0pBr/MHjX/v6uqaMt48q/Hzp5xySqR00TyatkFMIUj101yyttwnH/3xfdr41k4eRnFEtcaijKHv7u6ekq+7uzuWp49GrpRWrEu5ZppHfdVZILdMMK34ttN8/G3FBdTMBT48PGw9PT1Tyu3p6WlbDMLa24rPXUFVIeojcW9AvTdAwwQ7bcFJwhpL6gYVVm7UeuVHFiIYiXsAw8PDDYU9zKIs8uNvUq6lsHKj3hg1AkSIYBqJe0cHVJsJgAYFGAcGBhgbG2NiYoKxsbFMZjxsZ1KtpAKqQfubGX19fQCRJhYr2sRpUdCkaCJRglS/mQUYA/4ArMG/gwBzgZXAw/7fOWHlZGW5Nxo2SM59u+26hurtP9kf7TyJhL1E1Uwba5+KKpXKjNhAns9LsyiWIOKApNwyvrgfMC3tEmCpv74UuDisnLTEfborpZG/Pe+uljh80bWjhKbf6NoRmtpyo7axnuh1d3dbpVIprAusHooliDhIW9zXA/P99fnA+rBykhD36UK+ZMmSGaLR09MzYzhgUaynOH3RYULcan+00sZOET3FEkQcJCnujwK/B1YDg37atprtrvb/afsOAqPA6MKFC2M94EYuh+lLpVIpZGA0ThFMyj3VShs7RfQ65SYmkiVJcT/Y/3sgcC+waLqYA8+ElRO35d7IEo0qGnkdFROnz7aZ/mpFdJppY7OusrKJnnzuIg4SE/cpBcFFwAXkwC0TZok2Kxp5vwDjuvGEBUHbsZwbtTHIv17GAGo98mo4iOKQiLgDewP71qz/X2AxcClTA6qXhJWVluUeNWjYSY/O7QRBWyWovqK6yoRIm6TE/TDfFXMvcB+wzE+vAD/HGwr5M2BuWFlxi3uQxb1kyZJIotEp/t9a0nxa6cT+FSJOUnHLtLOkMVomrYBgGUjLXdCp/StEXHSkuLfL8PBw3eBeWf2/WZD3mIYQeaeRuHf09ANBjIyMMDg4yNatW6ekVyqVhq/Ki2gMDAxEmopACNE8zhP/bOnv77fR0dGsm/EqfX19jI+Pz0ivVquMjY2l3yAhhKiDc261mfXX2ybLvQ5BXzkKShdCiLwhca9DmWciFEJ0BhL3OuhblEKIoiNxr4MCfUKIoqOAqhBCFBQFVIUQosOQuAshRAmRuAshRAmRuAshRAmRuAshRAnJxWgZ59wWYOb7/uXkAOCprBuRc9RH4aiPwumEPqqa2bx6G3Ih7p2Ec240aOiS8FAfhaM+CqfT+0huGSGEKCESdyGEKCES9/QZyroBBUB9FI76KJyO7iP53IUQooTIchdCiBIicRdCiBIicY8Z59xc59xK59zD/t85AfnO9vM87Jw7uyb9F8659c65Nf5yYHqtTxbn3GL/2B5xzi2ts322c+47/va7nXN9Ndsu9NPXO+fek2rDU6TVPnLO9TnnXqz53fxb6o1PkSb6aZFz7vfOuV3OuTOnbat77ZWOoC9na2ltAS4BlvrrS4GL6+SZC/zR/zvHX5/jb/sF0J/1cSTQL13ABuAwoAe4FzhqWp5zgX/z188CvuOvH+Xnnw0c6pfTlfUx5ayP+oB1WR9DjvqpD/ivwA3AmTXpgdde2RZZ7vHzQeB6f/164LQ6ed4DrDSzp83sGWAlsDid5mXGccAjZvZHM3sZuAmvr2qp7bubgROdc85Pv8nMdprZo8Ajfnllo50+6iRC+8nMxsxsLTAxbd+OufYk7vFzkJlt8tefBA6qk+dg4PGa/5/w0yb5lv9o/fclunDDjnlKHjPbBTwLVJrctwy000cAhzrn/p9z7pfOuXcl3dgMaef30Cm/JfbIugFFxDn3M+C/1Nm0rPYfMzPnXNSxpgNmttE5ty/wH8DH8B4thWjEJmChmW11zr0N+KFz7mgzey7rholskOXeAmb2V2Z2TJ3lFuBPzrn5AP7fzXWK2AgcUvP/Aj8NM5v8+zzwbcrjfgg85np5nHN7AK8Dtja5bxlouY98l9VWADNbjeeTPjLxFmdDO7+HTvktSdwTYAUwGYE/G7ilTp7bgZOdc3P80TQnA7c75/Zwzh0A4JzrBt4PrEuhzWnwO+AI59yhzrkevGDgiml5avvuTGCVeVGwFcBZ/kiRQ4EjgHtSaneatNxHzrl5zrkuAOfcYXh99MeU2p02zfRTEHWvvYTamS1ZR3TLtuD5P38OPAz8DJjrp/cDV9fkOwcvMPgI8Ek/bW9gNbAWuA/4GiUaFQKcAjyEZ1Uu89O+BHzAX98T+J7fJ/cAh9Xsu8zfbz3w3qyPJW99BJzh/2bWAL8HTs36WDLup7fj+dNfwHv6u69m3xnXXhkXTT8ghBAlRG4ZIYQoIRJ3IYQoIRJ3IYQoIRJ3IYQoIRJ3IYQoIRJ3IYQoIRJ3IYQoIf8fLPL8lsgdAxcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X_testset, y_testset, color='black')\n",
    "plt.plot(X_testset, LinReg.predict(X_testset), color='blue', linewidth=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id = \"eva\"></a>\n",
    "\n",
    "\n",
    "### <span style=\"color:#0b486b\">1.2 Evaluation</span>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, you will learn the about the different evaluation models and metrics. You will be able to identify the strengths and weaknesses of each model and how to incorporate underfitting or overfilling them also referred to as the Bias-Variance trade-off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><b> Here's a list of useful functions: </b><br>\n",
    "    mean -> np.mean()<br>\n",
    "    exponent -> **<br>\n",
    "    absolute value -> abs()\n",
    "    \n",
    "    \n",
    "We use three evaluation metrics:\n",
    "\n",
    "$$ MAE = \\frac{\\sum_{j=1}^n|y_i-\\hat y_i|}{n} $$\n",
    "\n",
    "$$ MSE = \\frac{\\sum_{j=1}^n (y_i-\\hat y_i)^2}{n} $$\n",
    "\n",
    "$$ RMSE = \\sqrt{\\frac{\\sum_{j=1}^n (y_i-\\hat y_i)^2}{n}} $$\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49.13321801160823\n"
     ]
    }
   ],
   "source": [
    "# ss_X = StandardScaler()\n",
    "# ss_y = StandardScaler()\n",
    "# X_trainset = ss_X.fit_transform(X_trainset)\n",
    "# X_testset = ss_X.transform(X_testset)\n",
    "# y_trainset = ss_y.fit_transform(y_trainset)\n",
    "# y_testset = ss_y.transform(y_testset)\n",
    "print(np.mean(abs(LinReg.predict(X_testset) - y_testset)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3563.8152601287006\n"
     ]
    }
   ],
   "source": [
    "# ss_X = StandardScaler()\n",
    "# ss_y = StandardScaler()\n",
    "# X_trainset = ss_X.fit_transform(X_trainset)\n",
    "# X_testset = ss_X.transform(X_testset)\n",
    "# y_trainset = ss_y.fit_transform(y_trainset)\n",
    "# y_testset = ss_y.transform(y_testset)\n",
    "print(np.mean((LinReg.predict(X_testset) - y_testset) ** 2) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "59.697698951707515\n"
     ]
    }
   ],
   "source": [
    "# ss_X = StandardScaler()\n",
    "# ss_y = StandardScaler()\n",
    "# X_trainset = ss_X.fit_transform(X_trainset)\n",
    "# X_testset = ss_X.transform(X_testset)\n",
    "# y_trainset = ss_y.fit_transform(y_trainset)\n",
    "# y_testset = ss_y.transform(y_testset)\n",
    "print(np.mean((LinReg.predict(X_testset) - y_testset) ** 2) ** (0.5) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">2. Classification</span>\n",
    "\n",
    "<a id = \"cls\"></a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"data\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.1 Skulls dataset</span>\n",
    "\n",
    "In this section, we will take a closer look at a data set.\n",
    "\n",
    "Everything starts off with how the data is stored. We will be working with .csv files, or comma separated value files. As the name implies, each attribute (or column) in the data is separated by commas.\n",
    "\n",
    "Next, a little information about the dataset. We are using a dataset called skulls.csv, which contains the measurements made on Egyptian skulls from five epochs.\n",
    "\n",
    "#### The attributes of the data are as follows: \n",
    "\n",
    "<img src = \"image/Skull.png\", align = 'left'>\n",
    "\n",
    "<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>\n",
    "\n",
    "<b>epoch</b> - The epoch the skull as assigned to, a factor with levels c4000BC c3300BC, c1850BC, c200BC, and cAD150, where the years are only given approximately.\n",
    "\n",
    "<b>mb</b> - Maximal Breadth of the skull.\n",
    "\n",
    "<b>bh</b> - Basiregmatic Heights of the skull.\n",
    "\n",
    "<b>bl</b> - Basilveolar Length of the skull.\n",
    "\n",
    "<b>nh</b> - Nasal Heights of the skull.\n",
    "\n",
    "#### Importing Libraries\n",
    "\n",
    "Before we begin, we need to import some libraries, as they have useful functions that will be used later on.<br>\n",
    "\n",
    "If you look at the imports below, you will notice the return of **numpy**! Remember that numpy is homogeneous multidimensional array (ndarray)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "We need the **pandas** library for a function to read .csv files\n",
    "<ul>\n",
    "    <li> <b>pandas.read_csv</b> - Reads data into DataFrame </li>\n",
    "    <li> The read_csv function takes in <i>2 parameters</i>: </li>\n",
    "    <ul>\n",
    "        <li> The .csv file as the first parameter </li>\n",
    "        <li> The delimiter as the second parameter </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "\n",
    "-----------------------------\n",
    "Save the \"<b> skulls.csv </b>\" data file into a variable called <b> my_data </b>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>mb</th>\n",
       "      <th>bh</th>\n",
       "      <th>bl</th>\n",
       "      <th>nh</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>75.500000</td>\n",
       "      <td>133.973333</td>\n",
       "      <td>132.546667</td>\n",
       "      <td>96.460000</td>\n",
       "      <td>50.933333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>43.445368</td>\n",
       "      <td>4.890680</td>\n",
       "      <td>4.939346</td>\n",
       "      <td>5.377844</td>\n",
       "      <td>3.207932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>119.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>44.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>38.250000</td>\n",
       "      <td>131.000000</td>\n",
       "      <td>129.000000</td>\n",
       "      <td>93.000000</td>\n",
       "      <td>49.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>75.500000</td>\n",
       "      <td>134.000000</td>\n",
       "      <td>133.000000</td>\n",
       "      <td>96.000000</td>\n",
       "      <td>51.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>112.750000</td>\n",
       "      <td>137.000000</td>\n",
       "      <td>136.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>53.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>150.000000</td>\n",
       "      <td>148.000000</td>\n",
       "      <td>145.000000</td>\n",
       "      <td>114.000000</td>\n",
       "      <td>60.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0          mb          bh          bl          nh\n",
       "count  150.000000  150.000000  150.000000  150.000000  150.000000\n",
       "mean    75.500000  133.973333  132.546667   96.460000   50.933333\n",
       "std     43.445368    4.890680    4.939346    5.377844    3.207932\n",
       "min      1.000000  119.000000  120.000000   81.000000   44.000000\n",
       "25%     38.250000  131.000000  129.000000   93.000000   49.000000\n",
       "50%     75.500000  134.000000  133.000000   96.000000   51.000000\n",
       "75%    112.750000  137.000000  136.000000  100.000000   53.000000\n",
       "max    150.000000  148.000000  145.000000  114.000000   60.000000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Note！！！！！！！！！！！！！！！！！！: change to your local file location\n",
    "my_data = pandas.read_csv(\"D:/filp/filp00/Notebook/Data/skulls.csv\", delimiter=\",\")\n",
    "my_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the data in <b> my_data </b>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     Unnamed: 0    epoch   mb   bh   bl  nh\n",
      "0             1  c4000BC  131  138   89  49\n",
      "1             2  c4000BC  125  131   92  48\n",
      "2             3  c4000BC  131  132   99  50\n",
      "3             4  c4000BC  119  132   96  44\n",
      "4             5  c4000BC  136  143  100  54\n",
      "..          ...      ...  ...  ...  ...  ..\n",
      "145         146   cAD150  137  125   85  57\n",
      "146         147   cAD150  129  128   81  52\n",
      "147         148   cAD150  140  135  103  48\n",
      "148         149   cAD150  147  129   87  48\n",
      "149         150   cAD150  136  133   97  51\n",
      "\n",
      "[150 rows x 6 columns]\n"
     ]
    }
   ],
   "source": [
    "print (my_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the type of <b> my_data </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "print(type(my_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "There are various functions that the **pandas** library has to look at the data\n",
    "<ul>\n",
    "    <li> <font color = \"red\"> [DataFrame Data].columns </font> - Displays the Header of the Data </li>\n",
    "    <ul> \n",
    "        <li> Type: pandas.indexes.base.Index </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    <li> <font color = \"red\"> [DataFrame Data].values </font> (or <font color = \"red\"> [DataFrame Data].as_matrix() </font>) - Displays the values of the data (without headers) </li>\n",
    "    <ul>\n",
    "        <li> Type: numpy.ndarray </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    <li> <font color = \"red\"> [DataFrame Data].shape </font> - Displays the dimensions of the data (rows x columns) </li>\n",
    "    <ul>\n",
    "        <li> Type: tuple </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "\n",
    "----------\n",
    "Using the <b> my_data </b> variable containing the DataFrame data, retrieve the <b> header </b> data, data <b> values </b>, and <b> shape </b> of the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Unnamed: 0', 'epoch', 'mb', 'bh', 'bl', 'nh'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print (my_data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 'c4000BC' 131 138 89 49]\n",
      " [2 'c4000BC' 125 131 92 48]\n",
      " [3 'c4000BC' 131 132 99 50]\n",
      " [4 'c4000BC' 119 132 96 44]\n",
      " [5 'c4000BC' 136 143 100 54]\n",
      " [6 'c4000BC' 138 137 89 56]\n",
      " [7 'c4000BC' 139 130 108 48]\n",
      " [8 'c4000BC' 125 136 93 48]\n",
      " [9 'c4000BC' 131 134 102 51]\n",
      " [10 'c4000BC' 134 134 99 51]\n",
      " [11 'c4000BC' 129 138 95 50]\n",
      " [12 'c4000BC' 134 121 95 53]\n",
      " [13 'c4000BC' 126 129 109 51]\n",
      " [14 'c4000BC' 132 136 100 50]\n",
      " [15 'c4000BC' 141 140 100 51]\n",
      " [16 'c4000BC' 131 134 97 54]\n",
      " [17 'c4000BC' 135 137 103 50]\n",
      " [18 'c4000BC' 132 133 93 53]\n",
      " [19 'c4000BC' 139 136 96 50]\n",
      " [20 'c4000BC' 132 131 101 49]\n",
      " [21 'c4000BC' 126 133 102 51]\n",
      " [22 'c4000BC' 135 135 103 47]\n",
      " [23 'c4000BC' 134 124 93 53]\n",
      " [24 'c4000BC' 128 134 103 50]\n",
      " [25 'c4000BC' 130 130 104 49]\n",
      " [26 'c4000BC' 138 135 100 55]\n",
      " [27 'c4000BC' 128 132 93 53]\n",
      " [28 'c4000BC' 127 129 106 48]\n",
      " [29 'c4000BC' 131 136 114 54]\n",
      " [30 'c4000BC' 124 138 101 46]\n",
      " [31 'c3300BC' 124 138 101 48]\n",
      " [32 'c3300BC' 133 134 97 48]\n",
      " [33 'c3300BC' 138 134 98 45]\n",
      " [34 'c3300BC' 148 129 104 51]\n",
      " [35 'c3300BC' 126 124 95 45]\n",
      " [36 'c3300BC' 135 136 98 52]\n",
      " [37 'c3300BC' 132 145 100 54]\n",
      " [38 'c3300BC' 133 130 102 48]\n",
      " [39 'c3300BC' 131 134 96 50]\n",
      " [40 'c3300BC' 133 125 94 46]\n",
      " [41 'c3300BC' 133 136 103 53]\n",
      " [42 'c3300BC' 131 139 98 51]\n",
      " [43 'c3300BC' 131 136 99 56]\n",
      " [44 'c3300BC' 138 134 98 49]\n",
      " [45 'c3300BC' 130 136 104 53]\n",
      " [46 'c3300BC' 131 128 98 45]\n",
      " [47 'c3300BC' 138 129 107 53]\n",
      " [48 'c3300BC' 123 131 101 51]\n",
      " [49 'c3300BC' 130 129 105 47]\n",
      " [50 'c3300BC' 134 130 93 54]\n",
      " [51 'c3300BC' 137 136 106 49]\n",
      " [52 'c3300BC' 126 131 100 48]\n",
      " [53 'c3300BC' 135 136 97 52]\n",
      " [54 'c3300BC' 129 126 91 50]\n",
      " [55 'c3300BC' 134 139 101 49]\n",
      " [56 'c3300BC' 131 134 90 53]\n",
      " [57 'c3300BC' 132 130 104 50]\n",
      " [58 'c3300BC' 130 132 93 52]\n",
      " [59 'c3300BC' 135 132 98 54]\n",
      " [60 'c3300BC' 130 128 101 51]\n",
      " [61 'c1850BC' 137 141 96 52]\n",
      " [62 'c1850BC' 129 133 93 47]\n",
      " [63 'c1850BC' 132 138 87 48]\n",
      " [64 'c1850BC' 130 134 106 50]\n",
      " [65 'c1850BC' 134 134 96 45]\n",
      " [66 'c1850BC' 140 133 98 50]\n",
      " [67 'c1850BC' 138 138 95 47]\n",
      " [68 'c1850BC' 136 145 99 55]\n",
      " [69 'c1850BC' 136 131 92 46]\n",
      " [70 'c1850BC' 126 136 95 56]\n",
      " [71 'c1850BC' 137 129 100 53]\n",
      " [72 'c1850BC' 137 139 97 50]\n",
      " [73 'c1850BC' 136 126 101 50]\n",
      " [74 'c1850BC' 137 133 90 49]\n",
      " [75 'c1850BC' 129 142 104 47]\n",
      " [76 'c1850BC' 135 138 102 55]\n",
      " [77 'c1850BC' 129 135 92 50]\n",
      " [78 'c1850BC' 134 125 90 60]\n",
      " [79 'c1850BC' 138 134 96 51]\n",
      " [80 'c1850BC' 136 135 94 53]\n",
      " [81 'c1850BC' 132 130 91 52]\n",
      " [82 'c1850BC' 133 131 100 50]\n",
      " [83 'c1850BC' 138 137 94 51]\n",
      " [84 'c1850BC' 130 127 99 45]\n",
      " [85 'c1850BC' 136 133 91 49]\n",
      " [86 'c1850BC' 134 123 95 52]\n",
      " [87 'c1850BC' 136 137 101 54]\n",
      " [88 'c1850BC' 133 131 96 49]\n",
      " [89 'c1850BC' 138 133 100 55]\n",
      " [90 'c1850BC' 138 133 91 46]\n",
      " [91 'c200BC' 137 134 107 54]\n",
      " [92 'c200BC' 141 128 95 53]\n",
      " [93 'c200BC' 141 130 87 49]\n",
      " [94 'c200BC' 135 131 99 51]\n",
      " [95 'c200BC' 133 120 91 46]\n",
      " [96 'c200BC' 131 135 90 50]\n",
      " [97 'c200BC' 140 137 94 60]\n",
      " [98 'c200BC' 139 130 90 48]\n",
      " [99 'c200BC' 140 134 90 51]\n",
      " [100 'c200BC' 138 140 100 52]\n",
      " [101 'c200BC' 132 133 90 53]\n",
      " [102 'c200BC' 134 134 97 54]\n",
      " [103 'c200BC' 135 135 99 50]\n",
      " [104 'c200BC' 133 136 95 52]\n",
      " [105 'c200BC' 136 130 99 55]\n",
      " [106 'c200BC' 134 137 93 52]\n",
      " [107 'c200BC' 131 141 99 55]\n",
      " [108 'c200BC' 129 135 95 47]\n",
      " [109 'c200BC' 136 128 93 54]\n",
      " [110 'c200BC' 131 125 88 48]\n",
      " [111 'c200BC' 139 130 94 53]\n",
      " [112 'c200BC' 144 124 86 50]\n",
      " [113 'c200BC' 141 131 97 53]\n",
      " [114 'c200BC' 130 131 98 53]\n",
      " [115 'c200BC' 133 128 92 51]\n",
      " [116 'c200BC' 138 126 97 54]\n",
      " [117 'c200BC' 131 142 95 53]\n",
      " [118 'c200BC' 136 138 94 55]\n",
      " [119 'c200BC' 132 136 92 52]\n",
      " [120 'c200BC' 135 130 100 51]\n",
      " [121 'cAD150' 137 123 91 50]\n",
      " [122 'cAD150' 136 131 95 49]\n",
      " [123 'cAD150' 128 126 91 57]\n",
      " [124 'cAD150' 130 134 92 52]\n",
      " [125 'cAD150' 138 127 86 47]\n",
      " [126 'cAD150' 126 138 101 52]\n",
      " [127 'cAD150' 136 138 97 58]\n",
      " [128 'cAD150' 126 126 92 45]\n",
      " [129 'cAD150' 132 132 99 55]\n",
      " [130 'cAD150' 139 135 92 54]\n",
      " [131 'cAD150' 143 120 95 51]\n",
      " [132 'cAD150' 141 136 101 54]\n",
      " [133 'cAD150' 135 135 95 56]\n",
      " [134 'cAD150' 137 134 93 53]\n",
      " [135 'cAD150' 142 135 96 52]\n",
      " [136 'cAD150' 139 134 95 47]\n",
      " [137 'cAD150' 138 125 99 51]\n",
      " [138 'cAD150' 137 135 96 54]\n",
      " [139 'cAD150' 133 125 92 50]\n",
      " [140 'cAD150' 145 129 89 47]\n",
      " [141 'cAD150' 138 136 92 46]\n",
      " [142 'cAD150' 131 129 97 44]\n",
      " [143 'cAD150' 143 126 88 54]\n",
      " [144 'cAD150' 134 124 91 55]\n",
      " [145 'cAD150' 132 127 97 52]\n",
      " [146 'cAD150' 137 125 85 57]\n",
      " [147 'cAD150' 129 128 81 52]\n",
      " [148 'cAD150' 140 135 103 48]\n",
      " [149 'cAD150' 147 129 87 48]\n",
      " [150 'cAD150' 136 133 97 51]]\n"
     ]
    }
   ],
   "source": [
    "print (my_data.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 6)\n"
     ]
    }
   ],
   "source": [
    "print (my_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"datapre\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.2 Data Preprocessing</span>\n",
    "\n",
    "When we train a model, the model requires two inputs, X and y\n",
    "<ul>\n",
    "    <li> X: Feature Matrix, or array that contains the data. </li>\n",
    "    <li> y: Response Vector, or 1-D array that contains the classification categories </li>\n",
    "</ul>\n",
    "\n",
    "\n",
    "\n",
    "------------\n",
    "There are some problems with the data in my_data:\n",
    "<ul>\n",
    "    <li> There is a header on the data (Unnamed: 0    epoch   mb   bh   bl  nh) </li>\n",
    "    <li> The data needs to be in numpy.ndarray format in order to use it in the machine learning model </li>\n",
    "    <li> There is non-numeric data within the dataset </li>\n",
    "    <li> There are row numbers associated with each row that affect the model </li>\n",
    "</ul>\n",
    "\n",
    "To resolve these problems, I have created a function that fixes these for us:\n",
    "<b> removeColumns(pandasArray, column) </b>\n",
    "\n",
    "This function produces one output and requires two inputs.\n",
    "<ul>\n",
    "    <li> 1st Input: A pandas array. The pandas array we have been using is my_data </li>\n",
    "    <li> 2nd Input: Any number of integer values (order doesn't matter) that represent the columns that we want to remove. (Look at the data again and find which column contains the non-numeric values). We also want to remove the first column because that only contains the row number, which is irrelevant to our analysis.</li>\n",
    "    <ul>\n",
    "        <li> Note: Remember that Python is zero-indexed, therefore the first column would be 0. </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the column containing the target name since it doesn't contain numeric values.\n",
    "# Also remove the column that contains the row number\n",
    "# axis=1 means we are removing columns instead of rows.\n",
    "# Function takes in a pandas array and column numbers and returns a numpy array without\n",
    "# the stated columns\n",
    "def removeColumns(pandasArray, *column):\n",
    "    return pandasArray.drop(pandasArray.columns[[column]], axis=1).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------\n",
    "Using the function, store the values from the DataFrame data into a variable called new_data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\filp\\filp00\\anaconda\\envs\\tensorflow\\lib\\site-packages\\pandas\\core\\indexes\\base.py:4107: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n",
      "  result = getitem(key)\n"
     ]
    }
   ],
   "source": [
    "new_data = removeColumns(my_data, 0, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the data in <b> new_data </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[131 138  89  49]\n",
      " [125 131  92  48]\n",
      " [131 132  99  50]\n",
      " [119 132  96  44]\n",
      " [136 143 100  54]\n",
      " [138 137  89  56]\n",
      " [139 130 108  48]\n",
      " [125 136  93  48]\n",
      " [131 134 102  51]\n",
      " [134 134  99  51]\n",
      " [129 138  95  50]\n",
      " [134 121  95  53]\n",
      " [126 129 109  51]\n",
      " [132 136 100  50]\n",
      " [141 140 100  51]\n",
      " [131 134  97  54]\n",
      " [135 137 103  50]\n",
      " [132 133  93  53]\n",
      " [139 136  96  50]\n",
      " [132 131 101  49]\n",
      " [126 133 102  51]\n",
      " [135 135 103  47]\n",
      " [134 124  93  53]\n",
      " [128 134 103  50]\n",
      " [130 130 104  49]\n",
      " [138 135 100  55]\n",
      " [128 132  93  53]\n",
      " [127 129 106  48]\n",
      " [131 136 114  54]\n",
      " [124 138 101  46]\n",
      " [124 138 101  48]\n",
      " [133 134  97  48]\n",
      " [138 134  98  45]\n",
      " [148 129 104  51]\n",
      " [126 124  95  45]\n",
      " [135 136  98  52]\n",
      " [132 145 100  54]\n",
      " [133 130 102  48]\n",
      " [131 134  96  50]\n",
      " [133 125  94  46]\n",
      " [133 136 103  53]\n",
      " [131 139  98  51]\n",
      " [131 136  99  56]\n",
      " [138 134  98  49]\n",
      " [130 136 104  53]\n",
      " [131 128  98  45]\n",
      " [138 129 107  53]\n",
      " [123 131 101  51]\n",
      " [130 129 105  47]\n",
      " [134 130  93  54]\n",
      " [137 136 106  49]\n",
      " [126 131 100  48]\n",
      " [135 136  97  52]\n",
      " [129 126  91  50]\n",
      " [134 139 101  49]\n",
      " [131 134  90  53]\n",
      " [132 130 104  50]\n",
      " [130 132  93  52]\n",
      " [135 132  98  54]\n",
      " [130 128 101  51]\n",
      " [137 141  96  52]\n",
      " [129 133  93  47]\n",
      " [132 138  87  48]\n",
      " [130 134 106  50]\n",
      " [134 134  96  45]\n",
      " [140 133  98  50]\n",
      " [138 138  95  47]\n",
      " [136 145  99  55]\n",
      " [136 131  92  46]\n",
      " [126 136  95  56]\n",
      " [137 129 100  53]\n",
      " [137 139  97  50]\n",
      " [136 126 101  50]\n",
      " [137 133  90  49]\n",
      " [129 142 104  47]\n",
      " [135 138 102  55]\n",
      " [129 135  92  50]\n",
      " [134 125  90  60]\n",
      " [138 134  96  51]\n",
      " [136 135  94  53]\n",
      " [132 130  91  52]\n",
      " [133 131 100  50]\n",
      " [138 137  94  51]\n",
      " [130 127  99  45]\n",
      " [136 133  91  49]\n",
      " [134 123  95  52]\n",
      " [136 137 101  54]\n",
      " [133 131  96  49]\n",
      " [138 133 100  55]\n",
      " [138 133  91  46]\n",
      " [137 134 107  54]\n",
      " [141 128  95  53]\n",
      " [141 130  87  49]\n",
      " [135 131  99  51]\n",
      " [133 120  91  46]\n",
      " [131 135  90  50]\n",
      " [140 137  94  60]\n",
      " [139 130  90  48]\n",
      " [140 134  90  51]\n",
      " [138 140 100  52]\n",
      " [132 133  90  53]\n",
      " [134 134  97  54]\n",
      " [135 135  99  50]\n",
      " [133 136  95  52]\n",
      " [136 130  99  55]\n",
      " [134 137  93  52]\n",
      " [131 141  99  55]\n",
      " [129 135  95  47]\n",
      " [136 128  93  54]\n",
      " [131 125  88  48]\n",
      " [139 130  94  53]\n",
      " [144 124  86  50]\n",
      " [141 131  97  53]\n",
      " [130 131  98  53]\n",
      " [133 128  92  51]\n",
      " [138 126  97  54]\n",
      " [131 142  95  53]\n",
      " [136 138  94  55]\n",
      " [132 136  92  52]\n",
      " [135 130 100  51]\n",
      " [137 123  91  50]\n",
      " [136 131  95  49]\n",
      " [128 126  91  57]\n",
      " [130 134  92  52]\n",
      " [138 127  86  47]\n",
      " [126 138 101  52]\n",
      " [136 138  97  58]\n",
      " [126 126  92  45]\n",
      " [132 132  99  55]\n",
      " [139 135  92  54]\n",
      " [143 120  95  51]\n",
      " [141 136 101  54]\n",
      " [135 135  95  56]\n",
      " [137 134  93  53]\n",
      " [142 135  96  52]\n",
      " [139 134  95  47]\n",
      " [138 125  99  51]\n",
      " [137 135  96  54]\n",
      " [133 125  92  50]\n",
      " [145 129  89  47]\n",
      " [138 136  92  46]\n",
      " [131 129  97  44]\n",
      " [143 126  88  54]\n",
      " [134 124  91  55]\n",
      " [132 127  97  52]\n",
      " [137 125  85  57]\n",
      " [129 128  81  52]\n",
      " [140 135 103  48]\n",
      " [147 129  87  48]\n",
      " [136 133  97  51]]\n"
     ]
    }
   ],
   "source": [
    "print(new_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------\n",
    "Now, we have one half of the required data to fit a model, which is X or new_data\n",
    "\n",
    "Next, we need to get the response vector y. Since we cannot use .target and .target_names, I have created a function that will do this for us.\n",
    "\n",
    "<b> targetAndtargetNames(numpyArray, targetColumnIndex) </b>\n",
    "\n",
    "This function produces two outputs, and requires two inputs.\n",
    "<ul>\n",
    "    <li> <font size = 3.5><b><i>1st Input</i></b></font>: A numpy array. The numpy array you will use is my_data.values (or my_data.as_matrix())</li>\n",
    "    <ul>\n",
    "        <li> Note: DO NOT USE <b> new_data </b> here. We need the original .csv data file without the headers </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "<ul>\n",
    "    <li> <font size = 3.5><b><i>2nd Input</i></b></font>: An integer value that represents the target column . (Look at the data again and find which column contains the non-numeric values. This is the target column)</li>\n",
    "    <ul>\n",
    "        <li> Note: Remember that Python is zero-indexed, therefore the first column would be 0. </li>\n",
    "   </ul>\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    <li> <font size = 3.5><b><i>1st Output</i></b></font>: The response vector (target) </li>\n",
    "    <li> <font size = 3.5><b><i>2nd Output</i></b></font>: The target names (target_names) </li>\n",
    "</ul>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def targetAndtargetNames(numpyArray, targetColumnIndex):\n",
    "    target_dict = dict()\n",
    "    target = list()\n",
    "    target_names = list()\n",
    "    count = -1\n",
    "    for i in range(len(my_data.values)):\n",
    "        if my_data.values[i][targetColumnIndex] not in target_dict:\n",
    "            count += 1\n",
    "            target_dict[my_data.values[i][targetColumnIndex]] = count\n",
    "        target.append(target_dict[my_data.values[i][targetColumnIndex]])\n",
    "    # Since a dictionary is not ordered, we need to order it and output it to a list so the\n",
    "    # target names will match the target.\n",
    "    for targetName in sorted(target_dict, key=target_dict.get):\n",
    "        target_names.append(targetName)\n",
    "    return np.asarray(target), target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the targetAndtargetNames function, create two variables called <b>target</b> and <b>target_names</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "y, targetNames = targetAndtargetNames(my_data, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the <b>y</b> and <b>targetNames</b> variables you created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3\n",
      " 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4\n",
      " 4 4]\n",
      "['c4000BC', 'c3300BC', 'c1850BC', 'c200BC', 'cAD150']\n"
     ]
    }
   ],
   "source": [
    "print(y)\n",
    "print(targetNames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the two required variables to fit the data, a sneak peak at how to fit data will be shown in the cell below.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"knn\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.3 KNN</span>\n",
    "\n",
    "**K-Nearest Neighbors** is an algorithm for supervised learning. Where the data is 'trained' with data points corresponding to their classification. Once a point is to be predicted, it takes into account the 'K' nearest points to it to determine it's classification.\n",
    "\n",
    "#### Here's an visualization of the K-Nearest Neighbors algorithm.\n",
    "\n",
    "<img src = \"image/KNN.png\">\n",
    "\n",
    "In this case, we have data points of Class A and B. We want to predict what the star (test data point) is. If we consider a k value of 3 (3 nearest data points) we will obtain a prediction of Class B. Yet if we consider a k value of 6, we will obtain a prediction of Class A.\n",
    "\n",
    "In this sense, it is important to consider the value of k. But hopefully from this diagram, you should get a sense of what the K-Nearest Neighbors algorithm is. It considers the 'K' Nearest Neighbors (points) when it predicts the classification of the test point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 4)\n",
      "(150,)\n"
     ]
    }
   ],
   "source": [
    "# X = removeColumns(my_data, 0, 1)\n",
    "# y = target(my_data, 1)\n",
    "\n",
    "X = new_data\n",
    "\n",
    "print (X.shape)\n",
    "print (y.shape)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to perform <b>train/test split</b> we have to split the <b>X</b> and <b>y</b> into two different sets: The <b>training</b> and <b>testing</b> set. Luckily there is a sklearn function for just that!\n",
    "\n",
    "Import the <b>train_test_split</b> from <b>sklearn.cross_validation</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from sklearn.cross_validation import train_test_split\n",
    "#Cross has been abandoned in sklearn_ Validation, which integrates the contents into the model_ In selection\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now <b>train_test_split</b> will return <b>4</b> different parameters. We will name this <b>X_trainset</b>, <b>X_testset</b>, <b>y_trainset</b>, <b>y_testset</b>. The <b>train_test_split</b> will need the parameters <b>X</b>, <b>y</b>, <b>test_size=0.3</b>, and <b>random_state=7</b>. The <b>X</b> and <b>y</b> are the arrays required before the split, the <b>test_size</b> represents the ratio of the testing dataset, and the <b>random_state</b> ensures we obtain the same splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_trainset, X_testset, y_trainset, y_testset = train_test_split(X, y, test_size=0.3, random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's print the shape of the training sets to see if they match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(105, 4)\n",
      "(105,)\n"
     ]
    }
   ],
   "source": [
    "print (X_trainset.shape)\n",
    "print (y_trainset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the same with the testing sets! They should both match up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(45, 4)\n",
      "(45,)\n"
     ]
    }
   ],
   "source": [
    "print (X_testset.shape)\n",
    "print (y_testset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now similarly with the last lab, let's create declarations of KNeighborsClassifier. Except we will create 3 different ones:\n",
    "\n",
    "- neigh -> n_neighbors = 1 \n",
    "- neigh23 -> n_neighbors = 23 \n",
    "- neigh90 -> n_neighbors = 90 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "neigh = KNeighborsClassifier(n_neighbors = 1)\n",
    "neigh23 = KNeighborsClassifier(n_neighbors = 23)\n",
    "neigh90 = KNeighborsClassifier(n_neighbors = 90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will fit each instance of <b>KNeighborsClassifier</b> with the <b>X_trainset</b> and <b>y_trainset</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(n_neighbors=90)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neigh.fit(X_trainset, y_trainset)\n",
    "neigh23.fit(X_trainset, y_trainset)\n",
    "neigh90.fit(X_trainset, y_trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you are able to predict with <b>multiple</b> datapoints. We can do this by just passing in the <b>y_testset</b> which contains multiple test points into a <b>predict</b> function of <b>KNeighborsClassifier</b>.\n",
    "\n",
    "Let's pass the <b>y_testset</b> in the <b>predict</b> function each instance of <b>KNeighborsClassifier</b> but store it's returned value into <b>pred</b>, <b>pred23</b>, <b>pred90</b> (corresponding to each of their names)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = neigh.predict(X_testset)\n",
    "pred23 = neigh23.predict(X_testset)\n",
    "pred90 = neigh90.predict(X_testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome! Now let's compute neigh's <b>prediction accuracy</b>. We can do this by using the <b>metrics.accuracy_score</b> function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Neigh's Accuracy: \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None, 0.2222222222222222)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print(\"Neigh's Accuracy: \"), metrics.accuracy_score(y_testset, pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting! Let's do the same for the other instances of KNeighborsClassifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Neigh23's Accuracy: \n",
      "Neigh90's Accuracy: \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None, 0.13333333333333333)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Neigh23's Accuracy: \"), metrics.accuracy_score(y_testset, pred23)\n",
    "print(\"Neigh90's Accuracy: \"), metrics.accuracy_score(y_testset, pred90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown, the accuracy of <b>neigh23</b> is the highest. When <b>n_neighbors = 1</b>, the model was <b>overfit</b> to the training data (<i>too specific</i>) and when <b>n_neighbors = 90</b>, the model was <b>underfit</b> (<i>too generalized</i>). In comparison, <b>n_neighbors = 23</b> had a <b>good balance</b> between <b>Bias</b> and <b>Variance</b>, creating a generalized model that neither <b>underfit</b> the data nor <b>overfit</b> it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"dt\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.4 Decision Tree</span>\n",
    "\n",
    "In this section, you will learn <b>decision trees</b> and <b>random forests</b>. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <b> getFeatureNames </b> is a function made to get the attribute names for specific columns\n",
    "\n",
    "This function produces one output and requires two inputs:\n",
    "<ul>\n",
    "    <li> <b>1st Input</b>: A pandas array. The pandas array we have been using is <b>my_data</b>. </li>\n",
    "    <li> <b>2nd Input</b>: Any number of integer values (order doesn't matter) that represent the columns that we want to include. In our case we want <b>columns 2-5</b>. </li>\n",
    "    <ul> <li> Note: Remember that Python is zero-indexed, therefore the first column would be 0. </li> </ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getFeatureNames(pandasArray, *column):\n",
    "    actualColumns = list()\n",
    "    allColumns = list(pandasArray.columns.values)\n",
    "    for i in sorted(column):\n",
    "        actualColumns.append(allColumns[i])\n",
    "    return actualColumns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we prepare the data for decision tree construction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#X = removeColumns(my_data, 0, 1) \n",
    "#y, targetNames = targetAndtargetNames(my_data, 1) \n",
    "featureNames = getFeatureNames(my_data, 2,3,4,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out <b>y</b>, <b>targetNames</b>, and <b>featureNames</b> to use in the next example. Remember that the numbers correspond to the names, 0 being the first name,1 being the second name, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3\n",
      " 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4\n",
      " 4 4]\n"
     ]
    }
   ],
   "source": [
    "print (y) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['c4000BC', 'c3300BC', 'c1850BC', 'c200BC', 'cAD150']\n"
     ]
    }
   ],
   "source": [
    "print (targetNames) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['mb', 'bh', 'bl', 'nh']\n"
     ]
    }
   ],
   "source": [
    "print (featureNames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first create an instance of the <b>DecisionTreeClassifier</b> called <b>skullsTree</b>.<br>\n",
    "Inside of the classifier, specify <i> criterion=\"entropy\" </i> so we can see the information gain of each node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "skullsTree = DecisionTreeClassifier(criterion=\"entropy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(criterion='entropy')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skullsTree.fit(X_trainset,y_trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make some <b>predictions</b> on the testing dataset and store it into a variable called <b>predTree</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "predTree = skullsTree.predict(X_testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can print out <b>predTree</b> and <b>y_testset</b> if you want to visually compare the prediction to the actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 2 4 1 0 1 2 4 1 4 4 1 1 1 3 1 2 3 1 4 3 0 3 1 3 2 3 1 2 1 4 4 4 0 2 0\n",
      " 3 3 2 2 0 3 1 4]\n",
      "[4 2 1 2 3 1 1 3 0 1 2 2 1 3 0 2 3 4 1 1 2 4 2 3 4 4 3 2 3 4 4 2 0 4 2 1 1\n",
      " 0 0 3 4 2 4 3 3]\n"
     ]
    }
   ],
   "source": [
    "print (predTree)\n",
    "print (y_testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's import metrics from sklearn and check the accuracy of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTrees's Accuracy: \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None, 0.15555555555555556)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print(\"DecisionTrees's Accuracy: \"), metrics.accuracy_score(y_testset, predTree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can visualize the tree constructed.\n",
    "\n",
    "However, it should be noted that the following code may not work in all Python2 environment. You can try to see packages like <b>pydot</b>, <b>pydot2</b>, <b>pydot2</b>, <b>pydotplus</b>, etc., and see which one works in your platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pydot2 in d:\\filp\\filp00\\anaconda\\envs\\tensorflow\\lib\\site-packages (1.0.33)\n",
      "Requirement already satisfied: setuptools in d:\\filp\\filp00\\anaconda\\envs\\tensorflow\\lib\\site-packages (from pydot2) (49.6.0.post20200814)\n",
      "Requirement already satisfied: pyparsing in d:\\filp\\filp00\\anaconda\\envs\\tensorflow\\lib\\site-packages (from pydot2) (2.4.7)\n"
     ]
    }
   ],
   "source": [
    "#!pip install pydotplus\n",
    "!pip install pydot2\n",
    "\n",
    "#conda install pydot2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'create_png'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-65-57b598b1198f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     16\u001b[0m \u001b[0mgraph\u001b[0m\u001b[1;33m=\u001b[0m \u001b[0mpydot\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgraph_from_dot_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdot_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 17\u001b[1;33m \u001b[0mImage\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgraph\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcreate_png\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'list' object has no attribute 'create_png'"
     ]
    }
   ],
   "source": [
    "from IPython.display import Image  \n",
    "from six import StringIO\n",
    "from sklearn import tree\n",
    "import pydot\n",
    "#import pydotplus\n",
    "\n",
    "dot_data = StringIO()  \n",
    "\n",
    "tree.export_graphviz(skullsTree, out_file=dot_data, \n",
    "feature_names=featureNames, \n",
    "class_names=targetNames, \n",
    "filled=True, rounded=True, \n",
    "special_characters=True, \n",
    "leaves_parallel=True)\n",
    "\n",
    "graph= pydot.graph_from_dot_data(dot_data.getvalue())\n",
    "Image(graph.create_png())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"rf\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.5 Random Forest</span>\n",
    "\n",
    "Import the <b>RandomForestClassifier</b> class from <b>sklearn.ensemble</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an instance of the <b>RandomForestClassifier()</b> called <b>skullsForest</b>, where the forest has <b>10 decision tree estimators</b> (<i>n_estimators=10</i>) and the <b>criterion is entropy</b> (<i>criterion=\"entropy\"</i>)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "skullsForest=RandomForestClassifier(n_estimators=10, criterion=\"entropy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the same <b>X_trainset</b>, <b>y_trainset</b> datasets that we made when dealing with the <b>Decision Trees</b> above to fit <b>skullsForest</b>.\n",
    "<br> <br>\n",
    "<b>Note</b>: Make sure you have ran through the Decision Trees section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(criterion='entropy', n_estimators=10)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skullsForest.fit(X_trainset, y_trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now create a variable called <b>predForest</b> using a predict on <b>X_testset</b> with <b>skullsForest</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "predForest = skullsForest.predict(X_testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can print out <b>predForest</b> and <b>y_testset</b> if you want to visually compare the prediction to the actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 2 1 4 2 0 1 1 3 0 4 4 1 1 1 3 1 4 3 0 3 2 0 3 4 1 2 3 1 2 2 4 4 4 0 4 0\n",
      " 3 3 0 3 0 1 1 1]\n",
      "[4 2 1 2 3 1 1 3 0 1 2 2 1 3 0 2 3 4 1 1 2 4 2 3 4 4 3 2 3 4 4 2 0 4 2 1 1\n",
      " 0 0 3 4 2 4 3 3]\n"
     ]
    }
   ],
   "source": [
    "print (predForest) \n",
    "print (y_testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the accuracy of our model. <br>\n",
    "\n",
    "Note: Make sure you have metrics imported from sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForests's Accuracy: \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None, 0.17777777777777778)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"RandomForests's Accuracy: \"), metrics.accuracy_score(y_testset, predForest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see what trees are in our <b> skullsForest </b> variable by using the <b> .estimators_ </b> attribute. This attribute is indexable, so we can look at any individual tree we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=1138831049), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=89552963), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=1172072420), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=864324396), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=656040751), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=1225253542), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=1640145679), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=670150349), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=1907525103), DecisionTreeClassifier(criterion='entropy', max_features='auto',\n",
      "                       random_state=1876534791)]\n"
     ]
    }
   ],
   "source": [
    "print(skullsForest.estimators_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can choose to view any tree by using the code below. Replace the <i>\"&\"</i> in <b>skullsForest[&]</b> with the tree you want to see.\n",
    "\n",
    "The following block may not work in your Python enrionment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'create_png'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-58-3d1c9f5496b1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     14\u001b[0m                      leaves_parallel=True)\n\u001b[0;32m     15\u001b[0m \u001b[0mgraph\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpydot\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgraph_from_dot_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdot_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m \u001b[0mImage\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgraph\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcreate_png\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'list' object has no attribute 'create_png'"
     ]
    }
   ],
   "source": [
    "from IPython.display import Image  \n",
    "# from sklearn.externals.six import StringIO  \n",
    "from six import StringIO  \n",
    "import pydot\n",
    "from sklearn import tree\n",
    "dot_data = StringIO()\n",
    "\n",
    "#Replace the '&' below with the tree number\n",
    "tree.export_graphviz(skullsForest[1], out_file=dot_data,\n",
    "                     feature_names=featureNames,\n",
    "                     class_names=targetNames,\n",
    "                     filled=True, rounded=True,\n",
    "                     special_characters=True,\n",
    "                     leaves_parallel=True)\n",
    "graph = pydot.graph_from_dot_data(dot_data.getvalue())\n",
    "Image(graph.create_png())  "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}