exercises/Predicting_No_Shows.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/chris-lovejoy/MLmedics/blob/master/exercises/Predicting_No_Shows.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dmy8Lfi8OAmi"
   },
   "source": [
    "# Predicting Hospital Non-Attendance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In this exercise, we will train different machine learning models (including a neural network) to predict whether somebody will attend their upcoming hospital appointment.**\n",
    "\n",
    "Our models will be trained on the publicly-available [\"Medical Appointment No Shows\"](https://www.kaggle.com/datasets/joniarroba/noshowappointments) dataset.\n",
    "\n",
    "This is a *classification* problem, because we're asking the algorithm to classify individuals into two classes: **will attend** or **won't attend**. We'll train a Nearest Neighbours Classifier for this task. You're then invited to try out other classifications algorithms, such as:\n",
    "\n",
    "- Random Forest\n",
    "- Support Vector Machine \n",
    "- Neural networks\n",
    "\n",
    "In this exercise, we'll learn how to:\n",
    "\n",
    "- **Download data** and **load it into our Jupyter Notebook**\n",
    "- **Import useful libraries** like pandas and sci-kit learn\n",
    "- **Clean our data**\n",
    "- **Engineer new features**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Downloading and importing our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a machine learning model, the first thing we need is data.\n",
    "\n",
    "There are various open-source datasets available on the internet. Great sources of datasets include [Kaggle](https://www.kaggle.com/), [Papers with Code](https://paperswithcode.com/datasets) and [data.world](https://data.world/datasets/health).\n",
    "\n",
    "For this exercise, we're using a dataset available on Kaggle. You can view information about the dataset and download it [here](https://www.kaggle.com/datasets/joniarroba/noshowappointments). (You may need to create a Kaggle account, which is definitely worth doing - Kaggle is great.)\n",
    "\n",
    "*Note: we want version 5 of the dataset (which is the default - at the time of writing).*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading and moving our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we've downloaded the data, we're looking for the *.csv* file (it may be within a .zip file, which needs unzipping). CSV stands for 'comma-separated value' and refers to the fact that each row of data is stored with values that are separated by commas. You can open the file in a 'plain text' editor (like Notepad (windows) or TextEdit (Mac)) to see what this looks like. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where we put that file will depend on whether we're running Jupyter Notebook locally or in Google Colab. These options are discussed in the [Jupyter Notebook setup exercise](https://github.com/chris-lovejoy/CodingForMedicine/blob/main/exercises/Setting_up_Jupyter_Notebook.ipynb).\n",
    "\n",
    "**If you are using Google Colab**, you need to:\n",
    "1. Make sure you are connected to a runtime (click Connect in the top right, if you aren't)\n",
    "2. Select the 'Files' folder on the left hand tab\n",
    "3. Drag our downloaded Breast Cancer 'data.csv' file into the Files tab (it should show the file uploading in the bottom left, and then you'll see 'data.csv' within the Files tab).\n",
    "\n",
    "**If you are running Jupyter Notebook on your local computer**, you can simply:\n",
    "1. Drag our 'data.csv' file into the same folder (aka. directory) as this Jupyter Notebook. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing the 'pandas' library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load our data, we're going to use a popular library called [\"Pandas\"](https://pandas.pydata.org). A library is a collection of code with ready-made functions that we can use. We can import it with one line of code and then use it for a wide range of functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*If you are running on your local computer and get the following error, it means you need to install pandas on your computer. Instructions for doing so are available [here](https://pypi.org/project/pandas/). Drop me a message if any difficulties (message form at the bottom of this document).*\n",
    "\n",
    "\n",
    "> \"ModuleNotFoundError: No module named 'pandas'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're also going to need another popular library called numpy, so let's import that too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing our data into the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have our data in the correct place and pandas imported, we can load it with the following command. This uses one of the *pandas* functions **read_csv()**, which lets us load csv files and save them as a **'DataFrame'** (which is basically a table)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows = pd.read_csv('./no-shows-data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our table should now be saved in the 'df' variable, so we should be able to see it by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PatientId</th>\n",
       "      <th>AppointmentID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>ScheduledDay</th>\n",
       "      <th>AppointmentDay</th>\n",
       "      <th>Age</th>\n",
       "      <th>Neighbourhood</th>\n",
       "      <th>Scholarship</th>\n",
       "      <th>Hipertension</th>\n",
       "      <th>Diabetes</th>\n",
       "      <th>Alcoholism</th>\n",
       "      <th>Handcap</th>\n",
       "      <th>SMS_received</th>\n",
       "      <th>No-show</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.987250e+13</td>\n",
       "      <td>5642903</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T18:38:08Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>62</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.589978e+14</td>\n",
       "      <td>5642503</td>\n",
       "      <td>M</td>\n",
       "      <td>2016-04-29T16:08:27Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.262962e+12</td>\n",
       "      <td>5642549</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T16:19:04Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>62</td>\n",
       "      <td>MATA DA PRAIA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.679512e+11</td>\n",
       "      <td>5642828</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T17:29:31Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>8</td>\n",
       "      <td>PONTAL DE CAMBURI</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.841186e+12</td>\n",
       "      <td>5642494</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T16:07:23Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110522</th>\n",
       "      <td>2.572134e+12</td>\n",
       "      <td>5651768</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-05-03T09:15:35Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110523</th>\n",
       "      <td>3.596266e+12</td>\n",
       "      <td>5650093</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-05-03T07:27:33Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>51</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110524</th>\n",
       "      <td>1.557663e+13</td>\n",
       "      <td>5630692</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-27T16:03:52Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>21</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110525</th>\n",
       "      <td>9.213493e+13</td>\n",
       "      <td>5630323</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-27T15:09:23Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>38</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110526</th>\n",
       "      <td>3.775115e+14</td>\n",
       "      <td>5629448</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-27T13:30:56Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>54</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>110527 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           PatientId  AppointmentID Gender          ScheduledDay  \\\n",
       "0       2.987250e+13        5642903      F  2016-04-29T18:38:08Z   \n",
       "1       5.589978e+14        5642503      M  2016-04-29T16:08:27Z   \n",
       "2       4.262962e+12        5642549      F  2016-04-29T16:19:04Z   \n",
       "3       8.679512e+11        5642828      F  2016-04-29T17:29:31Z   \n",
       "4       8.841186e+12        5642494      F  2016-04-29T16:07:23Z   \n",
       "...              ...            ...    ...                   ...   \n",
       "110522  2.572134e+12        5651768      F  2016-05-03T09:15:35Z   \n",
       "110523  3.596266e+12        5650093      F  2016-05-03T07:27:33Z   \n",
       "110524  1.557663e+13        5630692      F  2016-04-27T16:03:52Z   \n",
       "110525  9.213493e+13        5630323      F  2016-04-27T15:09:23Z   \n",
       "110526  3.775115e+14        5629448      F  2016-04-27T13:30:56Z   \n",
       "\n",
       "              AppointmentDay  Age      Neighbourhood  Scholarship  \\\n",
       "0       2016-04-29T00:00:00Z   62    JARDIM DA PENHA            0   \n",
       "1       2016-04-29T00:00:00Z   56    JARDIM DA PENHA            0   \n",
       "2       2016-04-29T00:00:00Z   62      MATA DA PRAIA            0   \n",
       "3       2016-04-29T00:00:00Z    8  PONTAL DE CAMBURI            0   \n",
       "4       2016-04-29T00:00:00Z   56    JARDIM DA PENHA            0   \n",
       "...                      ...  ...                ...          ...   \n",
       "110522  2016-06-07T00:00:00Z   56        MARIA ORTIZ            0   \n",
       "110523  2016-06-07T00:00:00Z   51        MARIA ORTIZ            0   \n",
       "110524  2016-06-07T00:00:00Z   21        MARIA ORTIZ            0   \n",
       "110525  2016-06-07T00:00:00Z   38        MARIA ORTIZ            0   \n",
       "110526  2016-06-07T00:00:00Z   54        MARIA ORTIZ            0   \n",
       "\n",
       "        Hipertension  Diabetes  Alcoholism  Handcap  SMS_received No-show  \n",
       "0                  1         0           0        0             0      No  \n",
       "1                  0         0           0        0             0      No  \n",
       "2                  0         0           0        0             0      No  \n",
       "3                  0         0           0        0             0      No  \n",
       "4                  1         1           0        0             0      No  \n",
       "...              ...       ...         ...      ...           ...     ...  \n",
       "110522             0         0           0        0             1      No  \n",
       "110523             0         0           0        0             1      No  \n",
       "110524             0         0           0        0             1      No  \n",
       "110525             0         0           0        0             1      No  \n",
       "110526             0         0           0        0             1      No  \n",
       "\n",
       "[110527 rows x 14 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Visualising our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand our data, we can use functions such as:\n",
    "\n",
    "- head()\n",
    "- tail()\n",
    "- describe() [to be used on specific variables]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PatientId</th>\n",
       "      <th>AppointmentID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>ScheduledDay</th>\n",
       "      <th>AppointmentDay</th>\n",
       "      <th>Age</th>\n",
       "      <th>Neighbourhood</th>\n",
       "      <th>Scholarship</th>\n",
       "      <th>Hipertension</th>\n",
       "      <th>Diabetes</th>\n",
       "      <th>Alcoholism</th>\n",
       "      <th>Handcap</th>\n",
       "      <th>SMS_received</th>\n",
       "      <th>No-show</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.987250e+13</td>\n",
       "      <td>5642903</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T18:38:08Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>62</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.589978e+14</td>\n",
       "      <td>5642503</td>\n",
       "      <td>M</td>\n",
       "      <td>2016-04-29T16:08:27Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.262962e+12</td>\n",
       "      <td>5642549</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T16:19:04Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>62</td>\n",
       "      <td>MATA DA PRAIA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.679512e+11</td>\n",
       "      <td>5642828</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T17:29:31Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>8</td>\n",
       "      <td>PONTAL DE CAMBURI</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.841186e+12</td>\n",
       "      <td>5642494</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T16:07:23Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      PatientId  AppointmentID Gender          ScheduledDay  \\\n",
       "0  2.987250e+13        5642903      F  2016-04-29T18:38:08Z   \n",
       "1  5.589978e+14        5642503      M  2016-04-29T16:08:27Z   \n",
       "2  4.262962e+12        5642549      F  2016-04-29T16:19:04Z   \n",
       "3  8.679512e+11        5642828      F  2016-04-29T17:29:31Z   \n",
       "4  8.841186e+12        5642494      F  2016-04-29T16:07:23Z   \n",
       "\n",
       "         AppointmentDay  Age      Neighbourhood  Scholarship  Hipertension  \\\n",
       "0  2016-04-29T00:00:00Z   62    JARDIM DA PENHA            0             1   \n",
       "1  2016-04-29T00:00:00Z   56    JARDIM DA PENHA            0             0   \n",
       "2  2016-04-29T00:00:00Z   62      MATA DA PRAIA            0             0   \n",
       "3  2016-04-29T00:00:00Z    8  PONTAL DE CAMBURI            0             0   \n",
       "4  2016-04-29T00:00:00Z   56    JARDIM DA PENHA            0             1   \n",
       "\n",
       "   Diabetes  Alcoholism  Handcap  SMS_received No-show  \n",
       "0         0           0        0             0      No  \n",
       "1         0           0        0             0      No  \n",
       "2         0           0        0             0      No  \n",
       "3         0           0        0             0      No  \n",
       "4         1           0        0             0      No  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    110527.000000\n",
       "mean         37.088874\n",
       "std          23.110205\n",
       "min          -1.000000\n",
       "25%          18.000000\n",
       "50%          37.000000\n",
       "75%          55.000000\n",
       "max         115.000000\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows.Age.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do a quick screen to see all the unique values for each variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Age: [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 102, 115]\n",
      "Gender: ['F' 'M']\n",
      "Diabetes: [0 1]\n",
      "Alchoholism: [0 1]\n",
      "Hypertension: [1 0]\n",
      "Handicap: [0 1 2 3 4]\n",
      "Scholarship: [0 1]\n",
      "SMS_received: [0 1]\n",
      "Neighbourhood: ['JARDIM DA PENHA' 'MATA DA PRAIA' 'PONTAL DE CAMBURI' 'REPÚBLICA'\n",
      " 'GOIABEIRAS' 'ANDORINHAS' 'CONQUISTA' 'NOVA PALESTINA' 'DA PENHA'\n",
      " 'TABUAZEIRO' 'BENTO FERREIRA' 'SÃO PEDRO' 'SANTA MARTHA' 'SÃO CRISTÓVÃO'\n",
      " 'MARUÍPE' 'GRANDE VITÓRIA' 'SÃO BENEDITO' 'ILHA DAS CAIEIRAS'\n",
      " 'SANTO ANDRÉ' 'SOLON BORGES' 'BONFIM' 'JARDIM CAMBURI' 'MARIA ORTIZ'\n",
      " 'JABOUR' 'ANTÔNIO HONÓRIO' 'RESISTÊNCIA' 'ILHA DE SANTA MARIA'\n",
      " 'JUCUTUQUARA' 'MONTE BELO' 'MÁRIO CYPRESTE' 'SANTO ANTÔNIO' 'BELA VISTA'\n",
      " 'PRAIA DO SUÁ' 'SANTA HELENA' 'ITARARÉ' 'INHANGUETÁ' 'UNIVERSITÁRIO'\n",
      " 'SÃO JOSÉ' 'REDENÇÃO' 'SANTA CLARA' 'CENTRO' 'PARQUE MOSCOSO'\n",
      " 'DO MOSCOSO' 'SANTOS DUMONT' 'CARATOÍRA' 'ARIOVALDO FAVALESSA'\n",
      " 'ILHA DO FRADE' 'GURIGICA' 'JOANA D´ARC' 'CONSOLAÇÃO' 'PRAIA DO CANTO'\n",
      " 'BOA VISTA' 'MORADA DE CAMBURI' 'SANTA LUÍZA' 'SANTA LÚCIA'\n",
      " 'BARRO VERMELHO' 'ESTRELINHA' 'FORTE SÃO JOÃO' 'FONTE GRANDE'\n",
      " 'ENSEADA DO SUÁ' 'SANTOS REIS' 'PIEDADE' 'JESUS DE NAZARETH'\n",
      " 'SANTA TEREZA' 'CRUZAMENTO' 'ILHA DO PRÍNCIPE' 'ROMÃO' 'COMDUSA'\n",
      " 'SANTA CECÍLIA' 'VILA RUBIM' 'DE LOURDES' 'DO QUADRO' 'DO CABRAL' 'HORTO'\n",
      " 'SEGURANÇA DO LAR' 'ILHA DO BOI' 'FRADINHOS' 'NAZARETH' 'AEROPORTO'\n",
      " 'ILHAS OCEÂNICAS DE TRINDADE' 'PARQUE INDUSTRIAL']\n"
     ]
    }
   ],
   "source": [
    "print('Age:',sorted(noShows.Age.unique()))\n",
    "print('Gender:',noShows.Gender.unique())\n",
    "print('Diabetes:',noShows.Diabetes.unique())\n",
    "print('Alchoholism:',noShows.Alcoholism.unique())\n",
    "print('Hypertension:',noShows.Hipertension.unique())\n",
    "print('Handicap:',noShows.Handcap.unique())\n",
    "print('Scholarship:',noShows.Scholarship.unique())\n",
    "print('SMS_received:',noShows.SMS_received.unique())\n",
    "print('Neighbourhood:',noShows.Neighbourhood.unique())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK**: Use the cell below to explore and better understand the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Cleaning our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From looking at what df.Age.describe() returns, we can already see some problems. For example, the minimum age is -1 and the maximum age is 115.\n",
    "\n",
    "Let's remove all datapoints outside of a particular range.\n",
    "\n",
    "To do this, we can use the **.loc** functionality of pandas to identify datapoints that meet particular conditions.\n",
    "\n",
    "Let's set the top cut-off as 90 years and the lower cut-off as 5 years:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PatientId</th>\n",
       "      <th>AppointmentID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>ScheduledDay</th>\n",
       "      <th>AppointmentDay</th>\n",
       "      <th>Age</th>\n",
       "      <th>Neighbourhood</th>\n",
       "      <th>Scholarship</th>\n",
       "      <th>Hipertension</th>\n",
       "      <th>Diabetes</th>\n",
       "      <th>Alcoholism</th>\n",
       "      <th>Handcap</th>\n",
       "      <th>SMS_received</th>\n",
       "      <th>No-show</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.987250e+13</td>\n",
       "      <td>5642903</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T18:38:08Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>62</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.589978e+14</td>\n",
       "      <td>5642503</td>\n",
       "      <td>M</td>\n",
       "      <td>2016-04-29T16:08:27Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.262962e+12</td>\n",
       "      <td>5642549</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T16:19:04Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>62</td>\n",
       "      <td>MATA DA PRAIA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.679512e+11</td>\n",
       "      <td>5642828</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T17:29:31Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>8</td>\n",
       "      <td>PONTAL DE CAMBURI</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.841186e+12</td>\n",
       "      <td>5642494</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-29T16:07:23Z</td>\n",
       "      <td>2016-04-29T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>JARDIM DA PENHA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110522</th>\n",
       "      <td>2.572134e+12</td>\n",
       "      <td>5651768</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-05-03T09:15:35Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>56</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110523</th>\n",
       "      <td>3.596266e+12</td>\n",
       "      <td>5650093</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-05-03T07:27:33Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>51</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110524</th>\n",
       "      <td>1.557663e+13</td>\n",
       "      <td>5630692</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-27T16:03:52Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>21</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110525</th>\n",
       "      <td>9.213493e+13</td>\n",
       "      <td>5630323</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-27T15:09:23Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>38</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110526</th>\n",
       "      <td>3.775115e+14</td>\n",
       "      <td>5629448</td>\n",
       "      <td>F</td>\n",
       "      <td>2016-04-27T13:30:56Z</td>\n",
       "      <td>2016-06-07T00:00:00Z</td>\n",
       "      <td>54</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>98378 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           PatientId  AppointmentID Gender          ScheduledDay  \\\n",
       "0       2.987250e+13        5642903      F  2016-04-29T18:38:08Z   \n",
       "1       5.589978e+14        5642503      M  2016-04-29T16:08:27Z   \n",
       "2       4.262962e+12        5642549      F  2016-04-29T16:19:04Z   \n",
       "3       8.679512e+11        5642828      F  2016-04-29T17:29:31Z   \n",
       "4       8.841186e+12        5642494      F  2016-04-29T16:07:23Z   \n",
       "...              ...            ...    ...                   ...   \n",
       "110522  2.572134e+12        5651768      F  2016-05-03T09:15:35Z   \n",
       "110523  3.596266e+12        5650093      F  2016-05-03T07:27:33Z   \n",
       "110524  1.557663e+13        5630692      F  2016-04-27T16:03:52Z   \n",
       "110525  9.213493e+13        5630323      F  2016-04-27T15:09:23Z   \n",
       "110526  3.775115e+14        5629448      F  2016-04-27T13:30:56Z   \n",
       "\n",
       "              AppointmentDay  Age      Neighbourhood  Scholarship  \\\n",
       "0       2016-04-29T00:00:00Z   62    JARDIM DA PENHA            0   \n",
       "1       2016-04-29T00:00:00Z   56    JARDIM DA PENHA            0   \n",
       "2       2016-04-29T00:00:00Z   62      MATA DA PRAIA            0   \n",
       "3       2016-04-29T00:00:00Z    8  PONTAL DE CAMBURI            0   \n",
       "4       2016-04-29T00:00:00Z   56    JARDIM DA PENHA            0   \n",
       "...                      ...  ...                ...          ...   \n",
       "110522  2016-06-07T00:00:00Z   56        MARIA ORTIZ            0   \n",
       "110523  2016-06-07T00:00:00Z   51        MARIA ORTIZ            0   \n",
       "110524  2016-06-07T00:00:00Z   21        MARIA ORTIZ            0   \n",
       "110525  2016-06-07T00:00:00Z   38        MARIA ORTIZ            0   \n",
       "110526  2016-06-07T00:00:00Z   54        MARIA ORTIZ            0   \n",
       "\n",
       "        Hipertension  Diabetes  Alcoholism  Handcap  SMS_received No-show  \n",
       "0                  1         0           0        0             0      No  \n",
       "1                  0         0           0        0             0      No  \n",
       "2                  0         0           0        0             0      No  \n",
       "3                  0         0           0        0             0      No  \n",
       "4                  1         1           0        0             0      No  \n",
       "...              ...       ...         ...      ...           ...     ...  \n",
       "110522             0         0           0        0             1      No  \n",
       "110523             0         0           0        0             1      No  \n",
       "110524             0         0           0        0             1      No  \n",
       "110525             0         0           0        0             1      No  \n",
       "110526             0         0           0        0             1      No  \n",
       "\n",
       "[98378 rows x 14 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows.loc[(noShows.Age < 90) & (noShows.Age > 5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this yields 98,378 rows - out of our original 110,527. Gives that our dataset is large, this may not be of concern. If we wanted to minimise data that is removed, we could do further manual inspection of this dataset to understand the age distribution and consider a more refined removal criteria.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we're happy, we can go ahead and update the dataset by declaring this into the variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows = noShows.loc[(noShows.Age < 90) & (noShows.Age > 5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK**: Have a look at other variables, to see whether there are other outliers or erroneous data points that should be removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correct spelling mistakes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also some simple spelling mistakes in the column names. We can correct this using the **.rename()** function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xo-CV0gdZurM"
   },
   "outputs": [],
   "source": [
    "noShows = noShows.rename(columns = {'Hipertension': 'Hypertension',\n",
    "                         'Handcap': 'Handicap'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing unhelpful columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some columns that won't be helpful for our analysis. We can remove those by using the **.drop()** function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows = noShows.drop('PatientId', axis=1)\n",
    "noShows = noShows.drop('AppointmentID', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting variables to binary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computers like to work with binary values of 0 or 1 much more than with text. So let's convert our 'diagnosis' column values from 'M' for malignant and 'B' for binary into 1 and 0, using the **.map()** function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows['No-show'] = noShows['No-show'].map({'Yes':1, 'No':0})\n",
    "noShows['Gender'] = noShows['Gender'].map({'F':1, 'M':0})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK:** Have a look through the dataset and see if there's any further cleaning that it requires."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, the variables that we have in our table aren't the best ones we want to feed into our models for training. In this case, we may want to manually \"engineer features\" for our model. This could involve transforming existing features and/or combining multiple.\n",
    "\n",
    "A simple example in our case here, could be to extract the waiting time for each individual (ie. the time between their booking being made and the date of their appointmenet). This feature doesn't yet exist, but it seems reasonable that this would be relevant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Engineering the \"waiting time\" variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add the waiting time, we'll want to calculate the difference between values in the \"ScheduledDay\" and \"AppointmentDay\" columns.\n",
    "\n",
    "However, on inspection we can notice a problem: The values in both columns are 'string' variables. (See the notebook on [Python Principles](./Python_principles.ipynb) if you're not sure what a string is.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         2016-04-29T18:38:08Z\n",
       "1         2016-04-29T16:08:27Z\n",
       "2         2016-04-29T16:19:04Z\n",
       "3         2016-04-29T17:29:31Z\n",
       "4         2016-04-29T16:07:23Z\n",
       "                  ...         \n",
       "110522    2016-05-03T09:15:35Z\n",
       "110523    2016-05-03T07:27:33Z\n",
       "110524    2016-04-27T16:03:52Z\n",
       "110525    2016-04-27T15:09:23Z\n",
       "110526    2016-04-27T13:30:56Z\n",
       "Name: ScheduledDay, Length: 98378, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows.ScheduledDay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "lGb3AnokQbr4",
    "outputId": "a95ede53-5ad2-43ca-ea3f-4be3144522ba"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(noShows.AppointmentDay[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that they're currently strings, there's no way they can be easily subtracted. We therefore want to convert them into \"datetime\" format, which enables us to handle this type of data.\n",
    "\n",
    "We can do that using the **.apply()** function and the numpy **datetime64** format, as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows.ScheduledDay = noShows.ScheduledDay.apply(np.datetime64)\n",
    "noShows.AppointmentDay = noShows.AppointmentDay.apply(np.datetime64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make things simpler, we can focus only on the date - and not the time of date. We can update that as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows['ScheduledDay'] = noShows['ScheduledDay'].dt.date\n",
    "noShows['AppointmentDay'] = noShows['AppointmentDay'].dt.date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now go ahead and add a new column called \"WaitingTime\", using the pandas **to_timedelta()** function, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "noShows['WaitingTime'] = pd.to_timedelta((noShows['AppointmentDay'] - noShows['ScheduledDay'])).dt.days\n",
    "noShows['WaitingTime'] = noShows['WaitingTime'].apply(np.int64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note: It is always better to implement column-wise transformations in this way, rather than a loop - both for better efficiency as well reduced risk of bugs.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see our new variable \"WaitingTime\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "flQ9KizlVl63",
    "outputId": "acf8ae90-e163-490b-d1d5-c61923380a62"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>ScheduledDay</th>\n",
       "      <th>AppointmentDay</th>\n",
       "      <th>Age</th>\n",
       "      <th>Neighbourhood</th>\n",
       "      <th>Scholarship</th>\n",
       "      <th>Hypertension</th>\n",
       "      <th>Diabetes</th>\n",
       "      <th>Alcoholism</th>\n",
       "      <th>Handicap</th>\n",
       "      <th>SMS_received</th>\n",
       "      <th>No-show</th>\n",
       "      <th>WaitingTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>110511</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>14</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110512</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>41</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110514</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>58</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110515</th>\n",
       "      <td>0</td>\n",
       "      <td>2016-06-06</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>33</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110516</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>2016-06-08</td>\n",
       "      <td>37</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110517</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>19</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110518</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-27</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>50</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110519</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-27</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>22</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110520</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-05-03</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>42</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110521</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-05-03</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>53</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110522</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-05-03</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>56</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110523</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-05-03</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>51</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110524</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-27</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>21</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110525</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-27</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>38</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110526</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-27</td>\n",
       "      <td>2016-06-07</td>\n",
       "      <td>54</td>\n",
       "      <td>MARIA ORTIZ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Gender ScheduledDay AppointmentDay  Age Neighbourhood  Scholarship  \\\n",
       "110511       1   2016-06-08     2016-06-08   14   MARIA ORTIZ            0   \n",
       "110512       1   2016-06-08     2016-06-08   41   MARIA ORTIZ            0   \n",
       "110514       1   2016-06-08     2016-06-08   58   MARIA ORTIZ            0   \n",
       "110515       0   2016-06-06     2016-06-08   33   MARIA ORTIZ            0   \n",
       "110516       1   2016-06-07     2016-06-08   37   MARIA ORTIZ            0   \n",
       "110517       1   2016-06-07     2016-06-07   19   MARIA ORTIZ            0   \n",
       "110518       1   2016-04-27     2016-06-07   50   MARIA ORTIZ            0   \n",
       "110519       1   2016-04-27     2016-06-07   22   MARIA ORTIZ            0   \n",
       "110520       1   2016-05-03     2016-06-07   42   MARIA ORTIZ            0   \n",
       "110521       1   2016-05-03     2016-06-07   53   MARIA ORTIZ            0   \n",
       "110522       1   2016-05-03     2016-06-07   56   MARIA ORTIZ            0   \n",
       "110523       1   2016-05-03     2016-06-07   51   MARIA ORTIZ            0   \n",
       "110524       1   2016-04-27     2016-06-07   21   MARIA ORTIZ            0   \n",
       "110525       1   2016-04-27     2016-06-07   38   MARIA ORTIZ            0   \n",
       "110526       1   2016-04-27     2016-06-07   54   MARIA ORTIZ            0   \n",
       "\n",
       "        Hypertension  Diabetes  Alcoholism  Handicap  SMS_received  No-show  \\\n",
       "110511             0         0           0         0             0        0   \n",
       "110512             0         0           0         0             0        0   \n",
       "110514             0         0           0         0             0        0   \n",
       "110515             1         0           0         0             0        1   \n",
       "110516             0         0           0         0             0        1   \n",
       "110517             0         0           0         0             0        0   \n",
       "110518             0         0           0         0             1        0   \n",
       "110519             0         0           0         0             1        0   \n",
       "110520             0         0           0         0             1        0   \n",
       "110521             0         0           0         0             1        0   \n",
       "110522             0         0           0         0             1        0   \n",
       "110523             0         0           0         0             1        0   \n",
       "110524             0         0           0         0             1        0   \n",
       "110525             0         0           0         0             1        0   \n",
       "110526             0         0           0         0             1        0   \n",
       "\n",
       "        WaitingTime  \n",
       "110511            0  \n",
       "110512            0  \n",
       "110514            0  \n",
       "110515            2  \n",
       "110516            1  \n",
       "110517            0  \n",
       "110518           41  \n",
       "110519           41  \n",
       "110520           35  \n",
       "110521           35  \n",
       "110522           35  \n",
       "110523           35  \n",
       "110524           41  \n",
       "110525           41  \n",
       "110526           41  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows.tail(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK:** Look at the current variables and consider what other features you could extract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding dummy variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we'll want to convert a variable into multiple separate variables. For example, we might have categorical variables, which many models can't \"natively\" incorporate it. \n",
    "\n",
    "A solution here can be to generate \"dummy variables\". Rather than having one variable with all the categories, we create binary (yes/no) variables for each of the categories.\n",
    "\n",
    "In this exercise, we can do this for hte 'Neighbourhood' variable. We can use the pandas function **\"get_dummies\"** to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BuNHVR3PqXOG"
   },
   "outputs": [],
   "source": [
    "dummy_cols = ['Neighbourhood']\n",
    "noShows = pd.get_dummies(noShows, columns = dummy_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see **many** new columns, for each potential neighbourhood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 355
    },
    "colab_type": "code",
    "id": "pKvg1kftQd5l",
    "outputId": "8d105af1-30cd-45cd-d294-820b0782dbd8"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>ScheduledDay</th>\n",
       "      <th>AppointmentDay</th>\n",
       "      <th>Age</th>\n",
       "      <th>Scholarship</th>\n",
       "      <th>Hypertension</th>\n",
       "      <th>Diabetes</th>\n",
       "      <th>Alcoholism</th>\n",
       "      <th>Handicap</th>\n",
       "      <th>SMS_received</th>\n",
       "      <th>...</th>\n",
       "      <th>Neighbourhood_SANTOS REIS</th>\n",
       "      <th>Neighbourhood_SEGURANÇA DO LAR</th>\n",
       "      <th>Neighbourhood_SOLON BORGES</th>\n",
       "      <th>Neighbourhood_SÃO BENEDITO</th>\n",
       "      <th>Neighbourhood_SÃO CRISTÓVÃO</th>\n",
       "      <th>Neighbourhood_SÃO JOSÉ</th>\n",
       "      <th>Neighbourhood_SÃO PEDRO</th>\n",
       "      <th>Neighbourhood_TABUAZEIRO</th>\n",
       "      <th>Neighbourhood_UNIVERSITÁRIO</th>\n",
       "      <th>Neighbourhood_VILA RUBIM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>2016-04-29</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 93 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Gender ScheduledDay AppointmentDay  Age  Scholarship  Hypertension  \\\n",
       "0       1   2016-04-29     2016-04-29   62            0             1   \n",
       "1       0   2016-04-29     2016-04-29   56            0             0   \n",
       "2       1   2016-04-29     2016-04-29   62            0             0   \n",
       "3       1   2016-04-29     2016-04-29    8            0             0   \n",
       "4       1   2016-04-29     2016-04-29   56            0             1   \n",
       "\n",
       "   Diabetes  Alcoholism  Handicap  SMS_received  ...  \\\n",
       "0         0           0         0             0  ...   \n",
       "1         0           0         0             0  ...   \n",
       "2         0           0         0             0  ...   \n",
       "3         0           0         0             0  ...   \n",
       "4         1           0         0             0  ...   \n",
       "\n",
       "   Neighbourhood_SANTOS REIS  Neighbourhood_SEGURANÇA DO LAR  \\\n",
       "0                          0                               0   \n",
       "1                          0                               0   \n",
       "2                          0                               0   \n",
       "3                          0                               0   \n",
       "4                          0                               0   \n",
       "\n",
       "   Neighbourhood_SOLON BORGES  Neighbourhood_SÃO BENEDITO  \\\n",
       "0                           0                           0   \n",
       "1                           0                           0   \n",
       "2                           0                           0   \n",
       "3                           0                           0   \n",
       "4                           0                           0   \n",
       "\n",
       "   Neighbourhood_SÃO CRISTÓVÃO  Neighbourhood_SÃO JOSÉ  \\\n",
       "0                            0                       0   \n",
       "1                            0                       0   \n",
       "2                            0                       0   \n",
       "3                            0                       0   \n",
       "4                            0                       0   \n",
       "\n",
       "   Neighbourhood_SÃO PEDRO  Neighbourhood_TABUAZEIRO  \\\n",
       "0                        0                         0   \n",
       "1                        0                         0   \n",
       "2                        0                         0   \n",
       "3                        0                         0   \n",
       "4                        0                         0   \n",
       "\n",
       "   Neighbourhood_UNIVERSITÁRIO  Neighbourhood_VILA RUBIM  \n",
       "0                            0                         0  \n",
       "1                            0                         0  \n",
       "2                            0                         0  \n",
       "3                            0                         0  \n",
       "4                            0                         0  \n",
       "\n",
       "[5 rows x 93 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noShows.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the large number of neighbourhoods in our dataset, this may not actually be the most appropriate way to handle this variable. A more intelligent approach may be to extract further information about the neighbourhoods - such as geographical location or socioeconomic information about them. \n",
    "\n",
    "However, we don't have that information readily available, so will stick with this approach for demonstration purposes for now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RezU2yGqlaao"
   },
   "source": [
    "## Part 5: Preparing and training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now prepare and train our model. We'll start selecting all the variables we want to use for training.\n",
    "\n",
    "*(Not sure what a list is? Check out the [Python principles](./Python_principles.ipynb) exercise.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cbQnL1PQlfdD"
   },
   "outputs": [],
   "source": [
    "prediction_var = ['Gender','Age','Scholarship','Hypertension','Diabetes','Alcoholism','Handicap','SMS_received','WaitingTime']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll split our data into training and test data, using a function from sklearn.\n",
    "\n",
    "We do this so we can understand whether our model is actually helpful. In the real world, we want to use our model on data it's never seen before. \n",
    "\n",
    "When training it model, we only show it the 'training' data. We can then test it on the 'test' data, and use that to understand how well the model may perform on new (unseen) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mogf34ZcVJFK"
   },
   "outputs": [],
   "source": [
    " from sklearn.model_selection import train_test_split\n",
    " \n",
    " train, test = train_test_split(noShows, test_size = 0.15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create our input 'x' variables and our output 'y' variables for the training and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZyerVv62VJHs"
   },
   "outputs": [],
   "source": [
    "train_x = train[prediction_var]\n",
    "train_y = train['No-show']\n",
    "\n",
    "test_x = test[prediction_var]\n",
    "test_y = test['No-show']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll import our model and train it. Here we'll train a nearest neighbours classifier. Explaining this is beyond the scope of this exercise, but you can read about it [here](https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier()"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = KNeighborsClassifier()\n",
    "model.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate and assess predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can generate some predictions as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(test_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And have a quick look at those predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 1])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can quantify those predictions using the sklearn function **f1_score()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.23368606701940034"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(test_y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The closer to 1, the better the F1 score (and the closer to 0, the worse)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reality, just looking at one score is not a good way to assess the model. Assessing model performance (including a description of the F1 score) is covered in more detail in [this exercise](./Breast_cancer_features.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a go at modifying our pipeline. See if you can improve the F1 score. Potential variations include:\n",
    "\n",
    "- Different features (using the 'prediction_var' list)\n",
    "- Different parameters (such as different learning rates and neural network layers, or different train/test splits)\n",
    "- Different models (see below)\n",
    "\n",
    "Popular classifiers to try as alternative models include:\n",
    "\n",
    "- [Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n",
    "- [Neural networks](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html)\n",
    "- [Support vector machines (SVMs)](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)\n",
    "\n",
    "These links will take you to the documentation. See if you can work out how to import them and then train the model based on the description and examples that they provide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill out the form below and we'll provide feedback on your code.\n",
    "\n",
    "**Any feedback on the exercise? Any questions? Want feedback on your code? Please fill out the form [here](https://docs.google.com/forms/d/e/1FAIpQLSdoOjVom8YKf11LxJ_bWN40afFMsWcoJ-xOrKhMbfBzgxTS9A/viewform).**"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "Predicting No Shows",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "data_science",
   "language": "python",
   "name": "data_science"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}