RNN_Mortality Prediction.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "66e7f07ff8da05c0613f4335a5916f7b",
          "grade": false,
          "grade_id": "cell-67ef996f07aa3ec3",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "# Perform Mortality Prediction using RNN \n",
        "\n",
        "To deal with sequential data, the most commonly used architecture is recurrent neural network (RNN). \n",
        "\n",
        "Table of Contents:\n",
        "- Motivation\n",
        "- Implementation\n",
        "- Modern RNN Models\n",
        "\n",
        "Some contents of this lab are adapted from [Dive into Deep Learning](https://d2l.ai) and [Official PyTorch Tutorials](https://pytorch.org/tutorials/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.121633Z",
          "start_time": "2021-12-10T04:02:05.517588Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "e7886bdee48e2d70080bf8171b533d19",
          "grade": false,
          "grade_id": "cell-05b5cd4b51c7c115",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import random\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.128075Z",
          "start_time": "2021-12-10T04:02:06.123724Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "c20a6b2d5dcf1ff1b54a78787843f632",
          "grade": false,
          "grade_id": "cell-c136bff551290161",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "# set seed\n",
        "seed = 42\n",
        "random.seed(seed)\n",
        "np.random.seed(seed)\n",
        "torch.manual_seed(seed)\n",
        "os.environ[\"PYTHONHASHSEED\"] = str(seed)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.254468Z",
          "start_time": "2021-12-10T04:02:06.129895Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "285586abd7789a119de4008c0a716510",
          "grade": false,
          "grade_id": "cell-13c70c678cbe91b9",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "DATA_PATH = \"../LAB4-lib/data\"\n",
        "assert os.path.isdir(DATA_PATH)\n",
        "!ls {DATA_PATH}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "a240d9088d7848bd7975b4bfccd2460b",
          "grade": false,
          "grade_id": "cell-fccc65aaca1f8dfb",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "## 1. Motivation\n",
        "\n",
        "While CNNs can efficiently process spatial information, recurrent neural networks (RNNs) are designed to better handle sequential information. RNNs introduce state variables to store past information, together with the current inputs, to determine the current outputs.\n",
        "\n",
        "That is, let us say we have a sequence $x_1, x_2, \\dots, x_t$. RNN attempts to model the conditional probability: $P(x_t \\mid x_1, x_2, \\dots, x_{t-1})$. More specifically, RNN leverages a hidden state variable  that stores the sequence information up to time step $t−1$:\n",
        "\n",
        "$$P(x_t \\mid x_{1}, x_{2}, \\ldots, x_{t-1}) \\approx P(x_t \\mid h_{t-1}),$$\n",
        "\n",
        "where $h_{t-1}$ is the hidden state variable. In general, the hidden state at any time step $t$ could be computed based on both the current input $x_t$ and the previous hidden state $h_{t-1}$:\n",
        "\n",
        "$$h_t = f(x_{t}, h_{t-1}).$$ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "fe092d44760bec7df70df26af07e6c6a",
          "grade": false,
          "grade_id": "cell-85a8813f852c2ba8",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "## 2. Implementation\n",
        "\n",
        "Assume that we have a minibatch of inputs $\\mathbf{X}_t \\in \\mathbb{R}^{n \\times d}$ at time step $t$. In other words, for a minibatch of $n$ sequence examples, each row of $\\mathbf{X}_t$ corresponds to one example at time step $t$ from the sequence. Next, denote by $\\mathbf{H}_t \\in \\mathbb{R}^{n \\times h}$ the hidden variable of time step $t$. Further, we save the hidden variable $\\mathbf{H}_{t-1}$ from the previous time step and introduce a new weight parameter $\\mathbf{W}_{hh} \\in \\mathbb{R}^{h \\times h}$ to describe how to use the hidden variable of the previous time step in the current time step. Specifically, the calculation of the hidden variable of the current time step is determined by the input of the current time step together with the hidden variable of the previous time step:\n",
        "\n",
        "$$\\mathbf{H}_t = \\phi(\\mathbf{X}_t \\mathbf{W}_{xh} + \\mathbf{H}_{t-1} \\mathbf{W}_{hh}  + \\mathbf{b}_h).$$\n",
        "\n",
        "From the relationship between hidden variables $\\mathbf{H}_{t}$ and $\\mathbf{H}_{t-1}$ of adjacent time steps, we know that these variables captured and retained the sequence’s historical information up to their current time step, just like the state or memory of the neural network’s current time step. Therefore, such a hidden variable is called a hidden state. Since the hidden state uses the same definition of the previous time step in the current time step, the computation of $\\mathbf{H}_{t}$ is recurrent. Hence, neural networks with hidden states based on recurrent computation are named recurrent neural networks. Layers that perform the computation of $\\mathbf{H}_{t}$ in RNNs are called recurrent layers.\n",
        "\n",
        "For time step $t$, the output of the output layer is similar to the computation in the MLP:\n",
        "\n",
        "$$\\mathbf{O}_t = \\mathbf{H}_t \\mathbf{W}_{hq} + \\mathbf{b}_q.$$\n",
        "\n",
        "Parameters of the RNN include the weights $\\mathbf{W}_{xh} \\in \\mathbb{R}^{d \\times h}, \\mathbf{W}_{hh} \\in \\mathbb{R}^{h \\times h}$, and the bias $\\mathbf{b}_h \\in \\mathbb{R}^{1 \\times h}$ of the hidden layer, together with the weights $\\mathbf{W}_{hq} \\in \\mathbb{R}^{h \\times q}$ and the bias $\\mathbf{b}_q \\in \\mathbb{R}^{1 \\times q}$ of the output layer. It is worth mentioning that even at different time steps, RNNs always use these model parameters. Therefore, the parameterization cost of an RNN does not grow as the number of time steps increases.\n",
        "\n",
        "<img src='img/rnn.svg'>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "d1bd296b858a8733a760442ee5cdb78e",
          "grade": false,
          "grade_id": "cell-bd33ea245d64c9d5",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Implement the function calculating the current hidden state:\n",
        "\n",
        "$$\\mathbf{H}_t = \\mathbf{X}_t \\mathbf{W}_{xh} + \\mathbf{H}_{t-1} \\mathbf{W}_{hh}  + \\mathbf{b}_h.$$\n",
        "\n",
        "Note that here `X` is the input at a sinlge time step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.262258Z",
          "start_time": "2021-12-10T04:02:06.258101Z"
        },
        "deletable": false
      },
      "outputs": [],
      "source": [
        "def calculate_current_h(X, H, W_xh, W_hh, b_h):\n",
        "    \"\"\"\n",
        "    Params:\n",
        "        - X: (batch size, input dimension)\n",
        "        - H: (batch size, hidden dimension)\n",
        "        - W_xh: (input dimension, hidden dimension)\n",
        "        - W_hh: (hidden dimension, hidden dimension)\n",
        "        - b_h: (1, hidden dimension)\n",
        "    \"\"\"\n",
        "    # your code here\n",
        "    raise NotImplementedError"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.273316Z",
          "start_time": "2021-12-10T04:02:06.264326Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "114fe7beef2fac92227bb321c2369f16",
          "grade": true,
          "grade_id": "cell-1c5b169c773b6b5d",
          "locked": true,
          "points": 20,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "'''\n",
        "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
        "'''\n",
        "\n",
        "torch.manual_seed(42)\n",
        "\n",
        "n, d, h = 2, 8, 4 # batch size, input dimension, hidden dimension\n",
        "X, W_xh = torch.normal(0, 1, (n, d)), torch.normal(0, 1, (d, h))\n",
        "H, W_hh = torch.normal(0, 1, (n, h)), torch.normal(0, 1, (h, h))\n",
        "b_h = torch.normal(0, 1, (1, h))\n",
        "assert torch.allclose(calculate_current_h(X, H, W_xh, W_hh, b_h),\n",
        "                      torch.tensor([[ 5.1226, -0.6884,  3.1821,  6.6513],\n",
        "                                    [-0.6077,  2.2313, -1.4812,  0.4403]]), rtol=1e-2)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "1352b212df3fdaf5b6acc167e99d6b53",
          "grade": false,
          "grade_id": "cell-b1abb8914e8b246e",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Call the previous implemented `calculate_current_h` recursively to calculate the final hidden state.\n",
        "\n",
        "Note that here `inputs` is the entire sequence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.278148Z",
          "start_time": "2021-12-10T04:02:06.274869Z"
        },
        "deletable": false
      },
      "outputs": [],
      "source": [
        "def rnn(inputs, state, W_xh, W_hh, b_h):\n",
        "    \"\"\"\n",
        "    Params:\n",
        "        - inputs: (batch size, sequence length, input dimension)\n",
        "        - state: (batch size, hidden dimension)\n",
        "        - W_xh: (input dimension, hidden dimension)\n",
        "        - W_hh: (hidden dimension, hidden dimension)\n",
        "        - b_h: (1, hidden dimension)\n",
        "    \"\"\"\n",
        "    # your code here\n",
        "    raise NotImplementedError"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.286460Z",
          "start_time": "2021-12-10T04:02:06.279942Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "f05f83c0debb279aa50701c729c102e5",
          "grade": true,
          "grade_id": "cell-5f600972f8df6f63",
          "locked": true,
          "points": 20,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "'''\n",
        "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
        "'''\n",
        "\n",
        "torch.manual_seed(42)\n",
        "\n",
        "n, t, d, h = 2, 5, 8, 4 # batch size, sequence length, input dimension, hidden dimension\n",
        "\n",
        "inputs = torch.normal(0, 1, (n, t, d))\n",
        "initial_state  = torch.normal(0, 1, (n, h))\n",
        "\n",
        "W_xh = torch.normal(0, 1, (d, h))\n",
        "W_hh = torch.normal(0, 1, (h, h))\n",
        "b_h = torch.normal(0, 1, (1, h))\n",
        "assert torch.allclose(rnn(inputs, initial_state, W_xh, W_hh, b_h),\n",
        "                      torch.tensor([[ -46.5894,  -61.1859,   14.3644,   56.4997],\n",
        "                                    [ 166.5581,   84.0468,  -24.5756, -149.2971]]), rtol=1e-2)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "c8699df7036671404a41bfb7a47d260b",
          "grade": false,
          "grade_id": "cell-9a87f61f0aa956c7",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "## 3. Modern RNN Models\n",
        "\n",
        "We have introduced the basics of RNNs, which can better handle sequence data. However, such techniques may not be sufficient for practitioners when they face a wide range of sequence learning problems nowadays.\n",
        "\n",
        "For instance, a notable issue in practice is the numerical instability of RNNs. Although there are several implementation tricks such as gradient clipping, this issue can be alleviated further with more sophisticated designs of sequence models. Specifically, gated RNNs are much more common in practice. In this section, we will introduce you two of such widely-used networks, namely long short-term memory (LSTM) and gated recurrent units (GRUs).\n",
        "\n",
        "Here, we will only cover the basic concepts of GRU and LSTM. We won't discuss the architecture in details. If you are intereseted, you can refer to this [link](https://d2l.ai/chapter_recurrent-modern/index.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "aa2b1fefce743ade6fd9611aaf554eff",
          "grade": false,
          "grade_id": "cell-e719fae13d88cc4f",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### 3.1 Gated Recurrent Units (GRU)\n",
        "\n",
        "The key distinction between vanilla RNNs and GRUs is that the latter support gating of the hidden state. This means that we have dedicated mechanisms for when a hidden state should be updated and also when it should be reset. These mechanisms are learned and the network can selectively keep or forget information. For instance, if the first token is of great importance we will learn not to update the hidden state after the first observation. Likewise, we will learn to skip irrelevant temporary observations.\n",
        "\n",
        "<img src='img/gru.svg'>\n",
        "\n",
        "As shown in the figure above, the reset gates help capture short-term dependencies in sequences. And the update gates help capture long-term dependencies in sequences."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "d89e835d7f068564ef0af63df416a366",
          "grade": false,
          "grade_id": "cell-d5d014c4b87288c3",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "To initialize a [GRU](https://pytorch.org/docs/stable/generated/torch.nn.GRU.html#torch.nn.GRU) layer in PyTorch, try the following snippet:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.294503Z",
          "start_time": "2021-12-10T04:02:06.287977Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "8b4f7f2a8ac03c4cb14a1390db9c1865",
          "grade": false,
          "grade_id": "cell-26ab854626e7bccb",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "n, t, d, h = 2, 5, 8, 4 # batch size, sequence length, input dimension, hidden dimension\n",
        "\n",
        "# If batch_first=True, then the input and output tensors are \n",
        "# provided as (batch, seq, feature) instead of (seq, batch, feature).\n",
        "rnn = nn.GRU(input_size=d, hidden_size=h, batch_first=True)\n",
        "\n",
        "inputs = torch.randn(n, t, d)\n",
        "h0 = torch.randn(1, n, h) # the first dimension is the number of RNN layers (default value is 1)\n",
        "\n",
        "output, hn = rnn(inputs, h0)\n",
        "print('output:', output.shape)\n",
        "print('hn:', hn.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "f06eac258f0762775c5275b2d9f094da",
          "grade": false,
          "grade_id": "cell-fe2b5470e33ea8c1",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### 3.2 Long Short-Term Memory (LSTM)\n",
        "\n",
        "The challenge to address long-term information preservation and short-term input skipping in latent variable models has existed for a long time. One of the earliest approaches to address this was the long short-term memory (LSTM) ([Hochreiter & Schmidhuber, 1997](https://direct.mit.edu/neco/article/9/8/1735/6109/Long-Short-Term-Memory)). It shares many of the properties of the GRU. Interestingly, LSTMs have a slightly more complex design than GRUs but predates GRUs by almost two decades.\n",
        "\n",
        "LSTM introduces a memory cell (or cell for short) that has the same shape as the hidden state, engineered to record additional information. To control the memory cell we need a number of gates. One gate is needed to read out the entries from the cell. We will refer to this as the output gate. A second gate is needed to decide when to read data into the cell. We refer to this as the input gate. Last, we need a mechanism to reset the content of the cell, governed by a forget gate. The motivation for such a design is the same as that of GRUs, namely to be able to decide when to remember and when to ignore inputs in the hidden state via a dedicated mechanism.\n",
        "\n",
        "<img src='img/lstm.svg'>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "5ab990bc6e894e913ec1cde85a04e45a",
          "grade": false,
          "grade_id": "cell-5f293c477444d133",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "To initialize a [LSTM](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html) layer in PyTorch, try the following snippet:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.302741Z",
          "start_time": "2021-12-10T04:02:06.296450Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "358cda51006ffe8ba39bbaa6b653dc22",
          "grade": false,
          "grade_id": "cell-632ffc2ed588593d",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "n, t, d, h = 2, 5, 8, 4 # batch size, sequence length, input dimension, hidden dimension\n",
        "\n",
        "# If batch_first=True, then the input and output tensors are \n",
        "# provided as (batch, seq, feature) instead of (seq, batch, feature).\n",
        "rnn = nn.LSTM(input_size=d, hidden_size=h, batch_first=True)\n",
        "\n",
        "inputs = torch.randn(n, t, d)\n",
        "h0 = torch.randn(1, n, h) # the first dimension is the number of RNN layers (default value is 1)\n",
        "c0 = torch.randn(1, n, h)\n",
        "\n",
        "output, (hn, cn) = rnn(inputs, (h0, c0))\n",
        "print('output:', output.shape)\n",
        "print('hn:', hn.shape)\n",
        "print('cn:', cn.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "477df5d8dc31519136543cbf66a17a5c",
          "grade": false,
          "grade_id": "cell-7ce478b19a724c2d",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Using [MIMIC-III Demo](https://physionet.org/content/mimiciii-demo/) dataset, which contains all intensive care unit (ICU) stays for 100 patients. The task is Mortality Prediction."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "6250f0bbc4007d8005a314910f560d53",
          "grade": false,
          "grade_id": "cell-0922073208c3d91c",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### Load Data\n",
        "\n",
        "In the previous lab, we have preprocessed the data. Thus, for this lab, we will directly use the processed data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.434914Z",
          "start_time": "2021-12-10T04:02:06.307464Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "216b82ccd2e70cff91702086bc9ed1f9",
          "grade": false,
          "grade_id": "cell-db67447afa9a0fba",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "!ls {DATA_PATH}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "70e38c8aeb8bc703bb740c5713e7ae97",
          "grade": false,
          "grade_id": "cell-62c5dba8a899823d",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Here are the helper fuctions and CustomDataset from the previous lab. \n",
        "\n",
        "We will use the entire patient visit instead of only the last visit.\n",
        "\n",
        "Note that in this lab, we **do not** need to exclude patients with only one visit."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.441883Z",
          "start_time": "2021-12-10T04:02:06.437064Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "0d119834f8fa3599bc7b6baff31eb65a",
          "grade": false,
          "grade_id": "cell-9ec92c92777c8c9c",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "# two helper functions\n",
        "\n",
        "TOTAL_NUM_CODES = 271\n",
        "\n",
        "\n",
        "def read_csv(filename):\n",
        "    \"\"\" reading csv from filename \"\"\"\n",
        "    data = []\n",
        "    with open(filename, \"r\") as file:\n",
        "        csv_reader = csv.DictReader(file, delimiter=',')\n",
        "        for row in csv_reader:\n",
        "            data.append(row)\n",
        "    header = list(data[0].keys())\n",
        "    return header, data\n",
        "\n",
        "\n",
        "def to_one_hot(label, num_class):\n",
        "    \"\"\" convert to one hot label \"\"\"\n",
        "    one_hot_label = [0] * num_class\n",
        "    for i in label:\n",
        "        one_hot_label[i] = 1\n",
        "    return one_hot_label"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.451168Z",
          "start_time": "2021-12-10T04:02:06.443700Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "d40f5812beeadc025663e31978be2465",
          "grade": false,
          "grade_id": "cell-a186eb985edd96fd",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import Dataset\n",
        "\n",
        "\n",
        "class CustomDataset(Dataset):\n",
        "    \n",
        "    def __init__(self):\n",
        "        # read the csv\n",
        "        self._df = pd.read_csv(f'{DATA_PATH}/data.csv')\n",
        "        # split diagnosis code index by ';' and convert it to integer\n",
        "        self._df.icd9 = self._df.icd9.apply(lambda x: [int(i) for i in x.split(';')])\n",
        "        # build data dict\n",
        "        self._build_data_dict()\n",
        "        # a list of subject ids\n",
        "        self._subj_ids = list(self._data.keys())\n",
        "        # sort the subject ids to maintain a fixed order\n",
        "        self._subj_ids.sort()\n",
        "    \n",
        "    def _build_data_dict(self):\n",
        "        \"\"\" \n",
        "        build SUBJECT_ID to ADMISSION dict\n",
        "            - subject_id\n",
        "                - icd9: a list of ICD9 code index\n",
        "                - mortality: 0/1 morality label\n",
        "        \"\"\"\n",
        "        dict_data = {}\n",
        "        df = self._df.groupby('subject_id').agg({'mortality': lambda x: x.iloc[0], 'icd9': list}).reset_index()\n",
        "        for idx, row in df.iterrows():\n",
        "            subj_id = row.subject_id\n",
        "            dict_data[subj_id] = {}\n",
        "            dict_data[subj_id]['icd9'] = row.icd9\n",
        "            dict_data[subj_id]['mortality'] = row.mortality\n",
        "        self._data = dict_data\n",
        "    \n",
        "    def __len__(self):\n",
        "        \"\"\" return the number of samples (i.e. patients). \"\"\"\n",
        "        return len(self._subj_ids)\n",
        "    \n",
        "    def __getitem__(self, index):\n",
        "        \"\"\" generates one sample of data. \"\"\"\n",
        "        # obtain the subject id\n",
        "        subj_id = self._subj_ids[index]\n",
        "        # obtain the data dict by subject id\n",
        "        data = self._data[subj_id]\n",
        "        # convert last admission's diagnosis code index to one hot\n",
        "        x = torch.tensor([to_one_hot(visit, TOTAL_NUM_CODES) for visit in data['icd9']], dtype=torch.float32)\n",
        "        # mortality label\n",
        "        y = torch.tensor(data['mortality'], dtype=torch.float32)\n",
        "        return x, y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.473061Z",
          "start_time": "2021-12-10T04:02:06.452951Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "ed5d94e0397fab3b75e81d70c308de6d",
          "grade": false,
          "grade_id": "cell-fd6c0e9220bdbfe9",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "dataset = CustomDataset()\n",
        "print('Size of dataset:', len(dataset))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.478535Z",
          "start_time": "2021-12-10T04:02:06.474783Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "c521f9d2e6348b1b13eb25449bf51c4f",
          "grade": false,
          "grade_id": "cell-bc87693d6901919d",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "from torch.utils.data.dataset import random_split\n",
        "\n",
        "\n",
        "split = int(len(dataset)*0.7)\n",
        "\n",
        "lengths = [split, len(dataset) - split]\n",
        "train_dataset, test_dataset = random_split(dataset, lengths)\n",
        "\n",
        "print(\"Length of train dataset:\", len(train_dataset))\n",
        "print(\"Length of test dataset:\", len(test_dataset))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "ca01072280a0e9bb9a2e0cc50bbf4c1a",
          "grade": false,
          "grade_id": "cell-3d75144c592bf460",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Here is an example of $x$, and $y$. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.486930Z",
          "start_time": "2021-12-10T04:02:06.480283Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "15ffc3c9791dfdfd2aaa93e71b997316",
          "grade": false,
          "grade_id": "cell-bbf4c2c13369ead7",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "x, y = train_dataset[0]\n",
        "print(f'Example x (shape {x.shape}):\\n', x)\n",
        "print(f'Example y:\\n', y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "ca435d10887461911c04925fbe653f0a",
          "grade": false,
          "grade_id": "cell-748697ceb2fe7d42",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "We can see that $x$ is of shape $(2, 271)$, which means there are $271$ diagnosis codes in total, and this patient has two visits. It is in one-hot format. A $1$ in position $i$ means that diagnosis code of index $i$ appears in the that visit.\n",
        "\n",
        "And $y$ is either $0$ or $1$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "ce0efd0a8dc9c459cfeae1a266a14735",
          "grade": false,
          "grade_id": "cell-74d8c95ccf004960",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### Padding\n",
        "\n",
        "In the previous lab, we implement a collate function `collate_fn()` to pad the sequence into the same length. For RNN, we will do something similar.\n",
        "\n",
        "Moreover, we will keep a separate variable storing the length of each sequence. Later, we will use this length variable to select the mask out padding visits."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.493336Z",
          "start_time": "2021-12-10T04:02:06.488360Z"
        },
        "deletable": false
      },
      "outputs": [],
      "source": [
        "def collate_fn(data):\n",
        "    \"\"\"\n",
        "    TODO: Collate the the list of samples into batches. For each patient, you need to pad the diagnosis\n",
        "        sequences to the sample shape (max # visits, total # diagnosis codes). Further, you need to store \n",
        "        the true length of each sequence into l.\n",
        "    \n",
        "    Arguments:\n",
        "        data: a list of samples fetched from `CustomDataset`\n",
        "        \n",
        "    Outputs:\n",
        "        x: a tensor of shape (# patients, total # diagnosis codes, max # visits) of type torch.float\n",
        "        y: a tensor of shape (# patients) of type torch.float\n",
        "        l: a tensor of shape (# patients) of type torch.long\n",
        "        \n",
        "    Note that you can obtains the list of diagnosis codes and the list of mortality labels\n",
        "        using: `sequences, labels = zip(*data)`\n",
        "    \"\"\"\n",
        "\n",
        "    sequences, labels = zip(*data)\n",
        "\n",
        "    y = torch.tensor(labels, dtype=torch.float)\n",
        "    \n",
        "    num_patients = len(sequences)\n",
        "    num_visits = [patient.shape[0] for patient in sequences]\n",
        "    total_num_codes = sequences[0].shape[1]\n",
        "\n",
        "    max_num_visits = max(num_visits)\n",
        "    \n",
        "    x = torch.zeros((num_patients, max_num_visits, total_num_codes), dtype=torch.float)\n",
        "    l = None\n",
        "    \n",
        "    # your code here\n",
        "    raise NotImplementedError\n",
        "    \n",
        "    return x, y, l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.506392Z",
          "start_time": "2021-12-10T04:02:06.494976Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "88cf22df0c8affdcef7954649474da40",
          "grade": true,
          "grade_id": "cell-93a8f004ead104ab",
          "locked": true,
          "points": 20,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "'''\n",
        "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
        "'''\n",
        "\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "\n",
        "loader = DataLoader(train_dataset, batch_size=4, collate_fn=collate_fn)\n",
        "loader_iter = iter(loader)\n",
        "x, y, l = next(loader_iter)\n",
        "\n",
        "print(x.shape)\n",
        "print(y.shape)\n",
        "print(l)\n",
        "\n",
        "assert x.dtype == torch.float\n",
        "assert y.dtype == torch.float\n",
        "assert l.dtype == torch.long\n",
        "\n",
        "assert x.shape[0] == 4\n",
        "assert x.shape[-1] == 271\n",
        "assert y.shape == (4,)\n",
        "assert l.shape == (4,)\n",
        "\n",
        "for i in range(4):\n",
        "    real_x, real_y = train_dataset[i]\n",
        "    assert len(real_x) == l[i]\n",
        "    for j in range(real_x.shape[0]):\n",
        "        visit = real_x[j]\n",
        "        got = x[i, j, :]\n",
        "        assert all(visit == got)\n",
        "        assert real_y == y[i]\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "d27f6c9c098c23dbc396947903c99f8b",
          "grade": false,
          "grade_id": "cell-2e7975367aeb8d4c",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### Data Loader\n",
        "\n",
        "Now, we can load the dataset into the data loader."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.511775Z",
          "start_time": "2021-12-10T04:02:06.508001Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "68bfd6d30a647aa8dca5696f1f21dd72",
          "grade": false,
          "grade_id": "cell-969cadf4d62ef5d5",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "# how many samples per batch to load\n",
        "batch_size = 4\n",
        "\n",
        "# prepare dataloaders\n",
        "train_loader = DataLoader(train_dataset, batch_size=batch_size, collate_fn=collate_fn, shuffle=True)\n",
        "test_loader = DataLoader(test_dataset, batch_size=batch_size, collate_fn=collate_fn)\n",
        "\n",
        "print(\"# of train batches:\", len(train_loader))\n",
        "print(\"# of test batches:\", len(test_loader))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.518113Z",
          "start_time": "2021-12-10T04:02:06.513595Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "2528e316246f838836b129a33d931a46",
          "grade": false,
          "grade_id": "cell-e34dc46c974a1ea7",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "train_iter = iter(train_loader)\n",
        "x, y, l = next(train_iter)\n",
        "\n",
        "print('Shape of a batch x:', x.shape)\n",
        "print('Shape of a batch y:', y.shape)\n",
        "print('Shape of a batch l:', l.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "26b1372c8d9f9d60271b69bd658e9ea9",
          "grade": false,
          "grade_id": "cell-369f8724259879e3",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### Build the Model\n",
        "\n",
        "<img src='img/naive-rnn.png'>\n",
        "\n",
        "We will construct this simple RNN structure. So each input is a one-hot vector. At the 0-th visit, this has $\\boldsymbol{X}_0$, and at t-th visit, this has $\\boldsymbol{X}_t$.\n",
        "\n",
        "Each one of them will then map to a hidden state $\\boldsymbol{h}_t$. The hidden state $\\boldsymbol{h}_t$ can be determined by $\\boldsymbol{h}_{t-1}$ and the corresponding current input $\\boldsymbol{X}_t$.\n",
        "\n",
        "Finally, once we have the $\\boldsymbol{h}_T$, the hidden state of the last timestamp, then we can use this as feature vectors and train a NN to perform the classification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "b90084c0ccec7e3fd14c847f07363350",
          "grade": false,
          "grade_id": "cell-23394ecb0a77ee62",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Now, let us build this model. The forward steps will be:\n",
        "\n",
        "    1. Pass the inputs through the RNN layer;\n",
        "    2. Obtain the hidden state at the last visit;\n",
        "    3. Pass the hidden state through the linear and activation layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "13ee714d778005ee06addc89098e53fd",
          "grade": false,
          "grade_id": "cell-92ef22ca87069002",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "#### Mask Selection\n",
        "\n",
        "Importantly, you need to use `length` to mask out the paddings in step 2."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.522548Z",
          "start_time": "2021-12-10T04:02:06.519611Z"
        },
        "deletable": false
      },
      "outputs": [],
      "source": [
        "def get_last_visit(hidden_states, length):\n",
        "    \"\"\"\n",
        "    TODO: obtain the hidden state for the last true visit (not padding visits)\n",
        "\n",
        "    Arguments:\n",
        "        hidden_states: the hidden states of each visit of shape (batch_size, # visits, hidden_dim)\n",
        "        length: the true visit length of shape (batch_size,)\n",
        "\n",
        "    Outputs:\n",
        "        last_hidden_state: the hidden state for the last true visit of shape (batch_size, hidden_dim)\n",
        "        \n",
        "    NOTE: DO NOT use for loop.\n",
        "    \"\"\"\n",
        "    \n",
        "    # your code here\n",
        "    raise NotImplementedError"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.529890Z",
          "start_time": "2021-12-10T04:02:06.524301Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "83d69fc427060418c0b9399251dc4e8b",
          "grade": true,
          "grade_id": "cell-26114cc62b765649",
          "locked": true,
          "points": 20,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "'''\n",
        "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
        "'''\n",
        "\n",
        "max_num_visits = 10\n",
        "batch_size = 16\n",
        "hidden_dim = 100\n",
        "\n",
        "hidden_states = torch.randn((batch_size, max_num_visits, hidden_dim))\n",
        "lengths = torch.tensor([random.randint(1, max_num_visits) for _ in range(batch_size)])\n",
        "out = get_last_visit(hidden_states, lengths)\n",
        "\n",
        "assert out.shape == (batch_size, hidden_dim)\n",
        "\n",
        "for b in range(batch_size):\n",
        "    last_h = 0\n",
        "    last_h = hidden_states[b, lengths[b] - 1]\n",
        "    assert torch.allclose(out[b], last_h, atol=1e-4), \\\n",
        "    \"The last visit's hidden state of %d-th visit of the %d-th patient is wrong. \"%(v,b) +\\\n",
        "    \"Expect {} Got {} with your get_last_visit\".format(last_h, out[b])\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "556e1fb024ea6f911e687f83603cd129",
          "grade": false,
          "grade_id": "cell-1ec1d62fd10f3408",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "#### Build NaiveRNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.544515Z",
          "start_time": "2021-12-10T04:02:06.531252Z"
        },
        "deletable": false
      },
      "outputs": [],
      "source": [
        "class NaiveRNN(nn.Module):\n",
        "    \n",
        "    \"\"\"\n",
        "    TODO: implement the naive RNN model above.\n",
        "    \"\"\"\n",
        "    \n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        \"\"\"\n",
        "        TODO: \n",
        "            1. Define the RNN using `nn.GRU()`; Set `input_size` to `TOTAL_NUM_CODES`.\n",
        "               Set `hidden_size` to 32. Set `batch_first` to True.\n",
        "            2. Define the linear layers using `nn.Linear()`; Set `output_size` to 1.\n",
        "            3. Define the final activation layer using `nn.Sigmoid().\n",
        "        \"\"\"\n",
        "        \n",
        "        # your code here\n",
        "        raise NotImplementedError\n",
        "    \n",
        "    def forward(self, x, length):\n",
        "        \"\"\"\n",
        "        TODO:\n",
        "            1. Pass the inputs through the RNN layer;\n",
        "            2. Obtain the hidden state at the last visit.\n",
        "               Use `get_last_visit()`;\n",
        "            3. Pass the hidden state through the linear and activation layers.\n",
        "            \n",
        "        Arguments:\n",
        "            x: the diagnosis sequence of shape (batch_size, # visits, # diagnosis codes)\n",
        "            length: the true visit length of shape (batch_size,)\n",
        "\n",
        "        Outputs:\n",
        "            probs: probabilities of shape (batch_size)\n",
        "        \"\"\"\n",
        "        \n",
        "        # your code here\n",
        "        raise NotImplementedError\n",
        "    \n",
        "\n",
        "# load the model here\n",
        "model = NaiveRNN()\n",
        "model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.553716Z",
          "start_time": "2021-12-10T04:02:06.546876Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "7e7fedfad9f4ff312c101f1b134cd44c",
          "grade": true,
          "grade_id": "cell-f22f078c9d3401a0",
          "locked": true,
          "points": 20,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "'''\n",
        "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
        "'''\n",
        "\n",
        "layers_to_check = [nn.GRU, nn.Linear, nn.Sigmoid]\n",
        "for layer_to_check in layers_to_check:\n",
        "    no_layer = True\n",
        "    for child in model.children():\n",
        "        for layer in child.modules():\n",
        "            if(isinstance(layer, layer_to_check)):\n",
        "                no_layer = False\n",
        "    assert not no_layer, \"{} is missing in your RNN\".format(layer_to_check)\n",
        "\n",
        "loader = DataLoader(dataset, batch_size=10, collate_fn=collate_fn)\n",
        "loader_iter = iter(loader)\n",
        "x, y, l = next(loader_iter)\n",
        "model_output = model(x, l)\n",
        "assert model_output.shape == (10,), \"Your RNN's output shape is {}, expect {}\".format(model_output.shape, (10,))\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "09f53939afb0c1281cdf4c119622b14e",
          "grade": false,
          "grade_id": "cell-69d86cdb0ee89e46",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "### Train the Network\n",
        "\n",
        "In this step, you will train the CNN model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:06.557956Z",
          "start_time": "2021-12-10T04:02:06.555274Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "22f95d571e3dc179ead5e9369b5eb2ff",
          "grade": false,
          "grade_id": "cell-e048585cb94b4435",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "# Use Binary Cross Entropy as the loss function (`nn.BCELoss`)\n",
        "# Use Adam as the optimizer (`torch.optim.Adam`)\n",
        "criterion = nn.BCELoss()\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-06-17T10:41:10.932858Z",
          "start_time": "2021-06-17T10:41:10.924864Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "7d40074d948a21387ecb1996396fb380",
          "grade": false,
          "grade_id": "cell-152e9aafcaa7a0ff",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "Now we can train the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:07.085255Z",
          "start_time": "2021-12-10T04:02:06.559522Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "3aa35a55451633aa5464c21baffffc21",
          "grade": false,
          "grade_id": "cell-3b8eb35770a1bf5b",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import *\n",
        "\n",
        "#input: Y_score,Y_pred,Y_true\n",
        "#output: accuracy, auc, precision, recall, f1-score\n",
        "def classification_metrics(Y_score, Y_pred, Y_true):\n",
        "    acc, auc, precision, recall, f1score = accuracy_score(Y_true, Y_pred), \\\n",
        "                                           roc_auc_score(Y_true, Y_score), \\\n",
        "                                           precision_score(Y_true, Y_pred), \\\n",
        "                                           recall_score(Y_true, Y_pred), \\\n",
        "                                           f1_score(Y_true, Y_pred)\n",
        "    return acc, auc, precision, recall, f1score\n",
        "\n",
        "\n",
        "#input: model, loader\n",
        "def evaluate(model, loader):\n",
        "    model.eval()\n",
        "    all_y_true = torch.LongTensor()\n",
        "    all_y_pred = torch.LongTensor()\n",
        "    all_y_score = torch.FloatTensor()\n",
        "    for x, y, l in loader:\n",
        "        # pass the input through the model\n",
        "        y_hat = model(x, l)\n",
        "        # convert shape from [batch size, 1] to [batch size]\n",
        "        y_hat = y_hat.view(y_hat.shape[0])\n",
        "        y_pred = (y_hat > 0.5).type(torch.float)\n",
        "        all_y_true = torch.cat((all_y_true, y.to('cpu')), dim=0)\n",
        "        all_y_pred = torch.cat((all_y_pred,  y_pred.to('cpu')), dim=0)\n",
        "        all_y_score = torch.cat((all_y_score,  y_hat.to('cpu')), dim=0)\n",
        "        \n",
        "    acc, auc, precision, recall, f1 = classification_metrics(all_y_score.detach().numpy(), \n",
        "                                                             all_y_pred.detach().numpy(), \n",
        "                                                             all_y_true.detach().numpy())\n",
        "    print(f\"acc: {acc:.3f}, auc: {auc:.3f}, precision: {precision:.3f}, recall: {recall:.3f}, f1: {f1:.3f}\")\n",
        "    return"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:07.119395Z",
          "start_time": "2021-12-10T04:02:07.086650Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "953bcca02953fe4bb35432d9cdc88a57",
          "grade": false,
          "grade_id": "cell-129173e0dec74bf8",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "print(\"model perfomance before training:\")\n",
        "evaluate(model, train_loader)\n",
        "evaluate(model, test_loader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-12-10T04:02:07.912476Z",
          "start_time": "2021-12-10T04:02:07.121481Z"
        },
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "667864a66702032edc23b209a217f554",
          "grade": false,
          "grade_id": "cell-b23c9926657ef7f6",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "# number of epochs to train the model\n",
        "# feel free to change this\n",
        "n_epochs = 15\n",
        "\n",
        "# prep model for training\n",
        "model.train()\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    \n",
        "    train_loss = 0\n",
        "    for x, y, l in train_loader:\n",
        "        \"\"\" Step 1. clear gradients \"\"\"\n",
        "        optimizer.zero_grad()\n",
        "        \"\"\"  Step 2. perform forward pass using `model`, save the output to y_hat \"\"\"\n",
        "        y_hat = model(x, l)\n",
        "        \"\"\" Step 3. calculate the loss using `criterion`, save the output to loss. \"\"\"\n",
        "        # convert shape from [batch size, 1] to [batch size]\n",
        "        y_hat = y_hat.view(y_hat.shape[0])\n",
        "        loss = criterion(y_hat, y)\n",
        "        \"\"\" Step 4. backward pass \"\"\"\n",
        "        loss.backward()\n",
        "        \"\"\" Step 5. optimization \"\"\"\n",
        "        optimizer.step()\n",
        "        \"\"\" Step 6. record loss \"\"\"\n",
        "        train_loss += loss.item()\n",
        "        \n",
        "    train_loss = train_loss / len(train_loader)\n",
        "    print('Epoch: {} \\tTraining Loss: {:.6f}'.format(epoch+1, train_loss))\n",
        "    evaluate(model, train_loader)\n",
        "    evaluate(model, test_loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "3ebe391699848bbb41c374f47fae453e",
          "grade": false,
          "grade_id": "cell-25544e26b197fe35",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "source": [
        "The result is bad due to very limited data. The model overfits the training data very fast.\n",
        "\n",
        "You are encouraged to try this on the whole MIMIC-III dataset. The result will be much more promising!"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": false,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {
        "height": "calc(100% - 180px)",
        "left": "10px",
        "top": "150px",
        "width": "358.390625px"
      },
      "toc_section_display": true,
      "toc_window_display": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}