nlp-course-lecture02-huawei-noahs-ark-lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seminar 2\n",
    "\n",
    "### Intro to PyTorch\n",
    "\n",
    "based on official [PyTorch Blitz Tutorial](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is PyTorch?\n",
    "\n",
    "* It's a package for scientific computations, basically, a replacement for NumPy, that supports GPUs.\n",
    "* It's a deep learning research platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensors\n",
    "\n",
    "Tensors are similar to NumPy's ndarrays, with the exception of being able to be operated with using GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To construct a 5x3 matrix, uninitialized:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[8.4489e-39, 9.6428e-39, 1.1112e-38],\n",
      "        [9.5511e-39, 1.0102e-38, 1.0286e-38],\n",
      "        [1.0194e-38, 9.6429e-39, 9.2755e-39],\n",
      "        [9.1837e-39, 9.3674e-39, 1.0745e-38],\n",
      "        [1.0653e-38, 9.5510e-39, 1.0561e-38]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.empty(5, 3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NB! An uninitialized matrix is declared, but does not contain definite known values before it is used. When an uninitialized matrix is created, whatever values were in the allocated memory at the time will appear as the initial values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To construct a randomly initialized matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2705, 0.8561, 0.7783],\n",
      "        [0.4793, 0.1794, 0.7983],\n",
      "        [0.1836, 0.3935, 0.6301],\n",
      "        [0.9729, 0.8827, 0.1801],\n",
      "        [0.4169, 0.5151, 0.9043]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(5, 3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To construct a matrix, filled with zeros and data-type long:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.zeros(5, 3, dtype=torch.long)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tensor may be initialized directly from data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5.5000, 3.0000])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([5.5, 3])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tensor may be created using an existing tensor. The new one will inherit all the properties of the one, that was passed as a parameter, apart from those, that were parametrized explicitly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.]], dtype=torch.float64)\n",
      "tensor([[-1.6111, -0.7489, -1.0613],\n",
      "        [-0.0936, -1.0655,  0.2892],\n",
      "        [ 0.4850, -0.1741,  1.5787],\n",
      "        [ 0.7381, -1.0141,  1.4165],\n",
      "        [-1.2311, -1.3559, -0.6673]])\n"
     ]
    }
   ],
   "source": [
    "x = x.new_ones(5, 3, dtype=torch.double)      # new_* methods take in sizes\n",
    "print(x)\n",
    "\n",
    "x = torch.randn_like(x, dtype=torch.float)    # override dtype!\n",
    "print(x)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check the size of a tensor we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([5, 3])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NB! The type torch.Size is an abstraction from a mere tuple, so it supports all the tuple operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operations\n",
    "\n",
    "PyTorch is so pythonic, that it implements operations on tensors in many different syntaxes to match everyones needs and tastes. Let us take a look at the addition operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.5159, -0.4743, -0.6127],\n",
      "        [ 0.7744, -0.9076,  0.4138],\n",
      "        [ 0.8662,  0.4577,  2.1337],\n",
      "        [ 1.5621, -1.0055,  1.8085],\n",
      "        [-0.4157, -0.8873,  0.1187]])\n"
     ]
    }
   ],
   "source": [
    "y = torch.rand(5, 3)\n",
    "print(x + y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.5159, -0.4743, -0.6127],\n",
      "        [ 0.7744, -0.9076,  0.4138],\n",
      "        [ 0.8662,  0.4577,  2.1337],\n",
      "        [ 1.5621, -1.0055,  1.8085],\n",
      "        [-0.4157, -0.8873,  0.1187]])\n"
     ]
    }
   ],
   "source": [
    "print(torch.add(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you need it, you can pass an out variable as a parameter to any operation like add:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.5159, -0.4743, -0.6127],\n",
      "        [ 0.7744, -0.9076,  0.4138],\n",
      "        [ 0.8662,  0.4577,  2.1337],\n",
      "        [ 1.5621, -1.0055,  1.8085],\n",
      "        [-0.4157, -0.8873,  0.1187]])\n"
     ]
    }
   ],
   "source": [
    "result = torch.empty(5, 3)\n",
    "torch.add(x, y, out=result)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor objects support all the operations as methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.5159, -0.4743, -0.6127],\n",
       "        [ 0.7744, -0.9076,  0.4138],\n",
       "        [ 0.8662,  0.4577,  2.1337],\n",
       "        [ 1.5621, -1.0055,  1.8085],\n",
       "        [-0.4157, -0.8873,  0.1187]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.add(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you need to perform an operation in-place, you use the operation_ syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.5159, -0.4743, -0.6127],\n",
       "        [ 0.7744, -0.9076,  0.4138],\n",
       "        [ 0.8662,  0.4577,  2.1337],\n",
       "        [ 1.5621, -1.0055,  1.8085],\n",
       "        [-0.4157, -0.8873,  0.1187]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.add_(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of an in-place operation is stored in the left operand object, in this particular case in x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.5159, -0.4743, -0.6127],\n",
       "        [ 0.7744, -0.9076,  0.4138],\n",
       "        [ 0.8662,  0.4577,  2.1337],\n",
       "        [ 1.5621, -1.0055,  1.8085],\n",
       "        [-0.4157, -0.8873,  0.1187]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sugarish NumPy indexing syntax is also supported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-0.4743, -0.9076,  0.4577, -1.0055, -0.8873])\n"
     ]
    }
   ],
   "source": [
    "print(x[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case there is a need to resize (*reshape*) a tensor, the ``` view ``` method comes into action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(4, 4)\n",
    "y = x.view(16)\n",
    "z = x.view(-1, 8)  # the size -1 denotes the original dimension size\n",
    "print(x.size(), y.size(), z.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the number out of the tensor use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.4692])\n",
      "0.4692341387271881\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(1)\n",
    "print(x)\n",
    "print(x.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5520794987678528"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[1].item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case we need to check, if CUDA is available, we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let us run this cell only if CUDA is available\n",
    "# We will use ``torch.device`` objects to move tensors in and out of GPU\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")          # a CUDA device object\n",
    "    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU\n",
    "    x = x.to(device)                       # or just use strings ``.to(\"cuda\")``\n",
    "    z = x + y\n",
    "    print(z)\n",
    "    print(z.to(\"cpu\", torch.double))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Autograd\n",
    "\n",
    "The next thing that is worth looking at is the automatic gradient computation module of pyTorch. It is called\n",
    "*torch.autograd* . This module does all the *magic* that is connected with gradient computations, using a sofisticated computation graph architecture, that is going to be covered later. For now we will get to know only basic concepts of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To include a `Tensor` into the computation graph, its `.requires_grad` attribute should be set to `True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After any operation is applied (in this particular case - addition), a `Function` object is assigned to the `.grad_fn` attribute of the tensor `y` and added to the computation graph for backward propagation of the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "y = x + 2\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<AddBackward0 object at 0x0000023866D6DF98>\n"
     ]
    }
   ],
   "source": [
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward0>) tensor(27., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z, out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `.grad_fn` attribute can be changed on the fly. See the difference: if a tensor does not require gradient, it is not included into the computation graph, hence it does not store any backward function. However, once `.grad_fn` changed to `True`, all the operations start to be tracked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n",
      "<SumBackward0 object at 0x0000023866D82940>\n"
     ]
    }
   ],
   "source": [
    "a = torch.randn(2, 2)\n",
    "a = ((a * 3) / (a - 1))\n",
    "print(a.requires_grad)\n",
    "a.requires_grad_(True)\n",
    "print(a.requires_grad)\n",
    "b = (a * a).sum()\n",
    "print(b.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most important things in the torch framework is the `.backward()` method. It triggers the calculation of the gradients for all the nodes (e.g. neural net parameters) in the computation graph that are chained to the callee node. \n",
    "\n",
    "NB! `.backward()` when called on a \\[1, 1\\] tensor, requires no arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4.5000, 4.5000],\n",
      "        [4.5000, 4.5000]])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([  369.1664, -1172.1536,   752.2024], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there is a need to stop autograd from tracking history on Tensors you can use either context manager:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "print((x ** 2).requires_grad)\n",
    "\n",
    "with torch.no_grad():\n",
    "    print((x ** 2).requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or `.detach()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n",
      "tensor(True)\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "y = x.detach()\n",
    "print(y.requires_grad)\n",
    "print(x.eq(y).all())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression Using PyTorch\n",
    "### based on [this](https://blog.goodaudience.com/awesome-introduction-to-logistic-regression-in-pytorch-d13883ceaa90) blogpost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, most of pyTorch modeling can be broken down into these steps:\n",
    "* loading the dataset\n",
    "* making the dataset iterable\n",
    "* instantiating the **model** class\n",
    "* instantiating the **loss** class\n",
    "* instantiating the **optimizer** class\n",
    "* training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torchtext import data\n",
    "from torch.nn import functional as F\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "    DEVICE = torch.device(\"cuda\")\n",
    "else:\n",
    "    DEVICE = torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "SEED = 1234\n",
    "\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package movie_reviews to\n",
      "[nltk_data]     C:\\Users\\18397891\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package movie_reviews is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download(\"movie_reviews\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "POS = \"pos\"\n",
    "NEG = \"neg\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text_sentiments = (POS, NEG)\n",
    "\n",
    "train_data_list = []\n",
    "test_data_list = []\n",
    "\n",
    "examples = []\n",
    "\n",
    "for sentiment in text_sentiments:\n",
    "    for filename in os.listdir(os.path.join(nltk.corpus.movie_reviews.root.path, sentiment)):\n",
    "        with open(os.path.join(nltk.corpus.movie_reviews.root.path, sentiment, filename), \"r\", encoding=\"utf-8\") as file:\n",
    "            examples.append({\"text\": file.read().strip(),\n",
    "                             \"sentiment\": int(sentiment == POS)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "examples_df = pd.DataFrame(examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples_df = examples_df.sample(frac=1)\n",
    "train_df = examples_df.sample(frac=0.7)\n",
    "test_df = examples_df.drop(train_df.index)\n",
    "train_texts, train_labels = train_df.text.values, train_df.sentiment.values\n",
    "test_texts, test_labels = test_df.text.values, test_df.sentiment.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(600, 600, 600)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_df.text.values), len(test_df.sentiment.values), len(test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from typing import List, Dict, Any, Iterable\n",
    "from collections import Counter, OrderedDict\n",
    "import math\n",
    "from itertools import islice\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class TfIdfVectorizer:\n",
    "\n",
    "            \n",
    "    def __init__(self, lower=True, tokenizer_pattern=r\"(?i)\\b[a-z]{2,}\\b\"):\n",
    "        self.lower = lower\n",
    "        self.tokenizer_pattern = re.compile(tokenizer_pattern)\n",
    "        self.vocab_df = OrderedDict()\n",
    "        \n",
    "    def __tokenize(self, text: str) -> List[str]:\n",
    "        return self.tokenizer_pattern.findall(text.lower() if self.lower else text)\n",
    "    \n",
    "    def fit(self, texts: Iterable[str]):\n",
    "        term_id = 0\n",
    "        for doc_idx, doc in enumerate(texts):\n",
    "            tokenized = self.__tokenize(doc)\n",
    "            for term in tokenized:\n",
    "                if term not in self.vocab_df:\n",
    "                    self.vocab_df[term] = {}\n",
    "                    self.vocab_df[term][\"doc_ids\"] = {doc_idx}\n",
    "                    self.vocab_df[term][\"doc_count\"] = 1\n",
    "                    self.vocab_df[term][\"id\"] = term_id\n",
    "                    term_id += 1\n",
    "                elif doc_idx not in self.vocab_df[term][\"doc_ids\"]:\n",
    "                    self.vocab_df[term][\"doc_ids\"].add(doc_idx)\n",
    "                    self.vocab_df[term][\"doc_count\"] += 1\n",
    "        texts_len = len(texts)\n",
    "        for term in self.vocab_df:\n",
    "            self.vocab_df[term][\"idf\"] = math.log(texts_len / self.vocab_df[term][\"doc_count\"])\n",
    "        \n",
    "        \n",
    "    def transform(self, texts: Iterable[str]) -> torch.sparse.LongTensor:\n",
    "        values = []\n",
    "        doc_indices = []\n",
    "        term_indices = []\n",
    "        for doc_idx, raw_doc in enumerate(texts):\n",
    "            term_counter = {}\n",
    "            for token in self.__tokenize(raw_doc):\n",
    "                if token in self.vocab_df:\n",
    "                    term = self.vocab_df[token]\n",
    "                    term_idx = term[\"id\"]\n",
    "                    term_idf = term[\"idf\"]\n",
    "                    if term_idx not in term_counter:\n",
    "                        term_counter[term_idx] = term_idf\n",
    "                    else:\n",
    "                        term_counter[term_idx] += term_idf\n",
    "            term_indices.extend(term_counter.keys())\n",
    "            values.extend(term_counter.values())\n",
    "            doc_indices.extend([doc_idx] * len(term_counter))\n",
    "        indices = torch.LongTensor([doc_indices, term_indices], device = DEVICE)\n",
    "        values_tensor = torch.LongTensor(values, device = DEVICE)\n",
    "        self.tf_idf = torch.sparse.LongTensor(indices, values_tensor, torch.Size([len(texts), len(self.vocab_df)]), device = DEVICE)\n",
    "        return self.tf_idf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.34 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "vectorizer = TfIdfVectorizer()\n",
    "vectorizer.fit(train_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.23 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "train_data = vectorizer.transform(train_texts)\n",
    "test_data = vectorizer.transform(test_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make the dataset iterable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader, Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data_loader = DataLoader(train_data, batch_size=64)\n",
    "test_data_loader = DataLoader(test_data, batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def batch(iterable, n=1):\n",
    "    l = len(iterable)\n",
    "    for ndx in range(0, l, n):\n",
    "        yield iterable[ndx:min(ndx + n, l)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class LogisticRegressionModel(nn.Module):\n",
    "\n",
    "    def __init__(self, input_dim, output_dim):\n",
    "        super(LogisticRegressionModel, self).__init__()\n",
    "        self.linear = nn.Linear(input_dim, output_dim)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = F.softmax(self.linear(x))\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = LogisticRegressionModel(len(vectorizer.vocab_df), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.001\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object Module.parameters at 0x000002386E91B468>\n",
      "2\n",
      "torch.Size([2, 33991])\n",
      "torch.Size([2])\n"
     ]
    }
   ],
   "source": [
    "# Type of parameter object\n",
    "print(model.parameters())\n",
    "\n",
    "# Length of parameters\n",
    "print(len(list(model.parameters())))\n",
    "\n",
    "# FC 1 Parameters\n",
    "print(list(model.parameters())[0].size())\n",
    "\n",
    "# FC 1 Bias Parameters\n",
    "print(list(model.parameters())[1].size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_epochs = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch #0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:11: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  # This is added back by InteractiveShellApp.init_path()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch #1\n",
      "Epoch #2\n",
      "Iteration: 50. Loss: 0.6934151649475098. Accuracy: 45\n",
      "Epoch #3\n",
      "Epoch #4\n",
      "Iteration: 100. Loss: 0.6931420564651489. Accuracy: 46\n"
     ]
    }
   ],
   "source": [
    "iteration = 0\n",
    "for epoch in range(num_epochs):\n",
    "    print(f\"Epoch #{epoch}\")\n",
    "    for i, (texts, labels) in enumerate(zip(train_data_loader, batch(train_labels, 64))):\n",
    "        labels = torch.LongTensor(labels)\n",
    "        texts = F.normalize(texts.to(torch.float).to_dense()).requires_grad_()\n",
    "#         print(texts.size(), labels.size(0))\n",
    "\n",
    "        # Clear gradients w.r.t. parameters\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Forward pass to get output/logits\n",
    "        outputs = model(texts)\n",
    "\n",
    "        # Calculate Loss: softmax --> cross entropy loss\n",
    "        loss = criterion(outputs, labels)\n",
    "\n",
    "        # Getting gradients w.r.t. parameters\n",
    "        loss.backward()\n",
    "\n",
    "        # Updating parameters\n",
    "        optimizer.step()\n",
    "\n",
    "        iteration += 1\n",
    "\n",
    "        if iteration % 50 == 0:\n",
    "            # Calculate Accuracy         \n",
    "            correct = 0\n",
    "            total = 0\n",
    "            # Iterate through test dataset\n",
    "            for test_texts, test_labels_batch in zip(test_data_loader, batch(test_labels, 64)):\n",
    "                # Load value to a Torch Variable\n",
    "                test_texts = F.normalize(test_texts.to(torch.float).to_dense())\n",
    "                test_labels_batch = torch.Tensor(test_labels_batch)\n",
    "                # Forward pass only to get logits/output\n",
    "                outputs = model(test_texts)\n",
    "\n",
    "                # Get predictions from the maximum value\n",
    "                _, predicted = torch.max(outputs.data, 1)\n",
    "\n",
    "                # Total number of labels\n",
    "                total += test_labels_batch.size(0)\n",
    "\n",
    "                # Total correct predictions\n",
    "                correct += (predicted == test_labels_batch).sum()\n",
    "\n",
    "            accuracy = 100 * correct / total\n",
    "\n",
    "            # Print Loss\n",
    "            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iteration, loss.item(), accuracy))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}