chapters/dl-representations_simple.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-12-02T17:30:29.181539",
     "start_time": "2016-12-02T17:30:29.172204"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'theme': 'white',\n",
       " 'transition': 'none',\n",
       " 'controls': 'false',\n",
       " 'progress': 'true'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reveal.js\n",
    "from notebook.services.config import ConfigManager\n",
    "cm = ConfigManager()\n",
    "cm.update('livereveal', {\n",
    "        'theme': 'white',\n",
    "        'transition': 'none',\n",
    "        'controls': 'false',\n",
    "        'progress': 'true',\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Representation Learning\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-12-02T16:54:38.546624",
     "start_time": "2016-12-02T16:54:38.540051"
    },
    "cell_style": "center",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Representations of Words\n",
    "    * Motivation\n",
    "    * Sparse Binary Representations\n",
    "    * Dense Continuous Representations\n",
    "* Unsupervised Learning of Word Representations\n",
    "    * Motivation\n",
    "    * Sparse Co-occurence Representations\n",
    "    * Neural Word Representations\n",
    "* From Word Representations to Sentences and Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Feed-forward Neural Networks\n",
    "<center><img src=\"../img/mlp.svg\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why talk about representations? ##\n",
    "\n",
    "* Machine Learning, features are representations of the input data\n",
    "    * Language is special\n",
    "* Better representations, better performance\n",
    "* Representation learning (neural networks, \"deep learning\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What makes a good representation? ##\n",
    "\n",
    "1. Representations are **distinct**\n",
    "2. **Similar** words have **similar** representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Formal Task ##\n",
    "\n",
    "* Words: $w$\n",
    "* Vocabulary: $\\mathbb{V} (\\forall_{i} w_{i} \\in \\mathbb{V})$\n",
    "* Find representation function: $f(w_{i}) = r_{i}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sparse Binary Representations ##\n",
    "\n",
    "* Map words to unique positive non-zero integers\n",
    "    * $f_{id}(w) \\mapsto \\mathbb{N^{*}}$\n",
    "    * $g(w, i) = {\\left\\lbrace\n",
    "    \\begin{array}{ll}\n",
    "        1 & \\textrm{if }~i = f_{id}(w) \\\\\n",
    "        0 & \\textrm{otherwise} \\\\\n",
    "    \\end{array}\\right.}$\n",
    "* Word representations as \"one-hot\" vectors\n",
    "    * $f_{sb}(w) = (g(w, 1), \\ldots, g(w, |V|))$\n",
    "    * $f_{sb}(w) \\mapsto \\{0,1\\}^{|V|}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example ###\n",
    "\n",
    "* $\\mathbb{V} = \\{\\textrm{apple}, \\textrm{orange}, \\textrm{rabbit}\\}$\n",
    "* $f_{id}(\\textrm{apple}) = 1, \\ldots$, $f_{id}(\\textrm{rabbit}) = 3$\n",
    "* $f_{sb}(\\textrm{apple}) = (1, 0, 0)$\n",
    "* $f_{sb}(\\textrm{orange}) = (0, 1, 0)$\n",
    "* $f_{sb}(\\textrm{rabbit}) = (0, 0, 1)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Sparse Binary Visualised ##\n",
    "\n",
    "![Sparse binary representations visualised](../img/sparse_binary.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Cosine Similarity ##\n",
    "\n",
    "* $cos(u, v) = \\frac{u \\cdot v}{||u|| \\cdot ||v||}$\n",
    "* $cos(u, v) \\mapsto [-1, 1]$\n",
    "* $cos(u, v) = 1$; identical\n",
    "* $cos(u, v) = -1$; opposites\n",
    "* $cos(u, v) = 0$; orthogonal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Note the different formulation in SciPy:\n",
    "$$cos(u, v) = 1 - \\frac{u \\cdot v}{||u|| \\cdot||v||}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Cosine Similarity Visualised ##\n",
    "\n",
    "<center><img src=\"http://blog.christianperone.com/wp-content/uploads/2013/09/cosinesimilarityfq1.png\" width=\"110%\"></center>\n",
    "\n",
    "http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/quiz_time.png?0.797147932631911\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/quiz_time.png'+'?'+str(random.random()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## [ucph.page.link/cos](https://ucph.page.link/cos)\n",
    "([Responses](https://docs.google.com/forms/d/1mbVCSbDXpA9qY5DkfR-fnX3c3MFXPiLAqeTrd2P0U90/edit#responses))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dense Continuous Representations ##\n",
    "\n",
    "* $f_{id}(w) \\mapsto \\mathbb{N}^{*}$\n",
    "* \"Embed\" words as matrix rows\n",
    "* Dimensionality: $d$ (hyperparameter)\n",
    "* $W \\in \\mathbb{R}^{|\\mathbb{V}| \\times d}$\n",
    "* $f_{dc}(w) = W_{f_{id}(w), :}$\n",
    "* $f_{dc}(w) \\mapsto \\mathbb{R}^{d}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example ###\n",
    "\n",
    "* $\\mathbb{V} = \\{\\textrm{apple}, \\textrm{orange}, \\textrm{rabbit}\\}$\n",
    "* $d = 2$\n",
    "* $W \\in \\mathbb{R}^{3 \\times 2}$\n",
    "* $f_{id}(\\textrm{apple}) = 1, \\ldots, f_{id}(\\textrm{rabbit}) = 3$\n",
    "* $f_{dc}(\\textrm{apple}) = (1.0, 1.0)$\n",
    "* $f_{dc}(\\textrm{orange}) = (0.9, 1.0)$\n",
    "* $f_{dc}(\\textrm{rabbit}) = (0.1, 0.5)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dense Continuous Visualised ##\n",
    "\n",
    "<center><img src=\"../img/dense_continuous.svg\" width=\"80%\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dense Continuous Similarities ##\n",
    "\n",
    "* $cos(f_{dc}(\\textrm{apple}),f_{dc}(\\textrm{rabbit})) \\approx 0.83$\n",
    "* $cos(f_{dc}(\\textrm{apple}),f_{dc}(\\textrm{orange})) \\approx 1.0$\n",
    "* $cos(f_{dc}(\\textrm{orange}),f_{dc}(\\textrm{rabbit})) \\approx 0.86$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Unsupervised Learning of Word Representations #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Why not supervised? ##\n",
    "\n",
    "<center><img src=\"../img/annotated_vs_unannotated_data.svg\" width=\"40%\"></center>\n",
    "\n",
    "\n",
    "* Current gains from large data sets / more compute\n",
    "    * Lack of comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Linguistic Inspirations ##\n",
    "\n",
    "* \"Oculist and eye-doctor … occur in almost the same environments. … If $A$ and $B$ have almost identical environments we say that they are synonyms.\" – Zellig Harris (1954)\n",
    "* \"You shall know a word by the company it keeps.\" – John Rupert Firth (1957)\n",
    "* Akin to \"meaning is use\" – Wittgenstein (1953)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Sparse Co-occurence Representations ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Co-occurences ##\n",
    "\n",
    "* Collected from a large collection of *raw* text\n",
    "* E.g. Wikipedia, crawled news data, tweets, ...\n",
    "\n",
    "1. \"…comparing an **apple** to an **orange**…\"\n",
    "2. \"…an **apple** and **orange** from Florida…\"\n",
    "3. \"…my **rabbit** is not shaped like an **orange**…\" (yes, there is always **noise** in the data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Sparse Co-occurence Representations ##\n",
    "\n",
    "* The number of times words co-occur in a text collection\n",
    "* $C \\in \\mathbb{N}^{|V| \\times |V|}$\n",
    "* $f_{id}(\\textrm{apple}) = 1, \\ldots, f_{id}(\\textrm{rabbit}) = 3$\n",
    "* $C = \\begin{pmatrix}\n",
    "        2 & 2 & 0 \\\\\n",
    "        2 & 3 & 1 \\\\\n",
    "        0 & 1 & 1 \\\\\n",
    "    \\end{pmatrix}$\n",
    "* $f_{cs}(w) = C_{f_{id}(w), :}$\n",
    "* $f_{cs}(w) \\mapsto \\mathbb{N}^{|V|}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example ###\n",
    "\n",
    "* $\\mathbb{V} = \\{\\textrm{apple}, \\textrm{orange}, \\textrm{rabbit}\\}$\n",
    "* $f_{id}(\\textrm{apple}) = 1, \\ldots, f_{id}(\\textrm{rabbit}) = 3$\n",
    "* $f_{cs}(\\textrm{apple}) = (2, 2, 0)$\n",
    "* $f_{cs}(\\textrm{orange}) = (2, 3, 1)$\n",
    "* $f_{cs}(\\textrm{rabbit}) = (0, 1, 1)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Sparse Co-occurence Similarities ##\n",
    "\n",
    "* $cos(f_{cs}(\\textrm{apple}), f_{cs}(\\textrm{rabbit})) \\approx 0.50$\n",
    "* $cos(f_{cs}(\\textrm{apple}), f_{cs}(\\textrm{orange})) \\approx 0.94$\n",
    "* $cos(f_{cs}(\\textrm{orange}), f_{cs}(\\textrm{rabbit})) \\approx 0.76$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Neural Word Representations #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Learning by Slot Filling ##\n",
    "\n",
    "* \"…I had some **_____** for breakfast today…\"\n",
    "* Good: *cereal*\n",
    "* Bad: *airplanes*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img width=1000 src=\"../img/cbow_sg2.png\"></center>\n",
    "\n",
    "<div style=\"text-align: right;\">\n",
    "    (word2vec: <a href=\"https://arxiv.org/abs/1301.3781\">Mikolov et al., 2013</a>)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Creating Positive Training Instances ##\n",
    "\n",
    "\n",
    "\"I had some cereal for breakfast today\"\n",
    "\n",
    "* **I** had some cereal for breakfast today -> (**I**, had); (**I**, some)\n",
    "* I **had** some cereal for breakfast today -> (**had**, I); (**had**, some); (**had**, cereal)\n",
    "* I had **some** cereal for breakfast today -> (**some**, I); (**some**, had); (**some**, cereal); (**some**, for)\n",
    "* I had some **cereal** for breakfast today -> (**cereal**, had); (**cereal**, some); (**cereal**, for); (**cereal**, breakfast)\n",
    "* ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Training instance: target word $w \\in \\mathbb{V}$; context word $c \\in \\mathbb{V}$\n",
    "\n",
    "$D = ((c, w),\\ldots)$; observed co-occurences\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Sampling Negative Training Instances ##\n",
    "\n",
    "\n",
    "\"I had some cereal for breakfast today\"\n",
    "\n",
    "* **Lecture** had some cereal for breakfast today -> (**Lecture**, had); (**Lecture**, some)\n",
    "* I **computer** some cereal for breakfast today -> (**computer**, I); (**computer**, some); (**computer**, cereal)\n",
    "* I had **word** cereal for breakfast today -> (**word**, I); (**word**, had); (**word**, cereal); (**word**, for)\n",
    "* I had some **books** for breakfast today -> (**books**, had); (**books**, some); (**books**, for); (**books**, breakfast)\n",
    "* ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "Create $D' = ((c, w),\\ldots)$; \"noise samples\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Unsupervised Loss Function ##\n",
    "\n",
    "* $w \\in \\mathbb{V}$; $c \\in \\mathbb{V}$\n",
    "* $D = ((c, w),\\ldots)$; observed co-occurences\n",
    "* $D' = ((c, w),\\ldots)$; \"noise samples\"\n",
    "* $\\textrm{max}~p((c, w) \\in D | W) - p((c, w) \\in D' | W)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Idea: maximise difference between the score for true instances and negative samples\n",
    "* How do we get $p(c, w)$?\n",
    "   * using a neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Neural Skip-Gram Model ##\n",
    "\n",
    "<center><img width=650 src=\"../img/skip-gram_architecture.png\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* $W^{w} \\in \\mathbb{R}^{|\\mathbb{V}| \\times d}$; $W^{c} \\in \\mathbb{R}^{|\\mathbb{V}| \\times d}$\n",
    "* $D = ((c, w),\\ldots)$; $D' = ((c, w),\\ldots)$\n",
    "* $\\sigma(x) = \\frac{1}{1 + \\textrm{exp}(-x)}$\n",
    "* $p((c, w) \\in D | W^{w}, W^{c}) = \\sigma(W^{c}_{f_{id}(c),:} \\cdot W^{w}_{f_{id}(w),:})$\n",
    "* $\\arg\\max\\limits_{W^{w},W^{c}} \\sum\\limits_{(w,c) \\in D} \\log \\sigma(W^{c}_{f_{id}(c),:} \\cdot W^{w}_{f_{id}(w),:})  + \\sum\\limits_{(w,c) \\in D'} \\log \\sigma(-W^{c}_{f_{id}(c),:} \\cdot W^{w}_{f_{id}(w),:})$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Neural Representation ##\n",
    "\n",
    "* Learned using [word2vec](https://code.google.com/p/word2vec/)\n",
    "* Google News data, $~1,000,000,000$ words\n",
    "* $|\\mathbb{V}| = 3,000,000$\n",
    "* $d = 300$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Neural Representation Example ##\n",
    "\n",
    "* $f_{n}(\\textrm{apple}) = (-0.06, -0.16, \\ldots, 0.34)$\n",
    "* $f_{n}(\\textrm{orange}) = (-0.10, -0.18, \\ldots, 0.08)$\n",
    "* $f_{n}(\\textrm{rabbit}) = (0.02, 0.11, \\ldots, 0.11)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Neural Representation Similarities ##\n",
    "\n",
    "* $cos(f_{n}(\\textrm{apple}), f_{n}(\\textrm{rabbit})) \\approx 0.34$\n",
    "* $cos(f_{n}(\\textrm{apple}), f_{n}(\\textrm{orange})) \\approx 0.39$\n",
    "* $cos(f_{n}(\\textrm{orange}), f_{n}(\\textrm{rabbit})) \\approx 0.20$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Neural Representations Visualised ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/word_representations.svg?0.18778573578323954\" width=\"1200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/word_representations.svg'+'?'+str(random.random()), width=1200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Dimensionality reduction using [t-SNE](https://lvdmaaten.github.io/tsne/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Neural Representations Visualised (zoomed) ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/word_representations_zoom.svg?0.1130807191996096\" width=\"1200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/word_representations_zoom.svg'+'?'+str(random.random()), width=1200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* Dimensionality reduction using [t-SNE](https://lvdmaaten.github.io/tsne/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/quiz_time.png?0.7587317573482633\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/quiz_time.png'+'?'+str(random.random()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## [ucph.page.link/wordrep](https://ucph.page.link/wordrep)\n",
    "([Responses](https://docs.google.com/forms/d/1eJJKaSUc8MfxtAvwbLArg6cHneV4m_WS6xqc1CmNSvY/edit#responses))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Word Representation Algebra ##\n",
    "\n",
    "* $f_{n}(\\textrm{king}) - f_{n}(\\textrm{man}) + f_{n}(\\textrm{woman}) \\approx f_{n}(\\textrm{queen})$\n",
    "* $f_{n}(\\textrm{Paris}) - f_{n}(\\textrm{France}) + f_{n}(\\textrm{Italy}) \\approx f_{n}(\\textrm{Rome})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/regularities.png?0.5868496187301693\" width=\"1000\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/regularities.png'+'?'+str(random.random()), width=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "(It's not really that simple... see [Drozd et al. (2016)](https://aclanthology.org/C16-1332.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Granularities of representations #\n",
    "\n",
    "## Output\n",
    "* Word representations\n",
    "* Sentence representations\n",
    "* Document representations\n",
    "\n",
    "## Input\n",
    "* Words\n",
    "* Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# From Words to Sentences to Documents\n",
    "\n",
    "* Standard pooling approaches of word embeddings\n",
    "    * Sum, Mean, Max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/embedding_sum.jpg?0.8622647983897516\" width=\"500\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/embedding_sum.jpg'+'?'+str(random.random()), width=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# From Words to Sentences to Documents\n",
    "\n",
    "* Standard pooling approaches of word embeddings\n",
    "    * Sum, Mean, Max\n",
    "    * TF-IDF weighting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/embedding_weighted.jpg?0.21418794608659253\" width=\"500\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/embedding_weighted.jpg'+'?'+str(random.random()), width=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Building Sentence and Document Representations\n",
    "\n",
    "* Sentence representations from scratch (e.g. with RNNs in next lecture)\n",
    "\n",
    "* Doc2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Doc2vec\n",
    "\n",
    "\n",
    "* Simple extension of word2vec (cbow)\n",
    "* Paragraph vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/doc2vec_0.png?0.05223337074153245\" width=\"1000\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/doc2vec_0.png'+'?'+str(random.random()), width=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Source: https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Doc2vec\n",
    "\n",
    "\n",
    "* Simple extension of word2vec (cbow)\n",
    "* Paragraph vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../img/doc2vec_1.png?0.2755562451691236\" width=\"1000\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='../img/doc2vec_1.png'+'?'+str(random.random()), width=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Source: https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary #\n",
    "\n",
    "* Moving from features to \"representations\"\n",
    "* Representations limits what we can learn and our generalisation power\n",
    "* Many ways to learn representations (there are many more than what we covered)\n",
    "* Neural representations most widely used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Additional Reading #\n",
    "\n",
    "* [\"Word Representations: A Simple and General Method for Semi-Supervised Learning\"](http://www.aclweb.org/anthology/P/P10/P10-1040.pdf) by Turian et al. (2010)\n",
    "* [\"Representation Learning: A Review and New Perspectives\"](https://arxiv.org/abs/1206.5538) by Bengio et al. (2012)\n",
    "* [\"Linguistic Regularities in Continuous Space Word Representations\"](http://www.aclweb.org/anthology/N/N13/N13-1090.pdf) by Mikolov et al. (2013a) ([video](http://techtalks.tv/talks/linguistic-regularities-in-continuous-space-word-representations/58471/))\n",
    "* [\"Distributed Representations of Words and Phrases and their Compositionality\"](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality) by Mikolov et al. (2013b)\n",
    "* [\"word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method\"](https://arxiv.org/abs/1402.3722) by Goldberg and Levy (2014)\n",
    "* [\"Neural Word Embedding as Implicit Matrix Factorization\"](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization) by Levy and Goldberg (2014)\n",
    "* [Demystifying Neural Network in Skip-Gram Language Modeling](https://aegis4048.github.io/demystifying_neural_network_in_skip_gram_language_modeling)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}