run_topics_and_transformations.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Topics and Transformations\n",
    "\n",
    "Introduces transformations and demonstrates their use on a toy corpus.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:23:30.875969200Z",
     "start_time": "2024-05-07T05:23:30.864833800Z"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, I will show how to transform documents from one vector representation\n",
    "into another. This process serves two goals:\n",
    "\n",
    "1. To bring out hidden structure in the corpus, discover relationships between\n",
    "   words and use them to describe the documents in a new and\n",
    "   (hopefully) more semantic way.\n",
    "2. To make the document representation more compact. This both improves efficiency\n",
    "   (new representation consumes less resources) and efficacy (marginal data\n",
    "   trends are ignored, noise-reduction).\n",
    "\n",
    "## Creating the Corpus\n",
    "\n",
    "First, we need to create a corpus to work with.\n",
    "This step is the same as in the previous tutorial;\n",
    "if you completed it, feel free to skip to the next section.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:24:03.197270200Z",
     "start_time": "2024-05-07T05:24:01.788220Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:24:03,167 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2024-05-07 13:24:03,169 : INFO : built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)\n",
      "2024-05-07 13:24:03,173 : INFO : Dictionary lifecycle event {'msg': \"built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)\", 'datetime': '2024-05-07T13:24:03.172664', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'created'}\n"
     ]
    }
   ],
   "source": [
    "from collections import defaultdict\n",
    "from gensim import corpora\n",
    "\n",
    "documents = [\n",
    "    \"Human machine interface for lab abc computer applications\",\n",
    "    \"A survey of user opinion of computer system response time\",\n",
    "    \"The EPS user interface management system\",\n",
    "    \"System and human system engineering testing of EPS\",\n",
    "    \"Relation of user perceived response time to error measurement\",\n",
    "    \"The generation of random binary unordered trees\",\n",
    "    \"The intersection graph of paths in trees\",\n",
    "    \"Graph minors IV Widths of trees and well quasi ordering\",\n",
    "    \"Graph minors A survey\",\n",
    "]\n",
    "\n",
    "# remove common words and tokenize\n",
    "stoplist = set('for a of the and to in'.split())\n",
    "texts = [\n",
    "    [word for word in document.lower().split() if word not in stoplist]\n",
    "    for document in documents\n",
    "]\n",
    "\n",
    "# remove words that appear only once\n",
    "frequency = defaultdict(int)\n",
    "for text in texts:\n",
    "    for token in text:\n",
    "        frequency[token] += 1\n",
    "\n",
    "texts = [\n",
    "    [token for token in text if frequency[token] > 1]\n",
    "    for text in texts\n",
    "]\n",
    "\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a transformation\n",
    "\n",
    "The transformations are standard Python objects, typically initialized by means of\n",
    "a :dfn:`training corpus`:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:24:11.971314400Z",
     "start_time": "2024-05-07T05:24:11.876182900Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:24:11,875 : INFO : collecting document frequencies\n",
      "2024-05-07 13:24:11,876 : INFO : PROGRESS: processing document #0\n",
      "2024-05-07 13:24:11,879 : INFO : TfidfModel lifecycle event {'msg': 'calculated IDF weights for 9 documents and 12 features (28 matrix non-zeros)', 'datetime': '2024-05-07T13:24:11.879218', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'initialize'}\n"
     ]
    }
   ],
   "source": [
    "from gensim import models\n",
    "\n",
    "tfidf = models.TfidfModel(corpus)  # step 1 -- initialize a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We used our old corpus from tutorial 1 to initialize (train) the transformation model. Different\n",
    "transformations may require different initialization parameters; in case of TfIdf, the\n",
    "\"training\" consists simply of going through the supplied corpus once and computing document frequencies\n",
    "of all its features. Training other models, such as Latent Semantic Analysis or Latent Dirichlet\n",
    "Allocation, is much more involved and, consequently, takes much more time.\n",
    "\n",
    "<div class=\"alert alert-info\"><h4>Note</h4><p>Transformations always convert between two specific vector\n",
    "  spaces. The same vector space (= the same set of feature ids) must be used for training\n",
    "  as well as for subsequent vector transformations. Failure to use the same input\n",
    "  feature space, such as applying a different string preprocessing, using different\n",
    "  feature ids, or using bag-of-words input vectors where TfIdf vectors are expected, will\n",
    "  result in feature mismatch during transformation calls and consequently in either\n",
    "  garbage output and/or runtime exceptions.</p></div>\n",
    "\n",
    "\n",
    "### Transforming vectors\n",
    "\n",
    "From now on, ``tfidf`` is treated as a read-only object that can be used to convert\n",
    "any vector from the old representation (bag-of-words integer counts) to the new representation\n",
    "(TfIdf real-valued weights):\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:24:59.655551500Z",
     "start_time": "2024-05-07T05:24:59.569596100Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.7071067811865476), (1, 0.7071067811865476)]\n"
     ]
    }
   ],
   "source": [
    "doc_bow = [(0, 1), (1, 1)]\n",
    "print(tfidf[doc_bow])  # step 2 -- use the model to transform vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or to apply a transformation to a whole corpus:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:25:38.190997800Z",
     "start_time": "2024-05-07T05:25:38.167014400Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.5773502691896257), (1, 0.5773502691896257), (2, 0.5773502691896257)]\n",
      "[(0, 0.44424552527467476), (3, 0.44424552527467476), (4, 0.44424552527467476), (5, 0.3244870206138555), (6, 0.44424552527467476), (7, 0.3244870206138555)]\n",
      "[(2, 0.5710059809418182), (5, 0.4170757362022777), (7, 0.4170757362022777), (8, 0.5710059809418182)]\n",
      "[(1, 0.49182558987264147), (5, 0.7184811607083769), (8, 0.49182558987264147)]\n",
      "[(3, 0.6282580468670046), (6, 0.6282580468670046), (7, 0.45889394536615247)]\n",
      "[(9, 1.0)]\n",
      "[(9, 0.7071067811865475), (10, 0.7071067811865475)]\n",
      "[(9, 0.5080429008916749), (10, 0.5080429008916749), (11, 0.695546419520037)]\n",
      "[(4, 0.6282580468670046), (10, 0.45889394536615247), (11, 0.6282580468670046)]\n"
     ]
    }
   ],
   "source": [
    "corpus_tfidf = tfidf[corpus]\n",
    "for doc in corpus_tfidf:\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular case, we are transforming the same corpus that we used\n",
    "for training, but this is only incidental. Once the transformation model has been initialized,\n",
    "it can be used on any vectors (provided they come from the same vector space, of course),\n",
    "even if they were not used in the training corpus at all. This is achieved by a process called\n",
    "folding-in for LSA, by topic inference for LDA etc.\n",
    "\n",
    "<div class=\"alert alert-info\"><h4>Note</h4><p>Calling ``model[corpus]`` only creates a wrapper around the old ``corpus``\n",
    "  document stream -- actual conversions are done on-the-fly, during document iteration.\n",
    "  We cannot convert the entire corpus at the time of calling ``corpus_transformed = model[corpus]``,\n",
    "  because that would mean storing the result in main memory, and that contradicts gensim's objective of memory-indepedence.\n",
    "  If you will be iterating over the transformed ``corpus_transformed`` multiple times, and the\n",
    "  transformation is costly, `serialize the resulting corpus to disk first <corpus-formats>` and continue\n",
    "  using that.</p></div>\n",
    "\n",
    "Transformations can also be serialized, one on top of another, in a sort of chain:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:30:16.256305100Z",
     "start_time": "2024-05-07T05:30:15.974197600Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:30:15,975 : INFO : using serial LSI version on this node\n",
      "2024-05-07 13:30:15,976 : INFO : updating model with new documents\n",
      "2024-05-07 13:30:15,977 : INFO : preparing a new chunk of documents\n",
      "2024-05-07 13:30:15,978 : INFO : using 100 extra samples and 2 power iterations\n",
      "2024-05-07 13:30:15,979 : INFO : 1st phase: constructing (12, 102) action matrix\n",
      "2024-05-07 13:30:15,990 : INFO : orthonormalizing (12, 102) action matrix\n",
      "2024-05-07 13:30:16,111 : INFO : 2nd phase: running dense svd on (12, 9) matrix\n",
      "2024-05-07 13:30:16,117 : INFO : computing the final decomposition\n",
      "2024-05-07 13:30:16,119 : INFO : keeping 2 factors (discarding 47.565% of energy spectrum)\n",
      "2024-05-07 13:30:16,124 : INFO : processed documents up to #9\n",
      "2024-05-07 13:30:16,126 : INFO : topic #0(1.594): 0.703*\"trees\" + 0.538*\"graph\" + 0.402*\"minors\" + 0.187*\"survey\" + 0.061*\"system\" + 0.060*\"time\" + 0.060*\"response\" + 0.058*\"user\" + 0.049*\"computer\" + 0.035*\"interface\"\n",
      "2024-05-07 13:30:16,126 : INFO : topic #1(1.476): -0.460*\"system\" + -0.373*\"user\" + -0.332*\"eps\" + -0.328*\"interface\" + -0.320*\"time\" + -0.320*\"response\" + -0.293*\"computer\" + -0.280*\"human\" + -0.171*\"survey\" + 0.161*\"trees\"\n",
      "2024-05-07 13:30:16,126 : INFO : LsiModel lifecycle event {'msg': 'trained LsiModel(num_terms=12, num_topics=2, decay=1.0, chunksize=20000) in 0.15s', 'datetime': '2024-05-07T13:30:16.126596', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'created'}\n"
     ]
    }
   ],
   "source": [
    "lsi_model = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=2)  # initialize an LSI transformation\n",
    "corpus_lsi = lsi_model[corpus_tfidf]  # create a double wrapper over the original corpus: bow->tfidf->fold-in-lsi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we transformed our Tf-Idf corpus via `Latent Semantic Indexing <http://en.wikipedia.org/wiki/Latent_semantic_indexing>`_\n",
    "into a latent 2-D space (2-D because we set ``num_topics=2``). Now you're probably wondering: what do these two latent\n",
    "dimensions stand for? Let's inspect with :func:`models.LsiModel.print_topics`:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:30:30.013154900Z",
     "start_time": "2024-05-07T05:30:29.743737100Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:30:29,742 : INFO : topic #0(1.594): 0.703*\"trees\" + 0.538*\"graph\" + 0.402*\"minors\" + 0.187*\"survey\" + 0.061*\"system\" + 0.060*\"time\" + 0.060*\"response\" + 0.058*\"user\" + 0.049*\"computer\" + 0.035*\"interface\"\n",
      "2024-05-07 13:30:29,745 : INFO : topic #1(1.476): -0.460*\"system\" + -0.373*\"user\" + -0.332*\"eps\" + -0.328*\"interface\" + -0.320*\"time\" + -0.320*\"response\" + -0.293*\"computer\" + -0.280*\"human\" + -0.171*\"survey\" + 0.161*\"trees\"\n"
     ]
    },
    {
     "data": {
      "text/plain": "[(0,\n  '0.703*\"trees\" + 0.538*\"graph\" + 0.402*\"minors\" + 0.187*\"survey\" + 0.061*\"system\" + 0.060*\"time\" + 0.060*\"response\" + 0.058*\"user\" + 0.049*\"computer\" + 0.035*\"interface\"'),\n (1,\n  '-0.460*\"system\" + -0.373*\"user\" + -0.332*\"eps\" + -0.328*\"interface\" + -0.320*\"time\" + -0.320*\"response\" + -0.293*\"computer\" + -0.280*\"human\" + -0.171*\"survey\" + 0.161*\"trees\"')]"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi_model.print_topics(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(the topics are printed to log -- see the note at the top of this page about activating\n",
    "logging)\n",
    "\n",
    "It appears that according to LSI, \"trees\", \"graph\" and \"minors\" are all related\n",
    "words (and contribute the most to the direction of the first topic), while the\n",
    "second topic practically concerns itself with all the other words. As expected,\n",
    "the first five documents are more strongly related to the second topic while the\n",
    "remaining four documents to the first topic:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:30:54.331655100Z",
     "start_time": "2024-05-07T05:30:54.306724300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.06600783396090183), (1, -0.5200703306361856)] Human machine interface for lab abc computer applications\n",
      "[(0, 0.19667592859142435), (1, -0.7609563167700053)] A survey of user opinion of computer system response time\n",
      "[(0, 0.08992639972446285), (1, -0.7241860626752512)] The EPS user interface management system\n",
      "[(0, 0.07585847652178036), (1, -0.6320551586003432)] System and human system engineering testing of EPS\n",
      "[(0, 0.10150299184980095), (1, -0.5737308483002959)] Relation of user perceived response time to error measurement\n",
      "[(0, 0.7032108939378313), (1, 0.16115180214025673)] The generation of random binary unordered trees\n",
      "[(0, 0.8774787673119833), (1, 0.1675890686465927)] The intersection graph of paths in trees\n",
      "[(0, 0.9098624686818579), (1, 0.14086553628718884)] Graph minors IV Widths of trees and well quasi ordering\n",
      "[(0, 0.6165825350569283), (1, -0.053929075663894564)] Graph minors A survey\n"
     ]
    }
   ],
   "source": [
    "# both bow->tfidf and tfidf->lsi transformations are actually executed here, on the fly\n",
    "for doc, as_text in zip(corpus_lsi, documents):\n",
    "    print(doc, as_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model persistency is achieved with the :func:`save` and :func:`load` functions:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:32:00.149461700Z",
     "start_time": "2024-05-07T05:32:00.042696300Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:32:00,043 : INFO : Projection lifecycle event {'fname_or_handle': 'C:\\\\Users\\\\017731~1\\\\AppData\\\\Local\\\\Temp\\\\model-kgiqdd7z.lsi.projection', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2024-05-07T13:32:00.043623', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'saving'}\n",
      "2024-05-07 13:32:00,045 : INFO : saved C:\\Users\\017731~1\\AppData\\Local\\Temp\\model-kgiqdd7z.lsi.projection\n",
      "2024-05-07 13:32:00,046 : INFO : LsiModel lifecycle event {'fname_or_handle': 'C:\\\\Users\\\\017731~1\\\\AppData\\\\Local\\\\Temp\\\\model-kgiqdd7z.lsi', 'separately': 'None', 'sep_limit': 10485760, 'ignore': ['projection', 'dispatcher'], 'datetime': '2024-05-07T13:32:00.046002', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'saving'}\n",
      "2024-05-07 13:32:00,046 : INFO : not storing attribute projection\n",
      "2024-05-07 13:32:00,047 : INFO : not storing attribute dispatcher\n",
      "2024-05-07 13:32:00,050 : INFO : saved C:\\Users\\017731~1\\AppData\\Local\\Temp\\model-kgiqdd7z.lsi\n",
      "2024-05-07 13:32:00,053 : INFO : loading LsiModel object from C:\\Users\\017731~1\\AppData\\Local\\Temp\\model-kgiqdd7z.lsi\n",
      "2024-05-07 13:32:00,059 : INFO : loading id2word recursively from C:\\Users\\017731~1\\AppData\\Local\\Temp\\model-kgiqdd7z.lsi.id2word.* with mmap=None\n",
      "2024-05-07 13:32:00,061 : INFO : setting ignored attribute projection to None\n",
      "2024-05-07 13:32:00,062 : INFO : setting ignored attribute dispatcher to None\n",
      "2024-05-07 13:32:00,063 : INFO : LsiModel lifecycle event {'fname': 'C:\\\\Users\\\\017731~1\\\\AppData\\\\Local\\\\Temp\\\\model-kgiqdd7z.lsi', 'datetime': '2024-05-07T13:32:00.063922', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'loaded'}\n",
      "2024-05-07 13:32:00,064 : INFO : loading LsiModel object from C:\\Users\\017731~1\\AppData\\Local\\Temp\\model-kgiqdd7z.lsi.projection\n",
      "2024-05-07 13:32:00,066 : INFO : Projection lifecycle event {'fname': 'C:\\\\Users\\\\017731~1\\\\AppData\\\\Local\\\\Temp\\\\model-kgiqdd7z.lsi.projection', 'datetime': '2024-05-07T13:32:00.066911', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'loaded'}\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import tempfile\n",
    "\n",
    "with tempfile.NamedTemporaryFile(prefix='model-', suffix='.lsi', delete=False) as tmp:\n",
    "    lsi_model.save(tmp.name)  # same for tfidf, lda, ...\n",
    "\n",
    "loaded_lsi_model = models.LsiModel.load(tmp.name)\n",
    "\n",
    "os.unlink(tmp.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next question might be: just how exactly similar are those documents to each other?\n",
    "Is there a way to formalize the similarity, so that for a given input document, we can\n",
    "order some other set of documents according to their similarity? Similarity queries\n",
    "are covered in the next tutorial (`sphx_glr_auto_examples_core_run_similarity_queries.py`).\n",
    "\n",
    "\n",
    "## Available transformations\n",
    "\n",
    "Gensim implements several popular Vector Space Model algorithms:\n",
    "\n",
    "* `Term Frequency * Inverse Document Frequency, Tf-Idf <http://en.wikipedia.org/wiki/Tf%E2%80%93idf>`_\n",
    "  expects a bag-of-words (integer values) training corpus during initialization.\n",
    "  During transformation, it will take a vector and return another vector of the\n",
    "  same dimensionality, except that features which were rare in the training corpus\n",
    "  will have their value increased.\n",
    "  It therefore converts integer-valued vectors into real-valued ones, while leaving\n",
    "  the number of dimensions intact. It can also optionally normalize the resulting\n",
    "  vectors to (Euclidean) unit length.\n",
    "\n",
    " .. sourcecode:: pycon\n",
    "\n",
    "    model = models.TfidfModel(corpus, normalize=True)\n",
    "\n",
    "* `Okapi Best Matching, Okapi BM25 <https://en.wikipedia.org/wiki/Okapi_BM25>`_\n",
    "  expects a bag-of-words (integer values) training corpus during initialization.\n",
    "  During transformation, it will take a vector and return another vector of the\n",
    "  same dimensionality, except that features which were rare in the training corpus\n",
    "  will have their value increased. It therefore converts integer-valued\n",
    "  vectors into real-valued ones, while leaving the number of dimensions intact.\n",
    "\n",
    "  Okapi BM25 is the standard ranking function used by search engines to estimate\n",
    "  the relevance of documents to a given search query.\n",
    "\n",
    " .. sourcecode:: pycon\n",
    "\n",
    "    model = models.OkapiBM25Model(corpus)\n",
    "\n",
    "* `Latent Semantic Indexing, LSI (or sometimes LSA) <http://en.wikipedia.org/wiki/Latent_semantic_indexing>`_\n",
    "  transforms documents from either bag-of-words or (preferrably) TfIdf-weighted space into\n",
    "  a latent space of a lower dimensionality. For the toy corpus above we used only\n",
    "  2 latent dimensions, but on real corpora, target dimensionality of 200--500 is recommended\n",
    "  as a \"golden standard\" [1]_.\n",
    "\n",
    "  .. sourcecode:: pycon\n",
    "\n",
    "    model = models.LsiModel(tfidf_corpus, id2word=dictionary, num_topics=300)\n",
    "\n",
    "  LSI training is unique in that we can continue \"training\" at any point, simply\n",
    "  by providing more training documents. This is done by incremental updates to\n",
    "  the underlying model, in a process called `online training`. Because of this feature, the\n",
    "  input document stream may even be infinite -- just keep feeding LSI new documents\n",
    "  as they arrive, while using the computed transformation model as read-only in the meanwhile!\n",
    "\n",
    "  .. sourcecode:: pycon\n",
    "\n",
    "    model.add_documents(another_tfidf_corpus)  # now LSI has been trained on tfidf_corpus + another_tfidf_corpus\n",
    "    lsi_vec = model[tfidf_vec]  # convert some new document into the LSI space, without affecting the model\n",
    "\n",
    "    model.add_documents(more_documents)  # tfidf_corpus + another_tfidf_corpus + more_documents\n",
    "    lsi_vec = model[tfidf_vec]\n",
    "\n",
    "  See the :mod:`gensim.models.lsimodel` documentation for details on how to make\n",
    "  LSI gradually \"forget\" old observations in infinite streams. If you want to get dirty,\n",
    "  there are also parameters you can tweak that affect speed vs. memory footprint vs. numerical\n",
    "  precision of the LSI algorithm.\n",
    "\n",
    "  `gensim` uses a novel online incremental streamed distributed training algorithm (quite a mouthful!),\n",
    "  which I published in [5]_. `gensim` also executes a stochastic multi-pass algorithm\n",
    "  from Halko et al. [4]_ internally, to accelerate in-core part\n",
    "  of the computations.\n",
    "  See also `wiki` for further speed-ups by distributing the computation across\n",
    "  a cluster of computers.\n",
    "\n",
    "* `Random Projections, RP <http://www.cis.hut.fi/ella/publications/randproj_kdd.pdf>`_ aim to\n",
    "  reduce vector space dimensionality. This is a very efficient (both memory- and\n",
    "  CPU-friendly) approach to approximating TfIdf distances between documents, by throwing in a little randomness.\n",
    "  Recommended target dimensionality is again in the hundreds/thousands, depending on your dataset.\n",
    "\n",
    "  .. sourcecode:: pycon\n",
    "\n",
    "    model = models.RpModel(tfidf_corpus, num_topics=500)\n",
    "\n",
    "* `Latent Dirichlet Allocation, LDA <http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation>`_\n",
    "  is yet another transformation from bag-of-words counts into a topic space of lower\n",
    "  dimensionality. LDA is a probabilistic extension of LSA (also called multinomial PCA),\n",
    "  so LDA's topics can be interpreted as probability distributions over words. These distributions are,\n",
    "  just like with LSA, inferred automatically from a training corpus. Documents\n",
    "  are in turn interpreted as a (soft) mixture of these topics (again, just like with LSA).\n",
    "\n",
    "  .. sourcecode:: pycon\n",
    "\n",
    "    model = models.LdaModel(corpus, id2word=dictionary, num_topics=100)\n",
    "\n",
    "  `gensim` uses a fast implementation of online LDA parameter estimation based on [2]_,\n",
    "  modified to run in `distributed mode <distributed>` on a cluster of computers.\n",
    "\n",
    "* `Hierarchical Dirichlet Process, HDP <http://jmlr.csail.mit.edu/proceedings/papers/v15/wang11a/wang11a.pdf>`_\n",
    "  is a non-parametric bayesian method (note the missing number of requested topics):\n",
    "\n",
    "  .. sourcecode:: pycon\n",
    "\n",
    "    model = models.HdpModel(corpus, id2word=dictionary)\n",
    "\n",
    "  `gensim` uses a fast, online implementation based on [3]_.\n",
    "  The HDP model is a new addition to `gensim`, and still rough around its academic edges -- use with care.\n",
    "\n",
    "Adding new :abbr:`VSM (Vector Space Model)` transformations (such as different weighting schemes) is rather trivial;\n",
    "see the `apiref` or directly the `Python code <https://github.com/piskvorky/gensim/blob/develop/gensim/models/tfidfmodel.py>`_\n",
    "for more info and examples.\n",
    "\n",
    "It is worth repeating that these are all unique, **incremental** implementations,\n",
    "which do not require the whole training corpus to be present in main memory all at once.\n",
    "With memory taken care of, I am now improving `distributed`,\n",
    "to improve CPU efficiency, too.\n",
    "If you feel you could contribute by testing, providing use-cases or code, see the `Gensim Developer guide <https://github.com/RaRe-Technologies/gensim/wiki/Developer-page>`__.\n",
    "\n",
    "## What Next?\n",
    "\n",
    "Continue on to the next tutorial on `sphx_glr_auto_examples_core_run_similarity_queries.py`.\n",
    "\n",
    "## References\n",
    "\n",
    ".. [1] Bradford. 2008. An empirical study of required dimensionality for large-scale latent semantic indexing applications.\n",
    "\n",
    ".. [2] Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation.\n",
    "\n",
    ".. [3] Wang, Paisley, Blei. 2011. Online variational inference for the hierarchical Dirichlet process.\n",
    "\n",
    ".. [4] Halko, Martinsson, Tropp. 2009. Finding structure with randomness.\n",
    "\n",
    ".. [5] Řehůřek. 2011. Subspace tracking for Latent Semantic Analysis.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:34:05.601398600Z",
     "start_time": "2024-05-07T05:34:03.651431100Z"
    }
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'run_topics_and_transformations.png'",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mFileNotFoundError\u001B[0m                         Traceback (most recent call last)",
      "\u001B[1;32m~\\AppData\\Local\\Temp\\ipykernel_7320\\2613515143.py\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[0;32m      1\u001B[0m \u001B[1;32mimport\u001B[0m \u001B[0mmatplotlib\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mpyplot\u001B[0m \u001B[1;32mas\u001B[0m \u001B[0mplt\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      2\u001B[0m \u001B[1;32mimport\u001B[0m \u001B[0mmatplotlib\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mimage\u001B[0m \u001B[1;32mas\u001B[0m \u001B[0mmpimg\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m----> 3\u001B[1;33m \u001B[0mimg\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mmpimg\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mimread\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;34m'run_topics_and_transformations.png'\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m      4\u001B[0m \u001B[0mimgplot\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mplt\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mimshow\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mimg\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      5\u001B[0m \u001B[0m_\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mplt\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0maxis\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;34m'off'\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;32m~\\anaconda3\\lib\\site-packages\\matplotlib\\image.py\u001B[0m in \u001B[0;36mimread\u001B[1;34m(fname, format)\u001B[0m\n\u001B[0;32m   1558\u001B[0m                     \u001B[0mresponse\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mio\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mBytesIO\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mresponse\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mread\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m   1559\u001B[0m                 \u001B[1;32mreturn\u001B[0m \u001B[0mimread\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mresponse\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mformat\u001B[0m\u001B[1;33m=\u001B[0m\u001B[0mext\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m-> 1560\u001B[1;33m     \u001B[1;32mwith\u001B[0m \u001B[0mimg_open\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mfname\u001B[0m\u001B[1;33m)\u001B[0m \u001B[1;32mas\u001B[0m \u001B[0mimage\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m   1561\u001B[0m         return (_pil_png_to_float_array(image)\n\u001B[0;32m   1562\u001B[0m                 \u001B[1;32mif\u001B[0m \u001B[0misinstance\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mimage\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mPIL\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mPngImagePlugin\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mPngImageFile\u001B[0m\u001B[1;33m)\u001B[0m \u001B[1;32melse\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;32m~\\anaconda3\\lib\\site-packages\\PIL\\ImageFile.py\u001B[0m in \u001B[0;36m__init__\u001B[1;34m(self, fp, filename)\u001B[0m\n\u001B[0;32m    102\u001B[0m         \u001B[1;32mif\u001B[0m \u001B[0mis_path\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mfp\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m    103\u001B[0m             \u001B[1;31m# filename\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m--> 104\u001B[1;33m             \u001B[0mself\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mfp\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mopen\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mfp\u001B[0m\u001B[1;33m,\u001B[0m \u001B[1;34m\"rb\"\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m    105\u001B[0m             \u001B[0mself\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mfilename\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mfp\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m    106\u001B[0m             \u001B[0mself\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0m_exclusive_fp\u001B[0m \u001B[1;33m=\u001B[0m \u001B[1;32mTrue\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;31mFileNotFoundError\u001B[0m: [Errno 2] No such file or directory: 'run_topics_and_transformations.png'"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "img = mpimg.imread('run_topics_and_transformations.png')\n",
    "imgplot = plt.imshow(img)\n",
    "_ = plt.axis('off')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}