run_corpora_and_vector_spaces.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T04:46:37.495563900Z",
     "start_time": "2024-05-07T04:46:25.332700700Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Corpora and Vector Spaces\n",
    "\n",
    "Demonstrates transforming text into a vector space representation.\n",
    "\n",
    "Also introduces corpus streaming and persistence to disk in various formats.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T04:52:34.904757400Z",
     "start_time": "2024-05-07T04:52:34.884780100Z"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let’s create a small corpus of nine short documents [1]_:\n",
    "\n",
    "\n",
    "## From Strings to Vectors\n",
    "\n",
    "This time, let's start from documents represented as strings:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T04:54:30.306647300Z",
     "start_time": "2024-05-07T04:54:30.067800200Z"
    }
   },
   "outputs": [],
   "source": [
    "documents = [\n",
    "    \"Human machine interface for lab abc computer applications\",\n",
    "    \"A survey of user opinion of computer system response time\",\n",
    "    \"The EPS user interface management system\",\n",
    "    \"System and human system engineering testing of EPS\",\n",
    "    \"Relation of user perceived response time to error measurement\",\n",
    "    \"The generation of random binary unordered trees\",\n",
    "    \"The intersection graph of paths in trees\",\n",
    "    \"Graph minors IV Widths of trees and well quasi ordering\",\n",
    "    \"Graph minors A survey\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tiny corpus of nine documents, each consisting of only a single sentence.\n",
    "\n",
    "First, let's tokenize the documents, remove common words (using a toy stoplist)\n",
    "as well as words that only appear once in the corpus:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T04:54:33.159435900Z",
     "start_time": "2024-05-07T04:54:33.106611800Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['human', 'interface', 'computer'],\n",
      " ['survey', 'user', 'computer', 'system', 'response', 'time'],\n",
      " ['eps', 'user', 'interface', 'system'],\n",
      " ['system', 'human', 'system', 'eps'],\n",
      " ['user', 'response', 'time'],\n",
      " ['trees'],\n",
      " ['graph', 'trees'],\n",
      " ['graph', 'minors', 'trees'],\n",
      " ['graph', 'minors', 'survey']]\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint  # pretty-printer\n",
    "from collections import defaultdict\n",
    "\n",
    "# remove common words and tokenize\n",
    "stoplist = set('for a of the and to in'.split())\n",
    "texts = [\n",
    "    [word for word in document.lower().split() if word not in stoplist]\n",
    "    for document in documents\n",
    "]\n",
    "\n",
    "# remove words that appear only once\n",
    "frequency = defaultdict(int)\n",
    "for text in texts:\n",
    "    for token in text:\n",
    "        frequency[token] += 1\n",
    "\n",
    "texts = [\n",
    "    [token for token in text if frequency[token] > 1]\n",
    "    for text in texts\n",
    "]\n",
    "\n",
    "pprint(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your way of processing the documents will likely vary; here, I only split on whitespace\n",
    "to tokenize, followed by lowercasing each word. In fact, I use this particular\n",
    "(simplistic and inefficient) setup to mimic the experiment done in Deerwester et al.'s\n",
    "original LSA article [1]_.\n",
    "\n",
    "The ways to process documents are so varied and application- and language-dependent that I\n",
    "decided to *not* constrain them by any interface. Instead, a document is represented\n",
    "by the features extracted from it, not by its \"surface\" string form: how you get to\n",
    "the features is up to you. Below I describe one common, general-purpose approach (called\n",
    ":dfn:`bag-of-words`), but keep in mind that different application domains call for\n",
    "different features, and, as always, it's `garbage in, garbage out <http://en.wikipedia.org/wiki/Garbage_In,_Garbage_Out>`_...\n",
    "\n",
    "To convert documents to vectors, we'll use a document representation called\n",
    "`bag-of-words <http://en.wikipedia.org/wiki/Bag_of_words>`_. In this representation,\n",
    "each document is represented by one vector where each vector element represents\n",
    "a question-answer pair, in the style of:\n",
    "\n",
    "- Question: How many times does the word `system` appear in the document?\n",
    "- Answer: Once.\n",
    "\n",
    "It is advantageous to represent the questions only by their (integer) ids. The mapping\n",
    "between the questions and ids is called a dictionary:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T04:54:43.036858500Z",
     "start_time": "2024-05-07T04:54:39.795324100Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 12:54:42,894 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2024-05-07 12:54:42,895 : INFO : built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)\n",
      "2024-05-07 12:54:42,895 : INFO : Dictionary lifecycle event {'msg': \"built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)\", 'datetime': '2024-05-07T12:54:42.895954', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'created'}\n",
      "2024-05-07 12:54:42,895 : INFO : Dictionary lifecycle event {'fname_or_handle': '/tmp/deerwester.dict', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2024-05-07T12:54:42.895954', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'saving'}\n",
      "2024-05-07 12:54:42,897 : INFO : saved /tmp/deerwester.dict\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...)\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "dictionary.save('/tmp/deerwester.dict')  # store the dictionary, for future reference\n",
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we assigned a unique integer id to all words appearing in the corpus with the\n",
    ":class:`gensim.corpora.dictionary.Dictionary` class. This sweeps across the texts, collecting word counts\n",
    "and relevant statistics. In the end, we see there are twelve distinct words in the\n",
    "processed corpus, which means each document will be represented by twelve numbers (ie., by a 12-D vector).\n",
    "To see the mapping between words and their ids:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:03:46.297456Z",
     "start_time": "2024-05-07T05:03:46.015275700Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'computer': 0, 'human': 1, 'interface': 2, 'response': 3, 'survey': 4, 'system': 5, 'time': 6, 'user': 7, 'eps': 8, 'trees': 9, 'graph': 10, 'minors': 11}\n"
     ]
    }
   ],
   "source": [
    "print(dictionary.token2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To actually convert tokenized documents to vectors:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:06:23.478424900Z",
     "start_time": "2024-05-07T05:06:23.425978500Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 1), (1, 1)]\n"
     ]
    }
   ],
   "source": [
    "new_doc = \"Human computer interaction\"\n",
    "new_vec = dictionary.doc2bow(new_doc.lower().split())\n",
    "print(new_vec)  # the word \"interaction\" does not appear in the dictionary and is ignored"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function :func:`doc2bow` simply counts the number of occurrences of\n",
    "each distinct word, converts the word to its integer word id\n",
    "and returns the result as a sparse vector. The sparse vector ``[(0, 1), (1, 1)]``\n",
    "therefore reads: in the document `\"Human computer interaction\"`, the words `computer`\n",
    "(id 0) and `human` (id 1) appear once; the other ten dictionary words appear (implicitly) zero times.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:06:26.915579700Z",
     "start_time": "2024-05-07T05:06:26.809420100Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:06:26,800 : INFO : storing corpus in Matrix Market format to /tmp/deerwester.mm\n",
      "2024-05-07 13:06:26,807 : INFO : saving sparse matrix to /tmp/deerwester.mm\n",
      "2024-05-07 13:06:26,808 : INFO : PROGRESS: saving document #0\n",
      "2024-05-07 13:06:26,811 : INFO : saved 9x12 matrix, density=25.926% (28/108)\n",
      "2024-05-07 13:06:26,813 : INFO : saving MmCorpus index to /tmp/deerwester.mm.index\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[(0, 1), (1, 1), (2, 1)], [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)], [(2, 1), (5, 1), (7, 1), (8, 1)], [(1, 1), (5, 2), (8, 1)], [(3, 1), (6, 1), (7, 1)], [(9, 1)], [(9, 1), (10, 1)], [(9, 1), (10, 1), (11, 1)], [(4, 1), (10, 1), (11, 1)]]\n"
     ]
    }
   ],
   "source": [
    "corpus = [dictionary.doc2bow(text) for text in texts]\n",
    "corpora.MmCorpus.serialize('/tmp/deerwester.mm', corpus)  # store to disk, for later use\n",
    "print(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By now it should be clear that the vector feature with ``id=10`` stands for the question \"How many\n",
    "times does the word `graph` appear in the document?\" and that the answer is \"zero\" for\n",
    "the first six documents and \"one\" for the remaining three.\n",
    "\n",
    "\n",
    "## Corpus Streaming -- One Document at a Time\n",
    "\n",
    "Note that `corpus` above resides fully in memory, as a plain Python list.\n",
    "In this simple example, it doesn't matter much, but just to make things clear,\n",
    "let's assume there are millions of documents in the corpus. Storing all of them in RAM won't do.\n",
    "Instead, let's assume the documents are stored in a file on disk, one document per line. Gensim\n",
    "only requires that a corpus must be able to return one document vector at a time:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:07:15.217080500Z",
     "start_time": "2024-05-07T05:07:15.177188500Z"
    }
   },
   "outputs": [],
   "source": [
    "from smart_open import open  # for transparently opening remote files\n",
    "\n",
    "\n",
    "class MyCorpus:\n",
    "    def __iter__(self):\n",
    "        for line in open('https://radimrehurek.com/mycorpus.txt'):\n",
    "            # assume there's one document per line, tokens separated by whitespace\n",
    "            yield dictionary.doc2bow(line.lower().split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full power of Gensim comes from the fact that a corpus doesn't have to be\n",
    "a ``list``, or a ``NumPy`` array, or a ``Pandas`` dataframe, or whatever.\n",
    "Gensim *accepts any object that, when iterated over, successively yields\n",
    "documents*.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This flexibility allows you to create your own corpus classes that stream the\n",
    "# documents directly from disk, network, database, dataframes... The models\n",
    "# in Gensim are implemented such that they don't require all vectors to reside\n",
    "# in RAM at once. You can even create the documents on the fly!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the sample `mycorpus.txt file here <https://radimrehurek.com/mycorpus.txt>`_. The assumption that\n",
    "each document occupies one line in a single file is not important; you can mold\n",
    "the `__iter__` function to fit your input format, whatever it is.\n",
    "Walking directories, parsing XML, accessing the network...\n",
    "Just parse your input to retrieve a clean list of tokens in each document,\n",
    "then convert the tokens via a dictionary to their ids and yield the resulting sparse vector inside `__iter__`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:09:16.384296300Z",
     "start_time": "2024-05-07T05:09:16.343899400Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<__main__.MyCorpus object at 0x000001F1170BC610>\n"
     ]
    }
   ],
   "source": [
    "corpus_memory_friendly = MyCorpus()  # doesn't load the corpus into memory!\n",
    "print(corpus_memory_friendly)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corpus is now an object. We didn't define any way to print it, so `print` just outputs address\n",
    "of the object in memory. Not very useful. To see the constituent vectors, let's\n",
    "iterate over the corpus and print each document vector (one at a time):\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:10:11.173311900Z",
     "start_time": "2024-05-07T05:10:09.968688600Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 1), (1, 1), (2, 1)]\n",
      "[(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)]\n",
      "[(2, 1), (5, 1), (7, 1), (8, 1)]\n",
      "[(1, 1), (5, 2), (8, 1)]\n",
      "[(3, 1), (6, 1), (7, 1)]\n",
      "[(9, 1)]\n",
      "[(9, 1), (10, 1)]\n",
      "[(9, 1), (10, 1), (11, 1)]\n",
      "[(4, 1), (10, 1), (11, 1)]\n"
     ]
    }
   ],
   "source": [
    "for vector in corpus_memory_friendly:  # load one vector into memory at a time\n",
    "    print(vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the output is the same as for the plain Python list, the corpus is now much\n",
    "more memory friendly, because at most one vector resides in RAM at a time. Your\n",
    "corpus can now be as large as you want.\n",
    "\n",
    "Similarly, to construct the dictionary without loading all texts into memory:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# collect statistics about all tokens\n",
    "dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/mycorpus.txt'))\n",
    "# remove stop words and words that appear only once\n",
    "stop_ids = [\n",
    "    dictionary.token2id[stopword]\n",
    "    for stopword in stoplist\n",
    "    if stopword in dictionary.token2id\n",
    "]\n",
    "once_ids = [tokenid for tokenid, docfreq in dictionary.dfs.items() if docfreq == 1]\n",
    "dictionary.filter_tokens(stop_ids + once_ids)  # remove stop words and words that appear only once\n",
    "dictionary.compactify()  # remove gaps in id sequence after words that were removed\n",
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that is all there is to it! At least as far as bag-of-words representation is concerned.\n",
    "Of course, what we do with such a corpus is another question; it is not at all clear\n",
    "how counting the frequency of distinct words could be useful. As it turns out, it isn't, and\n",
    "we will need to apply a transformation on this simple representation first, before\n",
    "we can use it to compute any meaningful document vs. document similarities.\n",
    "Transformations are covered in the next tutorial\n",
    "(`sphx_glr_auto_examples_core_run_topics_and_transformations.py`),\n",
    "but before that, let's briefly turn our attention to *corpus persistency*.\n",
    "\n",
    "\n",
    "## Corpus Formats\n",
    "\n",
    "There exist several file formats for serializing a Vector Space corpus (~sequence of vectors) to disk.\n",
    "`Gensim` implements them via the *streaming corpus interface* mentioned earlier:\n",
    "documents are read from (resp. stored to) disk in a lazy fashion, one document at\n",
    "a time, without the whole corpus being read into main memory at once.\n",
    "\n",
    "One of the more notable file formats is the `Market Matrix format <http://math.nist.gov/MatrixMarket/formats.html>`_.\n",
    "To save a corpus in the Matrix Market format:\n",
    "\n",
    "create a toy corpus of 2 documents, as a plain Python list\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:18:54.044911700Z",
     "start_time": "2024-05-07T05:18:53.944444200Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:18:53,935 : INFO : storing corpus in Matrix Market format to /tmp/corpus.mm\n",
      "2024-05-07 13:18:53,939 : INFO : saving sparse matrix to /tmp/corpus.mm\n",
      "2024-05-07 13:18:53,942 : INFO : PROGRESS: saving document #0\n",
      "2024-05-07 13:18:53,946 : INFO : saved 2x2 matrix, density=25.000% (1/4)\n",
      "2024-05-07 13:18:53,949 : INFO : saving MmCorpus index to /tmp/corpus.mm.index\n"
     ]
    }
   ],
   "source": [
    "corpus = [[(1, 0.5)], []]  # make one document empty, for the heck of it\n",
    "\n",
    "corpora.MmCorpus.serialize('/tmp/corpus.mm', corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other formats include `Joachim's SVMlight format <http://svmlight.joachims.org/>`_,\n",
    "`Blei's LDA-C format <https://github.com/blei-lab/lda-c>`_ and\n",
    "`GibbsLDA++ format <http://gibbslda.sourceforge.net/>`_.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:19:21.430018700Z",
     "start_time": "2024-05-07T05:19:21.302916900Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:19:21,299 : INFO : converting corpus to SVMlight format: /tmp/corpus.svmlight\n",
      "2024-05-07 13:19:21,302 : INFO : saving SvmLightCorpus index to /tmp/corpus.svmlight.index\n",
      "2024-05-07 13:19:21,306 : INFO : no word id mapping provided; initializing from corpus\n",
      "2024-05-07 13:19:21,307 : INFO : storing corpus in Blei's LDA-C format into /tmp/corpus.lda-c\n",
      "2024-05-07 13:19:21,309 : INFO : saving vocabulary of 2 words to /tmp/corpus.lda-c.vocab\n",
      "2024-05-07 13:19:21,311 : INFO : saving BleiCorpus index to /tmp/corpus.lda-c.index\n",
      "2024-05-07 13:19:21,317 : INFO : no word id mapping provided; initializing from corpus\n",
      "2024-05-07 13:19:21,320 : INFO : storing corpus in List-Of-Words format into /tmp/corpus.low\n",
      "2024-05-07 13:19:21,331 : WARNING : List-of-words format can only save vectors with integer elements; 1 float entries were truncated to integer value\n",
      "2024-05-07 13:19:21,332 : INFO : saving LowCorpus index to /tmp/corpus.low.index\n"
     ]
    }
   ],
   "source": [
    "corpora.SvmLightCorpus.serialize('/tmp/corpus.svmlight', corpus)\n",
    "corpora.BleiCorpus.serialize('/tmp/corpus.lda-c', corpus)\n",
    "corpora.LowCorpus.serialize('/tmp/corpus.low', corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conversely, to load a corpus iterator from a Matrix Market file:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:19:28.661258300Z",
     "start_time": "2024-05-07T05:19:28.580242500Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:19:28,579 : INFO : loaded corpus index from /tmp/corpus.mm.index\n",
      "2024-05-07 13:19:28,580 : INFO : initializing cython corpus reader from /tmp/corpus.mm\n",
      "2024-05-07 13:19:28,583 : INFO : accepted corpus with 2 documents, 2 features, 1 non-zero entries\n"
     ]
    }
   ],
   "source": [
    "corpus = corpora.MmCorpus('/tmp/corpus.mm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corpus objects are streams, so typically you won't be able to print them directly:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:20:22.535258600Z",
     "start_time": "2024-05-07T05:20:22.508304200Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MmCorpus(2 documents, 2 features, 1 non-zero entries)\n"
     ]
    }
   ],
   "source": [
    "print(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, to view the contents of a corpus:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:20:25.567371300Z",
     "start_time": "2024-05-07T05:20:25.395616600Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[(1, 0.5)], []]\n"
     ]
    }
   ],
   "source": [
    "# one way of printing a corpus: load it entirely into memory\n",
    "print(list(corpus))  # calling list() will convert any sequence to a plain Python list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:20:38.481180Z",
     "start_time": "2024-05-07T05:20:38.231296200Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1, 0.5)]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# another way of doing it: print one document at a time, making use of the streaming interface\n",
    "for doc in corpus:\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second way is obviously more memory-friendly, but for testing and development\n",
    "purposes, nothing beats the simplicity of calling ``list(corpus)``.\n",
    "\n",
    "To save the same Matrix Market document stream in Blei's LDA-C format,\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:20:54.675077300Z",
     "start_time": "2024-05-07T05:20:54.571924Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:20:54,569 : INFO : no word id mapping provided; initializing from corpus\n",
      "2024-05-07 13:20:54,570 : INFO : storing corpus in Blei's LDA-C format into /tmp/corpus.lda-c\n",
      "2024-05-07 13:20:54,575 : INFO : saving vocabulary of 2 words to /tmp/corpus.lda-c.vocab\n",
      "2024-05-07 13:20:54,577 : INFO : saving BleiCorpus index to /tmp/corpus.lda-c.index\n"
     ]
    }
   ],
   "source": [
    "corpora.BleiCorpus.serialize('/tmp/corpus.lda-c', corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this way, `gensim` can also be used as a memory-efficient **I/O format conversion tool**:\n",
    "just load a document stream using one format and immediately save it in another format.\n",
    "Adding new formats is dead easy, check out the `code for the SVMlight corpus\n",
    "<https://github.com/piskvorky/gensim/blob/develop/gensim/corpora/svmlightcorpus.py>`_ for an example.\n",
    "\n",
    "## Compatibility with NumPy and SciPy\n",
    "\n",
    "Gensim also contains `efficient utility functions <http://radimrehurek.com/gensim/matutils.html>`_\n",
    "to help converting from/to numpy matrices\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:21:08.589252400Z",
     "start_time": "2024-05-07T05:21:08.566665800Z"
    }
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "import numpy as np\n",
    "numpy_matrix = np.random.randint(10, size=[5, 2])  # random matrix as an example\n",
    "corpus = gensim.matutils.Dense2Corpus(numpy_matrix)\n",
    "# numpy_matrix = gensim.matutils.corpus2dense(corpus, num_terms=number_of_corpus_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and from/to `scipy.sparse` matrices\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:21:26.269274500Z",
     "start_time": "2024-05-07T05:21:26.246343700Z"
    }
   },
   "outputs": [],
   "source": [
    "import scipy.sparse\n",
    "scipy_sparse_matrix = scipy.sparse.random(5, 2)  # random sparse matrix as example\n",
    "corpus = gensim.matutils.Sparse2Corpus(scipy_sparse_matrix)\n",
    "scipy_csc_matrix = gensim.matutils.corpus2csc(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Next\n",
    "\n",
    "Read about `sphx_glr_auto_examples_core_run_topics_and_transformations.py`.\n",
    "\n",
    "## References\n",
    "\n",
    "For a complete reference (Want to prune the dictionary to a smaller size?\n",
    "Optimize converting between corpora and NumPy/SciPy arrays?), see the `apiref`.\n",
    "\n",
    ".. [1] This is the same corpus as used in\n",
    "       `Deerwester et al. (1990): Indexing by Latent Semantic Analysis <http://www.cs.bham.ac.uk/~pxt/IDA/lsa_ind.pdf>`_, Table 2.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "img = mpimg.imread('run_corpora_and_vector_spaces.png')\n",
    "imgplot = plt.imshow(img)\n",
    "_ = plt.axis('off')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}