run_similarity_queries.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Similarity Queries\n",
    "==================\n",
    "\n",
    "Demonstrates querying a corpus for similar documents.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:37:48.781078100Z",
     "start_time": "2024-05-07T05:37:48.770001500Z"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the Corpus\n",
    "-------------------\n",
    "\n",
    "First, we need to create a corpus to work with.\n",
    "This step is the same as in the previous tutorial;\n",
    "if you completed it, feel free to skip to the next section.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:37:50.989402100Z",
     "start_time": "2024-05-07T05:37:49.545852500Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:37:50,973 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2024-05-07 13:37:50,974 : INFO : built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)\n",
      "2024-05-07 13:37:50,975 : INFO : Dictionary lifecycle event {'msg': \"built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)\", 'datetime': '2024-05-07T13:37:50.975497', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'created'}\n"
     ]
    }
   ],
   "source": [
    "from collections import defaultdict\n",
    "from gensim import corpora\n",
    "\n",
    "documents = [\n",
    "    \"Human machine interface for lab abc computer applications\",\n",
    "    \"A survey of user opinion of computer system response time\",\n",
    "    \"The EPS user interface management system\",\n",
    "    \"System and human system engineering testing of EPS\",\n",
    "    \"Relation of user perceived response time to error measurement\",\n",
    "    \"The generation of random binary unordered trees\",\n",
    "    \"The intersection graph of paths in trees\",\n",
    "    \"Graph minors IV Widths of trees and well quasi ordering\",\n",
    "    \"Graph minors A survey\",\n",
    "]\n",
    "\n",
    "# remove common words and tokenize\n",
    "stoplist = set('for a of the and to in'.split())\n",
    "texts = [\n",
    "    [word for word in document.lower().split() if word not in stoplist]\n",
    "    for document in documents\n",
    "]\n",
    "\n",
    "# remove words that appear only once\n",
    "frequency = defaultdict(int)\n",
    "for text in texts:\n",
    "    for token in text:\n",
    "        frequency[token] += 1\n",
    "\n",
    "texts = [\n",
    "    [token for token in text if frequency[token] > 1]\n",
    "    for text in texts\n",
    "]\n",
    "\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarity interface\n",
    "--------------------\n",
    "\n",
    "In the previous tutorials on\n",
    "`sphx_glr_auto_examples_core_run_corpora_and_vector_spaces.py`\n",
    "and\n",
    "`sphx_glr_auto_examples_core_run_topics_and_transformations.py`,\n",
    "we covered what it means to create a corpus in the Vector Space Model and how\n",
    "to transform it between different vector spaces. A common reason for such a\n",
    "charade is that we want to determine **similarity between pairs of\n",
    "documents**, or the **similarity between a specific document and a set of\n",
    "other documents** (such as a user query vs. indexed documents).\n",
    "\n",
    "To show how this can be done in gensim, let us consider the same corpus as in the\n",
    "previous examples (which really originally comes from Deerwester et al.'s\n",
    "`\"Indexing by Latent Semantic Analysis\" <http://www.cs.bham.ac.uk/~pxt/IDA/lsa_ind.pdf>`_\n",
    "seminal 1990 article).\n",
    "To follow Deerwester's example, we first use this tiny corpus to define a 2-dimensional\n",
    "LSI space:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:38:14.614692500Z",
     "start_time": "2024-05-07T05:38:14.517320300Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:38:14,511 : INFO : using serial LSI version on this node\n",
      "2024-05-07 13:38:14,514 : INFO : updating model with new documents\n",
      "2024-05-07 13:38:14,516 : INFO : preparing a new chunk of documents\n",
      "2024-05-07 13:38:14,518 : INFO : using 100 extra samples and 2 power iterations\n",
      "2024-05-07 13:38:14,518 : INFO : 1st phase: constructing (12, 102) action matrix\n",
      "2024-05-07 13:38:14,519 : INFO : orthonormalizing (12, 102) action matrix\n",
      "2024-05-07 13:38:14,540 : INFO : 2nd phase: running dense svd on (12, 9) matrix\n",
      "2024-05-07 13:38:14,542 : INFO : computing the final decomposition\n",
      "2024-05-07 13:38:14,544 : INFO : keeping 2 factors (discarding 43.156% of energy spectrum)\n",
      "2024-05-07 13:38:14,546 : INFO : processed documents up to #9\n",
      "2024-05-07 13:38:14,547 : INFO : topic #0(3.341): 0.644*\"system\" + 0.404*\"user\" + 0.301*\"eps\" + 0.265*\"time\" + 0.265*\"response\" + 0.240*\"computer\" + 0.221*\"human\" + 0.206*\"survey\" + 0.198*\"interface\" + 0.036*\"graph\"\n",
      "2024-05-07 13:38:14,548 : INFO : topic #1(2.542): 0.623*\"graph\" + 0.490*\"trees\" + 0.451*\"minors\" + 0.274*\"survey\" + -0.167*\"system\" + -0.141*\"eps\" + -0.113*\"human\" + 0.107*\"time\" + 0.107*\"response\" + -0.072*\"interface\"\n",
      "2024-05-07 13:38:14,549 : INFO : LsiModel lifecycle event {'msg': 'trained LsiModel(num_terms=12, num_topics=2, decay=1.0, chunksize=20000) in 0.03s', 'datetime': '2024-05-07T13:38:14.549657', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'created'}\n"
     ]
    }
   ],
   "source": [
    "from gensim import models\n",
    "lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purposes of this tutorial, there are only two things you need to know about LSI.\n",
    "First, it's just another transformation: it transforms vectors from one space to another.\n",
    "Second, the benefit of LSI is that enables identifying patterns and relationships between terms (in our case, words in a document) and topics.\n",
    "Our LSI space is two-dimensional (`num_topics = 2`) so there are two topics, but this is arbitrary.\n",
    "If you're interested, you can read more about LSI here: `Latent Semantic Indexing <https://en.wikipedia.org/wiki/Latent_semantic_indexing>`_:\n",
    "\n",
    "Now suppose a user typed in the query `\"Human computer interaction\"`. We would\n",
    "like to sort our nine corpus documents in decreasing order of relevance to this query.\n",
    "Unlike modern search engines, here we only concentrate on a single aspect of possible\n",
    "similarities---on apparent semantic relatedness of their texts (words). No hyperlinks,\n",
    "no random-walk static ranks, just a semantic extension over the boolean keyword match:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:38:43.451385300Z",
     "start_time": "2024-05-07T05:38:43.381305800Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.46182100453271563), (1, -0.07002766527900038)]\n"
     ]
    }
   ],
   "source": [
    "doc = \"Human computer interaction\"\n",
    "vec_bow = dictionary.doc2bow(doc.lower().split())\n",
    "vec_lsi = lsi[vec_bow]  # convert the query to LSI space\n",
    "print(vec_lsi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, we will be considering `cosine similarity <http://en.wikipedia.org/wiki/Cosine_similarity>`_\n",
    "to determine the similarity of two vectors. Cosine similarity is a standard measure\n",
    "in Vector Space Modeling, but wherever the vectors represent probability distributions,\n",
    "`different similarity measures <http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence>`_\n",
    "may be more appropriate.\n",
    "\n",
    "Initializing query structures\n",
    "++++++++++++++++++++++++++++++++\n",
    "\n",
    "To prepare for similarity queries, we need to enter all documents which we want\n",
    "to compare against subsequent queries. In our case, they are the same nine documents\n",
    "used for training LSI, converted to 2-D LSA space. But that's only incidental, we\n",
    "might also be indexing a different corpus altogether.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:45:07.094607900Z",
     "start_time": "2024-05-07T05:45:06.835936400Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:45:06,833 : WARNING : scanning corpus to determine the number of features (consider setting `num_features` explicitly)\n",
      "2024-05-07 13:45:06,842 : INFO : creating matrix with 9 documents and 2 features\n"
     ]
    }
   ],
   "source": [
    "from gensim import similarities\n",
    "index = similarities.MatrixSimilarity(lsi[corpus])  # transform corpus to LSI space and index it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger\"><h4>Warning</h4><p>The class :class:`similarities.MatrixSimilarity` is only appropriate when the whole\n",
    "  set of vectors fits into memory. For example, a corpus of one million documents\n",
    "  would require 2GB of RAM in a 256-dimensional LSI space, when used with this class.\n",
    "\n",
    "  Without 2GB of free RAM, you would need to use the :class:`similarities.Similarity` class.\n",
    "  This class operates in fixed memory, by splitting the index across multiple files on disk, called shards.\n",
    "  It uses :class:`similarities.MatrixSimilarity` and :class:`similarities.SparseMatrixSimilarity` internally,\n",
    "  so it is still fast, although slightly more complex.</p></div>\n",
    "\n",
    "Index persistency is handled via the standard :func:`save` and :func:`load` functions:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:46:30.701400500Z",
     "start_time": "2024-05-07T05:46:30.590269400Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-07 13:46:30,585 : INFO : MatrixSimilarity lifecycle event {'fname_or_handle': '/tmp/deerwester.index', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2024-05-07T13:46:30.585450', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'saving'}\n",
      "2024-05-07 13:46:30,588 : INFO : saved /tmp/deerwester.index\n",
      "2024-05-07 13:46:30,589 : INFO : loading MatrixSimilarity object from /tmp/deerwester.index\n",
      "2024-05-07 13:46:30,592 : INFO : MatrixSimilarity lifecycle event {'fname': '/tmp/deerwester.index', 'datetime': '2024-05-07T13:46:30.592262', 'gensim': '4.1.2', 'python': '3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]', 'platform': 'Windows-10-10.0.19044-SP0', 'event': 'loaded'}\n"
     ]
    }
   ],
   "source": [
    "index.save('/tmp/deerwester.index')\n",
    "index = similarities.MatrixSimilarity.load('/tmp/deerwester.index')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is true for all similarity indexing classes (:class:`similarities.Similarity`,\n",
    ":class:`similarities.MatrixSimilarity` and :class:`similarities.SparseMatrixSimilarity`).\n",
    "Also in the following, `index` can be an object of any of these. When in doubt,\n",
    "use :class:`similarities.Similarity`, as it is the most scalable version, and it also\n",
    "supports adding more documents to the index later.\n",
    "\n",
    "Performing queries\n",
    "++++++++++++++++++\n",
    "\n",
    "To obtain similarities of our query document against the nine indexed documents:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:46:47.235951700Z",
     "start_time": "2024-05-07T05:46:47.162391600Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.998093), (1, 0.93748635), (2, 0.9984453), (3, 0.9865886), (4, 0.90755945), (5, -0.12416792), (6, -0.10639259), (7, -0.09879464), (8, 0.050041765)]\n"
     ]
    }
   ],
   "source": [
    "sims = index[vec_lsi]  # perform a similarity query against the corpus\n",
    "print(list(enumerate(sims)))  # print (document_number, document_similarity) 2-tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cosine measure returns similarities in the range `<-1, 1>` (the greater, the more similar),\n",
    "so that the first document has a score of 0.99809301 etc.\n",
    "\n",
    "With some standard Python magic we sort these similarities into descending\n",
    "order, and obtain the final answer to the query `\"Human computer interaction\"`:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-07T05:47:23.425015800Z",
     "start_time": "2024-05-07T05:47:23.229410100Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9984453 The EPS user interface management system\n",
      "0.998093 Human machine interface for lab abc computer applications\n",
      "0.9865886 System and human system engineering testing of EPS\n",
      "0.93748635 A survey of user opinion of computer system response time\n",
      "0.90755945 Relation of user perceived response time to error measurement\n",
      "0.050041765 Graph minors A survey\n",
      "-0.09879464 Graph minors IV Widths of trees and well quasi ordering\n",
      "-0.10639259 The intersection graph of paths in trees\n",
      "-0.12416792 The generation of random binary unordered trees\n"
     ]
    }
   ],
   "source": [
    "sims = sorted(enumerate(sims), key=lambda item: -item[1])\n",
    "for doc_position, doc_score in sims:\n",
    "    print(doc_score, documents[doc_position])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The thing to note here is that documents no. 2 (``\"The EPS user interface management system\"``)\n",
    "and 4 (``\"Relation of user perceived response time to error measurement\"``) would never be returned by\n",
    "a standard boolean fulltext search, because they do not share any common words with ``\"Human\n",
    "computer interaction\"``. However, after applying LSI, we can observe that both of\n",
    "them received quite high similarity scores (no. 2 is actually the most similar!),\n",
    "which corresponds better to our intuition of\n",
    "them sharing a \"computer-human\" related topic with the query. In fact, this semantic\n",
    "generalization is the reason why we apply transformations and do topic modelling\n",
    "in the first place.\n",
    "\n",
    "Where next?\n",
    "------------\n",
    "\n",
    "Congratulations, you have finished the tutorials -- now you know how gensim works :-)\n",
    "To delve into more details, you can browse through the `apiref`,\n",
    "see the `wiki` or perhaps check out `distributed` in `gensim`.\n",
    "\n",
    "Gensim is a fairly mature package that has been used successfully by many individuals and companies, both for rapid prototyping and in production.\n",
    "That doesn't mean it's perfect though:\n",
    "\n",
    "* there are parts that could be implemented more efficiently (in C, for example), or make better use of parallelism (multiple machines cores)\n",
    "* new algorithms are published all the time; help gensim keep up by `discussing them <http://groups.google.com/group/gensim>`_ and `contributing code <https://github.com/piskvorky/gensim/wiki/Developer-page>`_\n",
    "* your **feedback is most welcome** and appreciated (and it's not just the code!):\n",
    "  `bug reports <https://github.com/piskvorky/gensim/issues>`_ or\n",
    "  `user stories and general questions <http://groups.google.com/group/gensim/topics>`_.\n",
    "\n",
    "Gensim has no ambition to become an all-encompassing framework, across all NLP (or even Machine Learning) subfields.\n",
    "Its mission is to help NLP practitioners try out popular topic modelling algorithms\n",
    "on large datasets easily, and to facilitate prototyping of new algorithms for researchers.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "img = mpimg.imread('run_similarity_queries.png')\n",
    "imgplot = plt.imshow(img)\n",
    "_ = plt.axis('off')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}