nlp-course-lecture03-huawei-noahs-ark-lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Gensim** is billed as a Natural Language Processing package that does ‘Topic Modeling for Humans’. But it is practically much more than that. It is a leading and a state-of-the-art package for processing texts, working with *word vector models* and for building *topic models*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the width and scope of facilities to build and evaluate topic models are unparalleled in gensim, plus many more convenient facilities for text processing.\n",
    "\n",
    "Also, another significant advantage with gensim is: it lets you handle large text files without having to load the entire file in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim import corpora\n",
    "import gensim.downloader as api\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.models.word2vec import Word2Vec\n",
    "from gensim.models.doc2vec import TaggedDocument, Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dictionary and Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dictionary** is an object that maps each word to a unique id.\n",
    "\n",
    "The dictionary object is typically used to create a ‘bag of words’ Corpus. It is this Dictionary and the bag-of-words (Corpus) that are used as inputs to topic modeling and other models that Gensim specializes in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*gensim.utils.simple_preprocess* Convert a document into a list of lowercase tokens, ignoring tokens that are too short or too long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from a list of sentences\n",
    "\n",
    "documents = [\"If you use a car frequently, the first step to cutting\",\n",
    "             \"down your emissions may well be to simply\", \n",
    "             \"fully consider the\", \n",
    "             \"alternatives available to you.\"\n",
    "             ]\n",
    "\n",
    "# Tokenize(split) the sentences into words\n",
    "texts = [[text for text in doc.split()] for doc in documents]\n",
    "\n",
    "# Create dictionary\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "\n",
    "print(dictionary)\n",
    "print(dictionary.token2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With simple_preprocess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary(simple_preprocess(line, deacc=True) for line in documents)\n",
    "print(dictionary)\n",
    "print(dictionary.token2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from document\n",
    "dictionary = corpora.Dictionary(simple_preprocess(line, deacc=True) for line in open('sample.txt', encoding='utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Corpus** object that contains the word id and its frequency in each document.\n",
    "\n",
    "The dictionary object is typically used to create a ‘bag of words’ Corpus. It is this Dictionary and the bag-of-words (Corpus) that are used as inputs to topic modeling and other models that Gensim specializes in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_docs = [\"Who let the dogs out?\",\n",
    "           \"Who? Who? Who? Who?\"]\n",
    "\n",
    "# Tokenize the docs\n",
    "tokenized_list = [simple_preprocess(doc) for doc in my_docs]\n",
    "\n",
    "# Create the Corpus\n",
    "dictionary = corpora.Dictionary()\n",
    "\n",
    "#allow_update=True - add new words to dictionary\n",
    "bow_corpus = [dictionary.doc2bow(doc, allow_update=True) for doc in tokenized_list]\n",
    "print(bow_corpus)\n",
    "\n",
    "print(\"Dictionary: \", dictionary.token2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The (0, 1) in line 1 means, the word with id=0 appears once in the 1st document.\n",
    "Likewise, the (4, 4) in the second list item means the word with id 4 appears 4 times in the second document. And so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- TfIdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim import models\n",
    "import numpy as np\n",
    "\n",
    "documents = [\"This is the first line\",\n",
    "             \"This is the second sentence\",\n",
    "             \"This third document\"]\n",
    "\n",
    "# Create the Dictionary and Corpus\n",
    "mydict = corpora.Dictionary([simple_preprocess(line) for line in documents])\n",
    "corpus = [mydict.doc2bow(simple_preprocess(line)) for line in documents]\n",
    "\n",
    "# Show the Word Weights in Corpus\n",
    "for doc in corpus:\n",
    "    print([[mydict[id], freq] for id, freq in doc])\n",
    "print()\n",
    "\n",
    "# Create the TF-IDF model\n",
    "tfidf = models.TfidfModel(corpus, smartirs='ntc')\n",
    "\n",
    "# Show the TF-IDF weights\n",
    "for doc in tfidf[corpus]:\n",
    "    print([[mydict[id], np.around(freq, decimals=2)] for id, freq in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save and load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the Dict and Corpus\n",
    "dictionary.save('mydict.dict')  # save dict to disk\n",
    "corpora.MmCorpus.serialize('bow_corpus.mm', bow_corpus)  # save corpus to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load them back\n",
    "loaded_dict = corpora.Dictionary.load('mydict.dict')\n",
    "\n",
    "corpus = corpora.MmCorpus('bow_corpus.mm')\n",
    "for line in corpus:\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim provides an inbuilt API to download popular text datasets and word embedding models.\n",
    "\n",
    "A comprehensive list of available datasets and models is maintained [here](https://raw.githubusercontent.com/RaRe-Technologies/gensim-data/master/list.json).\n",
    "\n",
    "Using the API to download the dataset is as simple as calling the ```api.load()``` method with the right data or model name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.downloader as api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Short description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.info('text8')\n",
    "api.info('glove-wiki-gigaword-50')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = api.load(\"text8\")\n",
    "data = [d for d in dataset]\n",
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretrained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w2v_model = api.load(\"glove-wiki-gigaword-50\")\n",
    "w2v_model.most_similar('blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A word embedding model is a model that can provide numerical vectors for a given word. Using the Gensim’s downloader API, you can download pre-built word embedding models like word2vec, fasttext, GloVe and ConceptNet. These are built on large corpuses of commonly occurring text data such as wikipedia, google news etc.\n",
    "\n",
    "The training algorithms in the Gensim package were actually ported from the original Word2Vec implementation by Google and extended with additional functionality.\n",
    "\n",
    "This module implements the word2vec family of algorithms, using *highly optimized* C routines, data streaming and Pythonic interfaces.\n",
    "\n",
    "**Parameters:**\n",
    "\n",
    "- ```sentences``` - (iterable of iterables, optional) – The sentences iterable can be simply a *list of lists of tokens*, but for larger corpora, consider an *iterable* that streams the sentences directly from disk/network.\n",
    "- ```corpus_file``` (str, optional) – Path to a corpus file in LineSentence format. You may use this argument instead of sentences to get performance boost. *Only one of sentences or corpus_file arguments need to be passed* \n",
    "- ```size``` = 100 - Dimensionality of the word vectors.\n",
    "- ```window``` = 5 - Maximum distance between the current and predicted word within a sentence.\n",
    "- ```min_count``` = 5 (int, optional) – Ignores all words with total frequency lower than this.\n",
    "- ```workers``` = 3 (int, optional) – Use these many worker threads to train the model (=faster training with multicore machines).\n",
    "- ```sg``` = 0 ({0, 1}, optional) – Training algorithm: 1 for skip-gram; otherwise CBOW.\n",
    "- ```hs``` = 0 ({0, 1}, optional) – If 1, hierarchical softmax will be used for model training. If 0, and negative is non-zero, negative sampling will be used.\n",
    "- ```negative``` = 5 (int, optional) – If > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). If set to 0, no negative sampling is used.\n",
    "- ```max_vocab_size``` = None (int, optional) Limits the RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones.\n",
    "- ```iter``` (int, optional) – Number of iterations (epochs) over the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging \n",
    "# Setting up the loggings to monitor gensim\n",
    "logging.basicConfig(format=\"%(levelname)s - %(asctime)s: %(message)s\", datefmt= '%H:%M:%S', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dataset = api.load(\"text8\")\n",
    "data = [d for d in dataset]\n",
    "\n",
    "# Train Word2Vec model\n",
    "model = Word2Vec(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Word2Vec``` without ```sentences``` or ```corpus``` is initialization only, should be trained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Word2Vec()\n",
    "model.train(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Save and load model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('w2v_newmodel')\n",
    "model = Word2Vec.load('w2v_newmodel')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **continue trainig**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train([[\"hello\", \"world\"]], total_examples=1, epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Word2vec input**\n",
    "\n",
    "\n",
    "1) Parameter ```sentences```\n",
    "\n",
    "Gensim’s word2vec expects a sequence of sentences as its input. Each sentence a list of words\n",
    "\n",
    "   1.1 *List of list of tokens*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "input1 = [['first', 'sentence'], ['second', 'sentence']]\n",
    "model1 = Word2Vec(input1, min_count=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.2 Gensim only requires that the input must provide sentences sequentially, when iterated over. *No need to keep everything in RAM*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MySentences(object):\n",
    "    def __init__(self, dirname):\n",
    "        self.dirname = dirname\n",
    " \n",
    "    def __iter__(self):\n",
    "        for fname in os.listdir(self.dirname):\n",
    "            for line in open(os.path.join(self.dirname, fname)):\n",
    "                yield line.split()\n",
    "                \n",
    "input2 = MySentences('/some/directory') # a memory-friendly iterator\n",
    "model = gensim.models.Word2Vec(input2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See [BrownCorpus](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.BrownCorpus), [Text8Corpus](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Text8Corpus) or [LineSentence](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.LineSentence) in word2vec module for such examples.\n",
    "\n",
    "BrownCorpus and Text8Corpus were implemented special for BrownCorpus and Text8 datasets. Text8 corpus, for example, consists of one line of cleaned and joined wikipedia articles.\n",
    "\n",
    "LineSentence iterate over a file that contains sentences: one line = one sentence. Words must be already preprocessed and separated by whitespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.test.utils import datapath\n",
    "from gensim.models.word2vec importLineSentence\n",
    "\n",
    "input3 = LineSentence(datapath('lee_background.cor'))\n",
    "model = gensim.models.Word2Vec(input3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Parameter ```corpus_file``` - path to a corpus file in LineSentence format\n",
    "\n",
    "If corpus is in right format, parameter corpus_file may be passed instead of last cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Extract the trained word vectors from model.wv:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.wv['topic']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.wv.most_similar(positive = ['topic'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Similarity between (walk, walking): ', model.wv.similarity('walk', 'walking'))\n",
    "print('Similarity between (duck, ducks): ', model.wv.similarity('duck', 'ducks'))\n",
    "print('Similarity between (banana, pear): ', model.wv.similarity('banana', 'pear'))\n",
    "print()\n",
    "print('Similarity between (banana, sky): ', model.wv.similarity('banana', 'sky'))\n",
    "print('Similarity between (walk, lie): ', model.wv.similarity('walk', 'lie'))\n",
    "print('Similarity between (dark, slow): ', model.wv.similarity('dark', 'slow'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Analogy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.wv.most_similar(positive = ['lower', 'tall'], negative = ['low'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.wv.most_similar(positive = ['mother', 'man'], negative = ['woman'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Matching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.wv.doesnt_match(['car', 'airplane', 'bed']), \" doesn't match to [car, airplane]\")\n",
    "print(model.wv.doesnt_match(['red', 'blue', 'roof']), \" doesn't match to [red, blue]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare with other pretrained embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext_model300 = api.load('fasttext-wiki-news-subwords-300')\n",
    "word2vec_model300 = api.load('word2vec-google-news-300')\n",
    "glove_model300 = api.load('glove-wiki-gigaword-300')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To define which one performs better using the respective model's evaluate_word_analogies() \n",
    "\n",
    "Compute performance of the model on an analogy test set. The accuracy is reported (printed to log and returned as a score) for each section separately, plus there’s one aggregate summary at the end. \n",
    "\n",
    "Input:\n",
    "- ```analogies``` (str) – Path to file, where lines are 4-tuples of words, split into sections by “: SECTION NAME” lines. See [this file](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/datasets/questions-words.txt) as example.\n",
    "\n",
    "Output:\n",
    "- ```score``` (float) – The overall evaluation score on the entire evaluation set\n",
    "\n",
    "- ```sections``` (list of dict of {str : str or list of tuple of (str, str, str, str)}) – Results broken down by each section of the evaluation set. Each dict contains the name of the section under the key ‘section’, and lists of correctly and incorrectly predicted 4-tuples of words under the keys ‘correct’ and ‘incorrect’."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('questions-words.txt', 'r')\n",
    "for i in range(5):\n",
    "    print(f.readline())\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2vec_model300.evaluate_word_analogies(analogies='questions-words.txt')[0]\n",
    "\n",
    "# fasttext_accuracy\n",
    "fasttext_model300.evaluate_word_analogies(analogies='questions-words.txt')[0]\n",
    "\n",
    "# GloVe accuracy\n",
    "glove_model300.evaluate_word_analogies(analogies='questions-words.txt')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Doc2Vec\n",
    "\n",
    "Unlike Word2Vec, a Doc2Vec model provides a vectorised representation of a group of words taken collectively as a single unit. It is not a simple average of the word vectors of the words in the sentence.\n",
    "\n",
    "The training data for ```Doc2Vec``` should be a list of ```TaggedDocuments```. To create one, we pass a list of words and a unique integer as input to the ```models.doc2vec.TaggedDocument()```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#prepare dataset\n",
    "def create_tagged_document(list_of_list_of_words):\n",
    "    for i, list_of_words in enumerate(list_of_list_of_words):\n",
    "        yield TaggedDocument(list_of_words, [i])\n",
    "\n",
    "dataset = api.load(\"text8\")\n",
    "data = [d for d in dataset]\n",
    "\n",
    "train_data = list(create_tagged_document(data))\n",
    "print(train_data[:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Train model\n",
    "model = Doc2Vec(vector_size=50, min_count=1, epochs=40)\n",
    "model.build_vocab(train_data)\n",
    "model.train(train_data, total_examples=model.corpus_count, epochs=model.epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Get document vector\n",
    "print(model.infer_vector('gensim is really awesome'.split(' ')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Used materials: \n",
    "    \n",
    "- https://www.machinelearningplus.com/nlp/gensim-tutorial/\n",
    "- https://radimrehurek.com/gensim/models/word2vec.html\n",
    "- https://radimrehurek.com/gensim/models/keyedvectors.html\n",
    "- https://rare-technologies.com/word2vec-tutorial/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}