Similarity.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Similarity.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Shin-kyoto/www/blob/develop/Similarity.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9aVVn6hB9Zei",
        "colab_type": "text"
      },
      "source": [
        "# Preprocessing and calculate similarity\n",
        "\n",
        "このノートの目標は自力で文書の類似度を計算できるようになること  \n",
        "最終的にWikipediaのデータを用いて国の類似度を測り  \n",
        "日本と似ている国を探す"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N5-315WVmN64",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 383
        },
        "outputId": "4e31e3c2-6dc9-4d23-b0ed-48bd92e4ec24"
      },
      "source": [
        "# 必要なパッケージのインストール\n",
        "!pip install nltk\n",
        "!pip install gensim"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (3.2.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk) (1.12.0)\n",
            "Requirement already satisfied: gensim in /usr/local/lib/python3.6/dist-packages (3.6.0)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (2.0.0)\n",
            "Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.18.5)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.4.1)\n",
            "Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.12.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (2.23.0)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (1.14.9)\n",
            "Requirement already satisfied: boto in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (2.49.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (2020.6.20)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (2.9)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (0.10.0)\n",
            "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (0.3.3)\n",
            "Requirement already satisfied: botocore<1.18.0,>=1.17.9 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (1.17.9)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.18.0,>=1.17.9->boto3->smart-open>=1.2.1->gensim) (2.8.1)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.18.0,>=1.17.9->boto3->smart-open>=1.2.1->gensim) (0.15.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M8iK7xBDTmtf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import nltk\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from scipy.spatial.distance import minkowski"
      ],
      "execution_count": 105,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iBUvXHj7mRls",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145
        },
        "outputId": "fa695209-e8d4-46e3-a18a-1bdd877ecd2a"
      },
      "source": [
        "nltk.download(\"stopwords\")\n",
        "nltk.download(\"wordnet\")\n",
        "nltk.download(\"punkt\")"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kpg2beFhKA2R",
        "colab_type": "text"
      },
      "source": [
        "## 1. Calculate similarity\n",
        "\n",
        "以下の三つの文を考える  \n",
        "Doc A : \"I like apples and a strawberries. I will buy an apple tomorrow. \"  \n",
        "Doc B : \"I bought some apples and strawberries. I will eat an apple tomorrow.\"  \n",
        "Doc C : \"I play basketball every day. I like Michael Jordan.\"  \n",
        "\n",
        "Doc AとDoc Bは似ていそうだが、Doc CはDoc AともDoc Bとも似ていなさそう  \n",
        "これを類似度を計算することで確かめる\n",
        "\n",
        "類似度の計算の仕方はいくつかある\n",
        "\n",
        "- 集合ベースの類似度\n",
        "  - Jaccard係数\n",
        "  - Dice係数\n",
        "  - Simpson係数\n",
        "- ベクトルベースの類似度\n",
        "  - ユークリッド距離\n",
        "  - コサイン類似度\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y1rpPCaUeOWE",
        "colab_type": "text"
      },
      "source": [
        "### 集合ベース\n",
        "\n",
        "文書を単語の集合に変換する  \n",
        "集合なので重複した単語は削除する  \n",
        "前処理は今回はスキップする   \n",
        "\n",
        "Doc A : \"I like apples and a strawberries. I will buy an apple tomorrow. \"  \n",
        "Doc B : \"I bought some apples and strawberries. I will eat an apple tomorrow.\"  \n",
        "Doc C : \"I play basketball every day. I like Michael Jordan.\"  \n",
        "↓    \n",
        "Set A : {'a', 'an', 'and', 'apple', 'apples', 'buy', 'i', 'like', 'strawberries', 'tomorrow', 'will'}  \n",
        "Set B : {'an', 'and', 'apple', 'apples', 'bought', 'eat', 'i', 'some', 'strawberries', 'tomorrow', 'will'}  \n",
        "Set C : {'basketball', 'day', 'every', 'i', 'jordan', 'like', 'michael', 'play'}  \n",
        "\n",
        "この集合が文書の特徴を表していると考える  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uts5Ns2eKDaW",
        "colab_type": "text"
      },
      "source": [
        "#### Jaccard係数\n",
        "Jaccard係数は二つの集合A,Bに対して定義される類似度である  \n",
        "計算式は以下の通り\n",
        "\n",
        "\\begin{equation}\n",
        "J(A,B)=\\dfrac{|A\\cap B|}{|A \\cup B|}\n",
        "\\end{equation}\n",
        "\n",
        "共通部分の割合が大きければその二つの文書は似ていると考える"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wqr10Mw-K5UQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def jaccard_similarity(set_a,set_b):\n",
        "  # 積集合の要素数を計算\n",
        "  num_intersection = len(set.intersection(set_a, set_b))\n",
        "  # 和集合の要素数を計算\n",
        "  num_union = len(set.union(set_a, set_b))\n",
        "  #Jaccard係数を算出　空集合の時は1を出力\n",
        "  try:\n",
        "      return float(num_intersection) / num_union\n",
        "  except ZeroDivisionError:\n",
        "      return 1.0 "
      ],
      "execution_count": 107,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZSFY5urK8eT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "e74717ef-99ae-4b97-85dc-ac99ed2fd3bf"
      },
      "source": [
        "set_a = set(['a', 'an', 'and', 'apple', 'apples', 'buy', 'i', 'like', 'strawberries', 'tomorrow', 'will'])\n",
        "set_b = set(['an', 'and', 'apple', 'apples', 'bought', 'eat', 'i', 'some', 'strawberries', 'tomorrow', 'will'])\n",
        "set_c = set(['basketball', 'day', 'every', 'i', 'jordan', 'like', 'michael', 'play'])\n",
        "\n",
        "print(\"jaccard(a, b) = \", jaccard_similarity(set_a, set_b)) #Jaccard係数を計算\n",
        "print(\"jaccard(a, c) = \", jaccard_similarity(set_a, set_c))\n",
        "print(\"jaccard(b, c) = \", jaccard_similarity(set_b, set_c))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "jaccard(a, b) =  0.5714285714285714\n",
            "jaccard(a, c) =  0.11764705882352941\n",
            "jaccard(b, c) =  0.05555555555555555\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXBk5SiTMmGf",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "nltkで実装されている  \n",
        "定義と同じように計算を行うので、入力は集合  \n",
        "距離になっているところには注意が必要"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lZQ9q8mFLJTO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "8526ea6d-d88c-469d-a793-56e64ae5a63c"
      },
      "source": [
        "from nltk.metrics import jaccard_distance\n",
        "\n",
        "set_a = set(['a', 'an', 'and', 'apple', 'apples', 'buy', 'i', 'like', 'strawberries', 'tomorrow', 'will'])\n",
        "set_b = set(['an', 'and', 'apple', 'apples', 'bought', 'eat', 'i', 'some', 'strawberries', 'tomorrow', 'will'])\n",
        "set_c = set(['basketball', 'day', 'every', 'i', 'jordan', 'like', 'michael', 'play'])\n",
        "\n",
        "# Jaccard距離になっているので、類似度に変換するときは1から引く\n",
        "print(\"jaccard(a, b) = \", 1 - jaccard_distance(set_a, set_b))\n",
        "print(\"jaccard(a, c) = \", 1 - jaccard_distance(set_a, set_c))\n",
        "print(\"jaccard(b, c) = \", 1 - jaccard_distance(set_b, set_c))\n"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "jaccard(a, b) =  0.5714285714285714\n",
            "jaccard(a, c) =  0.11764705882352944\n",
            "jaccard(b, c) =  0.05555555555555558\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2lyCVTfePwB2",
        "colab_type": "text"
      },
      "source": [
        "#### Sørensen-Dice係数\n",
        "\n",
        "Jaccard係数では分母はの和集合であったため  \n",
        "片方の集合がとても大きいと共通部分が大きくても係数の値が小さくなってしまうという問題がある  \n",
        "Sørensen-Dice係数では、分母を二つの集合の大きさの平均をとることで、その影響を緩和している  \n",
        "\n",
        "$\n",
        "DSC(A,B) = \\dfrac{|A\\cap B|}{\\dfrac{|A| + |B|}{2}} = \\dfrac{2|A\\cap B|}{|A| + |B|}\n",
        "$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0M0RikFPR3Qr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def dice_similarity(set_a, set_b):\n",
        "  num_intersection =  len(set.intersection(set_a, set_b))\n",
        "  sum_nums = len(set_a) + len(set_b)\n",
        "  try:\n",
        "    return 2 * num_intersection / sum_nums\n",
        "  except ZeroDivisionError:\n",
        "    return 1.0 "
      ],
      "execution_count": 109,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZZQFbXlESPWl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "e5248c67-0b7e-4e0a-e541-517f22d6b1a5"
      },
      "source": [
        "set_a = set(['a', 'an', 'and', 'apple', 'apples', 'buy', 'i', 'like', 'strawberries', 'tomorrow', 'will'])\n",
        "set_b = set(['an', 'and', 'apple', 'apples', 'bought', 'eat', 'i', 'some', 'strawberries', 'tomorrow', 'will'])\n",
        "set_c = set(['basketball', 'day', 'every', 'i', 'jordan', 'like', 'michael', 'play'])\n",
        "\n",
        "print(\"dice(a, b) = \", dice_similarity(set_a, set_b))\n",
        "print(\"dice(a, c) = \", dice_similarity(set_a, set_c))\n",
        "print(\"dice(b, c) = \", dice_similarity(set_b, set_c))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dice(a, b) =  0.7272727272727273\n",
            "dice(a, c) =  0.21052631578947367\n",
            "dice(b, c) =  0.10526315789473684\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DxtliI-HPwRE",
        "colab_type": "text"
      },
      "source": [
        "#### Szymkiewicz-Simpson係数\n",
        "\n",
        "差集合の要素数の影響を極限まで抑えたのがSzymkiewicz-Simpson係数    \n",
        "$\n",
        "overlap(𝐴,𝐵) = \\dfrac{|A\\cap B|}{\\min(|A|, |B|)}\n",
        "$\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgGJ5GhmUduH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def simpson_similarity(list_a, list_b):\n",
        "  num_intersection = len(set.intersection(set(list_a), set(list_b)))\n",
        "  min_num = min(len(set(list_a)), len(set(list_b)))\n",
        "  try:\n",
        "    return num_intersection / min_num\n",
        "  except ZeroDivisionError:\n",
        "    if num_intersection == 0:\n",
        "      return 1.0\n",
        "    else:\n",
        "      return 0"
      ],
      "execution_count": 110,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g_1Gjy4IV9eo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "bd752621-a182-4b35-d7f9-842de47d1a61"
      },
      "source": [
        "set_a = set(['a', 'an', 'and', 'apple', 'apples', 'buy', 'i', 'like', 'strawberries', 'tomorrow', 'will'])\n",
        "set_b = set(['an', 'and', 'apple', 'apples', 'bought', 'eat', 'i', 'some', 'strawberries', 'tomorrow', 'will'])\n",
        "set_c = set(['basketball', 'day', 'every', 'i', 'jordan', 'like', 'michael', 'play'])\n",
        "\n",
        "print(\"simpson(a, b) = \", simpson_similarity(set_a, set_b)) \n",
        "print(\"simpson(a, c) = \", simpson_similarity(set_a, set_c)) \n",
        "print(\"simpson(b, c) = \", simpson_similarity(set_b, set_c)) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "simpson(a, b) =  0.7272727272727273\n",
            "simpson(a, c) =  0.25\n",
            "simpson(b, c) =  0.125\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J-T32gmnZIyt",
        "colab_type": "text"
      },
      "source": [
        "#### Exercise 1\n",
        "色々な集合を作って集合ベース手法の比較をしよう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oJwJpZpdYtz9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "outputId": "b1b03cdb-10f6-48fd-9272-27e10daff2f4"
      },
      "source": [
        "set_a = set(['a', 'an', 'and', 'apple', 'apples', 'buy', 'i', 'like', 'strawberries', 'tomorrow', 'will'])\n",
        "set_b = set(['an', 'and', 'apple', 'apples', 'bought', 'eat', 'i', 'some', 'strawberries', 'tomorrow', 'will'])\n",
        "set_c = set(['basketball', 'day', 'every', 'i', 'jordan', 'like', 'michael', 'play'])\n",
        "set_d = set(['an','and','apple','i','jordan','like','micheal']) # 大きめの集合を作って試してみよう\n",
        "\n",
        "print(\"jaccard similarity:\")\n",
        "print(jaccard_similarity(set_d, set_a))\n",
        "print(jaccard_similarity(set_d, set_b))\n",
        "print(jaccard_similarity(set_d, set_c))\n",
        "\n",
        "print(\"dice similarity:\")\n",
        "print(dice_similarity(set_d, set_a))\n",
        "print(dice_similarity(set_d, set_b))\n",
        "print(dice_similarity(set_d, set_c))\n",
        "\n",
        "print(\"simpson similarity:\")\n",
        "print(simpson_similarity(set_d, set_a))\n",
        "print(simpson_similarity(set_d, set_b))\n",
        "print(simpson_similarity(set_d, set_c))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "jaccard similarity:\n",
            "0.38461538461538464\n",
            "0.2857142857142857\n",
            "0.25\n",
            "dice similarity:\n",
            "0.5555555555555556\n",
            "0.4444444444444444\n",
            "0.4\n",
            "simpson similarity:\n",
            "0.7142857142857143\n",
            "0.5714285714285714\n",
            "0.42857142857142855\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-Hm34tgXCbh",
        "colab_type": "text"
      },
      "source": [
        "### ベクトルベース \n",
        "\n",
        "\n",
        "文書をベクトルとして表現し類似度を計算する  \n",
        "ベクトル化の手法は色々あるが今回はBoW(Bag of Words)で説明する  \n",
        "\n",
        "BoWは文をベクトルで表現する方法の一つ  \n",
        "想定している単語の総数をNとすると、各次元が各単語に対応するN次元のベクトルを考える  \n",
        "各次元の値はその単語が文書中で出た回数\n",
        "\n",
        "例）  \n",
        "Doc A : \"I like apples and a strawberries. I will buy an apple tomorrow. \"  \n",
        "Doc B : \"I bought some apples and strawberries. I will eat an apple tomorrow.\"  \n",
        "Doc C : \"I play basketball every day. I like Michael Jordan.\"  \n",
        "↓  \n",
        "全単語は19個で、各次元の値は以下の単語の個数に対応するBoWを考える  \n",
        "['an', 'and', 'apple', 'apples', 'basketball', 'bought', 'buy', 'day', 'eat', 'every', 'i', 'jordan', 'like', 'michael', 'play', 'some', 'strawberries', 'tomorrow', 'will']  \n",
        "↓  \n",
        "BoW A : [1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 2, 0, 1, 0, 0, 0, 1, 1, 1]  \n",
        "BoW B : [1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 2, 0, 0, 0, 0, 1, 1, 1, 1]  \n",
        "BoW C : [0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 2, 1, 1, 1, 1, 0, 0, 0, 0]  \n",
        "\n",
        "このベクトルが文書の特徴を表していると考える"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kQOjHTfCeXjs",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "#### ユークリッド距離\n",
        "\n",
        "各文書をベクトルで表すことが出来たので  \n",
        "ユークリッド距離が計算できる  \n",
        "この距離が小さければ似ていると考えることが出来る\n",
        "\n",
        "\\begin{equation}\n",
        "d(v_1,v_2) =(\\sum_{i=1}^n (v_{1i}-v_{2i})^2)^{\\frac{1}{2}}\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YtEzuSDKZbpV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def euclidean_distance(list_a, list_b):\n",
        "  diff_vec = np.array(list_a) - np.array(list_b)\n",
        "  return np.linalg.norm(diff_vec)"
      ],
      "execution_count": 111,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Sr_ZYj7azLV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "150b64d4-7ccc-42e4-a893-bdeda965579e"
      },
      "source": [
        "bow_a = [1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 2, 0, 1, 0, 0, 0, 1, 1, 1]  \n",
        "bow_b = [1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 2, 0, 0, 0, 0, 1, 1, 1, 1]  \n",
        "bow_c = [0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 2, 1, 1, 1, 1, 0, 0, 0, 0]  \n",
        "\n",
        "print(\"euclidean_distance(bow_a, bow_b) = \",euclidean_distance(bow_a, bow_b))\n",
        "print(\"euclidean_distance(bow_a, bow_c) = \",euclidean_distance(bow_a, bow_c))\n",
        "print(\"euclidean_distance(bow_b, bow_c) = \",euclidean_distance(bow_b, bow_c))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "euclidean_distance(bow_a, bow_b) =  2.23606797749979\n",
            "euclidean_distance(bow_a, bow_c) =  3.7416573867739413\n",
            "euclidean_distance(bow_b, bow_c) =  4.123105625617661\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfKcQd5SiBb-",
        "colab_type": "text"
      },
      "source": [
        "#### ミンコフスキー距離\n",
        "\n",
        "ユークリッド距離を一般化した距離\n",
        "pの値を変えることで色々な距離を表現できる  \n",
        "\n",
        "\\begin{equation}\n",
        "d(v_1,v_2) = (\\sum_{i=1}^n |v_{1i}-v_{2i}|^p)^{\\frac{1}{p}}\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tx46KLSNhjI2",
        "colab_type": "text"
      },
      "source": [
        "#### Exercise 2\n",
        "ミンコフスキー距離を計算するプログラムを書いて  \n",
        "p=0,1,2,3で距離を計算してみよう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X6UMFh58bexH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# np.linalg.normについて調べよう\n",
        "def minkowski_distance(list_a, list_b, p):\n",
        "  diff_vec = np.array(list_a) - np.array(list_b)\n",
        "  #return minkowski(diff_vec,p)\n",
        "  return np.linalg.norm(diff_vec,ord=p)\n",
        "  "
      ],
      "execution_count": 113,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jKMIMbyJc8eH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 308
        },
        "outputId": "54dcb089-39a2-4145-c2b7-a940763a49cf"
      },
      "source": [
        "p=0\n",
        "print('p={}'.format(p))\n",
        "print(minkowski_distance(bow_a, bow_b, 0))\n",
        "print(minkowski_distance(bow_a, bow_c, 0))\n",
        "print(minkowski_distance(bow_b, bow_c, 0))\n",
        "p=1\n",
        "print('p={}'.format(p))\n",
        "print(minkowski_distance(bow_a, bow_b, 1))\n",
        "print(minkowski_distance(bow_a, bow_c, 1))\n",
        "print(minkowski_distance(bow_b, bow_c, 1))\n",
        "\n",
        "p=2\n",
        "print('p={}'.format(p))\n",
        "print(minkowski_distance(bow_a, bow_b, 2))\n",
        "print(minkowski_distance(bow_a, bow_c, 2))\n",
        "print(minkowski_distance(bow_b, bow_c, 2))\n",
        "\n",
        "p=3\n",
        "print('p={}'.format(p))\n",
        "print(minkowski_distance(bow_a, bow_b, 3))\n",
        "print(minkowski_distance(bow_a, bow_c, 3))\n",
        "print(minkowski_distance(bow_b, bow_c, 3))\n"
      ],
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "p=0\n",
            "5.0\n",
            "14.0\n",
            "17.0\n",
            "p=1\n",
            "5.0\n",
            "14.0\n",
            "17.0\n",
            "p=2\n",
            "2.23606797749979\n",
            "3.7416573867739413\n",
            "4.123105625617661\n",
            "p=3\n",
            "1.7099759466766968\n",
            "2.4101422641752297\n",
            "2.571281590658235\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KIyabGRTKGwA",
        "colab_type": "text"
      },
      "source": [
        "#### コサイン類似度\n",
        "\n",
        "ベクトルのなす角に着目して類似度を計算する  \n",
        "\n",
        "\\begin{equation}\n",
        "similarity(A, B)=cos(\\theta)=\\dfrac{\\sum_{i=1}^n A_iB_i}{{\\sqrt A}{\\sqrt B}}\n",
        "\\end{equation}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fWy619amlwp",
        "colab_type": "text"
      },
      "source": [
        "#### Exercise 3\n",
        "コサイン類似度を計算するプログラムを書いて計算しよう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xDhwqTN5yJGQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# numpy.array について調べよう\n",
        "def cosine_similarity(list_a, list_b):\n",
        "  # あとで消す\n",
        "  inner_prod = np.array(list_a).dot(np.array(list_b))\n",
        "  norm_a = np.linalg.norm(list_a)\n",
        "  norm_b = np.linalg.norm(list_b)\n",
        "  try:\n",
        "      return inner_prod / (norm_a*norm_b)\n",
        "  except ZeroDivisionError:\n",
        "      return 1.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sew3u-YezRrX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "45ed3498-e526-4ce5-96aa-ab942a9d5fb2"
      },
      "source": [
        "bow_a = [1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 2, 0, 1, 0, 0, 0, 1, 1, 1]\n",
        "bow_b = [1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 2, 0, 0, 0, 0, 1, 1, 1, 1]\n",
        "bow_c = [0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 2, 1, 1, 1, 1, 0, 0, 0, 0]\n",
        "\n",
        "print(\"cosine_similarity(bow_a, bow_b) = \",cosine_similarity(bow_a, bow_b))\n",
        "print(\"cosine_similarity(bow_a, bow_c) = \",cosine_similarity(bow_a, bow_c))\n",
        "print(\"cosine_similarity(bow_b, bow_c) = \",cosine_similarity(bow_b, bow_c))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cosine_similarity(bow_a, bow_b) =  0.8153742483272114\n",
            "cosine_similarity(bow_a, bow_c) =  0.41812100500354543\n",
            "cosine_similarity(bow_b, bow_c) =  0.3223291856101521\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ao0jywXqKIFS",
        "colab_type": "text"
      },
      "source": [
        "### 集合ベースとベクトルベースの比較\n",
        "\n",
        "集合演算の方は一つ一つの文書が小さいデータに対して性能が高い  \n",
        "文書がある程度大きくなるとベクトルベースの方が有用になる  \n",
        "その代わり、語彙集合が大きくなり計算量が大きくなってしまう\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u1IXMwIyhU-6",
        "colab_type": "text"
      },
      "source": [
        "### Exercise 4\n",
        "短い文章のデータセットと長い文章のデータセットを自分で作り    \n",
        "Jaccard係数とコサイン類似度を計算して比較してみよう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BpkcXVDBhZg8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "short_docs = []\n",
        "long_docs = []"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uuXSHk01KLza",
        "colab_type": "text"
      },
      "source": [
        "## 2. Preprocessing\n",
        "\n",
        "集合間の共通部分やベクトル間の距離や角度で類似度を測ることが出来た  \n",
        "集合やベクトルが文書の特徴を上手く表せていないと類似度が上手く測れない  \n",
        "文書からどのように集合やベクトルを作るかがとても大事  \n",
        " \n",
        "適切な前処理を行うことで特徴を捉えた類似度を測れるようになる    \n",
        "後半はベクトル化に絞って練習していく  \n",
        "\n",
        "1. Clearning\n",
        "2. Tokenize\n",
        "3. Stemming\n",
        "4. Remove stop words\n",
        "5. Vectorize"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4ispTSLKM-z",
        "colab_type": "text"
      },
      "source": [
        "### 2-1. Clearning\n",
        "\n",
        "上の例では綺麗な文章ばかり扱っていたが、実際はもっと汚い   \n",
        "Webから取ってきたデータだとhtmlタグが残っていたり、変な記号が入っていたりする  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZM-e5AZT316r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "documents=[\"I like apples and a strawberries. I will buy an apple tomorrow @Fresco.\",\n",
        "           \"I bought some apples and strawberries. I will eat an apple <b>tomorrow.</b>\",\n",
        "           \"I play basketball every day. I like Michael Jordan (born February 17, 1963).\"]"
      ],
      "execution_count": 133,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nh56eU_levzv",
        "colab_type": "text"
      },
      "source": [
        "今は三つなので手動で消せるが  \n",
        "大量のデータを扱うときには自動で綺麗にできないといけない  \n",
        "綺麗にするプログラムを作る"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "06coot4PnVgS",
        "colab_type": "text"
      },
      "source": [
        "#### Exercise 5\n",
        "\n",
        "正規表現を使ってテキストを綺麗にするプログラムを書こう\n",
        "\n",
        "参考: 正規表現 (https://uxmilk.jp/41416)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rt3NL8Ux5Q4E",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "aa04c74d-9f9b-444f-8b11-fbc6a6640bc7"
      },
      "source": [
        "import re\n",
        "\n",
        "def cleaning_text(text):\n",
        "    # @の削除\n",
        "    pattern1 = '@'\n",
        "    text = re.sub(pattern1, '', text)    \n",
        "    # <b>タグの削除\n",
        "    pattern2 = r'<(/)?b>' #(/)?は，/があってもなくても良いということ\n",
        "    text = re.sub(pattern2, '', text)    \n",
        "    # ()内を削除\n",
        "    pattern3 = r'\\([\\w\\s\\S]*\\)' #r'\\([\\w\\s\\S]*\\)'でもOK\n",
        "    #r'(.*?)'ではだめ．\n",
        "    #()は，エスケープが必要な文字（正規表現におけるグループ化の意味を持っている文字なので，そのまま使えない）．\n",
        "    text = re.sub(pattern3, '', text)\n",
        "    return text\n",
        "  \n",
        "\n",
        "for text in documents:\n",
        "    print(cleaning_text(text))"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I like apples and a strawberries. I will buy an apple tomorrow Fresco.\n",
            "I bought some apples and strawberries. I will eat an apple tomorrow.\n",
            "I play basketball every day. I like Michael Jordan .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcgcKWkRflR5",
        "colab_type": "text"
      },
      "source": [
        "#### Option 1\n",
        "\n",
        "以下のテキストを綺麗にするコードを書いてみよう\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GcsdHgV-h26y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "text = '<p><b>Natural language processing</b> (<b>NLP</b>) is a subfield of <a href=\"/wiki/Computer_science\" title=\"Computer science\">computer science</a>, <a href=\"/wiki/Information_engineering_(field)\" title=\"Information engineering (field)\">information engineering</a>, and <a href=\"/wiki/Artificial_intelligence\" title=\"Artificial intelligence\">artificial intelligence</a> concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of <a href=\"/wiki/Natural_language\" title=\"Natural language\">natural language</a> data.</p>'"
      ],
      "execution_count": 123,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Nn4KlnmvMLZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cleaning_text(text):\n",
        "    # @の削除\n",
        "    pattern1 = '@'\n",
        "    text = re.sub(pattern1, '', text)    \n",
        "    # <b>タグの削除\n",
        "    pattern2 = r'<(/)?.*?>'\n",
        "    #(/)?は，/があってもなくても良いということ\n",
        "    #b→.*?とすることで，どんなタグにも非貪欲にマッチするように修正\n",
        "    \n",
        "    text = re.sub(pattern2, '', text)    \n",
        "    # ()内を削除\n",
        "    pattern3 = r'\\(.*?\\)' #r'(.*?)'ではだめ．　()は，エスケープが必要な文字（正規表現における意味を持っている文字なので，そのまま使えない）．\n",
        "    text = re.sub(pattern3, '', text)\n",
        "    return text"
      ],
      "execution_count": 124,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yYst7oxnvVSr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "78731e2e-e2ac-45e5-bee5-7c20a296afa6"
      },
      "source": [
        "cleaning_text(text)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic": {
              "type": "string"
            },
            "text/plain": [
              "'Natural language processing  is a subfield of computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human  languages, in particular how to program computers to process and analyze large amounts of natural language data.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1doxCMAGr0_O",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "edJ80nTbQgLi",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "### 2-2. Tokenize\n",
        "\n",
        "まだ文字列のままなので、単語ごとに区切る  \n",
        "英語だと空白区切りでよいが日本語だと少し面倒  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JGbqTvkyYl2S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "54cc63ed-53ba-44ae-e297-b0504d4ec9ef"
      },
      "source": [
        "def tokenize_text(text):\n",
        "  text = re.sub('[.,]', '', text)\n",
        "  return text.split()\n",
        "\n",
        "for text in documents:\n",
        "  text = cleaning_text(text)\n",
        "  print(tokenize_text(text))"
      ],
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['I', 'like', 'apples', 'and', 'a', 'strawberries', 'I', 'will', 'buy', 'an', 'apple', 'tomorrow', 'Fresco']\n",
            "['I', 'bought', 'some', 'apples', 'and', 'strawberries', 'I', 'will', 'eat', 'an', 'apple', 'tomorrow']\n",
            "['I', 'play', 'basketball', 'every', 'day', 'I', 'like', 'Michael', 'Jordan']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K97zCvZBQik9",
        "colab_type": "text"
      },
      "source": [
        "### 2-3. Stemming, Lemmatize\n",
        "\n",
        "同じ意味の単語でも異なる形をしていることがある  \n",
        "それらを別の単語としてカウントするのは不自然  \n",
        "小文字に変換した後  \n",
        "StemmingやLemmatizeという処理で同じ形にする  \n",
        "今回はLemmatizeのみ"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P7R55rKxZtAz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.corpus import wordnet as wn #lemmatize関数のためのimport\n",
        "\n",
        "def lemmatize_word(word):\n",
        "    # make words lower  example: Python =>python\n",
        "    word=word.lower()\n",
        "    \n",
        "    # lemmatize  example: cooked=>cook\n",
        "    lemma = wn.morphy(word)\n",
        "    if lemma is None:\n",
        "        return word\n",
        "    else:\n",
        "      return lemma"
      ],
      "execution_count": 128,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_0sEVvmwaCqA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        },
        "outputId": "5c03faee-b64e-474a-a232-0fc6ff58cb26"
      },
      "source": [
        "for text in documents:\n",
        "  text = cleaning_text(text)\n",
        "  tokens = tokenize_text(text)\n",
        "  print([lemmatize_word(word) for word in tokens])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['i', 'like', 'apple', 'and', 'a', 'strawberry', 'i', 'will', 'buy', 'an', 'apple', 'tomorrow', 'fresco']\n",
            "['i', 'buy', 'some', 'apple', 'and', 'strawberry', 'i', 'will', 'eat', 'an', 'apple', 'tomorrow']\n",
            "['i', 'play', 'basketball', 'every', 'day', 'i', 'like', 'michael', 'jordan', '(born', 'february', '17', '1963)']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MOPIFqvjfxEg",
        "colab_type": "text"
      },
      "source": [
        "strawberries→strawberryのように語を標準形に変換出来た"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UpUxnB9kQlKv",
        "colab_type": "text"
      },
      "source": [
        "### 2-4. Remove stop words\n",
        "\n",
        "a, theなどの文章に寄らず一般的に使われる冠詞、代名詞、前置詞などを使っても意味がない  \n",
        "それらの単語はstop wordと呼ばれる  \n",
        "nltkには専門家が定義したstop wordのリストがあるのでそれを使う  \n",
        "必要に応じてstop wordは自分でカスタマイズするべき  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dbSLDay-a36N",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "94b8e0f9-738c-43a4-864b-3acd028336e2"
      },
      "source": [
        "#1 nltkのストップワードリスト\n",
        "en_stop = nltk.corpus.stopwords.words('english')\n",
        "print(en_stop)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nX8eIyxfbo6q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def remove_stopwords(word, stopwordset):\n",
        "  if word in stopwordset:\n",
        "    return None\n",
        "  else:\n",
        "    return word"
      ],
      "execution_count": 129,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WckIX0ThbQMy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        },
        "outputId": "fed475ab-07ff-47f5-849d-8bf9285176bd"
      },
      "source": [
        "for text in documents:\n",
        "  text = cleaning_text(text)\n",
        "  tokens = tokenize_text(text)\n",
        "  tokens = [lemmatize_word(word) for word in tokens]\n",
        "  print([remove_stopwords(word, en_stop) for word in tokens])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[None, 'like', 'apple', None, None, 'strawberry', None, None, 'buy', None, 'apple', 'tomorrow', 'fresco']\n",
            "[None, 'buy', None, 'apple', None, 'strawberry', None, None, 'eat', None, 'apple', 'tomorrow']\n",
            "[None, 'play', 'basketball', 'every', 'day', None, 'like', 'michael', 'jordan', '(born', 'february', '17', '1963)']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V4kNHhr5f8Vt",
        "colab_type": "text"
      },
      "source": [
        "今回はこれだけで終わりにするが単語の削除はかなり重要  \n",
        "出現頻度が極端に低い単語を削除したり、動詞と名詞に限定するなど色々ある"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BgjfOITFxwx_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "1db416e0-bcc4-4923-cf0d-8ad3a0a4c268"
      },
      "source": [
        "def preprocessing_text(text):\n",
        "  text = cleaning_text(text)\n",
        "  tokens = tokenize_text(text)\n",
        "  tokens = [lemmatize_word(word) for word in tokens]\n",
        "  tokens = [remove_stopwords(word, en_stop) for word in tokens]\n",
        "  tokens = [word for word in tokens if word is not None]\n",
        "  return tokens\n",
        "\n",
        "\n",
        "preprocessed_docs = [preprocessing_text(text) for text in documents]\n",
        "preprocessed_docs"
      ],
      "execution_count": 139,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['like', 'apple', 'strawberry', 'buy', 'apple', 'tomorrow', 'fresco'],\n",
              " ['buy', 'apple', 'strawberry', 'eat', 'apple', 'tomorrow'],\n",
              " ['play', 'basketball', 'every', 'day', 'like', 'michael', 'jordan']]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CVcnsI1p2PzI",
        "colab_type": "text"
      },
      "source": [
        "### 前処理関数の作成\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nCKK9jsI3OlQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocessing(documents):\n",
        "  doc2=[]\n",
        "  for text in documents:\n",
        "    text = cleaning_text(text)\n",
        "    tokens = tokenize_text(text)\n",
        "    #print([lemmatize_word(word) for word in tokens])\n",
        "    doc2.append([lemmatize_word(word) for word in tokens])\n",
        "  return doc2\n"
      ],
      "execution_count": 135,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_jCorM354nLu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 490
        },
        "outputId": "2bb95c30-0ad6-49a2-fb93-453f26cbd712"
      },
      "source": [
        "preprocessing(documents)"
      ],
      "execution_count": 136,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['i',\n",
              "  'like',\n",
              "  'apple',\n",
              "  'and',\n",
              "  'a',\n",
              "  'strawberry',\n",
              "  'i',\n",
              "  'will',\n",
              "  'buy',\n",
              "  'an',\n",
              "  'apple',\n",
              "  'tomorrow',\n",
              "  'fresco'],\n",
              " ['i',\n",
              "  'buy',\n",
              "  'some',\n",
              "  'apple',\n",
              "  'and',\n",
              "  'strawberry',\n",
              "  'i',\n",
              "  'will',\n",
              "  'eat',\n",
              "  'an',\n",
              "  'apple',\n",
              "  'tomorrow'],\n",
              " ['i', 'play', 'basketball', 'every', 'day', 'i', 'like', 'michael', 'jordan']]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FCd6YpYbKJSJ",
        "colab_type": "text"
      },
      "source": [
        "### 2-5. Vectorize\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmrBtgPxcwX0",
        "colab_type": "text"
      },
      "source": [
        "#### BoW(Bag of Words)\n",
        "\n",
        "\n",
        "テキストを単語の出現回数のベクトルで表したもの  \n",
        "人手で単語を数えたりするのは不可能なのでプログラムで処理を完結してしまおう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S69MsPjHcvut",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def bow_vectorizer(docs):\n",
        "  word2id = {}\n",
        "  for doc in docs:\n",
        "    for w in doc:\n",
        "      if w not in word2id:\n",
        "        word2id[w] = len(word2id)\n",
        "        \n",
        "  result_list = []\n",
        "  for doc in docs:\n",
        "    doc_vec = [0] * len(word2id)\n",
        "    for w in doc:\n",
        "      doc_vec[word2id[w]] += 1\n",
        "    result_list.append(doc_vec)\n",
        "  return result_list, word2id"
      ],
      "execution_count": 140,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zq3TpP7KO-XP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "1b7a49ad-ccae-4fa3-a935-bcd408d04819"
      },
      "source": [
        "bow_vec, word2id = bow_vectorizer(preprocessed_docs)\n",
        "print(bow_vec)"
      ],
      "execution_count": 141,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 2, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "udTTbKv8oFpG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "12da0fa4-d848-4805-ffc3-f601c40ca4c7"
      },
      "source": [
        "word2id.items()"
      ],
      "execution_count": 142,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_items([('like', 0), ('apple', 1), ('strawberry', 2), ('buy', 3), ('tomorrow', 4), ('fresco', 5), ('eat', 6), ('play', 7), ('basketball', 8), ('every', 9), ('day', 10), ('michael', 11), ('jordan', 12)])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 142
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6e8h1SMTcyD5",
        "colab_type": "text"
      },
      "source": [
        "### TF-IDF(Term Frequency - Inverse Document Frequency)\n",
        "\n",
        "BoWでは各単語の重みが同じだったが、単語によって重要度は変わる  \n",
        "単語の重要度を考慮したのがTF-IDF  \n",
        "\n",
        "TF(t, d) = ある単語(t)のある文書(d)における出現頻度  \n",
        "IDF(t) = ある単語(t)が全文書集合(D)中にどれだけの文書で出現したかの逆数  \n",
        "\n",
        "TF-IDF(t,d) = TF(t, d) * IDF(t)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iiKn2p7Bc0bN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def tfidf_vectorizer(docs):\n",
        "  def tf(word2id, doc):\n",
        "    term_counts = np.zeros(len(word2id))\n",
        "    for term in word2id.keys():\n",
        "      term_counts[word2id[term]] = doc.count(term)\n",
        "    tf_values = list(map(lambda x: x/sum(term_counts), term_counts))\n",
        "    return tf_values\n",
        "  \n",
        "  def idf(word2id, docs):\n",
        "    idf = np.zeros(len(word2id))\n",
        "    for term in word2id.keys():\n",
        "      idf[word2id[term]] = np.log(len(docs) / sum([bool(term in doc) for doc in docs]))\n",
        "    return idf\n",
        "  \n",
        "  word2id = {}\n",
        "  for doc in docs:\n",
        "    for w in doc:\n",
        "      if w not in word2id:\n",
        "        word2id[w] = len(word2id)\n",
        "  \n",
        "  return [[_tf*_idf for _tf, _idf in zip(tf(word2id, doc), idf(word2id, docs))] for doc in docs], word2id\n",
        "  "
      ],
      "execution_count": 154,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kz5YnQZclGfL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "outputId": "37580d2d-9b0d-4e56-be2a-0df44786e2a4"
      },
      "source": [
        "tfidf_vector, word2id = tfidf_vectorizer(preprocessed_docs)\n",
        "print(tfidf_vector)\n",
        "print(word2id.items())"
      ],
      "execution_count": 155,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0.05792358687259491, 0.11584717374518982, 0.05792358687259491, 0.05792358687259491, 0.05792358687259491, 0.15694461266687282, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.13515503603605478, 0.06757751801802739, 0.06757751801802739, 0.06757751801802739, 0.0, 0.1831020481113516, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05792358687259491, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.15694461266687282, 0.15694461266687282, 0.15694461266687282, 0.15694461266687282, 0.15694461266687282, 0.15694461266687282]]\n",
            "dict_items([('like', 0), ('apple', 1), ('strawberry', 2), ('buy', 3), ('tomorrow', 4), ('fresco', 5), ('eat', 6), ('play', 7), ('basketball', 8), ('every', 9), ('day', 10), ('michael', 11), ('jordan', 12)])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WxCJEOjXYSIP",
        "colab_type": "text"
      },
      "source": [
        "### Exercise 6\n",
        "BoWとTF-IDFでコサイン類似度をそれぞれ計算してみよう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QD4nID08s-21",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "463337f2-d590-483e-d1dd-ff9ae15fd73b"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tfidf\n",
            "0-1 similarity: 0.4719198681637555\n",
            "0-2 similarity: 0.03803869439363926\n",
            "1-2 similarity: 0.0\n",
            "bow\n",
            "0-1 similarity: 0.8249579113843053\n",
            "0-2 similarity: 0.1259881576697424\n",
            "1-2 similarity: 0.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6DAU606U6AKc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "359c2788-a3b6-49f9-be2c-a387a3545b82"
      },
      "source": [
        "list1=[0,0,1]\n",
        "list2=[1,2,2]\n",
        "\n",
        "for i,j in zip(list1,list2):\n",
        "  print('{}-{} similarity:'.format(i,j),cosine_similarity(tfidf_vector[i],tfidf_vector[j]))\n",
        "\n",
        "for i,j in zip(list1,list2):\n",
        "  print('{}-{} similarity:'.format(i,j),cosine_similarity(bow_vec[i],bow_vec[j]))"
      ],
      "execution_count": 156,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0-1 similarity: 0.4719198681637555\n",
            "0-2 similarity: 0.03803869439363926\n",
            "1-2 similarity: 0.0\n",
            "0-1 similarity: 0.8249579113843053\n",
            "0-2 similarity: 0.1259881576697424\n",
            "1-2 similarity: 0.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HAUfDUOmJMsp",
        "colab_type": "text"
      },
      "source": [
        "### Option 2\n",
        "scikit-learn, nltk gensimそれぞれにTF-IDFを計算する関数がある  \n",
        "それぞれでTF-IDFを計算してみよう"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f2tZLjHNxRDG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# scikit-learnのtfidf　あとで消す\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7x7nd2-ekIs-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "34a34a2d-2042-4819-81eb-6b31f26858de"
      },
      "source": [
        "#nltk のtf-idf　あとで消す\n",
        "collection = nltk.TextCollection(docs)\n",
        "terms = list(set(collection))\n",
        "nltk_vector = []\n",
        "for doc in docs:\n",
        "  tmp_vec = np.zeros(len(word2id))\n",
        "  for term in word2id.keys():\n",
        "    tmp_vec[word2id[term]] = collection.tf_idf(term, doc)\n",
        "  nltk_vector.append(list(tmp_vec))\n",
        "print(nltk_vector)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0.0, 0.033788759009013694, 0.033788759009013694, 0.033788759009013694, 0.0915510240556758, 0.033788759009013694, 0.033788759009013694, 0.0915510240556758, 0.033788759009013694, 0.033788759009013694, 0.033788759009013694, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.033788759009013694, 0.033788759009013694, 0.0, 0.033788759009013694, 0.033788759009013694, 0.0, 0.033788759009013694, 0.033788759009013694, 0.033788759009013694, 0.0915510240556758, 0.0915510240556758, 0.0915510240556758, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.04505167867868493, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12206803207423442, 0.12206803207423442, 0.12206803207423442, 0.12206803207423442, 0.12206803207423442, 0.12206803207423442]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tEyOY2eQkRMc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "0d93c47b-47ad-4e11-d3fd-62b19e24888e"
      },
      "source": [
        "#gensim tf-idf あとで消す\n",
        "from gensim import corpora\n",
        "from gensim import models\n",
        "\n",
        "dictionary = corpora.Dictionary(docs)\n",
        "print('===単語->idの変換辞書===')\n",
        "print(dictionary.token2id)\n",
        "print(word2id)\n",
        "\n",
        "corpus = list(map(dictionary.doc2bow, docs))\n",
        "test_model = models.TfidfModel(corpus)\n",
        "corpus_tfidf = test_model[corpus]\n",
        "\n",
        "print('===結果表示===')\n",
        "gensim_vector = []\n",
        "for doc in corpus_tfidf:\n",
        "  tmp_vec = [0] * len(word2id)\n",
        "  for word in doc:\n",
        "    key = dictionary[word[0]]\n",
        "    tmp_vec[word2id[key]] = word[1]\n",
        "  gensim_vector.append(tmp_vec)\n",
        "\n",
        "print(gensim_vector)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "===単語->idの変換辞書===\n",
            "{'a': 0, 'an': 1, 'and': 2, 'apple': 3, 'apples': 4, 'buy': 5, 'i': 6, 'like': 7, 'strawberries.': 8, 'tomorrow.': 9, 'will': 10, 'bought': 11, 'eat': 12, 'some': 13, 'basketball': 14, 'day.': 15, 'every': 16, 'jordan.': 17, 'michael': 18, 'play': 19}\n",
            "{'i': 0, 'like': 1, 'apples': 2, 'and': 3, 'a': 4, 'strawberries.': 5, 'will': 6, 'buy': 7, 'an': 8, 'apple': 9, 'tomorrow.': 10, 'bought': 11, 'some': 12, 'eat': 13, 'play': 14, 'basketball': 15, 'every': 16, 'day.': 17, 'michael': 18, 'jordan.': 19}\n",
            "===結果表示===\n",
            "[[0, 0.20996682609546996, 0.20996682609546996, 0.20996682609546996, 0.5689074861149032, 0.20996682609546996, 0.20996682609546996, 0.5689074861149032, 0.20996682609546996, 0.20996682609546996, 0.20996682609546996, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0.185617413417644, 0.185617413417644, 0, 0.185617413417644, 0.185617413417644, 0, 0.185617413417644, 0.185617413417644, 0.185617413417644, 0.5029324775265576, 0.5029324775265576, 0.5029324775265576, 0, 0, 0, 0, 0, 0], [0, 0.1489905855640844, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.40369167389095173, 0.40369167389095173, 0.40369167389095173, 0.40369167389095173, 0.40369167389095173, 0.40369167389095173]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tMjo7F6ZKOW1",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 7\n",
        "\n",
        "様々な国のWikipediaにおけるabstractを取り出したデータセットを用意した  \n",
        "https://drive.google.com/open?id=1i7tekPQRKaAwg-ze3kv5IsufMW13LkLo  \n",
        "このデータをダウンロードして使う  \n",
        "\n",
        "Cosine類似度の計算を行い、Japanに似ている国Top5を表示してみよう  \n",
        "前処理を自分なりに工夫すること  \n",
        "注）類似度はあまり高くならなくても良い  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1IbA30KKElT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 545
        },
        "outputId": "755886e1-6ce4-4547-fcd9-83f8cd760601"
      },
      "source": [
        "df = pd.read_csv(\"./nlp_country.csv\")\n",
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Name</th>\n",
              "      <th>Abstract</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Japan</td>\n",
              "      <td>Japan is an island country in East Asia. Locat...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>United States</td>\n",
              "      <td>The United States of America (USA), commonly k...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>England</td>\n",
              "      <td>England is a country that is part of the Unite...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>China</td>\n",
              "      <td>China, officially the People's Republic of Chi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>India</td>\n",
              "      <td>India, also known as the Republic of India,[19...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Korea</td>\n",
              "      <td>Korea is a region in East Asia.[3] Since 1948 ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Germany</td>\n",
              "      <td>Germany, officially the Federal Republic of Ge...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Russia</td>\n",
              "      <td>Russia, or the Russian Federation[12], is a tr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>France</td>\n",
              "      <td>France, officially the French Republic, is a c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>Italy</td>\n",
              "      <td>Italy, officially the Italian Republic,[10][11...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Brazil</td>\n",
              "      <td>Brazil officially the Federative Republic of B...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Canada</td>\n",
              "      <td>Canada is a country in the northern part of No...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Spain</td>\n",
              "      <td>Spain, officially the Kingdom of Spain[11][a][...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>Australia</td>\n",
              "      <td>Australia, officially the Commonwealth of Aust...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Indonesia</td>\n",
              "      <td>Indonesia, officially the Republic of Indonesi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Mexico</td>\n",
              "      <td>Mexico, officially the United Mexican States (...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             Name                                           Abstract\n",
              "0           Japan  Japan is an island country in East Asia. Locat...\n",
              "1   United States  The United States of America (USA), commonly k...\n",
              "2         England  England is a country that is part of the Unite...\n",
              "3           China  China, officially the People's Republic of Chi...\n",
              "4           India  India, also known as the Republic of India,[19...\n",
              "5           Korea  Korea is a region in East Asia.[3] Since 1948 ...\n",
              "6         Germany  Germany, officially the Federal Republic of Ge...\n",
              "7          Russia  Russia, or the Russian Federation[12], is a tr...\n",
              "8          France  France, officially the French Republic, is a c...\n",
              "9           Italy  Italy, officially the Italian Republic,[10][11...\n",
              "10         Brazil  Brazil officially the Federative Republic of B...\n",
              "11         Canada  Canada is a country in the northern part of No...\n",
              "12          Spain  Spain, officially the Kingdom of Spain[11][a][...\n",
              "13      Australia  Australia, officially the Commonwealth of Aust...\n",
              "14      Indonesia  Indonesia, officially the Republic of Indonesi...\n",
              "15         Mexico  Mexico, officially the United Mexican States (..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 137
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A2ZqrWJhBAO2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "411c6f83-dfa4-4751-b973-1cc1d27e2a9b"
      },
      "source": [
        "df.iloc[0][\"Abstract\"]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Japan is an island country in East Asia. Located in the Pacific Ocean, it lies off the eastern coast of the Asian continent and stretches from the Sea of Okhotsk in the north to the East China Sea and the Philippine Sea in the south. The kanji that make up Japan\\'s name mean \\'sun origin\\', and it is often called the \"Land of the Rising Sun\". Japan is a stratovolcanic archipelago consisting of about 6,852 islands. The four largest are Honshu, Hokkaido, Kyushu, and Shikoku, which make up about ninety-seven percent of Japan\\'s land area and often are referred to as home islands. The country is divided into 47 prefectures in eight regions, with Hokkaido being the northernmost prefecture and Okinawa being the southernmost one. Japan is the 2nd most populous island country. The population of 127 million is the world\\'s eleventh largest, of which 98.5% are ethnic Japanese. 90.7% of people live in cities, while 9.3% live in the countryside.[16] About 13.8 million people live in Tokyo,[17] the capital of Japan. The Greater Tokyo Area is the most populous metropolitan area in the world with over 38 million people.[18] Archaeological research indicates that Japan was inhabited as early as the Upper Paleolithic period. The first written mention of Japan is in Chinese history texts from the 1st century AD. Influence from other regions, mainly China, followed by periods of isolation, particularly from Western Europe, has characterized Japan\\'s history. From the 12th century until 1868, Japan was ruled by successive feudal military shōguns who ruled in the name of the Emperor. Japan entered into a long period of isolation in the early 17th century, which was ended in 1853 when a United States fleet pressured Japan to open to the West. After nearly two decades of internal conflict and insurrection, the Imperial Court regained its political power in 1868 through the help of several clans from Chōshū and Satsuma – and the Empire of Japan was established. In the late 19th and early 20th centuries, victories in the First Sino-Japanese War, the Russo-Japanese War and World War I allowed Japan to expand its empire during a period of increasing militarism. The Second Sino-Japanese War of 1937 expanded into part of World War II in 1941, which came to an end in 1945 following the Japanese surrender. Since adopting its revised constitution on May 3, 1947, during the occupation led by SCAP, the sovereign state of Japan has maintained a unitary parliamentary constitutional monarchy with an Emperor and an elected legislature called the National Diet. Japan is a member of the ASEAN Plus mechanism, UN, the OECD, the G7, the G8, and the G20, and is considered a great power.[19][20][21] Its economy is the world\\'s third-largest by nominal GDP and the fourth-largest by purchasing power parity. It is also the world\\'s fourth-largest exporter and fourth-largest importer. Japan benefits from a highly skilled and educated workforce; it has among the world\\'s largest proportion of citizens holding a tertiary education degree.[22] Although it has officially renounced its right to declare war, Japan maintains a modern military with the world\\'s eighth-largest military budget,[23] used for self-defense and peacekeeping roles; it ranked as the world\\'s fourth most-powerful military in 2015.[24] Japan is a highly developed country with a very high standard of living and Human Development Index. Its population enjoys the highest life expectancy and third lowest infant mortality rate in the world, but is experiencing issues due to an aging population and low birthrate. Japan is renowned for its historical and extensive cinema, influential music industry, anime, video gaming, rich cuisine and its major contributions to science and modern technology.[25][26]'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 138
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UpQzg57nSAT2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 後で消す\n",
        "def preprocessing_text(text):\n",
        "  def cleaning_text(text):\n",
        "    # @の削除\n",
        "    pattern1 = '@|%'\n",
        "    text = re.sub(pattern1, '', text)    \n",
        "    pattern2 = '\\[[0-9 ]*\\]'\n",
        "    text = re.sub(pattern2, '', text)    \n",
        "    # <b>タグの削除\n",
        "    pattern3 = '\\([a-z ]*\\)'\n",
        "    text = re.sub(pattern3, '', text)    \n",
        "    pattern4 = '[0-9]'\n",
        "    text = re.sub(pattern4, '', text)\n",
        "    return text\n",
        "  \n",
        "  def tokenize_text(text):\n",
        "    text = re.sub('[.,]', '', text)\n",
        "    return text.split()\n",
        "\n",
        "  def lemmatize_word(word):\n",
        "    # make words lower  example: Python =>python\n",
        "    word=word.lower()\n",
        "    \n",
        "    # lemmatize  example: cooked=>cook\n",
        "    lemma = wn.morphy(word)\n",
        "    if lemma is None:\n",
        "        return word\n",
        "    else:\n",
        "      return lemma\n",
        "    \n",
        "  text = cleaning_text(text)\n",
        "  tokens = tokenize_text(text)\n",
        "  tokens = [lemmatize_word(word) for word in tokens]\n",
        "  tokens = [remove_stopwords(word, en_stop) for word in tokens]\n",
        "  tokens = [word for word in tokens if word is not None]\n",
        "  return tokens\n",
        "  \n",
        "docs = df[\"Abstract\"].values\n",
        "pp_docs = [preprocessing_text(text) for text in docs]\n",
        "tfidf_vector, word2id = tfidf_vectorizer(pp_docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hNzPIvaxPOv5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "843e5418-5a17-4a1e-86ea-fc8e5e7c7984"
      },
      "source": [
        "word2id.items()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_items([('alibaba', 0), ('group', 1), ('holding', 2), ('limited', 3), ('chinese', 4), ('multinational', 5), ('conglomerate', 6), ('company', 7), ('specialize', 8), ('e-commerce', 9), ('retail', 10), ('internet', 11), ('technology', 12), ('found', 13), ('april', 14), ('provide', 15), ('consumer-to-consumer', 16), ('(cc)', 17), ('business-to-consumer', 18), ('(bc)', 19), ('business-to-business', 20), ('(bb)', 21), ('sales', 22), ('services', 23), ('via', 24), ('web', 25), ('portal', 26), ('well', 27), ('electronic', 28), ('payment', 29), ('shopping', 30), ('search', 31), ('engine', 32), ('cloud', 33), ('computing', 34), ('operate', 35), ('diverse', 36), ('array', 37), ('business', 38), ('around', 39), ('world', 40), ('numerous', 41), ('sector', 42), ('name', 43), ('one', 44), (\"world's\", 45), ('admire', 46), ('fortune', 47), ('closing', 48), ('time', 49), ('date', 50), ('initial', 51), ('public', 52), ('offering', 53), ('(ipo)', 54), ('–', 55), ('us$', 56), ('billion', 57), ('high', 58), ('history', 59), ('september', 60), (\"alibaba's\", 61), ('market', 62), ('value', 63), ('wa', 64), ('december', 65), ('cap', 66), ('stand', 67), ('top', 68), ('valuable', 69), ('big', 70), ('january', 71), ('become', 72), ('second', 73), ('asian', 74), ('break', 75), ('valuation', 76), ('mark', 77), ('competitor', 78), ('tencent', 79), ('ha', 80), ('th', 81), ('global', 82), ('brand', 83), ('large', 84), ('retailer', 85), ('ai', 86), ('venture', 87), ('capital', 88), ('firm', 89), ('investment', 90), ('corporation', 91), ('host', 92), ('bb', 93), ('(alibabacom)', 94), ('cc', 95), ('(taobao)', 96), ('bc', 97), ('(tmall)', 98), ('marketplace', 99), ('online', 100), ('profits', 101), ('surpass', 102), ('us', 103), ('(including', 104), ('walmart', 105), ('amazon', 106), ('ebay)', 107), ('combine', 108), ('since', 109), ('expand', 110), ('medium', 111), ('industry', 112), ('revenue', 113), ('rising', 114), ('triple', 115), ('percentage', 116), ('point', 117), ('year', 118), ('also', 119), ('set', 120), ('record', 121), ('edition', 122), (\"china's\", 123), (\"singles'\", 124), ('day', 125), ('offline', 126), ('amazoncom', 127), ('inc', 128), ('american', 129), ('base', 130), ('seattle', 131), ('washington', 132), ('focus', 133), ('digital', 134), ('streaming', 135), ('artificial', 136), ('intelligence', 137), ('consider', 138), ('four', 139), ('along', 140), ('google', 141), ('apple', 142), ('facebook', 143), ('know', 144), ('disruption', 145), ('well-established', 146), ('technological', 147), ('innovation', 148), ('mass', 149), ('scale', 150), ('assistant', 151), ('provider', 152), ('platform', 153), ('measure', 154), ('capitalization', 155), ('employer', 156), ('unite', 157), ('state', 158), ('jeff', 159), ('bezos', 160), ('july', 161), ('bellevue', 162), ('initially', 163), ('start', 164), ('book', 165), ('later', 166), ('sell', 167), ('electronics', 168), ('software', 169), ('video', 170), ('game', 171), ('apparel', 172), ('furniture', 173), ('food', 174), ('toy', 175), ('jewelry', 176), ('acquire', 177), ('whole', 178), ('$', 179), ('vastly', 180), ('increase', 181), (\"amazon's\", 182), ('presence', 183), ('brick-and-mortar', 184), ('announce', 185), ('two-day', 186), ('delivery', 187), ('service', 188), ('prime', 189), ('million', 190), ('subscriber', 191), ('worldwide', 192), ('distribute', 193), ('download', 194), ('music', 195), ('audiobook', 196), ('audible', 197), ('subsidiary', 198), ('publishing', 199), ('arm', 200), ('film', 201), ('television', 202), ('studio', 203), ('produce', 204), ('consumer', 205), ('include', 206), ('kindle', 207), ('e-readers', 208), ('fire', 209), ('tablet', 210), ('tv', 211), ('echo', 212), ('devices', 213), ('addition', 214), ('ring', 215), ('twitchtv', 216), ('imdb', 217), ('among', 218), ('various', 219), ('controversy', 220), ('criticise', 221), ('poor', 222), ('working', 223), ('conditions', 224), ('tax', 225), ('avoidance', 226), ('anti-competitive', 227), ('practice', 228), ('headquarter', 229), ('cupertino', 230), ('california', 231), ('design', 232), ('develop', 233), ('computer', 234), (\"company's\", 235), ('hardware', 236), ('product', 237), ('iphone', 238), ('smartphone', 239), ('ipad', 240), ('mac', 241), ('personal', 242), ('ipod', 243), ('portable', 244), ('player', 245), ('watch', 246), ('smartwatch', 247), ('airpods', 248), ('wireless', 249), ('earbuds', 250), ('homepod', 251), ('smart', 252), ('speaker', 253), (\"apple's\", 254), ('macos', 255), ('io', 256), ('ipados', 257), ('watchos', 258), ('tvos', 259), ('system', 260), ('itunes', 261), ('safari', 262), ('browser', 263), ('ilife', 264), ('iwork', 265), ('creativity', 266), ('productivity', 267), ('suite', 268), ('professional', 269), ('application', 270), ('like', 271), ('final', 272), ('cut', 273), ('pro', 274), ('logic', 275), ('xcode', 276), ('store', 277), ('app', 278), ('tv+', 279), ('imessage', 280), ('icloud', 281), ('genius', 282), ('bar', 283), ('applecare', 284), ('pay', 285), ('cash', 286), ('card', 287), ('steve', 288), ('job', 289), ('wozniak', 290), ('ronald', 291), ('wayne', 292), (\"wozniak's\", 293), ('though', 294), ('share', 295), ('back', 296), ('within', 297), ('days', 298), ('incorporate', 299), ('ii', 300), ('grow', 301), ('quickly', 302), ('years', 303), ('hire', 304), ('staff', 305), ('designer', 306), ('production', 307), ('line', 308), ('go', 309), ('instant', 310), ('financial', 311), ('success', 312), ('next', 313), ('ship', 314), ('new', 315), ('feature', 316), ('innovative', 317), ('graphical', 318), ('user', 319), ('interface', 320), ('original', 321), ('macintosh', 322), ('marketing', 323), ('advertisement', 324), ('receive', 325), ('widespread', 326), ('critical', 327), ('acclaim', 328), ('however', 329), ('price', 330), ('library', 331), ('cause', 332), ('problem', 333), ('power', 334), ('struggle', 335), ('executive', 336), ('departed', 337), ('amicably', 338), ('remain', 339), ('honorary', 340), ('employee', 341), ('others', 342), ('resign', 343), ('evolve', 344), ('lost', 345), ('lower-priced', 346), ('duopoly', 347), ('microsoft', 348), ('windows', 349), ('intel', 350), ('pc', 351), ('clone', 352), ('board', 353), ('recruit', 354), ('ceo', 355), ('gil', 356), ('amelio', 357), ('would', 358), ('-day', 359), ('charge', 360), ('rehabilitate', 361), ('financially', 362), ('trouble', 363), ('company—reshaping', 364), ('layoff', 365), ('restructure', 366), ('led', 367), ('buy', 368), ('solving', 369), ('desperately', 370), ('fail', 371), ('strategy', 372), ('bringing', 373), ('pensively', 374), ('regain', 375), ('leadership', 376), ('status', 377), ('swiftly', 378), ('return', 379), ('profitability', 380), ('revitalize', 381), ('think', 382), ('different', 383), ('campaign', 384), ('rebuild', 385), ('launching', 386), ('imac', 387), ('opening', 388), ('chain', 389), ('acquiring', 390), ('broaden', 391), ('portfolio', 392), ('rename', 393), ('reflect', 394), ('shift', 395), ('toward', 396), ('launch', 397), ('great', 398), ('august', 399), ('due', 400), ('health', 401), ('complication', 402), ('tim', 403), ('cook', 404), ('two', 405), ('month', 406), ('die', 407), ('marking', 408), ('end', 409), ('era', 410), ('size', 411), ('annual', 412), ('total', 413), ('fiscal', 414), ('third-largest', 415), ('mobile', 416), ('phone', 417), ('manufacturer', 418), ('samsung', 419), ('huawei', 420), ('first', 421), ('trillion', 422), ('employ', 423), ('full-time', 424), ('maintain', 425), ('country', 426), ('actively', 427), ('use', 428), ('level', 429), ('loyalty', 430), ('rank', 431), ('significant', 432), ('criticism', 433), ('regard', 434), ('labor', 435), ('contractor', 436), ('environmental', 437), ('unethical', 438), ('behavior', 439), ('origin', 440), ('source', 441), ('material', 442), ('social', 443), ('network', 444), ('menlo', 445), ('park', 446), ('zuckerberg', 447), ('fellow', 448), ('harvard', 449), ('college', 450), ('student', 451), ('roommate', 452), ('eduardo', 453), ('saverin', 454), ('andrew', 455), ('mccollum', 456), ('dustin', 457), ('moskovitz', 458), ('chris', 459), ('hughes', 460), ('founder', 461), (\"website's\", 462), ('membership', 463), ('subsequently', 464), ('columbia', 465), ('stanford', 466), ('yale', 467), ('eventually', 468), ('ivy', 469), ('league', 470), ('school', 471), ('mit', 472), ('higher', 473), ('education', 474), ('institution', 475), ('boston', 476), ('area', 477), ('university', 478), ('lastly', 479), ('anyone', 480), ('claim', 481), ('least', 482), ('old', 483), ('allow', 484), ('register', 485), ('may', 486), ('vary', 487), ('depend', 488), ('local', 489), ('laws', 490), ('come', 491), ('face', 492), ('directory', 493), ('often', 494), ('given', 495), ('hold', 496), ('february', 497), ('newly', 498), ('list', 499), ('make', 500), ('appear', 501), ('onscreen', 502), (\"users'\", 503), ('news', 504), ('feed', 505), ('access', 506), ('connectivity', 507), ('smartphones', 508), ('create', 509), ('customize', 510), ('profile', 511), ('revealing', 512), ('information', 513), ('post', 514), ('text', 515), ('photo', 516), ('multimedia', 517), ('agree', 518), ('\"friend\"', 519), ('embed', 520), ('apps', 521), ('join', 522), ('common-interest', 523), ('notification', 524), (\"friends'\", 525), ('activity', 526), ('monthly', 527), ('active', 528), ('fake', 529), ('account', 530), ('catch', 531), ('miss', 532), ('real', 533), ('many', 534), ('critic', 535), ('question', 536), ('whether', 537), ('actual', 538), ('prominent', 539), ('coverage', 540), ('involve', 541), ('privacy', 542), ('(as', 543), ('cambridge', 544), ('analytica', 545), ('data', 546), ('scandal)', 547), ('political', 548), ('manipulation', 549), ('elections)', 550), ('psychological', 551), ('effects', 552), ('addiction', 553), ('low', 554), ('self-esteem', 555), ('content', 556), ('find', 557), ('objectionable', 558), ('conspiracy', 559), ('theory', 560), ('copyright', 561), ('infringement', 562), ('doe', 563), ('remove', 564), ('false', 565), ('page', 566), ('bring', 567), ('continuous', 568), ('commentator', 569), ('help', 570), ('spread', 571), ('offer', 572), ('instagram', 573), ('whatsapp', 574), ('oculus', 575), ('grokstyle', 576), ('independently', 577), ('messenger', 578), ('llc', 579), ('internet-related', 580), ('advertising', 581), ('alongside', 582), ('larry', 583), ('sergey', 584), ('brin', 585), ('phd', 586), ('together', 587), ('percent', 588), ('control', 589), ('stockholder', 590), ('voting', 591), ('supervoting', 592), ('stock', 593), ('privately', 594), ('take', 595), ('place', 596), ('move', 597), ('headquarters', 598), ('mountain', 599), ('view', 600), ('nickname', 601), ('googleplex', 602), ('plan', 603), ('reorganize', 604), ('interest', 605), ('call', 606), ('alphabet', 607), (\"alphabet's\", 608), ('leading', 609), ('continue', 610), ('umbrella', 611), ('sundar', 612), ('pichai', 613), ('appoint', 614), ('replacing', 615), ('rapid', 616), ('growth', 617), ('incorporation', 618), ('trigger', 619), ('acquisition', 620), ('partnership', 621), ('beyond', 622), (\"google's\", 623), ('core', 624), ('(google', 625), ('search)', 626), ('work', 627), ('doc', 628), ('sheet', 629), ('slides)', 630), ('email', 631), ('(gmail/inbox)', 632), ('scheduling', 633), ('management', 634), ('calendar)', 635), ('storage', 636), ('drive)', 637), ('messaging', 638), ('chat', 639), ('allo', 640), ('duo', 641), ('hangouts)', 642), ('language', 643), ('translation', 644), ('translate)', 645), ('mapping', 646), ('navigation', 647), ('map', 648), ('waze', 649), ('earth', 650), ('street', 651), ('view)', 652), ('sharing', 653), ('(youtube)', 654), ('note-taking', 655), ('keep)', 656), ('organize', 657), ('editing', 658), ('photos)', 659), ('lead', 660), ('development', 661), ('android', 662), ('chrome', 663), ('os', 664), ('lightweight', 665), ('increasingly', 666), ('hardware;', 667), ('partner', 668), ('major', 669), ('nexus', 670), ('release', 671), ('multiple', 672), ('october', 673), ('pixel', 674), ('home', 675), ('wifi', 676), ('mesh', 677), ('router', 678), ('daydream', 679), ('virtual', 680), ('reality', 681), ('headset', 682), ('experiment', 683), ('carrier', 684), ('fiber', 685), ('fi', 686), ('station)', 687), ('googlecom', 688), ('visit', 689), ('website', 690), ('several', 691), ('figure', 692), ('youtube', 693), ('blogger', 694), ('issue', 695), ('concern', 696), ('antitrust', 697), ('censorship', 698), ('neutrality', 699), ('mission', 700), ('statement', 701), ('\"to', 702), ('universally', 703), ('accessible', 704), ('useful\"', 705), ('unofficial', 706), ('slogan', 707), ('\"don\\'t', 708), ('evil\"', 709), ('code', 710), ('conduct', 711), ('reinstate', 712), ('international', 713), ('machine', 714), ('(ibm)', 715), ('armonk', 716), ('york', 717), ('operations', 718), ('begin', 719), ('endicott', 720), ('computing-tabulating-recording', 721), ('(ctr)', 722), ('\"international', 723), ('machines\"', 724), ('ibm', 725), ('middleware', 726), ('consult', 727), ('range', 728), ('mainframe', 729), ('nanotechnology', 730), ('research', 731), ('organization', 732), ('patent', 733), ('generate', 734), (')', 735), ('consecutive', 736), ('invention', 737), ('automate', 738), ('teller', 739), ('(atm)', 740), ('floppy', 741), ('disk', 742), ('hard', 743), ('drive', 744), ('magnetic', 745), ('stripe', 746), ('relational', 747), ('database', 748), ('sql', 749), ('programming', 750), ('upc', 751), ('barcode', 752), ('dynamic', 753), ('random-access', 754), ('memory', 755), ('(dram)', 756), ('exemplify', 757), ('system/', 758), ('dominant', 759), ('continually', 760), ('focusing', 761), ('higher-value', 762), ('profitable', 763), ('spinning', 764), ('printer', 765), ('lexmark', 766), ('sale', 767), ('(thinkpad/thinkcentre)', 768), ('x-based', 769), ('server', 770), ('lenovo', 771), ('(in', 772), ('respectively)', 773), ('pwc', 774), ('()', 775), ('spss', 776), ('weather', 777), ('red', 778), ('hat', 779), ('(agreement', 780), ('half', 781), ('\"fabless\"', 782), ('semiconductor', 783), ('offload', 784), ('manufacturing', 785), ('globalfoundries', 786), ('blue', 787), ('dow', 788), ('jones', 789), ('industrial', 790), ('average', 791), ('\"ibmers\"', 792), ('ibmers', 793), ('outside', 794), ('number', 795), ('india', 796), ('award', 797), ('five', 798), ('nobel', 799), ('prize', 800), ('six', 801), ('turing', 802), ('ten', 803), ('national', 804), ('medal', 805), ('(usa)', 806), ('science', 807), ('redmond', 808), ('manufacture', 809), ('license', 810), ('support', 811), ('relate', 812), ('best', 813), ('office', 814), ('explorer', 815), ('edge', 816), ('flagship', 817), ('xbox', 818), ('console', 819), ('surface', 820), ('lineup', 821), ('touchscreen', 822), ('maker', 823), ('word', 824), ('\"microsoft\"', 825), ('portmanteau', 826), ('\"microcomputer\"', 827), ('\"software\"', 828), ('ranking', 829), ('bill', 830), ('gates', 831), ('paul', 832), ('allen', 833), ('basic', 834), ('interpreter', 835), ('altair', 836), ('rose', 837), ('dominate', 838), ('ms-dos', 839), ('mid-s', 840), ('follow', 841), ('subsequent', 842), ('rise', 843), ('three', 844), ('billionaire', 845), ('estimate', 846), ('millionaire', 847), ('diversify', 848), ('corporate', 849), ('linkedin', 850), ('skype', 851), ('market-dominant', 852), ('compatible', 853), ('although', 854), ('majority', 855), ('overall', 856), ('wide', 857), ('enterprise', 858), ('desktop', 859), ('laptop', 860), ('tab', 861), ('gadget', 862), ('etc', 863), ('(with', 864), ('bing)', 865), ('(through', 866), ('msn)', 867), ('mix', 868), ('(hololens)', 869), ('(azure)', 870), ('(visual', 871), ('studio)', 872), ('ballmer', 873), ('replace', 874), ('envision', 875), ('\"devices', 876), ('services\"', 877), ('danger', 878), ('entering', 879), ('june', 880), ('computers;', 881), ('form', 882), (\"nokia's\", 883), ('division', 884), ('satya', 885), ('nadella', 886), ('instead', 887), ('reach', 888), ('publicly', 889), ('trade', 890), ('dethrone', 891), ('tech', 892), ('giant', 893), ('third', 894), ('respectively', 895), ('trillion-dollar', 896), ('netflix', 897), ('media-services', 898), ('los', 899), ('gatos', 900), ('reed', 901), ('hastings', 902), ('marc', 903), ('randolph', 904), ('scott', 905), ('valley', 906), ('primary', 907), ('subscription-based', 908), ('ott', 909), ('program', 910), ('in-house', 911), ('subscription', 912), ('free', 913), ('trial', 914), ('available', 915), ('almost', 916), ('except', 917), ('mainland', 918), ('china', 919), ('syria', 920), ('north', 921), ('korea', 922), ('crimea', 923), ('(due', 924), ('sanctions)', 925), ('netherlands', 926), ('brazil', 927), ('japan', 928), ('south', 929), ('member', 930), ('motion', 931), ('picture', 932), ('association', 933), ('america', 934), ('(mpaa)', 935), (\"netflix's\", 936), ('model', 937), ('dvd', 938), ('rental', 939), ('mail', 940), ('abandon', 941), ('founding', 942), ('introduction', 943), ('retain', 944), ('blu-ray', 945), ('internationally', 946), ('canada', 947), ('latin', 948), ('caribbean', 949), ('enter', 950), ('content-production', 951), ('debut', 952), ('series', 953), ('lilyhammer', 954), ('role', 955), ('producer', 956), ('distributor', 957), ('variety', 958), ('\"netflix', 959), ('original\"', 960), ('cable', 961), ('channel', 962), ('effort', 963), ('secure', 964), ('right', 965), ('additional', 966), ('diversity', 967), ('result', 968), ('rack', 969), ('debt:', 970), ('previous', 971), ('long-term', 972), ('debt', 973), ('obligation', 974), ('raise', 975), ('another', 976), ('fund', 977), ('whose', 978), ('specialise', 979), ('entertainment', 980), ('globally', 981), ('twin-skyscrapers', 982), ('seafront', 983), ('tower', 984), ('(also', 985), ('binhai', 986), ('mansion)', 987), ('nanshan', 988), ('district', 989), ('shenzhen', 990), ('gaming', 991), ('multiplayer', 992), ('successful', 993), ('respective', 994), ('category', 995), ('qq', 996), ('qqcom', 997), ('(tencent', 998), ('entertainment)', 999), ('cross', 1000), ('emerge', 1001), (\"asia's\", 1002), ('credit', 1003), ('hundred', 1004), ('associate', 1005), ('broad', 1006), ('ownership', 1007), ('across', 1008), ('estate', 1009), ('ride-sharing', 1010), ('banking', 1011), ('fintech', 1012), ('automobile', 1013), ('movie', 1014), ('ticket', 1015), ('space', 1016), ('natural', 1017), ('resource', 1018), ('agriculture', 1019), ('medical', 1020), ('robotics', 1021), ('uavs', 1022), ('courier', 1023), ('e-book', 1024), ('renewable', 1025), ('energy', 1026), ('stakes', 1027), ('recent', 1028), ('start-ups', 1029), ('asia’s', 1030), ('burgeon', 1031), ('scene', 1032), ('toyota', 1033), ('motor', 1034), ('japanese', 1035), ('automotive', 1036), ('aichi', 1037), (\"toyota's\", 1038), ('structure', 1039), ('consist', 1040), ('sixth-largest', 1041), ('vehicle', 1042), ('per', 1043), ('report', 1044), ('-millionth', 1045), ('(worth', 1046), ('twice', 1047), ('much', 1048), ('-ranked', 1049), ('softbank)', 1050), ('leader', 1051), ('hybrid', 1052), ('electric', 1053), ('encourage', 1054), ('mass-market', 1055), ('adoption', 1056), ('globe', 1057), ('hydrogen', 1058), ('fuel-cell', 1059), ('cumulative', 1060), ('lexus', 1061), ('passenger', 1062), ('car', 1063), ('achieve', 1064), ('milestone', 1065), ('prius', 1066), ('family', 1067), ('selling', 1068), ('nameplate', 1069), ('unit', 1070), ('kiichiro', 1071), ('toyoda', 1072), ('spinoff', 1073), (\"father's\", 1074), ('earlier', 1075), ('still', 1076), ('department', 1077), ('type', 1078), ('aa', 1079), ('hino', 1080), ('ranz', 1081), ('daihatsu', 1082), ('stake', 1083), ('subaru', 1084), ('isuzu', 1085), ('mazda', 1086), ('joint-ventures', 1087), ('(gac', 1088), ('sichuan', 1089), ('faw', 1090), ('motor)', 1091), ('(toyota', 1092), ('kirloskar)', 1093), ('czech', 1094), ('republic', 1095), ('(tpca)', 1096), ('\"nonautomotive\"', 1097), ('tmc', 1098), ('part', 1099), ('london', 1100), ('exchange', 1101), ('tokyo', 1102), ('twitter', 1103), ('interact', 1104), ('message', 1105), ('\"tweets\"', 1106), ('tweet', 1107), ('originally', 1108), ('restrict', 1109), ('character', 1110), ('november', 1111), ('limit', 1112), ('double', 1113), ('korean', 1114), ('retweet', 1115), ('unregistered', 1116), ('read', 1117), ('short', 1118), ('(sms)', 1119), ('mobile-device', 1120), ('(\"app\")', 1121), ('san', 1122), ('francisco', 1123), ('march', 1124), ('jack', 1125), ('dorsey', 1126), ('noah', 1127), ('glass', 1128), ('biz', 1129), ('stone', 1130), ('evan', 1131), ('williams', 1132), ('rapidly', 1133), ('gain', 1134), ('popularity', 1135), ('handle', 1136), ('query', 1137), ('most-visited', 1138), ('describe', 1139), ('\"the', 1140), ('sm', 1141), ('internet\"', 1142), ('hotbed', 1143), ('debate', 1144), ('covering', 1145), ('politics', 1146), ('presidential', 1147), ('election', 1148), ('breaking', 1149), ('election-related', 1150), ('sent', 1151), (':', 1152), ('pm', 1153), ('(eastern', 1154), ('time)', 1155), ('hypermarket', 1156), ('discount', 1157), ('grocery', 1158), ('bentonville', 1159), ('arkansas', 1160), ('sam', 1161), ('walton', 1162), (\"sam's\", 1163), ('club', 1164), ('warehouse', 1165), ('names', 1166), ('de', 1167), ('méxico', 1168), ('centroamérica', 1169), ('mexico', 1170), ('central', 1171), ('asda', 1172), ('kingdom', 1173), ('seiyu', 1174), ('wholly', 1175), ('argentina', 1176), ('chile', 1177), ('africa', 1178), ('minority', 1179), ('brasil', 1180), ('private', 1181), ('equity', 1182), ('advent', 1183), ('revenue—over', 1184), ('accord', 1185), ('—as', 1186), ('family-owned', 1187), (\"walton's\", 1188), ('heir', 1189), ('individual', 1190), (\"walmart's\", 1191), ('terms', 1192), ('geographically', 1193), ('lower', 1194), ('midwest', 1195), ('early', 1196), ('coast', 1197), ('coast:', 1198), ('open', 1199), ('jersey', 1200), ('outlet', 1201), ('lancaster', 1202), ('pennsylvania', 1203), ('main', 1204), ('northeast', 1205), ('see', 1206), ('results:', 1207), ('highly', 1208), ('whereas', 1209), ('germany', 1210), ('yahoo!', 1211), ('sunnyvale', 1212), ('verizon', 1213), ('jerry', 1214), ('yang', 1215), ('david', 1216), ('filo', 1217), ('yahoo', 1218), ('pioneer', 1219), ('finance', 1220), ('answer', 1221), ('fantasy', 1222), ('sport', 1223), ('height', 1224), ('popular', 1225), ('site', 1226), ('third-party', 1227), ('analytics', 1228), ('alexa', 1229), ('similarweb', 1230), ('widely', 1231), ('sixth-most-visited', 1232), ('slowly', 1233), ('decline', 1234), ('starting', 1235), ('late', 1236), ('communications', 1237), (\"yahoo's\", 1238), ('exclude', 1239), ('transfer', 1240), ('successor', 1241), ('altaba', 1242), ('despite', 1243), ('prominence', 1244), ('domain', 1245), ('(commonly', 1246), ('stylize', 1247), ('intel)', 1248), ('santa', 1249), ('clara', 1250), ('silicon', 1251), ('chip', 1252), ('overtake', 1253), ('inventor', 1254), ('x', 1255), ('microprocessor', 1256), ('processor', 1257), ('(pcs)', 1258), ('supply', 1259), ('hp', 1260), ('dell', 1261), ('motherboard', 1262), ('chipsets', 1263), ('controller', 1264), ('integrate', 1265), ('circuit', 1266), ('flash', 1267), ('graphics', 1268), ('chips', 1269), ('robert', 1270), ('noyce', 1271), ('gordon', 1272), ('moore', 1273), ('(of', 1274), (\"moore's\", 1275), ('law)', 1276), ('vision', 1277), ('grove', 1278), ('conceive', 1279), ('words', 1280), ('co-founder', 1281), ('key', 1282), ('fact', 1283), ('\"intel\"', 1284), ('term', 1285), ('appropriate', 1286), ('developer', 1287), ('sram', 1288), ('dram', 1289), ('represent', 1290), ('commercial', 1291), ('(pc)', 1292), ('invest', 1293), ('heavily', 1294), ('fostering', 1295), ('period', 1296), ('supplier', 1297), ('aggressive', 1298), ('tactics', 1299), ('defense', 1300), ('position', 1301), ('particularly', 1302), ('advance', 1303), ('micro', 1304), ('(amd)', 1305), ('direction', 1306), ('center', 1307), ('powertop', 1308), ('latencytop', 1309), ('open-source', 1310), ('project', 1311), ('wayland', 1312), ('mesad', 1313), ('building', 1314), ('block', 1315), ('thread', 1316), ('(tbb)', 1317), ('xen', 1318), ('sony', 1319), ('kōnan', 1320), ('minato', 1321), ('industry[better', 1322), ('needed]', 1323), ('parent', 1324), ('engage', 1325), ('components:', 1326), ('(av', 1327), ('&', 1328), ('communication', 1329), ('business)', 1330), ('(movies', 1331), ('shows)', 1332), ('comprehensive', 1333), ('interactive', 1334), ('sony/atv', 1335), ('leaders', 1336), ('fifth-largest', 1337), ('lg', 1338), ('tcl', 1339), ('hisense', 1340), ('current', 1341), ('former', 1342), ('(–)', 1343), ('likenoother', 1344), ('makebelieve', 1345), ('weak', 1346), ('tie', 1347), ('sumitomo', 1348), ('mitsui', 1349), ('(smfg)', 1350)])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 131
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_p73OT5sPs9y",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "39e8babf-2332-423b-f0d5-343856d826d3"
      },
      "source": [
        "def calc_cosine(vector, vector_list):\n",
        "  result = {}\n",
        "  for i, x in enumerate(vector_list):\n",
        "    result[i] = cosine_similarity(vector, vector_list[i])\n",
        "    \n",
        "  return result\n",
        "\n",
        "print(\"tfidf\")\n",
        "res = calc_cosine(tfidf_vector[0],tfidf_vector)\n",
        "res"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tfidf\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 1.0,\n",
              " 1: 0.04945156965230687,\n",
              " 2: 0.03550026859810149,\n",
              " 3: 0.07494324927746153,\n",
              " 4: 0.02200165046387345,\n",
              " 5: 0.089213868005443,\n",
              " 6: 0.04329186935344452,\n",
              " 7: 0.04340970910393382,\n",
              " 8: 0.050616794433693456,\n",
              " 9: 0.05446867547327852,\n",
              " 10: 0.03479541972998953,\n",
              " 11: 0.03392463518350004,\n",
              " 12: 0.038469390607195876,\n",
              " 13: 0.05035814117836253,\n",
              " 14: 0.06794378321355649,\n",
              " 15: 0.029516361108928312}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 140
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RMGEF5UJUZSz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "f311182a-318b-4dae-bb89-b58693a0d2ec"
      },
      "source": [
        "sorted(res.items(), key=lambda x:x[1],reverse=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(0, 1.0),\n",
              " (5, 0.089213868005443),\n",
              " (3, 0.07494324927746153),\n",
              " (14, 0.06794378321355649),\n",
              " (9, 0.05446867547327852),\n",
              " (8, 0.050616794433693456),\n",
              " (13, 0.05035814117836253),\n",
              " (1, 0.04945156965230687),\n",
              " (7, 0.04340970910393382),\n",
              " (6, 0.04329186935344452),\n",
              " (12, 0.038469390607195876),\n",
              " (2, 0.03550026859810149),\n",
              " (10, 0.03479541972998953),\n",
              " (11, 0.03392463518350004),\n",
              " (15, 0.029516361108928312),\n",
              " (4, 0.02200165046387345)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 141
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9OWm8pnAMQH-",
        "colab_type": "text"
      },
      "source": [
        "## Option 3\n",
        "\n",
        "### Word2Vec & Doc2Vec\n",
        "\n",
        "Word2VecやDoc2Vecでは単語の意味を捉えられているかのような演算が出来る  \n",
        "King - Man + Woman = Queen など  \n",
        "詳細は講義スライドへ   \n",
        "\n",
        "学習済みのword2vecがgithub( https://github.com/Kyubyong/wordvectors )に上がっているので  \n",
        "日本と各国の類似度を計算してみよう  \n",
        "足し算や引き算が出来るのでそれも試してみよう  \n",
        "\n",
        "参考 : \"BOKU\"のITな日常 (https://arakan-pgm-ai.hatenablog.com/entry/2019/02/08/090000)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wWddZ1mZXFsv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}