02_a_Text_Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"right\" width=\"400\" src=\"https://www.fhnw.ch/de/++theme++web16theme/assets/media/img/fachhochschule-nordwestschweiz-fhnw-logo.svg\" alt=\"FHNW Logo\">\n",
    "\n",
    "\n",
    "# Text Cleaning\n",
    "\n",
    "by Fabian Märki\n",
    "\n",
    "## Summary\n",
    "The aim of this notebook is to show how text can be cleaned in order to prepare it as training data for a downstream nlp task. Additionally, it uses language detection to identify text that is written in an unexpected language (in this case other than german) so it can be removed from the data (since the trained model is supposed to work on german text).\n",
    "\n",
    "This notebook contains assigments: <font color='red'>Questions are written in red.</font>\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/markif/2024_FS_CAS_NLP_LAB_Notebooks/blob/master/02_a_Text_Cleaning.ipynb\">\n",
    "  <img align=\"left\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "!pip install 'fhnw-nlp-utils>=0.8.0,<0.9.0'\n",
    "\n",
    "from fhnw.nlp.utils.processing import parallelize_dataframe\n",
    "from fhnw.nlp.utils.processing import is_iterable\n",
    "from fhnw.nlp.utils.storage import download\n",
    "from fhnw.nlp.utils.storage import save_dataframe\n",
    "from fhnw.nlp.utils.storage import load_dataframe\n",
    "\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OS name: posix\n",
      "Platform name: Linux\n",
      "Platform release: 5.15.0-56-generic\n",
      "Python version: 3.8.10\n",
      "CPU cores: 6\n",
      "RAM: 31.12GB total and 23.35GB available\n",
      "Tensorflow version: 2.10.0\n",
      "GPU is available\n",
      "GPU is a NVIDIA GeForce RTX 2070 with Max-Q Design with 8192MiB\n"
     ]
    }
   ],
   "source": [
    "from fhnw.nlp.utils.system import set_log_level\n",
    "from fhnw.nlp.utils.system import system_info\n",
    "\n",
    "set_log_level()\n",
    "print(system_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download data and load it into a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "download(\"https://drive.switch.ch/index.php/s/PrVg8Ugfc2jDwT3/download\", \"data/german_doctor_reviews_original.parq\")\n",
    "data = load_dataframe(\"data/german_doctor_reviews_original.parq\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get a first impression..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(357899, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0\n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0\n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(data.shape)\n",
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check how many empty cells each column has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text_original 0\n",
      "rating 7658\n"
     ]
    }
   ],
   "source": [
    "for col in data.columns:\n",
    "    print(col, data[col].isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop all empty rows (in some situations in can also make sense to amend the data - training data is valuable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>**TASK: Drop all rows with an empty cell and store the result in `data`.**</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(350241, 2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO: !!! place your code here !!!\n",
    "####################################\n",
    "\n",
    "\n",
    "###################\n",
    "# TODO: !!! end !!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if text contains HTML tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Frau Dr. Vetter kenne ich seit vielen Jahren, ...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bei der Ärztin selbst fühle ich micht stets gu...</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Bin seit Januar bei Dr. Henning in Behandlung ...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1. Termin:&lt;br /&gt;\\n1 Stunde Wartezimmer + 2 min...</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357886</th>\n",
       "      <td>Vor paar Tagen hatte ich eine Ultraschallunter...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357887</th>\n",
       "      <td>Unglaublich unfreundliche Praxis.&lt;br /&gt;\\nDie S...</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357890</th>\n",
       "      <td>Ich war einige Jahre lang Patientin bei Dr. Ge...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357897</th>\n",
       "      <td>Ich bin schon länger bei Frau Dr. Sonntag in B...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357898</th>\n",
       "      <td>Ich habe lange gesucht nach einem Hautarzt, de...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>126625 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            text_original  rating\n",
       "1       Dieser Arzt ist das unmöglichste was mir in me...     6.0\n",
       "4       Frau Dr. Vetter kenne ich seit vielen Jahren, ...     1.0\n",
       "10      Bei der Ärztin selbst fühle ich micht stets gu...     2.0\n",
       "12      Bin seit Januar bei Dr. Henning in Behandlung ...     1.0\n",
       "13      1. Termin:<br />\\n1 Stunde Wartezimmer + 2 min...     6.0\n",
       "...                                                   ...     ...\n",
       "357886  Vor paar Tagen hatte ich eine Ultraschallunter...     4.0\n",
       "357887  Unglaublich unfreundliche Praxis.<br />\\nDie S...     5.0\n",
       "357890  Ich war einige Jahre lang Patientin bei Dr. Ge...     4.0\n",
       "357897  Ich bin schon länger bei Frau Dr. Sonntag in B...     4.0\n",
       "357898  Ich habe lange gesucht nach einem Hautarzt, de...     1.0\n",
       "\n",
       "[126625 rows x 2 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data[\"text_original\"].str.contains(\"/>\", na=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems the text needs cleaning (remove html tags etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>**TASK: Implement `clean_text` and provide the functionality as described in the function documentation.**</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hints:\n",
    "- focus on one functionality at a time\n",
    "- you do not need to implement all functionalities (there is a reference solution)\n",
    "\n",
    "Possible Inspirations:\n",
    "- [Text Data Preprocessing: A Walkthrough in Python](https://www.kdnuggets.com/2018/03/text-data-preprocessing-walkthrough-python.html)\n",
    "- [Beautiful Soup](https://beautiful-soup-4.readthedocs.io/en/latest) for html tag removal (see also [here](https://stackoverflow.com/a/12982689))\n",
    "\n",
    "Possible Improvements:\n",
    "- Replace numbers with a special token\n",
    "- Replace ä, ö, ü and ß with ae, oe, ue and ss (see [here](https://jdvala.github.io/blog.io/thesis/2018/05/11/German-Preprocessing.html))\n",
    "- Keep specific word forms only (e.g. all nouns and adjectives - see also [here](https://stackoverflow.com/a/15278298))\n",
    "- German has many compound words like *Behandlungsdauer*, *Behandlungsempfehlung*, *Behandlungserfolg*, *Behandlungsergebnis* etc. which count for different words (and are thus considered unrelated) in a bag-of-words (BOW) model. Splitting such words could bring some improvement (e.g. see [here](https://github.com/dtuggener/CharSplit) and [here](https://github.com/TimKam/compound-word-splitter))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_text(text, keep_punctuation=False):\n",
    "    \"\"\"Cleans text by removing html tags, non ascii chars, digits and optionally punctuation\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text : str\n",
    "        The text to clean\n",
    "    keep_punctuation : bool\n",
    "        Defines if punctuation should be kept\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    str\n",
    "        The cleaned text\n",
    "    \"\"\"\n",
    "        \n",
    "    # TODO: !!! place your code here !!!\n",
    "    ####################################\n",
    "    \n",
    "    \n",
    "    ###################\n",
    "    # TODO: !!! end !!!\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ein Arzt \n",
      "Ein Arzt.\n",
      "Ein Arzt \n",
      "Ein Arzt \n",
      "Ein Ârzt \n"
     ]
    }
   ],
   "source": [
    "print(clean_text(\"Ein Arzt.\"))\n",
    "print(clean_text(\"Ein Arzt.\", keep_punctuation=True))\n",
    "print(clean_text(\"Ein <br> Arzt.\"))\n",
    "print(clean_text(\"Ein 0815 Arzt.\"))\n",
    "print(clean_text(\"Ein Ârzt भारत\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean the text in *text_original* and store it in *text*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.68 s, sys: 386 ms, total: 2.07 s\n",
      "Wall time: 4.37 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# drop in case of re-execution\n",
    "data = data.drop([\"text\"], axis=1, errors='ignore')\n",
    "data = parallelize_dataframe(data, clean_text, field_read=\"text_original\", field_write=\"text\", keep_punctuation=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double check... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [text_original, rating, text]\n",
       "Index: []"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data[\"text\"].str.contains(\"/>\", na=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for text without characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>510</th>\n",
       "      <td>.......,,,,,,,,,,,,,,,,,,,,,,</td>\n",
       "      <td>1.0</td>\n",
       "      <td>.......,,,,,,,,,,,,,,,,,,,,,,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4763</th>\n",
       "      <td>----------------------------------------------...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4928</th>\n",
       "      <td>1111111111111111111111111111111111111111111111...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7489</th>\n",
       "      <td>.</td>\n",
       "      <td>1.0</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17622</th>\n",
       "      <td>.................................................</td>\n",
       "      <td>5.0</td>\n",
       "      <td>.................................................</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61490</th>\n",
       "      <td>.................................................</td>\n",
       "      <td>1.0</td>\n",
       "      <td>.................................................</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204685</th>\n",
       "      <td>:-)</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>305991</th>\n",
       "      <td>... ... ... ... ... ... ... ... ... ... ... .....</td>\n",
       "      <td>1.0</td>\n",
       "      <td>... ... ... ... ... ... ... ... ... ... ... .....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309063</th>\n",
       "      <td>....!</td>\n",
       "      <td>5.0</td>\n",
       "      <td>....!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            text_original  rating  \\\n",
       "510                         .......,,,,,,,,,,,,,,,,,,,,,,     1.0   \n",
       "4763    ----------------------------------------------...     3.0   \n",
       "4928    1111111111111111111111111111111111111111111111...     1.0   \n",
       "7489                                                    .     1.0   \n",
       "17622   .................................................     5.0   \n",
       "61490   .................................................     1.0   \n",
       "204685                                                :-)     1.0   \n",
       "305991  ... ... ... ... ... ... ... ... ... ... ... .....     1.0   \n",
       "309063                                              ....!     5.0   \n",
       "\n",
       "                                                     text  \n",
       "510                         .......,,,,,,,,,,,,,,,,,,,,,,  \n",
       "4763                                                       \n",
       "4928                                                       \n",
       "7489                                                    .  \n",
       "17622   .................................................  \n",
       "61490   .................................................  \n",
       "204685                                                     \n",
       "305991  ... ... ... ... ... ... ... ... ... ... ... .....  \n",
       "309063                                              ....!  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[~data[\"text\"].str.contains(\"[A-Za-z]\", na=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>**TASK: Drop all rows identified above and store the result in `data`.**</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(350232, 3)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO: !!! place your code here !!!\n",
    "####################################\n",
    "\n",
    "\n",
    "###################\n",
    "# TODO: !!! end !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double check... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [text_original, rating, text]\n",
       "Index: []"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[~data[\"text\"].str.contains(\"[A-Za-z]\", na=False)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We analyze german text. Let's try to identify non-german text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>**TASK: Use fastText to implement `predict_lang` and predict the language of the provided text.**</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Possible Inspirations:\n",
    "- [Identify text language](https://amitness.com/2019/07/identify-text-language-python)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: fasttext in /usr/local/lib/python3.8/dist-packages (0.9.2)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from fasttext) (1.22.1)\n",
      "Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from fasttext) (60.7.0)\n",
      "Requirement already satisfied: pybind11>=2.2 in /usr/local/lib/python3.8/dist-packages (from fasttext) (2.9.2)\n",
      "\u001b[33mWARNING: You are using pip version 20.2.4; however, version 22.1 is available.\n",
      "You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.\n"
     ]
    }
   ],
   "source": [
    "!pip install fasttext\n",
    "\n",
    "import fasttext\n",
    "\n",
    "pretrained_model = \"fasttext/supervised-models/lid.176.ftz\"\n",
    "download(url=\"https://dl.fbaipublicfiles.com/\"+pretrained_model, path = pretrained_model)\n",
    "model = fasttext.load_model(pretrained_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_lang(text):\n",
    "    \"\"\"Predicts the language of a sentence\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text : str\n",
    "        The text to predict the language\n",
    "    model: fasttext model\n",
    "        Fasttext model to predict the language\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    str\n",
    "        The predicted language (e.g. en, de, fr, it, es, ru, ...)\n",
    "    \"\"\"\n",
    "    \n",
    "    # TODO: !!! place your code here !!!\n",
    "    ####################################\n",
    "    # !!! this needs rework !!!\n",
    "    return \"en\"\n",
    "\n",
    "    ###################\n",
    "    # TODO: !!! end !!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.47 s, sys: 352 ms, total: 1.83 s\n",
      "Wall time: 6.51 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = parallelize_dataframe(data, predict_lang, n_jobs=6, field_read=\"text\", field_write=\"lang\")\n",
    "\n",
    "# In case you get an error message \"NameError: name 'model' is not defined\" on your local machine\n",
    "# you can also execute the above code as follows (without parallelization)\n",
    "# data = pd.concat([data, data[\"text\"].map(predict_lang).to_frame(\"lang\")], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>text</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>de</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>de</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>de</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                                text lang  \n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...   de  \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...   de  \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...   de  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "de    349901\n",
       "en       238\n",
       "pl        17\n",
       "nl        15\n",
       "it         9\n",
       "es         7\n",
       "sv         6\n",
       "fi         4\n",
       "hu         4\n",
       "al         3\n",
       "da         3\n",
       "fr         3\n",
       "eu         2\n",
       "ro         2\n",
       "jb         2\n",
       "sr         2\n",
       "ru         2\n",
       "la         2\n",
       "pt         2\n",
       "gl         1\n",
       "ba         1\n",
       "oc         1\n",
       "no         1\n",
       "nd         1\n",
       "sl         1\n",
       "tr         1\n",
       "ko         1\n",
       "Name: lang, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get a summary of all predicted languages\n",
    "data[\"lang\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>text</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Dieser Mann weiss was er tut.</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dieser Mann weiss was er tut.</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134</th>\n",
       "      <td>HERVORHAGEND::::::::::::::::::::::::::::::::::...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>HERVORHAGEND</td>\n",
       "      <td>hu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021</th>\n",
       "      <td>Bisher i.O.</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Bisher .</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2827</th>\n",
       "      <td>Katastrophe .........</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Katastrophe .........</td>\n",
       "      <td>fr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3060</th>\n",
       "      <td>Super Kerl!!!! Super Praxisteam!!!! Super Prop...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Super Kerl!!!! Super Praxisteam!!!! Super Prop...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351661</th>\n",
       "      <td>Perfect doctor.Very competent and friendly</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Perfect doctor Very competent and friendly</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>355577</th>\n",
       "      <td>Sehr Gut !</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Sehr Gut !</td>\n",
       "      <td>sr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356190</th>\n",
       "      <td>Dr Dudik is the best gynecologist I have had. ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dr Dudik is the best gynecologist have had. An...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356684</th>\n",
       "      <td>He is very good and calm. Took care very well</td>\n",
       "      <td>1.0</td>\n",
       "      <td>He is very good and calm. Took care very well</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357099</th>\n",
       "      <td>Gut</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gut</td>\n",
       "      <td>sr</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>331 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            text_original  rating  \\\n",
       "5                           Dieser Mann weiss was er tut.     1.0   \n",
       "1134    HERVORHAGEND::::::::::::::::::::::::::::::::::...     1.0   \n",
       "2021                                          Bisher i.O.     2.0   \n",
       "2827                                Katastrophe .........     6.0   \n",
       "3060    Super Kerl!!!! Super Praxisteam!!!! Super Prop...     1.0   \n",
       "...                                                   ...     ...   \n",
       "351661         Perfect doctor.Very competent and friendly     1.0   \n",
       "355577                                         Sehr Gut !     1.0   \n",
       "356190  Dr Dudik is the best gynecologist I have had. ...     1.0   \n",
       "356684      He is very good and calm. Took care very well     1.0   \n",
       "357099                                                Gut     1.0   \n",
       "\n",
       "                                                     text lang  \n",
       "5                           Dieser Mann weiss was er tut.   en  \n",
       "1134                                        HERVORHAGEND    hu  \n",
       "2021                                             Bisher .   en  \n",
       "2827                                Katastrophe .........   fr  \n",
       "3060    Super Kerl!!!! Super Praxisteam!!!! Super Prop...   en  \n",
       "...                                                   ...  ...  \n",
       "351661         Perfect doctor Very competent and friendly   en  \n",
       "355577                                         Sehr Gut !   sr  \n",
       "356190  Dr Dudik is the best gynecologist have had. An...   en  \n",
       "356684      He is very good and calm. Took care very well   en  \n",
       "357099                                                Gut   sr  \n",
       "\n",
       "[331 rows x 4 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#pd.options.display.max_rows = None\n",
    "data.loc[data[\"lang\"] != \"de\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check each individual row to see if it really contains non german text. Alternatively, you might just want to delete all non german predicted rows (but remember, training data is valuable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# handcrafted\n",
    "delete_rows = [23666,32836,36789,37056,38863,41619,52122,59717,69225,70454,71649,75865,82514,89963,91433,93172,94829,94897,94965,101568,108747,115867,118969,120108,136188,137354,138723,143134,144365,146786,146971,148011,151936,154651,156848,157620,159678,161340,161340,174741,175238,179085,181464,181997,183575,185626,187205,187743,188892,191245,191729,192939,195947,196182,201934,203209,204340,207086,209502,213146,216070,216527,220184,221750,229629,230812,230927,231132,232964,235550,237095,237743,237932,238321,239181,242175,242671,243060,243227,250396,253465,253790,253819,254323,254760,255987,257958,260406,260446,266847,268197,270367,270374,270815,271952,272747,272836,273203,273924,275617,277243,279420,279477,280796,281613,281673,286529,287505,287768,287890,289947,293835,297010,298277,302994,303077,303479,312723,340714,341344,342168,342185,342222,342257,342369,342830,343944,344666,345929,347115,347539,348001,348727,351661,356190,356684]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>text</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23666</th>\n",
       "      <td>Simply the best</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Simply the best</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32836</th>\n",
       "      <td>Ausprobieren! Beauty at its best.</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Ausprobieren! Beauty at its best.</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36789</th>\n",
       "      <td>Tooooooooooooop</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Tooooooooooooop</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37056</th>\n",
       "      <td>Exelente doctora.</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Exelente doctora.</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38863</th>\n",
       "      <td>Dr. Rösener has a professional staff, modern p...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dr. Rösener has professional staff, modern pra...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348001</th>\n",
       "      <td>Nothing I can complain about. Explanations wer...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Nothing can complain about. Explanations were ...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348727</th>\n",
       "      <td>Both doctor Kettner and doctor Harz are profes...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Both doctor Kettner and doctor Harz are profes...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351661</th>\n",
       "      <td>Perfect doctor.Very competent and friendly</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Perfect doctor Very competent and friendly</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356190</th>\n",
       "      <td>Dr Dudik is the best gynecologist I have had. ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dr Dudik is the best gynecologist have had. An...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356684</th>\n",
       "      <td>He is very good and calm. Took care very well</td>\n",
       "      <td>1.0</td>\n",
       "      <td>He is very good and calm. Took care very well</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>136 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            text_original  rating  \\\n",
       "23666                                     Simply the best     1.0   \n",
       "32836                   Ausprobieren! Beauty at its best.     1.0   \n",
       "36789                                     Tooooooooooooop     1.0   \n",
       "37056                                   Exelente doctora.     1.0   \n",
       "38863   Dr. Rösener has a professional staff, modern p...     1.0   \n",
       "...                                                   ...     ...   \n",
       "348001  Nothing I can complain about. Explanations wer...     1.0   \n",
       "348727  Both doctor Kettner and doctor Harz are profes...     1.0   \n",
       "351661         Perfect doctor.Very competent and friendly     1.0   \n",
       "356190  Dr Dudik is the best gynecologist I have had. ...     1.0   \n",
       "356684      He is very good and calm. Took care very well     1.0   \n",
       "\n",
       "                                                     text lang  \n",
       "23666                                     Simply the best   en  \n",
       "32836                   Ausprobieren! Beauty at its best.   en  \n",
       "36789                                     Tooooooooooooop   en  \n",
       "37056                                   Exelente doctora.   pt  \n",
       "38863   Dr. Rösener has professional staff, modern pra...   en  \n",
       "...                                                   ...  ...  \n",
       "348001  Nothing can complain about. Explanations were ...   en  \n",
       "348727  Both doctor Kettner and doctor Harz are profes...   en  \n",
       "351661         Perfect doctor Very competent and friendly   en  \n",
       "356190  Dr Dudik is the best gynecologist have had. An...   en  \n",
       "356684      He is very good and calm. Took care very well   en  \n",
       "\n",
       "[136 rows x 4 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[delete_rows]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(350097, 3)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data.drop(delete_rows)\n",
    "data = data.drop(columns=[\"lang\"])\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 21.9 s, sys: 324 ms, total: 22.2 s\n",
      "Wall time: 22.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "save_dataframe(data, \"data/german_doctor_reviews_cleaned.parq\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}