04_a_Text_Preprocessor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"right\" width=\"400\" src=\"https://www.fhnw.ch/de/++theme++web16theme/assets/media/img/fachhochschule-nordwestschweiz-fhnw-logo.svg\" alt=\"FHNW Logo\">\n",
    "\n",
    "\n",
    "# Text Preprocessor\n",
    "\n",
    "by Fabian Märki\n",
    "\n",
    "## Summary\n",
    "The aim of this notebook is to understand how to write a generic Preprocessor that allows for parallel processing and can be used in a [scikit-learn](https://scikit-learn.org) [Pipeline](https://scikit-learn.org/stable/modules/compose.html). A [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline) is useful to chain individual processing steps into e.g. one classifier. This is convenient since there is often a fixed sequence of steps involved in data processing like cleaning, normalization, tokenization, steaming/lemmantization, vectorization, classification.\n",
    "\n",
    "Preprocessing steps like cleaning, normalization, tokenization, steaming/lemmantization are are often done before model training starts (since these steps need to be done once only whereas model training is an iterative process). The aim of this generic Preprocessor is to allow for a seamless integration of these preprocessing steps into a production pipeline (where these preprocessing steps need to be applied) and thanks to the support for parallel execution also for faster batch processing.\n",
    "\n",
    "## Links\n",
    "- [Text Preprocessing Steps and Universal Reusable Pipeline](https://towardsdatascience.com/text-preprocessing-steps-and-universal-pipeline-94233cb6725a)\n",
    "- [Make Pandas Apply Functions Faster using Parallel Processing](https://towardsdatascience.com/make-your-own-super-pandas-using-multiproc-1c04f41944a1)\n",
    "- [Developer’s Guide](https://scikit-learn.org/stable/developers) and [Developing scikit-learn Estimators](https://scikit-learn.org/stable/developers/develop.html)\n",
    "\n",
    "This notebook contains assigments: <font color='red'>Questions are written in red.</font>\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/markif/2024_FS_CAS_NLP_LAB_Notebooks/blob/master/04_a_Text_Preprocessor.ipynb\">\n",
    "  <img align=\"left\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "!pip install 'fhnw-nlp-utils>=0.8.0,<0.9.0'\n",
    "\n",
    "from fhnw.nlp.utils.processing import is_iterable\n",
    "from fhnw.nlp.utils.storage import download\n",
    "from fhnw.nlp.utils.storage import save_dataframe\n",
    "from fhnw.nlp.utils.storage import load_dataframe\n",
    "\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OS name: posix\n",
      "Platform name: Linux\n",
      "Platform release: 5.13.0-41-generic\n",
      "Python version: 3.8.10\n",
      "CPU cores: 6\n",
      "RAM: 31.13GB total and 21.56GB available\n",
      "Tensorflow version: 2.8.0\n",
      "GPU is available\n",
      "GPU is a NVIDIA GeForce RTX 2070 with Max-Q Design with 8192MiB\n"
     ]
    }
   ],
   "source": [
    "from fhnw.nlp.utils.system import set_log_level\n",
    "from fhnw.nlp.utils.system import system_info\n",
    "\n",
    "set_log_level()\n",
    "print(system_info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "!pip install multiprocess\n",
    "!pip install scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>**TASK: Implement the function `transform` so that it allows for parallel processing.**</font>\n",
    "\n",
    "Possible Inspirations:\n",
    "- [Text Preprocessing Steps and Universal Reusable Pipeline](https://towardsdatascience.com/text-preprocessing-steps-and-universal-pipeline-94233cb6725a) (at the very end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.base import TransformerMixin, BaseEstimator\n",
    "from multiprocess import Pool\n",
    "\n",
    "\n",
    "class TextPreprocessor(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self,\n",
    "                 func,\n",
    "                 field_read,\n",
    "                 field_write,\n",
    "                 n_jobs = -1):\n",
    "        \"\"\"  \n",
    "        func : function\n",
    "            The preprocessing function to apply (e.g. tokenization)\n",
    "        field_read : str\n",
    "            Specifies the data column name to read\n",
    "        field_write : str\n",
    "            Specifies the column name to write the result of applying the preprocessing function\n",
    "        n_jobs : int \n",
    "            Specifies the number of parallel processes to spawn\n",
    "        \"\"\"\n",
    "        self.func = func\n",
    "        self.field_read = field_read\n",
    "        self.field_write = field_write\n",
    "        self.n_jobs = n_jobs\n",
    "\n",
    "    def fit(self, X, y=None):\n",
    "        # nothing to do\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, *_):\n",
    "        n_jobs = self.n_jobs\n",
    "        if n_jobs <= 0:\n",
    "            import psutil\n",
    "            n_jobs = psutil.cpu_count(logical=True)\n",
    "        \n",
    "        # TODO: !!! place your code here !!!\n",
    "        ####################################\n",
    "        # !!! this needs rework !!!\n",
    "        write_df = self.transform_sub_df(X)\n",
    "        \n",
    "        \n",
    "        ###################\n",
    "        # TODO: !!! end !!!\n",
    "        \n",
    "        return pd.concat([X, write_df], axis=1)\n",
    "        \n",
    "    def transform_sub_df(self, df):\n",
    "        # apply the pre-processing function to the cells of the column\n",
    "        series = df[self.field_read].map(\n",
    "            lambda x: self.func(x) if isinstance(x, str) or is_iterable(x) else raise_(TypeError(\"Only string or iterable (e.g. list) is supported. Received a \"+ str(type(x))))\n",
    "        )\n",
    "    \n",
    "        # convert the pd.Series into a pd.DataFrame\n",
    "        return series.to_frame(self.field_write)\n",
    "        \n",
    "    def raise_(ex):\n",
    "        # see https://stackoverflow.com/a/8294654\n",
    "        raise ex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download data and load it into a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(357899, 2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download(\"https://drive.switch.ch/index.php/s/PrVg8Ugfc2jDwT3/download\", \"data/german_doctor_reviews_original.parq\")\n",
    "data = load_dataframe(\"data/german_doctor_reviews_original.parq\")\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0\n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0\n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try a simple use case - tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "def tokenize(text):\n",
    "    \"\"\"Tokenizes a text\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text : str, iterable\n",
    "        The text either as string or iterable of tokens (in this case tokenization is not applied)\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    list\n",
    "        The tokenized text\n",
    "    \"\"\"\n",
    "    \n",
    "    if isinstance(text, str):\n",
    "        word_tokens = word_tokenize(text)\n",
    "    else:\n",
    "        raise TypeError(\"Only string is supported. Received a \"+ str(type(text)))\n",
    "\n",
    "    return word_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    }
   ],
   "source": [
    "%%capture\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "\n",
    "stopwords = set(stopwords.words(\"german\"))\n",
    "empty_stopwords = set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how long it takes on a single core..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.09 s, sys: 1.39 s, total: 5.48 s\n",
      "Wall time: 36.2 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[Ich, bin, franzose, und, bin, seit, ein, paar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[Dieser, Arzt, ist, das, unmöglichste, was, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Hatte, akute, Beschwerden, am, Rücken, ., Her...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [Ich, bin, franzose, und, bin, seit, ein, paar...  \n",
       "1  [Dieser, Arzt, ist, das, unmöglichste, was, mi...  \n",
       "2  [Hatte, akute, Beschwerden, am, Rücken, ., Her...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "preprocessor = TextPreprocessor(func=tokenize, field_read=\"text_original\", field_write=\"tokenized\", n_jobs=1)\n",
    "data_test = data.head(50000)\n",
    "data_test = preprocessor.transform(data_test)\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how long it takes on all available cores..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n"
     ]
    }
   ],
   "source": [
    "import psutil\n",
    "\n",
    "n_jobs = psutil.cpu_count(logical=False)\n",
    "print(n_jobs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 730 ms, sys: 227 ms, total: 956 ms\n",
      "Wall time: 6.98 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[Ich, bin, franzose, und, bin, seit, ein, paar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[Dieser, Arzt, ist, das, unmöglichste, was, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Hatte, akute, Beschwerden, am, Rücken, ., Her...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [Ich, bin, franzose, und, bin, seit, ein, paar...  \n",
       "1  [Dieser, Arzt, ist, das, unmöglichste, was, mi...  \n",
       "2  [Hatte, akute, Beschwerden, am, Rücken, ., Her...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "preprocessor = TextPreprocessor(func=tokenize, field_read=\"text_original\", field_write=\"tokenized\", n_jobs=n_jobs)\n",
    "data_test = data.head(50000)\n",
    "data_test = preprocessor.transform(data_test)\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a speedup if we take hyperthreading into account..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n"
     ]
    }
   ],
   "source": [
    "import psutil\n",
    "\n",
    "n_jobs = psutil.cpu_count(logical=True)\n",
    "print(n_jobs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 951 ms, sys: 504 ms, total: 1.46 s\n",
      "Wall time: 5.33 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[Ich, bin, franzose, und, bin, seit, ein, paar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[Dieser, Arzt, ist, das, unmöglichste, was, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Hatte, akute, Beschwerden, am, Rücken, ., Her...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [Ich, bin, franzose, und, bin, seit, ein, paar...  \n",
       "1  [Dieser, Arzt, ist, das, unmöglichste, was, mi...  \n",
       "2  [Hatte, akute, Beschwerden, am, Rücken, ., Her...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "preprocessor = TextPreprocessor(func=tokenize, field_read=\"text_original\", field_write=\"tokenized\", n_jobs=n_jobs)\n",
    "data_test = data.head(50000)\n",
    "data_test = preprocessor.transform(data_test)\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fhnw-nlp-utils\n",
    "\n",
    "**Preprocessor** and **parallelize_dataframe** are available through **fhnw-nlp-utils**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are two functions that preprocess text (as placeholder for any other text preprocessing function).\n",
    "The function **tokenize** only access one field of a row (i.e. it processes a single columns) whereas **tokenize_by_row** could potentially access all fields of a row (as an example of a more advanced function that needs to access several columns (per row) for its computation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text, stopwords):\n",
    "    \"\"\"Tokenizes a text and removes stopwords\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text : str, iterable\n",
    "        The text either as string or iterable of tokens (in this case tokenization is not applied)\n",
    "    stopwords : set\n",
    "        A set of stopword to remove from the tokens\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    list\n",
    "        The tokenized text\n",
    "    \"\"\"\n",
    "    from nltk.tokenize import word_tokenize\n",
    "    from fhnw.nlp.utils.processing import is_iterable\n",
    "\n",
    "    if isinstance(text, str):\n",
    "        word_tokens = word_tokenize(text)\n",
    "    elif is_iterable(text):\n",
    "        word_tokens = text\n",
    "    else:\n",
    "        raise TypeError(\"Only string or iterable (e.g. list) is supported. Received a \"+ str(type(text)))\n",
    "\n",
    "    return [word.lower() for word in word_tokens if word.lower() not in stopwords]\n",
    "\n",
    "def tokenize_by_row(row, stopwords):\n",
    "    \"\"\"Tokenizes a text and removes stopwords\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    row : pd.Series\n",
    "        A row of the pandas Dataframe\n",
    "    stopwords : set\n",
    "        A set of stopword to remove from the tokens\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    list\n",
    "        The tokenized text\n",
    "    \"\"\"\n",
    "    from nltk.tokenize import word_tokenize\n",
    "\n",
    "    word_tokens = word_tokenize(row[\"text_original\"])\n",
    "    return [word.lower() for word in word_tokens if word.lower() not in stopwords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this works..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first example uses a function (i.e. **tokenize**) that accesses a single field of a row (i.e. one column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fhnw.nlp.utils.processing import parallelize_dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 459 ms, sys: 238 ms, total: 696 ms\n",
      "Wall time: 5.57 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[franzose, seit, paar, wochen, muenchen, ., za...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[arzt, unmöglichste, leben, je, begegnet, ,, u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[akute, beschwerden, rücken, ., herr, magura, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [franzose, seit, paar, wochen, muenchen, ., za...  \n",
       "1  [arzt, unmöglichste, leben, je, begegnet, ,, u...  \n",
       "2  [akute, beschwerden, rücken, ., herr, magura, ...  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "data_test = data.head(50000)\n",
    "data_test = parallelize_dataframe(data_test, tokenize, stopwords=stopwords, field_read=\"text_original\", field_write=\"tokenized\")\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second example uses a function (i.e. **tokenize_by_row**) that potentially could access several fields of a row (note that the parameter *field_read* is not provided in this example which acts as the switch between the *single field* and the *complete row* mode)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 644 ms, sys: 297 ms, total: 940 ms\n",
      "Wall time: 6.64 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[ich, bin, franzose, und, bin, seit, ein, paar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[dieser, arzt, ist, das, unmöglichste, was, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[hatte, akute, beschwerden, am, rücken, ., her...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [ich, bin, franzose, und, bin, seit, ein, paar...  \n",
       "1  [dieser, arzt, ist, das, unmöglichste, was, mi...  \n",
       "2  [hatte, akute, beschwerden, am, rücken, ., her...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "data_test = data.head(50000)\n",
    "data_test = parallelize_dataframe(data_test, tokenize_by_row, stopwords=stopwords, field_write=\"tokenized\")\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrap this into a generic Preprocessor *Estimator* that can be used in a scikit-learn Pipeline. \n",
    "\n",
    "During inference (i.e. running live data in production) it can be handy to build a complete *processing pipeline*. This generic Preprocessor allows to re-use the same functions/setup as it was used during training without further modifications (simple to use and less error prone)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fhnw.nlp.utils.processing import Preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 906 ms, sys: 360 ms, total: 1.27 s\n",
      "Wall time: 7.21 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[franzose, seit, paar, wochen, muenchen, ., za...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[arzt, unmöglichste, leben, je, begegnet, ,, u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[akute, beschwerden, rücken, ., herr, magura, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [franzose, seit, paar, wochen, muenchen, ., za...  \n",
       "1  [arzt, unmöglichste, leben, je, begegnet, ,, u...  \n",
       "2  [akute, beschwerden, rücken, ., herr, magura, ...  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "preprocessor = Preprocessor(tokenize, stopwords=stopwords, field_read=\"text_original\", field_write=\"tokenized\")\n",
    "data_test = data.head(50000)\n",
    "data_test = preprocessor.transform(data_test)\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 601 ms, sys: 312 ms, total: 914 ms\n",
      "Wall time: 6.61 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_original</th>\n",
       "      <th>rating</th>\n",
       "      <th>tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[ich, bin, franzose, und, bin, seit, ein, paar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in me...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[dieser, arzt, ist, das, unmöglichste, was, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[hatte, akute, beschwerden, am, rücken, ., her...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       text_original  rating  \\\n",
       "0  Ich bin franzose und bin seit ein paar Wochen ...     2.0   \n",
       "1  Dieser Arzt ist das unmöglichste was mir in me...     6.0   \n",
       "2  Hatte akute Beschwerden am Rücken. Herr Magura...     1.0   \n",
       "\n",
       "                                           tokenized  \n",
       "0  [ich, bin, franzose, und, bin, seit, ein, paar...  \n",
       "1  [dieser, arzt, ist, das, unmöglichste, was, mi...  \n",
       "2  [hatte, akute, beschwerden, am, rücken, ., her...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "preprocessor = Preprocessor(tokenize_by_row, stopwords=stopwords, field_write=\"tokenized\")\n",
    "data_test = data.head(50000)\n",
    "data_test = preprocessor.transform(data_test)\n",
    "data_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}