Sklearn Basics 2 - Train a Classifier on a Star Multi-Table Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sklearn Basics 2: Train a Classifier on a Star Multi-Table Dataset\n",
    "\n",
    "In this notebook, we will learn how to train a classifier with a multi-table data composed of two tables (a root table and a secondary table). It is highly recommended to see the _Sklearn Basics 1_ lesson if you are not familiar with Khiops' sklearn estimators.\n",
    "\n",
    "\n",
    "We start by importing the sklearn estimator `KhiopsClassifier`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from khiops import core as kh\n",
    "from khiops.sklearn import KhiopsClassifier\n",
    "from khiops.utils.helpers import train_test_split_dataset\n",
    "from sklearn import metrics\n",
    "\n",
    "# If there are any issues you may Khiops status with the following command\n",
    "# kh.get_runner().print_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a Multi-Table Classifier\n",
    "We'll train a \"sarcasm detector\" using the dataset `HeadlineSarcasm`. In its raw form, the dataset contains a list of text headlines paired with a label that indicates whether its source is a sarcastic site (such as [The Onion](https://www.theonion.com)) or not.\n",
    "\n",
    "We have transformed this dataset into two tables such that the text-label record\n",
    "```\n",
    "\"groundbreaking study finds gratification can be deliberately postponed\"\tyes\n",
    "```\n",
    "is transformed to an entry in a table that contains (id, label) records\n",
    "```\n",
    "97 yes\n",
    "```\n",
    "and various entries in a secondary table linking a headline id to its words and positions\n",
    "```\n",
    "97\t0\tgroundbreaking\n",
    "97\t1\tstudy\n",
    "97\t2\tfinds\n",
    "97\t3\tgratification\n",
    "97\t4\tcan\n",
    "97\t5\tbe\n",
    "97\t6\tdeliberately\n",
    "97\t7\tpostponed\n",
    "```\n",
    "\n",
    "Thus, the `HeadlineSarcasm` dataset has the following multi-table schema\n",
    "```\n",
    " +-----------+\n",
    " |Headline   |\n",
    " +-----------+ +-------------+\n",
    " |HeadlineId*| |HeadlineWords|\n",
    " |IsSarcastic| +-------------+\n",
    " +-----------+ |HeadlineId*  |\n",
    "      |        |Position     |\n",
    "      +-1:n--->|Word         |\n",
    "               +-------------+\n",
    "```\n",
    "The `HeadlineId` variable is special because it is a _key_ that links a particular headline to its words (a `1:n` relation).\n",
    "\n",
    "*Note: There are other methods more appropriate for this text-mining problem. This multi-table setup is only intended for pedagogical purposes.*\n",
    "\n",
    "To train the `KhiopsClassifier` for this setup we must specify a multi-table dataset. Let's first check the content of the created tables: \n",
    "- The main table `Headline`\n",
    "- The secondary table `HeadlineWords`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sarcasm_dataset_dir = os.path.join(\"data\", \"HeadlineSarcasm\")\n",
    "headlines_file = os.path.join(sarcasm_dataset_dir, \"Headlines.txt\")\n",
    "headlines_df = pd.read_csv(headlines_file, sep=\"\\t\")\n",
    "print(\"Headlines table (first 10 rows)\")\n",
    "display(headlines_df.head(10))\n",
    "\n",
    "headlines_words_file = os.path.join(sarcasm_dataset_dir, \"HeadlineWords.txt\")\n",
    "headlines_words_df = pd.read_csv(headlines_words_file, sep=\"\\t\")\n",
    "print(\"HeadlineWords table (first 10 rows)\")\n",
    "display(headlines_words_df.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before training the classifier, we split the main table into a feature matrix (only the `HeadlineId` column) and a target vector containing the labels (the `IsSarcasm` column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headlines_main_df = headlines_df.drop(\"IsSarcasm\", axis=1)\n",
    "y_sarcasm = headlines_df[\"IsSarcasm\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may note that the feature matrix does not contain any *feature* but do not worry. The Khiops AutoML engine will automatically create features by aggregating the columns of `HeadlineWords` for each headline (more details about this below). \n",
    "\n",
    "Moreover, instead of passing an `X` table to the `fit` method, we pass a *multi-table dataset* specification which is dictionary with the following format:\n",
    "```\n",
    "X = {\n",
    "   \"main_table\": <name of the main table>,\n",
    "   \"tables\" : {\n",
    "       <name of table 1>: (<dataframe of table 1>, <key column names of table 1>),\n",
    "       <name of table 2>: (<dataframe of table 2>, <key column names of table 2>),\n",
    "       ...\n",
    "    }\n",
    "}\n",
    "```\n",
    "Note that the key columns of each table are specified as a single name or a tuple containing the column names composing the key.\n",
    "\n",
    "So for our `HeadlineSarcasm` case, we specify the dataset as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_sarcasm = {\n",
    "    \"main_table\": \"headlines\",\n",
    "    \"tables\": {\n",
    "        \"headlines\": (headlines_main_df, \"HeadlineId\"),\n",
    "        \"headline_words\": (headlines_words_df, \"HeadlineId\"),\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To separate this dataset into train and test, we user the `khiops-python` helper function `train_test_split_dataset`. This function allows to separate ``dict`` dataset specifications:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    X_sarcasm_train,\n",
    "    X_sarcasm_test,\n",
    "    y_sarcasm_train,\n",
    "    y_sarcasm_test,\n",
    ") = train_test_split_dataset(X_sarcasm, y_sarcasm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The call to the `KhiopsClassifier` `fit` method is very similar to the single-table case but this time we specify the additional parameter `n_features` which is the number of aggregates that Khiops AutoML engine will construct and analyze during the training. Some examples of the features it will create for `HeadlineSarcasm` are:\n",
    "- Number of different words in the headline\n",
    "- Most common word in the headline\n",
    "- Number of times the word 'the' appears\n",
    "- ...\n",
    "\n",
    "The Khiops AutoML engine will also evaluate, select and combine the these features to build a classifier. We'll here request for `1000` features (the default is `100`):\n",
    "\n",
    "*Note: By default Khiops builds 10 decision tree features. This is not necessary for this tutorial so we set `n_trees=0`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "khc_sarcasm = KhiopsClassifier(n_features=1000, n_trees=0)\n",
    "khc_sarcasm.fit(X_sarcasm_train, y_sarcasm_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We quickly check its train accuracy and auc as in the previous tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sarcasm_train_performance = (\n",
    "    khc_sarcasm.model_report_.train_evaluation_report.get_snb_performance()\n",
    ")\n",
    "print(f\"HeadlineSarcasm train accuracy: {sarcasm_train_performance.accuracy}\")\n",
    "print(f\"HeadlineSarcasm train auc     : {sarcasm_train_performance.auc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use our sarcasm classifier to obtain predictions and probabilities on the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_sarcasm_test_predicted = khc_sarcasm.predict(X_sarcasm_test)\n",
    "probas_sarcasm_test = khc_sarcasm.predict_proba(X_sarcasm_test)\n",
    "\n",
    "print(\"HeadlineSarcasm test predictions (first 10 values):\")\n",
    "display(y_sarcasm_test_predicted[:10])\n",
    "print(\"HeadlineSarcasm test prediction probabilities (first 10 values):\")\n",
    "display(probas_sarcasm_test[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we may estimate the accuracy and AUC for the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sarcasm_test_accuracy = metrics.accuracy_score(y_sarcasm_test, y_sarcasm_test_predicted)\n",
    "sarcasm_test_auc = metrics.roc_auc_score(y_sarcasm_test, probas_sarcasm_test[:, 1])\n",
    "\n",
    "print(f\"Sarcasm test accuracy: {sarcasm_test_accuracy}\")\n",
    "print(f\"Sarcasm test auc     : {sarcasm_test_auc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Repeat the previous steps with the `AccidentsSummary` dataset. This dataset describes the characteristics of traffic accidents that happened in France in 2018. It has two tables with the following schema:\n",
    "```\n",
    "+---------------+\n",
    "|Accidents      |\n",
    "+---------------+\n",
    "|AccidentId*    |\n",
    "|Gravity        |\n",
    "|Date           |\n",
    "|Hour           | +---------------+\n",
    "|Light          | |Vehicles       |\n",
    "|Department     | +---------------+\n",
    "|Commune        | |AccidentId*    |\n",
    "|InAgglomeration| |VehicleId*     |\n",
    "|...            | |Direction      |\n",
    "+---------------+ |Category       |\n",
    "       |          |PassengerNumber|\n",
    "       +---1:n--->|...            |\n",
    "                  +---------------+\n",
    "```\n",
    "For each accident, we have both its characteristics (such as `Gravity` or `Light` conditions) and those of each involved vehicle (its `Direction` or `PassengerNumber`). We first load the tables of the `AccidentsSummary` into dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "accidents_dataset_dir = os.path.join(kh.get_samples_dir(), \"AccidentsSummary\")\n",
    "\n",
    "accidents_file = os.path.join(accidents_dataset_dir, \"Accidents.txt\")\n",
    "accidents_df = pd.read_csv(accidents_file, sep=\"\\t\", encoding=\"latin1\")\n",
    "print(f\"Accidents dataframe (first 10 rows):\")\n",
    "display(accidents_df.head(10))\n",
    "print()\n",
    "\n",
    "vehicles_file = os.path.join(accidents_dataset_dir, \"Vehicles.txt\")\n",
    "vehicles_df = pd.read_csv(vehicles_file, sep=\"\\t\", encoding=\"latin1\")\n",
    "print(f\"Vehicles dataframe (first 10 rows):\")\n",
    "display(vehicles_df.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the main feature matrix and the target vector for `AccidentsSummary`\n",
    "\n",
    "Note that the target variable is `Gravity`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "accidents_main_df = accidents_df.drop(\"Gravity\", axis=1)\n",
    "y_accidents = accidents_df[\"Gravity\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the multi-table dataset specification\n",
    "\n",
    "Note the main table has one key `AccidentId` and the secondary table has two keys `AccidentId` and `VehicleId`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "X_accidents = {\n",
    "    \"main_table\": \"accidents\",\n",
    "    \"tables\": {\n",
    "        \"accidents\": (accidents_main_df, \"AccidentId\"),\n",
    "        \"vehicles\": (vehicles_df, [\"AccidentId\", \"VehicleId\"]),\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split the dataset into train and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "(\n",
    "    X_accidents_train,\n",
    "    X_accidents_test,\n",
    "    y_accidents_train,\n",
    "    y_accidents_test,\n",
    ") = train_test_split_dataset(X_accidents, y_accidents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train a classifier with this dataset\n",
    "\n",
    "- You may choose the number of features `n_features` to be created by the Khiops AutoML engine\n",
    "- Set the number of trees to zero (`n_trees=0`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "khc_accidents = KhiopsClassifier(n_trees=0, n_features=1000)\n",
    "khc_accidents.fit(X_accidents_train, y_accidents_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Print the train accuracy and auc of the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "accidents_train_performance = (\n",
    "    khc_accidents.model_report_.train_evaluation_report.get_snb_performance()\n",
    ")\n",
    "print(f\"AccidentsSummary train accuracy: {accidents_train_performance.accuracy}\")\n",
    "print(f\"AccidentsSummary train auc     : {accidents_train_performance.auc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the classifier to obtain predictions and its probabilites on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "y_accidents_test_predicted = khc_accidents.predict(X_accidents_test)\n",
    "probas_accidents_test = khc_accidents.predict_proba(X_accidents_test)\n",
    "\n",
    "print(\"Accidents test predictions (first 10 values):\")\n",
    "display(y_accidents_test_predicted[:10])\n",
    "print(\"Accidentns test prediction probabilities (first 10 values):\")\n",
    "display(probas_accidents_test[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Obtain the accuracy and AUC on the test dataset\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "accidents_test_accuracy = metrics.accuracy_score(\n",
    "    y_accidents_test, y_accidents_test_predicted\n",
    ")\n",
    "accidents_test_auc = metrics.roc_auc_score(\n",
    "    y_accidents_test, probas_accidents_test[:, 1]\n",
    ")\n",
    "\n",
    "print(f\"Accidents test accuracy: {accidents_test_accuracy}\")\n",
    "print(f\"Accidents test auc     : {accidents_test_auc}\")"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}