Sklearn Basics 4 - Train a Coclustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sklearn Basics 4: Train a Coclustering\n",
    "The steps to train a coclustering model with Khiops are very similar to what we have already seen in the basic classifier tutorials.\n",
    "\n",
    "We start by importing the sklearn estimator `KhiopsCoclustering` and defining a helper function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import platform\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "from khiops import core as kh\n",
    "from khiops.sklearn import KhiopsCoclustering\n",
    "\n",
    "\n",
    "def os_open(path):\n",
    "    \"\"\"Opens a file or directory with its default application\"\"\"\n",
    "    if platform.system() == \"Windows\":\n",
    "        os.startfile(path)\n",
    "    elif platform.system() == \"Darwin\":\n",
    "        subprocess.call([\"open\", path])\n",
    "    else:\n",
    "        subprocess.call([\"xdg-open\", path])\n",
    "\n",
    "\n",
    "# If there are any issues you may Khiops status with the following command\n",
    "# kh.get_runner().print_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial, we use the dataset `CountriesByOrganization` that contains the relation country-organization for a large number of countries and organizations (*it is bit outdated though*). The objective is to build a coclustering between Country and Organization and see which countries resemble the most in terms of organizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first load this dataset and check its content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries_data_file = os.path.join(\n",
    "    \"data\", \"CountriesByOrganization\", \"CountriesByOrganization.csv\"\n",
    ")\n",
    "X_countries = pd.read_csv(countries_data_file, sep=\";\")\n",
    "print(\"CountriesByOrganization dataset:\")\n",
    "display(X_countries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's build the coclustering model.\n",
    "\n",
    "Note that a coclustering model is learned in an unsupervised way and aims to cluster jointly rows and columns of a matrix. So we need to provide a column name to be able to deploy it on a specific column. We do this by setting the `fit` parameter `id_column`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "khcc_countries = KhiopsCoclustering()\n",
    "khcc_countries.fit(X_countries, id_column=\"Country\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's access the coclustering training report to obtain the cluster information of the `Country` dimension. Since in each dimension there is a hierarchical cluster, so we only access the leaf clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries_clusters = khcc_countries.model_report_.coclustering_report.get_dimension(\n",
    "    \"Country\"\n",
    ").clusters\n",
    "countries_leaf_clusters = [cluster for cluster in countries_clusters if cluster.is_leaf]\n",
    "print(f\"Number of leaf clusters: {len(countries_leaf_clusters)}:\")\n",
    "for index, cluster in enumerate(countries_leaf_clusters, start=1):\n",
    "    print(f\"cluster {index:02d}: {cluster.name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The composition of the clusters is also available. For the first one we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Members of the cluster {countries_leaf_clusters[0].name}:\")\n",
    "for value_obj in countries_clusters[0].leaf_part.values:\n",
    "    print(value_obj.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coclustering is a complex model, so it is better to visualize it with the Khiops Co-visualization app. Let's export the report to a `.khcj` file and open it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries_report = os.path.join(\"exercises\", \"countries.khcj\")\n",
    "khcc_countries.export_report_file(countries_report)\n",
    "# explorer_open(countries_report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's deploy the coclustering model on the training data `countries_df`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries_predictions = khcc_countries.predict(X_countries)\n",
    "print(\"Predicted clusters (first 10)\")\n",
    "display(countries_predictions[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "We'll build a coclustering model for the `Tokyo2021` dataset. It is extracted from the `Athletes` table of the [Tokyo 2021 Kaggle dataset](https://www.kaggle.com/arjunprasadsarkhel/2021-olympics-in-tokyo) and each record contains three variables:\n",
    "- `Name`: the name of a competing athlete\n",
    "- `Country`: the country (or organization) it represents\n",
    "- `Discipline`: the athlete's discipline\n",
    "\n",
    "The objective with this exercise is to make a coclustering between `Country` and `Discipline` and see which countries resemble the most in terms of the athletes they bring to the Olympics. We start by loading the contents into a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "tokyo_data_file = os.path.join(\"data\", \"Tokyo2021\", \"Athletes.csv\")\n",
    "X_tokyo = pd.read_csv(tokyo_data_file, encoding=\"latin1\")\n",
    "print(\"Tokyo2021 dataset (first 10 rows):\")\n",
    "display(X_tokyo.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train the coclustering for the variables `Country` and `Discipline`\n",
    "\n",
    "Call `fit` parameters with the following parameters:\n",
    "- `X=X_tokyo[[\"Country\", \"Discipline\"]]`\n",
    "- `id_column=\"Country\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "khcc_tokyo = KhiopsCoclustering()\n",
    "khcc_tokyo.fit(X_tokyo[[\"Country\", \"Discipline\"]], id_column=\"Country\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Obtain the number and names of the clusters of the `Country` dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "tokyo_clusters = khcc_tokyo.model_report_.coclustering_report.get_dimension(\n",
    "    \"Country\"\n",
    ").clusters\n",
    "tokyo_leaf_clusters = [cluster for cluster in tokyo_clusters if cluster.is_leaf]\n",
    "print(f\"Number of leaf clusters: {len(tokyo_leaf_clusters)}:\")\n",
    "for index, cluster in enumerate(tokyo_leaf_clusters):\n",
    "    print(f\"cluster {index:02d}: {cluster.name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Print the members of one of the clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "print(f\"Members of the cluster {tokyo_leaf_clusters[29].name}:\")\n",
    "for value_obj in tokyo_leaf_clusters[29].leaf_part.values:\n",
    "    print(value_obj.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Check the results with the covisualization app**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "tokyo_report = os.path.join(\"exercises\", \"tokyo.khcj\")\n",
    "khcc_tokyo.export_report_file(tokyo_report)\n",
    "\n",
    "# To visualize uncomment the line below\n",
    "# os_open(tokyo_report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the learned coclustering model on the training data and check the obtained clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "is_khiops_tutorial_solution": true
   },
   "outputs": [],
   "source": [
    "tokyo_predictions = khcc_tokyo.predict(X_tokyo[[\"Country\", \"Discipline\"]])\n",
    "print(\"Predicted clusters (first 10)\")\n",
    "display(tokyo_predictions[:10])"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}