05- Ensemble models-SVM/Lab/SVM_DT_RF_Lab.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning for Neuroscience, <br>Department of Brain Sciences, Faculty of Medicine, <br> Imperial College London\n",
    "### Contributors: Francesca Palermo, Nan Fletcher-Lloyd, Alex Capstick, Yu Chen\n",
    "**Winter 2022**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM, Decision Trees, and Random Forest:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial will focus on Support Vector Machines, Decision Trees, and Random Forest. We will be using [sci-kit learn's](https://scikit-learn.org/stable/index.html) package for these models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import sklearn objects\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.inspection import DecisionBoundaryDisplay\n",
    "\n",
    "# importing numpy and pandas\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# import plotting functions\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.colors\n",
    "from cycler import cycler\n",
    "binary_cmap = matplotlib.colors.LinearSegmentedColormap.from_list(\"\", ['#332288', 'white', '#AA4499'])\n",
    "plt.rcParams[\"axes.prop_cycle\"] = cycler(\n",
    "    color=['#332288','#88CCEE','#44AA99','#117733','#999933','#DDCC77','#CC6677','#882255','#AA4499']\n",
    "    )\n",
    "\n",
    "# class for holding the random state throughout the notebook.\n",
    "# this keeps results consistent\n",
    "class RandomState(object):\n",
    "    def __init__(self, random_state=None):\n",
    "        self.random_state = random_state\n",
    "    def next(self):\n",
    "        self.random_state,\\\n",
    "            out_state = np.random.default_rng(self.random_state).integers(0, 1e9, size=(2,))\n",
    "        return out_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_state = RandomState(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading the dataset\n",
    "bc_data = datasets.load_breast_cancer(as_frame=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing the data or the target from the dataset loaded above\n",
    "bc_features = bc_data.data\n",
    "bc_target = bc_data.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first 5 lines of this data look as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the targets are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_target.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will cut the dataset to the mean features to make thing slightly easier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_mean_features = bc_features[\n",
    "    ['mean radius', 'mean texture', 'mean perimeter', 'mean area',\n",
    "    'mean smoothness', 'mean compactness', 'mean concavity',\n",
    "    'mean concave points', 'mean symmetry', 'mean fractal dimension']\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use [TSNE](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) to visualise the data in two dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import TSNE\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# pre-processing the data\n",
    "scaler = StandardScaler()\n",
    "tsne = TSNE(n_components=2, learning_rate='auto', init='random', random_state=random_state.next())\n",
    "x = tsne.fit_transform(scaler.fit_transform(bc_mean_features))\n",
    "\n",
    "# setting the figure\n",
    "fig, ax = plt.subplots(1,1,figsize=(5,5))\n",
    "\n",
    "# plotting the data\n",
    "scatter = ax.scatter(\n",
    "    x=x[:,0], \n",
    "    y=x[:,1], \n",
    "    c=bc_target.astype(bool), \n",
    "    alpha=0.5, \n",
    "    cmap=binary_cmap,\n",
    "    )\n",
    "\n",
    "# adding the legend\n",
    "ax.legend(\n",
    "    scatter.legend_elements(num=1)[0],\n",
    "    ['Negative', 'Positive'],\n",
    "    loc=\"upper right\", \n",
    "    title=\"Diagnosis\",\n",
    "    )\n",
    "\n",
    "# set title and labels\n",
    "ax.set_title('TSNE Plot of Breast Cancer Dataset')\n",
    "ax.set_xlabel('Component 1')\n",
    "ax.set_ylabel('Component 2')\n",
    "\n",
    "# showing plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values of the features are distributed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting the plotting area\n",
    "fig, axes = plt.subplots(5,2, figsize=(8,12))\n",
    "\n",
    "# getting the column names\n",
    "column_names = bc_mean_features.columns\n",
    "# getting the colours to make the plot look nicer!\n",
    "colors = plt.rcParams[\"axes.prop_cycle\"]()\n",
    "\n",
    "# looping over the subplots and the column names together\n",
    "for ax, col in zip(np.ravel(axes), column_names):\n",
    "    # plotting a histogram\n",
    "    ax.hist(\n",
    "        bc_mean_features[col], # the data, accessed by the column name \n",
    "        color=next(colors)[\"color\"], # the colour to look nicer!\n",
    "        bins=20 # the number of bins\n",
    "        )\n",
    "    # setting the title and labels\n",
    "    ax.set_title(f\"{col.title()} Histogram\")\n",
    "    ax.set_ylabel('Frequency')\n",
    "    ax.set_xlabel('Value')\n",
    "\n",
    "# setting plotting formats\n",
    "fig.subplots_adjust(hspace=0.75, wspace=0.25)\n",
    "\n",
    "# showing plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Supprt Vector Machine](https://en.wikipedia.org/wiki/Support_vector_machine) is a classic machine learning classifier that attempts to separate classes of data using a [hyperplane](https://en.wikipedia.org/wiki/Hyperplane). \n",
    "\n",
    "It works by maximising the width of the gap between two categories, when they are linearly separable by minimising the [hinge loss](https://en.wikipedia.org/wiki/Hinge_loss)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different kernels can be used to learn different boundaries between classes, that might have different distributions. In the following examples, we will see where these different kernels will be helpful.\n",
    "\n",
    "See Also: https://scikit-learn.org/stable/modules/svm.html#support-vector-machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access the [SVM code from sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html), we need to import it as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What arguments can we supply this model and what are the defaults?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SVC().get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These arguments are explained in the documentation [here](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html). For the following experiments, we will be varying the kernel function, to see how it can affect the classification performance on different datasets:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by seeing how we can train, and evaluate the performance of our model, and understand the model's decision boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the train and test splits of a synthetic dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = datasets.make_moons(1000, noise=0.15, random_state=random_state.next())\n",
    "\n",
    "# train-test splits:\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, train_size=0.75, random_state=random_state.next()\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaling the data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1,2,figsize=(8,4))\n",
    "ax1, ax2 = axes\n",
    "\n",
    "ax1.scatter(x=X_train[:,0], y=X_train[:,1], c=y_train, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "ax2.scatter(x=X_test[:,0], y=X_test[:,1], c=y_test, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "\n",
    "ax1.set_title('Train')\n",
    "ax2.set_title('Test')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be fit as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start with linear kernel\n",
    "svc = SVC(kernel='linear', random_state=random_state.next())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate the model and see how well its decision boundary fit the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is {accuracy_score(y_test, svc.predict(X_test))*100}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,1,figsize=(5,5))\n",
    "\n",
    "dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "    estimator=svc,\n",
    "    X=X_test,\n",
    "    grid_resolution=200,\n",
    "    plot_method='contourf',\n",
    "    response_method='decision_function',\n",
    "    ax=ax,\n",
    "    cmap=binary_cmap,\n",
    "    alpha=0.5,\n",
    "    eps=0.3,\n",
    "    levels=100,\n",
    "    )\n",
    "\n",
    "ax.scatter(x=X_test[:,0], y=X_test[:,1], c=y_test, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "ax.set_title('Linear Kernel')\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This doesn't fit the data correctly, we can see that the linear kernel is not designed for this dataset. In the following, we will try many different kernels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kernels = ['linear', 'poly', 'rbf', 'sigmoid']\n",
    "\n",
    "fig, axes = plt.subplots(1,len(kernels),figsize=(len(kernels)*4,4))\n",
    "\n",
    "# looping over kernels\n",
    "for nk, kernel in enumerate(kernels):\n",
    "    ax = np.ravel(axes)[nk] # getting the current axis\n",
    "\n",
    "    # fitting the model\n",
    "    svc = SVC(kernel=kernel, random_state=random_state.next())\n",
    "    svc.fit(X_train, y_train)\n",
    "\n",
    "    # plotting the decision boundary\n",
    "    dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "        estimator=svc,\n",
    "        X=X_test,\n",
    "        grid_resolution=200,\n",
    "        plot_method='contourf',\n",
    "        response_method='decision_function',\n",
    "        ax=ax,\n",
    "        cmap=binary_cmap,\n",
    "        alpha=0.5,\n",
    "        eps=0.3,\n",
    "        levels=100,\n",
    "        )\n",
    "    \n",
    "    # plotting the data\n",
    "    ax.scatter(\n",
    "        x=X_test[:,0], y=X_test[:,1], c=y_test, \n",
    "        alpha=0.5, cmap=binary_cmap, edgecolor='black'\n",
    "        )\n",
    "\n",
    "    # title\n",
    "    ax.set_title(f'{kernel.title()} Kernel - '\\\n",
    "        f'accuracy {accuracy_score(y_test, svc.predict(X_test))*100:.2f}%')\n",
    "\n",
    "# figure title\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20, y=1.1)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, in this example, the RBF kernel was the best!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But in which cases are the different kernels better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generating datasets\n",
    "data_dict = {\n",
    "    'moons': datasets.make_moons(\n",
    "        1000, noise=0.15, random_state=random_state.next()\n",
    "        ),\n",
    "    'circles': datasets.make_circles(\n",
    "        1000, noise=0.15, factor=0.2, random_state=random_state.next()\n",
    "        ),\n",
    "    'blobs': datasets.make_blobs(\n",
    "        1000, centers=[[1, -1], [1, 1]], cluster_std=0.3, random_state=random_state.next(),\n",
    "        ),\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# kernel names\n",
    "kernels = ['linear', 'poly', 'rbf', 'sigmoid']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plotting figure\n",
    "fig, axes = plt.subplots(\n",
    "    len(data_dict), len(kernels), figsize=(len(kernels)*4,len(data_dict)*4),\n",
    "    )\n",
    "\n",
    "# looping over kernels\n",
    "for nd, data in enumerate(data_dict):\n",
    "    for nk, kernel in enumerate(kernels):\n",
    "        \n",
    "        # getting the current axis\n",
    "        ax = axes[nd, nk]\n",
    "\n",
    "        # getting the data\n",
    "        X, y = data_dict[data]\n",
    "\n",
    "        # train-test splits:\n",
    "        X_train, X_test, y_train, y_test = train_test_split(\n",
    "            X, y, train_size=0.75, random_state=random_state.next()\n",
    "            )\n",
    "\n",
    "        # scaling the data\n",
    "        scaler = StandardScaler()\n",
    "        X_train = scaler.fit_transform(X_train)\n",
    "        X_test = scaler.transform(X_test)\n",
    "\n",
    "        # fitting the model\n",
    "        svc = SVC(kernel=kernel, random_state=random_state.next())\n",
    "        svc.fit(X_train, y_train)\n",
    "\n",
    "        # plotting the decision boundary\n",
    "        dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "            estimator=svc,\n",
    "            X=X_test,\n",
    "            grid_resolution=200,\n",
    "            plot_method='contourf',\n",
    "            response_method='decision_function',\n",
    "            ax=ax,\n",
    "            cmap=binary_cmap,\n",
    "            alpha=0.5,\n",
    "            eps=0.3,\n",
    "            levels=100,\n",
    "            )\n",
    "        \n",
    "        # plotting the data\n",
    "        ax.scatter(\n",
    "            x=X_test[:,0], y=X_test[:,1], c=y_test, \n",
    "            alpha=0.5, cmap=binary_cmap, edgecolor='black'\n",
    "            )\n",
    "\n",
    "        # title\n",
    "        ax.set_title(f'{kernel.title()} Kernel - '\\\n",
    "            f'accuracy {accuracy_score(y_test, svc.predict(X_test))*100:.2f}%')\n",
    "\n",
    "# figure title\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20)\n",
    "# showing plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an understanding of how this can be used in our generated examples, let's try to use SVM to predict the classes on the breast cancer dataset that we introduced at the beginning of this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our features are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_mean_features.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our targets are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_target.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# turning data from table to arrays\n",
    "X, y = bc_mean_features.values, bc_target.values\n",
    "\n",
    "# train-test splits:\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, train_size=0.75, random_state=random_state.next()\n",
    "    )\n",
    "\n",
    "# scaling the data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use cross validation to understand which of the SVM models might be the best predictor of breast cancer on this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting up the cross validated grid search\n",
    "gscv = GridSearchCV(\n",
    "    estimator=SVC(random_state=random_state.next()), # the model\n",
    "    param_grid={'kernel': ['linear', 'poly', 'rbf', 'sigmoid']}, # the parameters to change in the search\n",
    "    scoring='accuracy',  # how to score the parameters\n",
    "    refit=True, # return the best model fitted on all of the training data\n",
    "    cv=5, # the number of cross-validated folds\n",
    "    verbose=4, # print lots of info as the code is running\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fitting the model on the training data, with cross validation\n",
    "gscv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('The results were:')\n",
    "pd.DataFrame(gscv.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters was {gscv.best_params_}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model = gscv.best_estimator_\n",
    "best_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of this model on the test data is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The test accuracy is {accuracy_score(y_test, best_model.predict(X_test))*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Decision Trees](https://en.wikipedia.org/wiki/Decision_tree_learning) is a classic machine learning classifier that attempts to separate classes of data by learning a rule based system on the features independently. Because of this, we actually do not need to scale the data, since all features are split separately.\n",
    "\n",
    "At each iteration, the next split is performed on the feature that optimises the criterion most. For example, when using [Gini Impurity](https://victorzhou.com/blog/gini-impurity/), we want to make a split where the Gini Impurity is reduced the most between before and after the split is made.\n",
    "\n",
    "See also: https://scikit-learn.org/stable/modules/tree.html#decision-trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decision tree classifier is easily imported from sklearn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"This has the default parameters:\\n {DecisionTreeClassifier().get_params()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What each of these parameters refers to can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier). We will investigate how this decision tree can be used to classify data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to earlier, we will start by seeing how we can train, and evaluate the performance of our model, and understand the model's decision boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the train and test splits of a synthetic dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = datasets.make_moons(1000, noise=0.15, random_state=random_state.next())\n",
    "\n",
    "# train-test splits:\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, train_size=0.75, random_state=random_state.next()\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaling the data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1,2,figsize=(8,4))\n",
    "ax1, ax2 = axes\n",
    "\n",
    "ax1.scatter(x=X_train[:,0], y=X_train[:,1], c=y_train, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "ax2.scatter(x=X_test[:,0], y=X_test[:,1], c=y_test, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "\n",
    "ax1.set_title('Train')\n",
    "ax2.set_title('Test')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be fit as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start with linear kernel\n",
    "dt = DecisionTreeClassifier(random_state=random_state.next())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate the model and see how well its decision boundary fit the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without any tuning, this model already performs much better than SVM. Let us try and see why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is {accuracy_score(y_test, dt.predict(X_test))*100}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,1,figsize=(5,5))\n",
    "\n",
    "dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "    estimator=dt,\n",
    "    X=X_test,\n",
    "    grid_resolution=200,\n",
    "    plot_method='contourf',\n",
    "    response_method='predict_proba',\n",
    "    ax=ax,\n",
    "    cmap=binary_cmap,\n",
    "    alpha=0.5,\n",
    "    eps=0.3,\n",
    "    levels=100,\n",
    "    )\n",
    "\n",
    "ax.scatter(x=X_test[:,0], y=X_test[:,1], c=y_test, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "ax.set_title('Default Parameters')\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this mostly fits the data, but has clearly tried to over fit to the few datapoints from the pink class that fall within the purple moon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's study how the max depth of the tree can effect the performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depths = [1, 2, 5, 10, 100]\n",
    "\n",
    "fig, axes = plt.subplots(1, len(max_depths), figsize=(len(max_depths)*4,4))\n",
    "\n",
    "# looping over max_depths\n",
    "for nmd, max_depth in enumerate(max_depths):\n",
    "    # getting the current axis\n",
    "    ax = np.ravel(axes)[nmd]\n",
    "\n",
    "    # fitting the model\n",
    "    dt = DecisionTreeClassifier(max_depth=max_depth, random_state=random_state.next())\n",
    "    dt.fit(X_train, y_train)\n",
    "\n",
    "    # plotting the decision boundary\n",
    "    dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "        estimator=dt,\n",
    "        X=X_test,\n",
    "        grid_resolution=200,\n",
    "        plot_method='contourf',\n",
    "        response_method='predict_proba',\n",
    "        ax=ax,\n",
    "        cmap=binary_cmap,\n",
    "        alpha=0.5,\n",
    "        eps=0.3,\n",
    "        levels=100,\n",
    "        )\n",
    "    \n",
    "    # plotting the data\n",
    "    ax.scatter(\n",
    "        x=X_test[:,0], y=X_test[:,1], c=y_test, \n",
    "        alpha=0.5, cmap=binary_cmap, edgecolor='black'\n",
    "        )\n",
    "\n",
    "    # title\n",
    "    ax.set_title(f'Tree Depth: {max_depth} - '\\\n",
    "        f'accuracy {accuracy_score(y_test, dt.predict(X_test))*100:.2f}%')\n",
    "\n",
    "# figure title\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20, y=1.1)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why might a tree depth of 100 and 10 produce the same results? Because any of `min_samples_split`, `min_samples_leaf`, or `min_weight_fraction_leaf` may have been met!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this model performs over different datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generating datasets\n",
    "data_dict = {\n",
    "    'moons': datasets.make_moons(\n",
    "        1000, noise=0.15, random_state=random_state.next()\n",
    "        ),\n",
    "    'circles': datasets.make_circles(\n",
    "        1000, noise=0.15, factor=0.2, random_state=random_state.next()\n",
    "        ),\n",
    "    'blobs': datasets.make_blobs(\n",
    "        1000, centers=[[1, -1], [1, 1]], cluster_std=0.3, random_state=random_state.next(),\n",
    "        ),\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# max depths\n",
    "max_depths = [1, 2, 5, 10, 100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plotting figure\n",
    "fig, axes = plt.subplots(\n",
    "    len(data_dict), len(max_depths), figsize=(len(max_depths)*4,len(data_dict)*4),\n",
    "    )\n",
    "\n",
    "# looping over max_depths\n",
    "for nd, data in enumerate(data_dict):\n",
    "    for nmd, max_depth in enumerate(max_depths):\n",
    "        \n",
    "        # getting the current axis\n",
    "        ax = axes[nd, nmd]\n",
    "\n",
    "        # getting the data\n",
    "        X, y = data_dict[data]\n",
    "\n",
    "        # train-test splits:\n",
    "        X_train, X_test, y_train, y_test = train_test_split(\n",
    "            X, y, train_size=0.75, random_state=random_state.next()\n",
    "            )\n",
    "\n",
    "        # scaling the data\n",
    "        scaler = StandardScaler()\n",
    "        X_train = scaler.fit_transform(X_train)\n",
    "        X_test = scaler.transform(X_test)\n",
    "\n",
    "        # fitting the model\n",
    "        dt = DecisionTreeClassifier(max_depth=max_depth, random_state=random_state.next())\n",
    "        dt.fit(X_train, y_train)\n",
    "\n",
    "        # plotting the decision boundary\n",
    "        dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "            estimator=dt,\n",
    "            X=X_test,\n",
    "            grid_resolution=200,\n",
    "            plot_method='contourf',\n",
    "            response_method='predict_proba',\n",
    "            ax=ax,\n",
    "            cmap=binary_cmap,\n",
    "            alpha=0.5,\n",
    "            eps=0.3,\n",
    "            levels=100,\n",
    "            )\n",
    "        \n",
    "        # plotting the data\n",
    "        ax.scatter(\n",
    "            x=X_test[:,0], y=X_test[:,1], c=y_test, \n",
    "            alpha=0.5, cmap=binary_cmap, edgecolor='black'\n",
    "            )\n",
    "\n",
    "        # title\n",
    "        ax.set_title(f'Tree Depth: {max_depth} - '\\\n",
    "            f'accuracy {accuracy_score(y_test, dt.predict(X_test))*100:.2f}%')\n",
    "\n",
    "# figure title\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20)\n",
    "# showing plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an understanding of how this can be used in our generated examples, let's try to use a Decision Tree to predict the classes on the breast cancer dataset that we introduced at the beginning of this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our features are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_mean_features.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our targets are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_target.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# turning data from table to arrays\n",
    "X, y = bc_mean_features.values, bc_target.values\n",
    "\n",
    "# train-test splits:\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, train_size=0.75, random_state=random_state.next()\n",
    "    )\n",
    "\n",
    "# scaling the data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use cross validation to understand which parameters in the DT models might be the best predictor of breast cancer on this dataset. We will test different max depths and criterions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting up the cross validated grid search\n",
    "gscv = GridSearchCV(\n",
    "    estimator=DecisionTreeClassifier(random_state=random_state.next()), # the model\n",
    "    param_grid={ # the parameters to change in the search\n",
    "        'max_depth': [1, 2, 5, 10, 20, 50, 100,], # max depth\n",
    "        'criterion': ['gini', 'entropy', 'log_loss'], # criterion\n",
    "        }, \n",
    "    scoring='accuracy',  # how to score the parameters\n",
    "    refit=True, # return the best model fitted on all of the training data\n",
    "    cv=5, # the number of cross-validated folds\n",
    "    verbose=1, # print lots of info as the code is running\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fitting the model on the training data, with cross validation\n",
    "gscv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('The results were:')\n",
    "pd.DataFrame(gscv.cv_results_).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters was {gscv.best_params_}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model = gscv.best_estimator_\n",
    "best_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of this model on the test data is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The test accuracy is {accuracy_score(y_test, best_model.predict(X_test))*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Random Forest](https://en.wikipedia.org/wiki/Random_forest) is a classifier that is built on top of the work done by Decision Trees and is a type of ensemble learning model. This is because it uses a \"forest\" of decision trees when classifying data. \n",
    "\n",
    "During training many decision trees are built based on different random splits of the features and data (depending on the parameters), and during testing, the predictions of these trees are combined to get a single prediction.\n",
    "\n",
    "See also https://scikit-learn.org/stable/modules/ensemble.html#forests-of-randomized-trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decision tree classifier is easily imported from sklearn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"This has the default parameters:\\n {RandomForestClassifier().get_params()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What each of these parameters refers to can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html). We will investigate how this random forest can be used to classify data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to earlier, we will start by seeing how we can train, and evaluate the performance of our model, and understand the model's decision boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the train and test splits of a synthetic dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = datasets.make_moons(1000, noise=0.15, random_state=random_state.next())\n",
    "\n",
    "# train-test splits:\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, train_size=0.75, random_state=random_state.next()\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaling the data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1,2,figsize=(8,4))\n",
    "ax1, ax2 = axes\n",
    "\n",
    "ax1.scatter(x=X_train[:,0], y=X_train[:,1], c=y_train, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "ax2.scatter(x=X_test[:,0], y=X_test[:,1], c=y_test, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "\n",
    "ax1.set_title('Train')\n",
    "ax2.set_title('Test')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be fit as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start with linear kernel\n",
    "rf = RandomForestClassifier(random_state=random_state.next())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate the model and see how well its decision boundary fit the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without any tuning, this model already performs much better than SVM. Let us try and see why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is {accuracy_score(y_test, rf.predict(X_test))*100}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,1,figsize=(5,5))\n",
    "\n",
    "dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "    estimator=rf,\n",
    "    X=X_test,\n",
    "    grid_resolution=200,\n",
    "    plot_method='contourf',\n",
    "    response_method='predict_proba',\n",
    "    ax=ax,\n",
    "    cmap=binary_cmap,\n",
    "    alpha=0.5,\n",
    "    eps=0.3,\n",
    "    levels=100,\n",
    "    )\n",
    "\n",
    "ax.scatter(x=X_test[:,0], y=X_test[:,1], c=y_test, alpha=0.5, cmap=binary_cmap, edgecolor='black')\n",
    "ax.set_title('Default Parameters')\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this fits the data much better than Decision Trees!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's study how the number of trees in the forest can effect the performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_trees = [1, 2, 5, 10, 100,]\n",
    "\n",
    "fig, axes = plt.subplots(1, len(n_trees), figsize=(len(n_trees)*4,4))\n",
    "\n",
    "# looping over n_trees\n",
    "for nmd, n_estimators in enumerate(n_trees):\n",
    "    # getting the current axis\n",
    "    ax = np.ravel(axes)[nmd]\n",
    "\n",
    "    # fitting the model\n",
    "    rf = RandomForestClassifier(n_estimators=n_estimators, random_state=random_state.next())\n",
    "    rf.fit(X_train, y_train)\n",
    "\n",
    "    # plotting the decision boundary\n",
    "    dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "        estimator=rf,\n",
    "        X=X_test,\n",
    "        grid_resolution=200,\n",
    "        plot_method='contourf',\n",
    "        response_method='predict_proba',\n",
    "        ax=ax,\n",
    "        cmap=binary_cmap,\n",
    "        alpha=0.5,\n",
    "        eps=0.3,\n",
    "        levels=100,\n",
    "        )\n",
    "    \n",
    "    # plotting the data\n",
    "    ax.scatter(\n",
    "        x=X_test[:,0], y=X_test[:,1], c=y_test, \n",
    "        alpha=0.5, cmap=binary_cmap, edgecolor='black'\n",
    "        )\n",
    "\n",
    "    # title\n",
    "    ax.set_title(f'No. Trees: {n_estimators} - '\\\n",
    "        f'accuracy {accuracy_score(y_test, rf.predict(X_test))*100:.2f}%')\n",
    "\n",
    "# figure title\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20, y=1.1)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, because multiple trees are used, and each of them is acting over a subset of the data, Random Forest is less likely to over-fit to the data as a single decision tree acting over all of the training data.\n",
    "\n",
    "We see that even with a small number of trees, the performance is good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this model performs over different datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generating datasets\n",
    "data_dict = {\n",
    "    'moons': datasets.make_moons(\n",
    "        1000, noise=0.15, random_state=random_state.next()\n",
    "        ),\n",
    "    'circles': datasets.make_circles(\n",
    "        1000, noise=0.15, factor=0.2, random_state=random_state.next()\n",
    "        ),\n",
    "    'blobs': datasets.make_blobs(\n",
    "        1000, centers=[[1, -1], [1, 1]], cluster_std=0.3, random_state=random_state.next(),\n",
    "        ),\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of trees\n",
    "n_trees = [1, 2, 5, 10, 100,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plotting figure\n",
    "fig, axes = plt.subplots(\n",
    "    len(data_dict), len(n_trees), figsize=(len(n_trees)*4,len(data_dict)*4),\n",
    "    )\n",
    "\n",
    "# looping over n_trees\n",
    "for nd, data in enumerate(data_dict):\n",
    "    for nmd, n_estimators in enumerate(n_trees):\n",
    "        \n",
    "        # getting the current axis\n",
    "        ax = axes[nd, nmd]\n",
    "\n",
    "        # getting the data\n",
    "        X, y = data_dict[data]\n",
    "\n",
    "        # train-test splits:\n",
    "        X_train, X_test, y_train, y_test = train_test_split(\n",
    "            X, y, train_size=0.75, random_state=random_state.next()\n",
    "            )\n",
    "\n",
    "        # scaling the data\n",
    "        scaler = StandardScaler()\n",
    "        X_train = scaler.fit_transform(X_train)\n",
    "        X_test = scaler.transform(X_test)\n",
    "\n",
    "        # fitting the model\n",
    "        rf = RandomForestClassifier(n_estimators=n_estimators, random_state=random_state.next())\n",
    "        rf.fit(X_train, y_train)\n",
    "\n",
    "        # plotting the decision boundary\n",
    "        dbd = DecisionBoundaryDisplay.from_estimator(\n",
    "            estimator=rf,\n",
    "            X=X_test,\n",
    "            grid_resolution=200,\n",
    "            plot_method='contourf',\n",
    "            response_method='predict_proba',\n",
    "            ax=ax,\n",
    "            cmap=binary_cmap,\n",
    "            alpha=0.5,\n",
    "            eps=0.3,\n",
    "            levels=100,\n",
    "            )\n",
    "        \n",
    "        # plotting the data\n",
    "        ax.scatter(\n",
    "            x=X_test[:,0], y=X_test[:,1], c=y_test, \n",
    "            alpha=0.5, cmap=binary_cmap, edgecolor='black'\n",
    "            )\n",
    "\n",
    "        # title\n",
    "        ax.set_title(f'No. Trees: {n_estimators} - '\\\n",
    "            f'accuracy {accuracy_score(y_test, rf.predict(X_test))*100:.2f}%')\n",
    "\n",
    "# figure title\n",
    "fig.suptitle('Boundaries on the Test Set', fontsize=20)\n",
    "# showing plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an understanding of how this can be used in our generated examples, let's try to use a Random Forest to predict the classes on the breast cancer dataset that we introduced at the beginning of this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our features are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_mean_features.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our targets are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc_target.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# turning data from table to arrays\n",
    "X, y = bc_mean_features.values, bc_target.values\n",
    "\n",
    "# train-test splits:\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, train_size=0.75, random_state=random_state.next()\n",
    "    )\n",
    "\n",
    "# scaling the data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use cross validation to understand which parameters in the DT models might be the best predictor of breast cancer on this dataset. We will test different max depths and criterions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting up the cross validated grid search\n",
    "gscv = GridSearchCV(\n",
    "    estimator=RandomForestClassifier(random_state=random_state.next()), # the model\n",
    "    param_grid={ # the parameters to change in the search\n",
    "        'max_depth': [1, 2, 5, 10, 20, 50, 100,], # max depth\n",
    "        'criterion': ['gini', 'entropy', 'log_loss'], # criterion\n",
    "        'n_estimators': [1, 2, 5, 10, 100, 200],\n",
    "        }, \n",
    "    scoring='accuracy',  # how to score the parameters\n",
    "    refit=True, # return the best model fitted on all of the training data\n",
    "    cv=5, # the number of cross-validated folds\n",
    "    verbose=1, # print lots of info as the code is running\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fitting the model on the training data, with cross validation\n",
    "gscv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('The results were:')\n",
    "pd.DataFrame(gscv.cv_results_).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters was {gscv.best_params_}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model = gscv.best_estimator_\n",
    "best_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of this model on the test data is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The test accuracy is {accuracy_score(y_test, best_model.predict(X_test))*100:.2f}%\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "15bf93136d40f5eaa63d32eecf0d3b269d46ff0cfb31e2703821c0a79f3e6d46"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 ('ml4n')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7 (default, Sep 16 2021, 13:09:58) \n[GCC 7.5.0]"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}