data/notebook/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h1 style=\"text-align: center\">Maratona Behind The Code</h1>\n",
    "<h2 style=\"text-align: center\">Final Challenge -  Machine Learning applied to Planet Exploration</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Short Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Astronomy has always fascinated mankind. Until we can trace ancient civilizations had looked to the sky and found some patterns in the dynamics of the night sky and whether a celestial body can emit light or just reflect it. Using this simple approach, the Greeks, the Egyptians and the Babylonians had mapped the planets until saturn.\n",
    "\n",
    "More than a thousand years had passed to the discovery of Uranus, and this was only possible grace of technological advance. And during the last three decades humanity has discovery more planets than ever and now it is time to put A.I to help astronomers to classify the celestials bodies that light and the gravitational disturb is meraly measerable. Are you up for the challenge ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Libs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade ibm-cos-sdk==2.7.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade --force-reinstall ibm_watson_machine_learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install scikit-learn --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install xgboost --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install imblearn --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ## Aquisição do conjunto de dados -->\n",
    "## Acquiring dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is necessary to insert the dataset as a dataframe on jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<<ISERT_YOUR_PANDAS_DATAFREAME_HERE>>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_dataset = df_data_1\n",
    "df_training_dataset.fillna(0., inplace=True)\n",
    "df_training_dataset.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have some astronomical data on this dataset and it is important you know some of them:\n",
    "\n",
    "- **TARGET**: The disposition in the literature towards this exoplanet candidate. One of CANDIDATE, FALSE POSITIVE or CONFIRMED.\n",
    "- **koi_pdisposition**: The disposition Kepler data analysis has towards this exoplanet candidate. One of FALSE POSITIVE and CANDIDATE.\n",
    "- **koi_score**: A value between 0 and 1 that indicates the confidence in the KOI disposition. For CANDIDATEs, a higher value indicates more confidence in its disposition, while for FALSE POSITIVEs, a higher value indicates less confidence in that disposition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_dataset.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_dataset.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Challenge Details: Multiclass Classification\n",
    "\n",
    "The proposal of the challenge is to classify data to enable machine to point if an amout of measures available on the dataset can be a planet, a candidate of planet that require more studies, or none which means it is not a planet. For this, we can use two approaches: supervised machine learning (classification) or unsupervised (clustering). In this challenge the classification will be applied, since a dataset is already available with \"labels\", or in other words, already with examples of data together with the target variable.\n",
    "\n",
    "In the scikit-learn library we have several algorithms for classification. The participant is free to use the framework he wishes to complete this challenge. The role notebook is prepared for sckit-learn deployment though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data exploration\n",
    "\n",
    "Use the cells below to explore the data, check which variables most influence the `TARGET` variable and the distribution of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-processing the dataset before training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Construction of the complete Pipeline for WML encapsulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preparing custom transformations for loading on WML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To integrate these types of custom transformations into Watson Machine Learning Pipelines, you must first package your custom code as a Python library. This can be done easily using the *setuptools* tool.\n",
    "\n",
    "On the following git repository: https://github.com/vnderlev/sklearn_transforms we have all the necessaries files to create a Python package, named **my_custom_sklearn_transforms**.\n",
    "This package has the following file structure:\n",
    "\n",
    "    /my_custom_sklearn_transforms.egg-info\n",
    "        dependency_links.txt\n",
    "        not-zip-safe\n",
    "        PKG-INFO\n",
    "        SOURCES.txt\n",
    "        top_level.txt\n",
    "    /my_custom_sklearn_transforms\n",
    "        __init__.py\n",
    "        sklearn_transformers.py\n",
    "    PKG-INFO\n",
    "    README.md\n",
    "    setup.cfg\n",
    "    setup.py\n",
    "    \n",
    "The main file, which will contain the code for our custom transforms, is the file **/my_custom_sklearn_transforms/sklearn_transformers.py**. If you access it in the repository, you will notice that it contains a class called `DropColumns()`, which has the necessary methods to remove columns from any dataset.\n",
    "\n",
    "    - DropColumns() custom transformation code:\n",
    "    \n",
    "    from sklearn.base import BaseEstimator, TransformerMixin\n",
    "    # All sklearn Transforms must have the `transform` and `fit` methods\n",
    "    class DropColumns(BaseEstimator, TransformerMixin):\n",
    "        def __init__(self, columns):\n",
    "            self.columns = columns\n",
    "        def fit(self, X, y=None):\n",
    "            return self\n",
    "        def transform(self, X):\n",
    "            # Primeiro realizamos a cópia do dataframe 'X' de entrada\n",
    "            data = X.copy()\n",
    "            # Retornamos um novo dataframe sem as colunas indesejadas\n",
    "            return data.drop(labels=self.columns, axis='columns')\n",
    "\n",
    "If you have declared your own transformations (in addition to the provided DropColumn), you must add all the classes of those transformations created by you in this same file. To do this, you must fork this repository, and add your custom classes in the file **sklearn_transformers.py**.\n",
    "\n",
    "If you only made use of the provided transformation (DropColumns), you can skip this fork step, and continue using the supplied base package! :)\n",
    "\n",
    "After preparing your Python package with your custom transforms, replace the git repository link in the cell below and run it. If you have not prepared any new transforms, execute the cell with the repository link already provided.\n",
    "\n",
    "<hr>\n",
    "    \n",
    "**PAY ATTENTION**\n",
    "\n",
    "If the execution of the cell below returns an error that the repository already exists, run the foolowing command:\n",
    "\n",
    "**!rm -r -f sklearn_transforms**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace the link below with the link from your git repository (if applicable)\n",
    "!git clone https://github.com/vnderlev/sklearn_transforms.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd sklearn_transforms\n",
    "!ls -ltr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!zip -r sklearn_transforms.zip sklearn_transforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sklearn_transforms.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from my_custom_sklearn_transforms.sklearn_transformers import DropColumns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a custom `` DropColumns`` Transform\n",
    "\n",
    "rm_columns = DropColumns(\n",
    "    columns=['rowid']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a `` SimpleImputer`` object\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "si = SimpleImputer(\n",
    "    missing_values=np.nan,  # the missing values are type `` np.nan`` (standard Pandas)\n",
    "    strategy='constant',  # the chosen strategy is to change the missing value by a constant\n",
    "    fill_value=0,  # the constant that will be used to fill in the missing values is an int64 = 0.\n",
    "    verbose=0,\n",
    "    copy=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting FEATURES and setting the TARGET variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_dataset.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = df_training_dataset[\n",
    "    [\n",
    "        'rowid', 'kepid', 'koi_pdisposition', 'koi_score', 'koi_fpflag_nt',\n",
    "        'koi_fpflag_ss', 'koi_fpflag_co', 'koi_fpflag_ec', 'koi_period',\n",
    "        'koi_period_err1', 'koi_period_err2', 'koi_time0bk', 'koi_time0bk_err1',\n",
    "        'koi_time0bk_err2', 'koi_impact', 'koi_impact_err1', 'koi_impact_err2',\n",
    "        'koi_duration', 'koi_duration_err1', 'koi_duration_err2', 'koi_depth',\n",
    "        'koi_depth_err1', 'koi_depth_err2', 'koi_prad', 'koi_prad_err1',\n",
    "        'koi_prad_err2', 'koi_teq', 'koi_insol', 'koi_insol_err1',\n",
    "        'koi_insol_err2', 'koi_model_snr', 'koi_tce_plnt_num', 'koi_steff',\n",
    "        'koi_steff_err1', 'koi_steff_err2', 'koi_slogg', 'koi_slogg_err1',\n",
    "        'koi_slogg_err2', 'koi_srad', 'koi_srad_err1', 'koi_srad_err2', 'ra',\n",
    "        'dec', 'koi_kepmag'\n",
    "    ]\n",
    "]\n",
    "target = df_training_dataset['TARGET']  ## DO NOT CHANGE THE NAME OF THE TARGET VARIABLE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preparing the arguments for the methods of the `` scikit-learn`` library\n",
    "X = features\n",
    "y = target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the dataset into train and test partition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Separation of data into a training set and a test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=337)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating our pipeline for storage at Watson Machine Learning:\n",
    "from sklearn.you import YourModel\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "my_pipeline = Pipeline(\n",
    "    steps=[\n",
    "        ('step_1_remove_columns', rm_columns),\n",
    "        ('step_2_imputer', si),\n",
    "        ('choosen_model', YourModel()),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pipeline initialization (pre-processing and model training)\n",
    "model = my_pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making predictions in the test sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = my_pipeline.predict(X_test)\n",
    "print(y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tip: use the `metrics` library in scikit-learn to get more information about your model's metrics.[ref](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing the quality of the model through the confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import itertools\n",
    "\n",
    "\n",
    "def plot_confusion_matrix(cm, target_names, title='Confusion matrix', cmap=None, normalize=True):\n",
    "    accuracy = np.trace(cm) / float(np.sum(cm))\n",
    "    misclass = 1 - accuracy\n",
    "    if cmap is None:\n",
    "        cmap = plt.get_cmap('Blues')\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    if target_names is not None:\n",
    "        tick_marks = np.arange(len(target_names))\n",
    "        plt.xticks(tick_marks, target_names, rotation=45)\n",
    "        plt.yticks(tick_marks, target_names)\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "    thresh = cm.max() / 1.5 if normalize else cm.max() / 2\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        if normalize:\n",
    "            plt.text(j, i, \"{:0.4f}\".format(cm[i, j]),\n",
    "                     horizontalalignment=\"center\",\n",
    "                     color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "        else:\n",
    "            plt.text(j, i, \"{:,}\".format(cm[i, j]),\n",
    "                     horizontalalignment=\"center\",\n",
    "                     color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label\\naccuracy={:0.4f}; misclass={:0.4f}'.format(accuracy, misclass))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "\n",
    "plot_confusion_matrix(confusion_matrix(y_test, y_pred), ['0', '1', '2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy to WML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**WARNING**: the model you deploy to Watson Machine Learning must receive as input for a prediction **ALL of the columns** provided in the dataset, **except the TARGET column**. Any operations with the columns, such as dropping, must be done via pipeline. If the model does not behave as expected, your submission will fail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the model running, now we can deploy it to Watson Machine Learning, a service available on the IBM Cloud capable of executing and making machine learning models available through an API in a dedicated environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning import APIClient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access Watson Machine Learning, you need to create an APIKEY. There are two ways to do this: via the IBM Cloud cli or via the IBM Cloud interface.\n",
    "\n",
    "If you want to create an APIKEY via API first download install the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli). Once installed, run the following commands to obtain the APIKEY\n",
    "\n",
    "ibmcloud login <br>\n",
    "ibmcloud iam api-key-create API_KEY_NAME\n",
    "\n",
    "Through the interface, just click on `Manage` and then on `Access(IAM)` as shown in the image below.\n",
    "\n",
    "![api-1](https://imgur.com/bS61qef.png \"api1\")\n",
    "\n",
    "As soon as the page loads, on the left side there is a menu. Click on API keys to create a new one, as shown in the image below.\n",
    "\n",
    "![api-2](https://imgur.com/XaOalxq.png \"api2\")\n",
    "\n",
    "The image below shows a panel with all its APIs created for the IBM Cloud platform, let's create a new one accessing the WML service by clicking on `Create an IBM Cloud API key`\n",
    "\n",
    "![api-3](https://imgur.com/0WKTanm.png \"api3\")\n",
    "\n",
    "A form will open where you simply name your API and click on `Create`. As soon as you click on the button your API will be created and just copy it, insert in `apikey` in the cell below.\n",
    "\n",
    "![api-4](https://imgur.com/3wCTLaH.png \"api4\")\n",
    "\n",
    "In addition to needing an APIKEY to access Watson Machine Learning, we need to know the URL where it is located, so be aware when creating the service in which region you are instantiating it. Each region has a specific URL and they are listed below.\n",
    "\n",
    "- Dallas: `https://us-south.ml.cloud.ibm.com`\n",
    "- London: `https://eu-gb.ml.cloud.ibm.com`\n",
    "- Frankfurt: `https://eu-de.ml.cloud.ibm.com`\n",
    "- Tokyo: `https://jp-tok.ml.cloud.ibm.com`\n",
    "\n",
    "With the WML properly located, just enter the correct URL in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wml_credentials = {\n",
    "  \"apikey\": \"YOUR_WML_APIKEY\",\n",
    "  \"url\": \"URL_REGION_OF_YOUR_WML\"\n",
    "}\n",
    "\n",
    "print(wml_credentials)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = APIClient(wml_credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the environment that will receive the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watson Machine Learning organizes the deployment of models in spaces, so that it is possible to use the organization's WML instance and divide it into small spaces dedicated to hosting the models that each department will build and make available. Therefore, after instantiating the WML, it is necessary to create a space to receive the model that we are going to create. To create a space in WML we need to go back to the Cloud Pak 4 Data home screen and click on `Deployments`, located on the left side, as shown in the image below.\n",
    "\n",
    "![img-01](https://imgur.com/Fhx5iKO.png \"deployment\")\n",
    "\n",
    "As soon as the page loads we are inside the deployment interface that constitutes a direct access to Watson Machine Learning. Now let's click on the `deployment space` button to create a new space, as shown in the image below.\n",
    "\n",
    "![img-02](https://imgur.com/DRFuLj6.png \"space\")\n",
    "\n",
    "Let's create an empty space to receive our model, as shown in the image below.\n",
    "\n",
    "![img-03](https://imgur.com/uxUf77y.png \"creat\")\n",
    "\n",
    "We must fill in some information now. We need to give the space a name, associate an Object Storage and Watson Machine Learning to the space. With the form completed just click on the `Create` button located in the lower right corner.\n",
    "\n",
    "<!-- ![img-04](https://i.imgur.com/trikImj.png \"form\") -->\n",
    "\n",
    "\n",
    "With the space created, we can deploy the created model and proceed with the execution of the cells of this notebbok."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.spaces.list(limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the spaces listed above, you should find your newly created space and copy the space id in the cell below to be stored in the `space_id` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "space_id = 'YOUR_SPACE_ID'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.set.default_space(space_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen during the creation of the Pipeline, we used a library external to the model creation framework (scikit-learn, tensorflow, keras, etc.) We need to upload our library so that the pipeline can use the methods contained there. For this, the cell below uploads the library so that the model can run correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_prop_pkg_extn = {\n",
    "    client.package_extensions.ConfigurationMetaNames.NAME: \"my_custom_sklearn_transforms\",\n",
    "    client.package_extensions.ConfigurationMetaNames.DESCRIPTION: \"Pkg extension for custom lib\",\n",
    "    client.package_extensions.ConfigurationMetaNames.TYPE: \"pip_zip\"\n",
    "}\n",
    "\n",
    "pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path=\"sklearn_transforms.zip\")\n",
    "pkg_extn_uid = client.package_extensions.get_uid(pkg_extn_details)\n",
    "pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "details = client.package_extensions.get_details(pkg_extn_uid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.software_specifications.ConfigurationMetaNames.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.software_specifications.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofware_spec_uid = client.software_specifications.get_id_by_name(\"default_py3.7\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the upload of the library, it must be made available by creating a specific software environment where it is available for use. The cell below creates an environment that does just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_prop_sw_spec = {\n",
    "    client.software_specifications.ConfigurationMetaNames.NAME: \"my_custom_sklearn_transforms\",\n",
    "    client.software_specifications.ConfigurationMetaNames.DESCRIPTION: \"Software specification for linalgnorm-0.1\",\n",
    "    client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {\"guid\": sofware_spec_uid}\n",
    "}\n",
    "\n",
    "sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)\n",
    "sw_spec_uid = client.software_specifications.get_uid(sw_spec_details)\n",
    "\n",
    "\n",
    "client.software_specifications.add_package_extension(sw_spec_uid, pkg_extn_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the environment created, just upload the model created to Watson Machine Learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\n",
    "            client.repository.ModelMetaNames.NAME: 'Final',\n",
    "            client.repository.ModelMetaNames.TYPE: 'scikit-learn_0.23',\n",
    "            client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: sw_spec_uid\n",
    "}\n",
    "\n",
    "published_model = client.repository.store_model(\n",
    "    model=model,\n",
    "    meta_props=metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.repository.list_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "saved_model_uid = client.repository.get_model_uid(published_model)\n",
    "model_details = client.repository.get_details(saved_model_uid)\n",
    "print(json.dumps(model_details, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a model stored in WML, it is now necessary to make the model available so that it is available to be accessed via an API call. To make a model available, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\n",
    "    client.deployments.ConfigurationMetaNames.NAME: \"champion\",\n",
    "    client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "}\n",
    "\n",
    "created_deployment = client.deployments.create(client.repository.get_model_uid(published_model), meta_props=metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deployment_uid = client.deployments.get_uid(created_deployment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_endpoint = client.deployments.get_scoring_href(created_deployment)\n",
    "print(scoring_endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.array(X.iloc[0].values).tolist())\n",
    "print(y.iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_payload = {\n",
    "    \"input_data\": [{\n",
    "        'fields': X.columns.to_list(),\n",
    "        'values': [[1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 9.48803557, 0.02775, -0.02775, 170.53875, 0.00216, -0.00216, 146.0, 318.0, -146.0, 0.0, 0.0819, -0.0819, 615.8, 19.5, 0.0, 2.26, 0.0, -0.15, 793.0, 93.59, 29.45, -16.65, 35.8, 1.0, 5455.0, 81.0, -81.0, 4467.0, 64.0, -96.0, 927.0, 105.0, -61.0, 291.93423, 48.141651, 15.347]]}]\n",
    "}\n",
    "scoring_payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = client.deployments.score(deployment_uid, scoring_payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(predictions, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Important things you will use next\n",
    "\n",
    "In the cell below you will find the necessaries credentials you must insert in the submission app you have deployed on Red Hat OpenShift and provide them at the submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('WML APIKEY: ', wml_credentials['apikey'])\n",
    "print('URL to make predictions: ', scoring_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "- [Cloud Pak 4 data docs](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/wml-ai.html)\n",
    "- [ibm-watson-machine-learning sdk docs](http://ibm-wml-api-pyclient.mybluemix.net)\n",
    "- [Watson Machine Learning REST API docs](https://cloud.ibm.com/apidocs/machine-learning)\n",
    "- [Watson Machine Learning tutorials](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-samples-overview.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}