examples/ML-IRIS-Extract-Models-From-Cache.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting the IRIS Models from Cache\n",
    "\n",
    "This notebook demonstrates how to extract the machine learning Models + Analysis from the redis cache **CACHE**\" and saved to disk as a compressed, string artifact file (Pickle + zlib compression). Once the file is saved, it is uploaded to the configured S3 Bucket for archiving and sharing.\n",
    "\n",
    "## Overview\n",
    "\n",
    "Extract the IRIS Regressor and Classification datasets from the redis **CACHE**. After extraction, compile a manifest for defining a cache mapping for all the Models + their respective Analysis. Once cached, the Models can be extract and shared + deployed on other Sci-pype instances by using something like this notebook or the command-line versions.\n",
    "\n",
    "### Command-line Versions\n",
    "\n",
    "I built this notebook from the extractor examples:\n",
    "\n",
    "https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/extractors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "source": [
    "### 1) Extract the IRIS Classifier Models + Analysis from the Cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Setup the Sci-pype environment\n",
    "import sys, os\n",
    "\n",
    "# Only redis is needed for this notebook:\n",
    "os.environ[\"ENV_DEPLOYMENT_TYPE\"] = \"JustRedis\"\n",
    "\n",
    "# Load the Sci-pype PyCore as a named-object called \"core\" and environment variables\n",
    "from src.common.load_ipython_env import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Setup the Request\n",
    "\n",
    "Extract the Models from the Cache with this request and upload them object files to the configured S3 Bucket.\n",
    "\n",
    "Please make sure the environment variables are set correctly and the S3 Bucket exists:\n",
    "\n",
    "   ```\n",
    "   ENV_AWS_KEY=<AWS API Key>\n",
    "   ENV_AWS_SECRET=<AWS API Secret>\n",
    "   ```\n",
    "\n",
    "For docker containers make sure to set these keys in the correct Jupyter env file and restart the container:\n",
    "\n",
    "   ```\n",
    "   <repo base dir>/justredis/jupyter.env\n",
    "   <repo base dir>/local/jupyter.env\n",
    "   <repo base dir>/test/jupyter.env\n",
    "   ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "- What's the dataset name?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds_name             = \"iris_classifier\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Where is the downloaded file getting stored?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_dir            = str(os.getenv(\"ENV_DATA_DST_DIR\", \"/opt/work/data/dst\"))\n",
    "if not os.path.exists(data_dir):\n",
    "    os.mkdir(data_dir, 0777)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What's the S3 Location (Unique Bucket Name + Key)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s3_bucket           = \"unique-bucket-name-for-datasets\" # name this something under your AWS Account (This might be open to the public in the future...stay tuned)\n",
    "s3_key              = \"dataset_\" + core.to_upper(ds_name) + \".cache.pickle.zlib\"\n",
    "s3_loc              = str(s3_bucket) + \":\" + str(s3_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Build the full request and run it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decompressing Analysis Dataset\n",
      "Finding ManifestKey(Accuracy) Records in RLoc(CACHE:_MD_IRIS_CLASSIFIER_Accuracy)\n",
      "Decompressing Key(Accuracy)\n",
      "Done Decompressing Key(Accuracy)\n",
      "Finding ManifestKey(PredictionsDF) Records in RLoc(CACHE:_MD_IRIS_CLASSIFIER_PredictionsDF)\n",
      "Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Analysis Dataset\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_SepalLength)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python2/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_0) Type(XGBClassifier) Target(SepalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_PetalLength)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_1) Type(XGBClassifier) Target(PetalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_PetalWidth)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_2) Type(XGBClassifier) Target(PetalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_SepalWidth)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_3) Type(XGBClassifier) Target(SepalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_ResultTargetValue)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_4) Type(XGBClassifier) Target(ResultTargetValue) FeatureNames(4)\n",
      "Sorting Predictions\n",
      "Done Decompressing Models(5)\n",
      "Found DSName(IRIS_CLASSIFIER) Analysis Created on Date(2017-01-09 08:02:07) Creating File(/opt/work/data/dst/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "Validating Serialization\n",
      "Checking Model Counts\n",
      "\u001b[32mDecompression Validation Passed - Models(5) == (5)\u001b[0m\n",
      "Done Creating File(/opt/work/data/dst/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "Uploading to DSName(IRIS_CLASSIFIER) Analysis to S3(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "\u001b[32mUploaded DSName(IRIS_CLASSIFIER) S3(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\u001b[0m\n",
      "Done Uploading to S3(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "Done Uploading Model and Analysis DSName(iris_classifier) S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "\n",
      "\u001b[32mExtract and Upload Completed\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cache_req           = {\n",
    "                        \"RAName\"        : \"CACHE\",      # Redis instance name holding the models\n",
    "                        \"DSName\"        : str(ds_name), # Dataset name for pulling out of the cache\n",
    "                        \"S3Loc\"         : str(s3_loc),  # S3 location to store the model file\n",
    "                        \"DeleteAfter\"   : False,        # Optional delete after upload\n",
    "                        \"SaveDir\"       : data_dir,     # Optional dir to save the model file - default is ENV_DATA_DST_DIR\n",
    "                        \"TrackingID\"    : \"\"            # Future support for using the tracking id\n",
    "                    }\n",
    "\n",
    "upload_results      = core.ml_upload_cached_dataset_to_s3(cache_req, core.get_rds(), core.get_dbs(), debug)\n",
    "if upload_results[\"Status\"] == \"SUCCESS\":\n",
    "    lg(\"Done Uploading Model and Analysis DSName(\" + str(ds_name) + \") S3Loc(\" + str(cache_req[\"S3Loc\"]) + \")\", 6)\n",
    "else:\n",
    "    lg(\"\", 6)\n",
    "    lg(\"ERROR: Failed Upload Model and Analysis Caches as file for DSName(\" + str(ds_name) + \")\", 6)\n",
    "    lg(upload_results[\"Error\"], 6)\n",
    "    lg(\"\", 6)\n",
    "# end of if extract + upload worked\n",
    "\n",
    "lg(\"\", 6)\n",
    "lg(\"Extract and Upload Completed\", 5)\n",
    "lg(\"\", 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Setup the Extract and Upload for the IRIS Regressor Models and Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds_name             = \"iris_regressor\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Build and Run the Extract + Upload Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decompressing Analysis Dataset\n",
      "Finding ManifestKey(Accuracy) Records in RLoc(CACHE:_MD_IRIS_REGRESSOR_Accuracy)\n",
      "Decompressing Key(Accuracy)\n",
      "Done Decompressing Key(Accuracy)\n",
      "Finding ManifestKey(PredictionsDF) Records in RLoc(CACHE:_MD_IRIS_REGRESSOR_PredictionsDF)\n",
      "Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Analysis Dataset\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalLength)\n",
      "Found Model(_MD_IRIS_REGRESSOR_e50b12_0) Type(XGBRegressor) Target(SepalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalLength)\n",
      "Found Model(_MD_IRIS_REGRESSOR_e50b12_1) Type(XGBRegressor) Target(PetalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalWidth)\n",
      "Found Model(_MD_IRIS_REGRESSOR_e50b12_2) Type(XGBRegressor) Target(PetalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalWidth)\n",
      "Found Model(_MD_IRIS_REGRESSOR_e50b12_3) Type(XGBRegressor) Target(SepalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_ResultTargetValue)\n",
      "Found Model(_MD_IRIS_REGRESSOR_e50b12_4) Type(XGBRegressor) Target(ResultTargetValue) FeatureNames(4)\n",
      "Sorting Predictions\n",
      "Done Decompressing Models(5)\n",
      "Found DSName(IRIS_REGRESSOR) Analysis Created on Date(2017-01-09 08:01:27) Creating File(/opt/work/data/dst/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "Validating Serialization\n",
      "Checking Model Counts\n",
      "\u001b[32mDecompression Validation Passed - Models(5) == (5)\u001b[0m\n",
      "Done Creating File(/opt/work/data/dst/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "Uploading to DSName(IRIS_REGRESSOR) Analysis to S3(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "\u001b[32mUploaded DSName(IRIS_REGRESSOR) S3(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\u001b[0m\n",
      "Done Uploading to S3(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "Done Uploading Model and Analysis DSName(iris_regressor) S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "\n",
      "\u001b[32mExtract and Upload Completed\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cache_req           = {\n",
    "                        \"RAName\"        : \"CACHE\",      # Redis instance name holding the models\n",
    "                        \"DSName\"        : str(ds_name), # Dataset name for pulling out of the cache\n",
    "                        \"S3Loc\"         : str(s3_loc),  # S3 location to store the model file\n",
    "                        \"DeleteAfter\"   : False,        # Optional delete after upload\n",
    "                        \"SaveDir\"       : data_dir,     # Optional dir to save the model file - default is ENV_DATA_DST_DIR\n",
    "                        \"TrackingID\"    : \"\"            # Future support for using the tracking id\n",
    "                    }\n",
    "\n",
    "upload_results      = core.ml_upload_cached_dataset_to_s3(cache_req, core.get_rds(), core.get_dbs(), debug)\n",
    "if upload_results[\"Status\"] == \"SUCCESS\":\n",
    "    lg(\"Done Uploading Model and Analysis DSName(\" + str(ds_name) + \") S3Loc(\" + str(cache_req[\"S3Loc\"]) + \")\", 6)\n",
    "else:\n",
    "    lg(\"\", 6)\n",
    "    lg(\"ERROR: Failed Upload Model and Analysis Caches as file for DSName(\" + str(ds_name) + \")\", 6)\n",
    "    lg(upload_results[\"Error\"], 6)\n",
    "    lg(\"\", 6)\n",
    "    sys.exit(1)\n",
    "# end of if extract + upload worked\n",
    "\n",
    "lg(\"\", 6)\n",
    "lg(\"Extract and Upload Completed\", 5)\n",
    "lg(\"\", 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automation with Lambda - Coming Soon\n",
    "\n",
    "Native lambda uploading support will be added in the future. Packaging and functionality still need to be figured out. For now, you can extend the command line versions for the extractors below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Command-line Versions\n",
    "\n",
    "I built this notebook from the extractor examples:\n",
    "\n",
    "https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/extractors"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}