Variable Selection in Azure ML Jupyter Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 1 Introduction\n",
    "Before starting this example, you should be familiar with Azure Machine Learning (Azure ML) experiments and the process for publishing Azure web service from Python notebooks. \n",
    "\n",
    "The purpose of this example is to demonstrate how to use a feature selection technique not available for Azure ML experiments. This process can be completed with Azure ML Jupyter notebook, a cloud-based service that's part of Azure ML environment.\n",
    "\n",
    "As [Figure 1][pic 1] shows, there are three Feature Selection options for an Azure ML experiment:\n",
    " 1. Filter Based Feature Selection\n",
    " 2. Fisher Linear Discriminant Analysis\n",
    " 3. Permutation Feature Importance\n",
    "\n",
    "Among these, Fisher Linear Discriminant Analysis is actually a dimension reduction technique for cases where the response variable is categorical. Filter Based Feature Selection uses univariate analysis to select features (using metrics such as Pearson Correlation, Chi Squared, etc). Permutation Feature Importance ranks variables by calculating the difference in model performance metrics before and after random permutations of feature values. As the paper [Dimension Reduction vs. Variable Selection][paper link] describes, there are a wide variety of feature selection and dimension reduction techniques. In what follows we'll illustrate how to use recursive feature elimination within an Azure ML Jupyter notebook. \n",
    "\n",
    "[![Figure 1][pic 1]][pic 1] Figure 1\n",
    "\n",
    "[paper link]: http://rd.springer.com/chapter/10.1007%2F11558958_113\n",
    "[pic 1]: https://az712634.vo.msecnd.net/notebooks/4e5ae462-71c1-11e5-8c01-a96769336daf.PNG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 2 Read data\n",
    "In this example we'll be using the Boston Housing dataset. There are 506 rows in the dataset. The target variable is median home price. There are 13 predictor variables including average number of rooms per dwelling, crime rate by town, etc. More information about this dataset can be found at [UCI][uci link].\n",
    "[uci link]: https://archive.ics.uci.edu/ml/datasets/Housing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(506, 13)\n",
      "(506,)\n",
      "['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'\n",
      " 'B' 'LSTAT']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()\n",
    "X = boston.data\n",
    "y = boston.target\n",
    "feature_names = boston.feature_names\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Recursive Feature Elimination\n",
    "The sklearn package has several feature selection methods and details can be found at the [documentation][feature link] site. In this example we'll be using the recursive feature elimination method. This process starts with all variables and assigns weights to them. Those with small weights are first removed. The remaining variables are used to refit the model and the process repeats. \n",
    "\n",
    "We start by importing the packages that'll be used. \n",
    "\n",
    "[feature link]: http://scikit-learn.org/stable/modules/feature_selection.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn import preprocessing\n",
    "from sklearn.feature_selection import RFECV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following lines of code first scale the features then identify the selected features with recursive feature elimination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal number of features : 6\n",
      "Features selected: ['CRIM', 'NOX', 'RM', 'DIS', 'PTRATIO', 'LSTAT']\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEPCAYAAABV6CMBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3WmYXGW57vH/nQSEGBMIYwATEowgbBRBQIYNDREIchDk\nOIBscZYNYlA8bhBlEzyIICKi4IADAgbxgLoZJBKmBsMYQ8KY0B0hJIRBkRAIkJB0nvPhXU1Xmh6q\nuqt71aq6f9dVV1ZVrar1dKe773qH9S5FBGZmZuUakncBZmZWLA4OMzOriIPDzMwq4uAwM7OKODjM\nzKwiDg4zM6tI7sEhabKk+ZJaJJ3czT5NkuZIeljSbYNdo5mZdVCe53FIGgK0AJOAp4FZwJERMb9k\nn1HAXcCBEbFE0sYR8XwuBZuZWe4tjt2A1oh4MiJWAVcCh3Xa5xPAHyJiCYBDw8wsX3kHx5bA4pL7\nT2WPlXonMFrSbZJmSfrkoFVnZmZvMizvAsowDNgZ2B94K3C3pLsjYkG+ZZmZNaa8g2MJMLbk/lbZ\nY6WeAp6PiBXACkl3AO8B3hQckrzwlplZhSJCleyfd1fVLOAdksZJWhc4Eri20z7XAHtLGippOLA7\nMK+7N4yIQt5OP/303Gtw/fnX4fqLeSty/X2Ra4sjItoknQDMIIXYryJinqRj09NxcUTMl3Qj8CDQ\nBlwcEY/mWLaZWUPLu6uKiPgLsG2nx37e6f73ge8PZl1mZta1vLuqLNPU1JR3Cf3i+vPl+vNV9Por\nlesJgNUmKerp6zEzG2iSiIINjpuZWcE4OMzMrCIODjMzq4iDw8zMKuLgMDOzijg4zMysIg4OMzOr\niIPDzMwq4uCwhtbWBjNmwMMP512JWXHkvlaVWR4eegguuwymTYMxY2DJEpg1C97+9rwrM6t9bnFY\nw3juOfjhD+G974UPfhDWWQduuQVmz4avfAWOPBJWrcq7SrPa57WqrK6tWAHXXptaFzNnwmGHwTHH\nQFMTDB3asd+aNXDoofCud8H3vQ6zNZC+rFXl4LC6EwF33ZXC4uqrUwvjU5+CD38YRozo/nX/+hfs\nsgtccEEKGLNG0Jfg8BiH1Y3HH4fLL0+3ddZJYTF3bvnjFhttBL//fWp57LgjTJgwsPWaFZVbHFZo\ny5bBVVel1sX8+Wmc4phjUstBFX2G6nDBBSl8Zs6E9darbr1mtcZdVQ6OhrB6Ndx0E1x6KUyfDh/4\nQAqLgw+Gddft//tHwEc+AptvDhdd1P/3M6tlDg4HR1174IHUsrjiChg3LoXFxz+eupiqbdmy1Go5\n88zUijGrVw4OB0dduvNO+NKXYOlS+OQn023bbXt/XX/NmQMHHpi6rAbjeGZ5cHA4OOrOww/DpEnw\nk5+kWVFDBvnMo4svhgsvhHvugeHDB/fYZoPBweHgqCuLF8Nee8HZZ8MnPpFPDRGphbPuuvDrX+dT\ng9lA8jXHrW4sXZoGu6dMyS80IM3M+tnPUovjkkvyq8OslrjFYTVnxQo46CDYeWf4wQ/6Pq22mh59\nFPbdF269NZ3jYVYv3FXl4Ci8trY0i2nIEPjd7wZ/TKMnl18O3/lOWgzxbW/Luxqz6nBwODgKLQJO\nPDGtXPuXv8Bb3pJ3RW/2hS/A8uVpSnAttITM+stjHFZo554Lzc3wpz/VZmgA/OhHMG9eGvcwa1Ru\ncVhNuPxy+Na30uKEW26ZdzU9a22FPfdMraJddsm7GrP+cVeVg6OQZsxIU15vuw223z7vaspz1VVw\nyinpWh4bbJB3NQZpJt4TT6TFLp94ouP21FMd12A58EAYPTrvSmuLg8PBUTj3359mUP3pT7D33nlX\nU5kpU9K5Jn/8o8c7BsNrr8HChWuHQmlIrFmTVjQeP77jNmFCWnPsvvvghhvg9tvTrLgPfjBN995p\np9qagJEHB4eDo1Aefxz+/d/hxz+GI47Iu5rKrVyZ6j/ySDjppLyrKb62ttQ66C4YXngBxo59czC0\nb48e3XuAr1gBd9yRQuSGG+Dll1OAHHwwHHBAY7YeHRwOjsJ4/vk0TnDiiWkdqqJauBB23z21mPbc\nM+9qiiUirXL885+nBSwXL4aNN+661TB+PGyxxdpXbayGBQvSCsvTp8Nf/5rOHTr44NQi2XHHxmhJ\nOjgcHIXwyitp/an994ezzsq7mv67/no4/vjU7bbxxnlXU/tefjktiX/hhWn23AknwD77pBWP87z+\nyauvpll906en1sjKlR0hMmkSjByZX20DycHh4Kh5q1enxQpHj4bf/KZ+PtGdfHL61HzDDe4z705r\nawqL3/42fWj48pdTV18t/gxEQEtLR4jcfTfsumvH2Mj229dm3X3h4HBw1LQI+OIXU5fEddely7vW\ni1Wr0h/DyZPhm9/Mu5rasWYN3HhjOv9l9mz4/OfhuOPKv5xvrXjllbTcTHuQRHSEyP7793wt+1rn\n4HBwdOmOO+Dmm9M1uLfZJr86pk5N3TrNzcX+RevOkiXwvvels8r32y/vavK1bFlqUV50Ufq/njIl\nTSKoh0vxRqSTQNtD5M47U0BKqbU5ZEjHdjUeGz48he5AcXA4ONYye3b69NvSkj4d/f73qS/5a18b\n/IHcX/wiLY9+112w2WaDe+zBdNNN8OlPp+/95pvnXc3gmz8/dUddcUU6Z+LLX04/a/XSrdOVtrZ0\nW7MmhcqaNWtv9/exiIE9v8nB4eAA0qeh005L/bLf+hZ87nPpehLLl6dPgeefD5tumqaQfvjDMGzY\nwNZz3XWpi+qOO2DixIE9Vi2YOjWdL3DTTQP/va0FbW3p0/ePfgQPPpjW8/rP/6z9FQAscXA0eHAs\nXAhnnAF//jN8/etpmmtXV61ra4NrroHzzoNnnoGvfAU++9mB6T665x449NBU0267Vf/9a1FbWxrr\n2H33dM3yevXii+niVhddlCY7TJkCH/tY7a4zZl3zIocN6tln0y/tLrukQcfW1hQc3V3qdOjQdMLd\nnXemLoW//hW23jotobFkSfXqeuwxOPzwNPWyUUID0vd32rTUups+Pe9qqu/RR9MA9/jxqUtu2rR0\nZvYnP+nQaBQOjgJbuhROPRV22CF1icybB9/+NowaVf57vP/9ad2l++5LSzrsuCMcc0yaWtofzz6b\nZpycdVYaX2k0m26aQvkzn0mzyIquvZU6aVK6bbZZCpBp09LPUD2PYdib5d5VJWky8ENSiP0qIs7p\nZr9dgbuAj0fEH7vZpyG6qpYvT/3J55+fxihOO6160xuXLoWLL07LgGy3XRpInzy5sj8ML70ETU0d\ntTWyc85Jf3Bvv71Y049feCG1XFtb0weSK65IYTFlCnzkI2nMzOpD4cY4JA0BWoBJwNPALODIiJjf\nxX43Aa8Bv27U4Fi5Mv1RP+us9If5jDPgne8cmGO9/nqahXXeeekchZNOgqOP7n065euvwyGHpGm/\nP/2pP4muWZMCdPbstHTGuHFpvaWxY9fezmN68osvdoRD6W3BgnSi5sSJ8I53pH8PPbSxuhsbSRGD\n4/3A6RFxcHb/FCA6tzoknQi8DuwKXN9owbF6dbpexRlnwL/9Wxpw3WmnwTl2RDrx6bzzYM6ctLTG\nccd1vbTGmjWpm+vll+EPf2iMGUXlaGuDJ59Mt0WL0q10e9EiWH/9rgOlfXuzzfp2RvqyZR1h0Dkg\nVq5MoVAaEO23TTZx6DeKvgRH3r/aWwKlPcBPAWt9rpG0BXB4ROwnqaE+86xZk5bsPu201Gc+bRrs\ntdfg1iB19Gs/8kjqHnvnO+HjH4evfnXtFs83vpFWM735ZodGqaFDU2tjwoSun49Iiz52DpS77+7Y\nXrYMttqq63AZOzad2dxVQLz66tqh0NSUpstOnJjCyOFgfVGEX+8fAieX3O/xR33q1KlvbDc1NdHU\n1DQgRQ2kiLRMwze/mX6xL7ggLfmc9y/5DjvAL38J3/lOmoK5996wxx5pHGTu3NSXf+ed3c/msq5J\n6RP+Jpt0f0XB115Lg+ylLZaZMzu2hw/vCIe9906D8hMnwpgx+f/cWG1pbm6mubm5X+9RC11VUyNi\ncnb/TV1Vkh5v3wQ2Bl4BvhgR13bxfoXvqpo5M82U+uc/U5fUEUfU7i/+q6+mqbbnn58G7O+6K03r\nNbPiKOIYx1DgMdLg+DPAfcBRETGvm/0vAa6rxzGOOXPSWd6PPJLGMv7jP6p/7YGBsmZNGofxTBuz\n4incGEdEtEk6AZhBx3TceZKOTU/HxZ1fMuhFDoJnnkmL4p15ZhrTKNpJVEOGODTMGknu53FUU1Fb\nHLfckkLjttvyrsTMGo2XHCmolpbGWPzPzOqDg6MGtLYO3Il8ZmbV5uCoAS0tDg4zKw4HRw1wcJhZ\nkXhwPGerVsHb3pYWBvTMJDMbbB4cL6AnnkhLSTg0zKwoHBw5czeVmRWNgyNnDg4zKxoHR858DoeZ\nFY2DI2c+h8PMisbBkTN3VZlZ0Xg6bo5eeSVdg2H58r5d3c3MrL88HbdgFixI1+Z2aJhZkfhPVo7c\nTWVmReTgyJGDw8yKyMGRIweHmRWRgyNHDg4zKyIHR4588p+ZFZGDIyf/+he0taXpuGZmReLgyEn7\nGeOqaPa0mVn+HBw58fiGmRWVgyMnDg4zKyoHR04cHGZWVA6OnDg4zKyovMhhDtasSdcZf/bZ9K+Z\nWV68yGFBPP00jBzp0DCzYnJw5MAXbzKzInNw5MDjG2ZWZA6OHDg4zKzIHBw5cHCYWZE5OHLg4DCz\nIvN03EG2alWaTbVsGbzlLXlXY2aNztNxC2DhQthyS4eGmRWXg2OQ+RocZlZ0w3p6UtIRPT0fEX+s\nbjn1z+MbZlZ0PQYHcGj276bAnsCt2f39gLsAB0eFWlth++3zrsLMrO96DI6I+AyApBnA9hHxTHZ/\nDPCbAa+uDrW0wOGH512FmVnflTvG8fb20Mg8B4wdgHrqnruqzKzoeuuqaneLpBuB32X3Pw7cPDAl\n1a9XX4V//hPe/va8KzEz67uygiMiTpD0YWCf7KGLI+JPA1dWfVqwALbZBoYOzbsSM7O+q2Q67v3A\nnyPiq8CNkqqyKLikyZLmS2qRdHIXz39C0gPZbaakHatx3Dy4m8rM6kFZwSHpC8DVwM+zh7YE/qe/\nB5c0BLgQOAjYAThK0naddnsc2Cci3gOcCfyiv8fNi8/hMLN6UG6L40vAXsBLABHRSpqi21+7Aa0R\n8WRErAKuBA4r3SEi7omIZdnde0ihVUhucZhZPSg3OFZGxOvtdyQNA6qxKNSWwOKS+0/RczB8Hphe\nhePmwsFhZvWg3FlVt0s6FVhf0gHA8cB1A1fWm0naD/gMsPdgHreafOU/M6sH5QbHKcDngIeAY4Eb\ngF9W4fhLWPt8kK2yx9Yi6d3AxcDkiFja0xtOnTr1je2mpiaampqqUGb/vfACvP46bFqNDj4zsz5q\nbm6mubm5X+/R67LqkoYCl0XE0f06Uvfv/RgwCXgGuA84KiLmlewzFrgF+GRE3NPL+9Xssur33gsn\nnACzZuVdiZlZh74sq95riyMi2iSNk7Ru6ThHNWTvfQIwgzTe8quImCfp2PR0XAycBowGfiJJwKqI\n2K2adQwGj2+YWb0o60JOki4D3gVcC7zS/nhE/GDgSqtcLbc4TjsNhg2D00/PuxIzsw4DeSGnvwPX\nZ/u/reRmZXKLw8zqRblLjpwx0IXUO5/8Z2b1otyuqk2A/yKd3b1e++MRsf/AlVa5Wu2qikjXGV+y\nBEaNyrsaM7MOA9lVNQ2YD4wHzgAWAp4fVKZnnoERIxwaZlYfyg2OjSLiV6QZTbdHxGeBmmpt1DKP\nb5hZPSn3BMBV2b/PSDoEeJo0RdbK4OAws3pSbnCcKWkU8DXgx8BI4KsDVlWdcXCYWT0pd1bV9dnm\nMmC/gSunPrW0wN6FXWHLzGxtZQWHpEvoYjXcbKzDeuEWh5nVk3K7qq4v2V4P+DBpnMN6sXo1LFwI\nEybkXYmZWXWU21X1h9L7kn4HzByQiurMwoUwZgyst16vu5qZFUIl1xwvNZHqXAGw7rmbyszqTblj\nHC+TxjiU/fsscPIA1lU3fPEmM6s35XZVeUHDPmppge22y7sKM7PqKbfFsXNPz0fE/dUpp/60tMCH\nPpR3FWZm1VPurKqfADsDD5K6q94N/A1YQeq68vIj3fAYh5nVm3IHx58GdomI90XELsB7gSURsV+t\nrZBbS157Df7xDxg7tvd9zcyKotzg2DYiHmq/ExEPk64IaD1YsADGj4ehQ/OuxMysesrtqnpQ0i+B\n32b3jyZ1W1kP3E1lZvWo3OD4DHAccGJ2/w7gpwNSUR1xcJhZPSp3Ou4K4HzgfEmjga2yx6wHra2w\n5555V2FmVl1ljXFIapY0MguN2cAvJJ0/sKUVn1scZlaPyh0cHxURLwFHAJdFxO7ApIErqz44OMys\nHpUbHMMkjQE+xtor5Vo3li6FFStgs83yrsTMrLrKDY5vAzcCCyJilqQJQOvAlVV87WtUSXlXYmZW\nXeUOjl8FXFVy/3Hgfw9UUfXA3VRmVq8qXlZdktelKkNLC0ycmHcVZmbV15frcbjzpQxucZhZvepL\ncPy56lXUIQeHmdUrRUTlL5KGAEdFxLTql9R3kqIvX0+1RcDIkbB4MWywQd7VmJl1TxIRUVFPUo8t\njuykv29IulDSgUpOAB4nTc21Ljz7LAwf7tAws/rU26yqy4GlwN3A54FTSWMch0fE3AGurbDcTWVm\n9ay34JgQETsCZKvjPgOM9TpVPXNwmFk9621wfFX7RkS0AU85NHrn4DCzetZbi+M9kl7KtgWsn90X\nEBExckCrK6iWFthjj7yrMDMbGD0GR0T42nV94BaHmdWzPk3HrVW1MB139WoYMSItcrj++rmWYmbW\nq6pPx7XKLVoEm2/u0DCz+uXgqDJ3U5lZvXNwVJmDw8zqnYOjyhwcZlbvcg8OSZMlzZfUIunkbvb5\nkaRWSXMl7TTYNVbCwWFm9S7X4MgWS7wQOAjYAThK0nad9jkY2CYiJgLHAj8b9EIr4OtwmFm9y7vF\nsRvQGhFPRsQq4ErgsE77HAZcBhAR9wKjJNXklbxfey0tcDhuXN6VmJkNnLyDY0tgccn9p7LHetpn\nSRf71IS//x3Gj4dhZV2Q18ysmOruT9zUqVPf2G5qaqKpqWnQju3xDTOrdc3NzTQ3N/frPXI9c1zS\n+4GpETE5u38KaQ2sc0r2+RlwW0T8Prs/H9g3Ip7r4v1yPXP8nHPg+efh3HNzK8HMrCJFPHN8FvAO\nSeMkrQscCVzbaZ9rgWPgjaB5savQqAVucZhZI8g1OLKl2k8AZgCPAFdGxDxJx0r6YrbPDcATkhYA\nPweOz63gXjg4zKwReJHDKtpsM5g7F8aMya0EM7OKFLGrqm68+CK8+mpa4NDMrJ45OKqktTWd+KeK\nctvMrHgcHFXi8Q0zaxQOjipxcJhZo3BwVImDw8wahYOjSlpbHRxm1hg8HbcKImDUKHjySdhww0E/\nvJlZn3k6bk6eew7WW8+hYWaNwcFRBR7fMLNG4uCoAl+8ycwaiYOjCtziMLNG4uCoAgeHmTUSB0cV\nODjMrJF4Om4/tbXBiBHwwguw/vqDemgzs37zdNwcLFoEm27q0DCzxuHg6Cd3U5lZo3Fw9JODw8wa\njYOjn3wOh5k1GgdHP7nFYWaNxsHRTw4OM2s0no7bDytWwAYbwPLlMGzYoB3WzKxqPB13kP3977D1\n1g4NM2ssDo5+8MWbzKwROTj6weMbZtaIHBz94OAws0bk4OgHn8NhZo3IwdEPbnGYWSNycPTRsmVp\nGu4WW+RdiZnZ4HJw9FFra+qmUkWzn83Mis/B0UfupjKzRuXg6COfw2FmjcrB0UducZhZo3Jw9JGD\nw8walYOjDyJ8DoeZNS4HRx/84x+wzjowenTelZiZDT4HRx+4m8rMGpmDow8cHGbWyBwcfeDgMLNG\n5uDoAweHmTUyB0cf+OQ/M2tkvuZ4hdraYMQI+Ne/YPjwAT2UmdmAK9Q1xyVtKGmGpMck3ShpVBf7\nbCXpVkmPSHpI0pQ8ai21eDFssolDw8waV55dVacAN0fEtsCtwDe62Gc1cFJE7ADsAXxJ0naDWOOb\n+MQ/M2t0eQbHYcCl2falwOGdd4iIZyNibra9HJgHbDloFXbBA+Nm1ujyDI5NI+I5SAEBbNrTzpK2\nBnYC7h3wynrg4DCzRjdsIN9c0k3AZqUPAQF8q4vdux3VljQCuBo4MWt5dGvq1KlvbDc1NdHU1FR+\nwWVoaYGDDqrqW5qZDZrm5maam5v79R65zaqSNA9oiojnJG0O3BYR7+piv2HA9cD0iLigl/cc8FlV\nEybAjTd6nMPM6kOhZlUB1wKfzrY/BVzTzX6/Bh7tLTQGw8qV8PTTsPXWeVdiZpafPIPjHOAASY8B\nk4CzASSNkXR9tr0XcDSwv6Q5ku6XNDmvgh9/HMaNSyvjmpk1qgEd4+hJRLwAfKCLx58B/le2fScw\ndJBL65YHxs3MvORIRXwOh5mZg6MibnGYmTk4KuLgMDNzcFTEwWFm5uAo20svpdsWW+RdiZlZvhwc\nZWptTQPjQ/wdM7MG5z+DZfLFm8zMEgdHmTy+YWaWODjK5OAwM0scHGXyyX9mZomDowwRbnGYmbVz\ncJRh1So47jjYaKO8KzEzy19u1+MYCINxPQ4zs3pStOtxmJlZATk4zMysIg4OMzOriIOjRvT34vF5\nc/35cv35Knr9lXJw1Iii/+C5/ny5/nwVvf5KOTjMzKwiDg4zM6tI3Z3HkXcNZmZFU+l5HHUVHGZm\nNvDcVWVmZhVxcJiZWUXqIjgkTZY0X1KLpJPzrqcSkraSdKukRyQ9JGlK3jVVStIQSfdLujbvWvpC\n0ihJV0mal/0/7J53TeWS9FVJD0t6UNI0SevmXVNPJP1K0nOSHix5bENJMyQ9JulGSaPyrLEn3dT/\nvexnZ66kP0gamWeNPemq/pLnviZpjaTRvb1P4YND0hDgQuAgYAfgKEnb5VtVRVYDJ0XEDsAewJcK\nVj/AicCjeRfRDxcAN0TEu4D3APNyrqcskrYAvgzsHBHvBoYBR+ZbVa8uIf2uljoFuDkitgVuBb4x\n6FWVr6v6ZwA7RMROQCvFqx9JWwEHAE+W8yaFDw5gN6A1Ip6MiFXAlcBhOddUtoh4NiLmZtvLSX+0\ntsy3qvJlP3AfBH6Zdy19kX06/PeIuAQgIlZHxEs5l1WJocBbJQ0DhgNP51xPjyJiJrC008OHAZdm\n25cChw9qURXoqv6IuDki1mR37wG2GvTCytTN9x/gfODr5b5PPQTHlsDikvtPUaA/vKUkbQ3sBNyb\nbyUVaf+BK+r0vPHA85IuybrbLpa0ft5FlSMingbOAxYBS4AXI+LmfKvqk00j4jlIH6SATXOupz8+\nC0zPu4hKSPoQsDgiHir3NfUQHHVB0gjgauDErOVR8yQdAjyXtZiU3YpmGLAzcFFE7Ay8Suo6qXmS\nNiB9Wh8HbAGMkPSJfKuqikJ+CJH0TWBVRFyRdy3lyj4knQqcXvpwb6+rh+BYAowtub9V9lhhZN0M\nVwOXR8Q1eddTgb2AD0l6HPgdsJ+ky3KuqVJPkT5t/S27fzUpSIrgA8DjEfFCRLQBfwT2zLmmvnhO\n0mYAkjYH/pFzPRWT9GlSl23RgnsbYGvgAUlPkP5+zpbUY6uvHoJjFvAOSeOyGSVHAkWb3fNr4NGI\nuCDvQioREadGxNiImED6vt8aEcfkXVclsi6SxZLaryg/ieIM9C8C3i9pPUki1V6Egf3OrdNrgU9n\n258Cav3D01r1S5pM6q79UESszK2q8r1Rf0Q8HBGbR8SEiBhP+iD13ojoMbwLHxzZJ60TSDMbHgGu\njIgi/PIAIGkv4Ghgf0lzsn72yXnX1WCmANMkzSXNqjor53rKEhH3kVpIc4AHSH8MLs61qF5IugK4\nC3inpEWSPgOcDRwg6TFS+J2dZ4096ab+HwMjgJuy39+f5FpkD7qpv1RQRleVlxwxM7OKFL7FYWZm\ng8vBYWZmFXFwmJlZRRwcZmZWEQeHmZlVxMFhZmYVcXBYVWXLMp9bcv9rkv67Su99iaQjqvFevRzn\nI5IelXRLF8+dmy1/f04f3vc9kg6uTpUDQ9LLfXzdYX1Z1bmvx7N8OTis2lYCR5Szpv9gkjS0gt0/\nB3w+IiZ18dwXgHdHRF+u+7ITaVmKimRnhQ+Wvp7YdTjpsgaDdTzLkYPDqm016ezlkzo/0bnF0P5p\nU9K+kpol/Y+kBZK+K+kTku6V9ICk8SVvc4CkWUoX7joke/2Q7GI692YX0/lCyfveIeka0qoCnes5\nKrsA0oOSvps9dhqwN/Crzq2K7H1GkNby+aikjSVdnR33Xkl7ZPvtKukuSbMlzZQ0UdI6wLeBj2Vn\nF39U0umSTip5/4ckjc2Wz5kv6VJJDwFbSToge8+/Sfq9pOHZa85WupDTXEnf6+Jr3KdkRYLZkt6a\nPf5/JN2Xve70zq/raR9Jx2T/L3OyGvcAPgR8LzvOeEkTJE3P/q9uV7aki6Sts6/jAUn/t6vjWgFE\nhG++Ve0GvET64/oE8Dbga8B/Z89dAhxRum/2777AC6TltNclrZdzevbcFOAHJa+/Idt+B2k5/XVJ\nrYBTs8fXJa1fNi5735eBsV3UOYZ00ZrRpA9Qt5DWGgK4jbReT5dfX8n2NGDPbPvtpPXGyL7+Idn2\nJODqbPtTwI9KXn866SJe7fcfJC3YOY4UwLtmj28E3A6sn93/L+BbWe3zS14/sot6rwX2yLaHk67f\ncQDw8+wxAdcBe3f6P+lyH2B7YD6wYfbcBt38394MbJNt7wbckm1fAxydbR9f+v30rTi3YZhVWUQs\nl3Qp6cqAr5X5slmRLawm6e+ktccAHgKaSvb7f9kxFmT7bQccCOwo6aPZPiOBicAq4L6IWNTF8XYF\nbouIF7JjTgP2oWOBzO66h0of/wDwrpKupBFZS2AD4DJJE0ldMeX+npW+95MRMSvbfj/pD/ad2bHW\nIa03tAx4TdIvgT8D13fxnncC52df3x8jYomkA0ktt/uzY76V9P2aWfK67vZ5K3BVRCwFiIgX3/RF\npFbNnsC0Q9gnAAACc0lEQVRVJd+bdbJ/9wLaW52XU8PrUln3HBw2UC4A7id9Em23mqx7NPuDUnp9\n7NJVRdeU3F/D2j+npX3iomNRti9HxE2lBUjaF3ilhxr7MnbQ+fi7R7ryZOlxLyKtFHyEpHGkFkxX\n3vh+ZNYr2S6tW8CMiDi68xtI2o3UqvkoabHPtcZlIuIcSdcDhwAzlRbQFPDdiPhF919m1/tIOqGH\n17QbAiyNdH2TzoKO72ERr99ieIzDqq99uealpNbB50qeWwi8L9s+jI5PoZX4qJJtSFfvewy4EThe\n6bomZGMKw3t5n/uAfSSNVho4PwpoLuP4pX/sZpBaVWTHfU+2OZKOa8KUrj76cvZcu4Vk1/6QtHP2\n9XR1nHuAvbKvGUnDs6/xraSuor+QxpTe/aZipQkR8UhEfA/4G7At6fv12ZLxji0kbdzpuF3tswnp\nmuAfVTb5QdKGnb+2iHgZeELSR0rqaK/tTtL3GtKq0FZADg6rttJP5OeR+ufbH/sFsK+kOaTul+5a\nAz3NtFlE+qP/Z+DYiHiddL3zR4H7s8Hkn5H68rsvMl2i9BRSWMwhdZW1d/X0dPzS504E3pcN9D4M\nHJs9fi5wtqTZrP07dhuwffvgOPAHYKOs5uNJIfim40TE86TrVfxO0gOkbqptSWNI12eP3QF8tYt6\nv5INus8FXgemZy2zK4C7JT0IXJW91xvH7WafERHxKPAd4Pbs//G87HVXAl/PBuDHk0Lhc9nA+sOk\nwXOArwBfymoe0+V32Gqel1U3M7OKuMVhZmYVcXCYmVlFHBxmZlYRB4eZmVXEwWFmZhVxcJiZWUUc\nHGZmVhEHh5mZVeT/AyO5gcVOQu20AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa58f986e90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# initiate the linear model\n",
    "lm = LinearRegression()\n",
    "\n",
    "# scale the features\n",
    "min_max_scaler = preprocessing.MinMaxScaler()\n",
    "x_scaled_minmax = min_max_scaler.fit_transform(X)\n",
    "x_scaled_minmax_df = pd.DataFrame(x_scaled_minmax, columns = feature_names)\n",
    "\n",
    "# recursive feature elimination with cross validation, using r-squared as metric\n",
    "rfecv = RFECV(estimator=lm, step=1, cv=5, scoring='r2')\n",
    "rfecv.fit(x_scaled_minmax_df, y)\n",
    "\n",
    "# print the optimal number of feature\n",
    "print(\"Optimal number of features : %d\" % rfecv.n_features_)\n",
    "\n",
    "# save the selected features\n",
    "feature_cols_selected = list(np.array(feature_names)[rfecv.support_])\n",
    "print(\"Features selected: \" + str(feature_cols_selected))\n",
    "\n",
    "# plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"R-squared\")\n",
    "plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the features to include are selected, we can use them to fit the model. While the scaled features were used in the RFE step above, we'll use the original values for fitting the model. This way we don't have to tranform the data when making predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The R-squared value is: 0.7113 \n",
      "\n",
      "    Features       Coef\n",
      "0  intercept  35.629382\n",
      "1       CRIM  -0.071002\n",
      "2        NOX -17.043777\n",
      "3         RM   4.246403\n",
      "4        DIS  -1.204040\n",
      "5    PTRATIO  -1.001903\n",
      "6      LSTAT  -0.554361\n"
     ]
    }
   ],
   "source": [
    "#%% fit model with selected features\n",
    "X_new = X[:,rfecv.support_]\n",
    "lm2 = LinearRegression()\n",
    "lm2.fit(X_new, y)\n",
    "\n",
    "# print the R-squared\n",
    "print(\"The R-squared value is: {0:0.4f} \\n\".format(lm2.score(X_new, y)))\n",
    "# save intercept and coefficients\n",
    "param_df = pd.DataFrame({\"Features\": ['intercept'] + feature_cols_selected, \"Coef\": [lm2.intercept_] + list(lm2.coef_)})\n",
    "cols = param_df.columns.tolist()\n",
    "cols = cols[-1:]+cols[:-1]\n",
    "param_df = param_df[cols]\n",
    "print(param_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll use the model to make predictions. Typically, predictions are done on a validation dataset. Here, however, the training dataset is used for illustraton purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>DIS</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "      <th>predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>15.3</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "      <td>31.365126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>17.8</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "      <td>26.018747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>17.8</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "      <td>32.095787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>18.7</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "      <td>29.872827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>18.7</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "      <td>29.178013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    NOX     RM     DIS  PTRATIO  LSTAT  MEDV  predicted\n",
       "0  0.00632  0.538  6.575  4.0900     15.3   4.98  24.0  31.365126\n",
       "1  0.02731  0.469  6.421  4.9671     17.8   9.14  21.6  26.018747\n",
       "2  0.02729  0.469  7.185  4.9671     17.8   4.03  34.7  32.095787\n",
       "3  0.03237  0.458  6.998  6.0622     18.7   2.94  33.4  29.872827\n",
       "4  0.06905  0.458  7.147  6.0622     18.7   5.33  36.2  29.178013"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# assign test data\n",
    "newX = X_new\n",
    "newY = y\n",
    "\n",
    "# make predictions\n",
    "predicted = lm2.predict(newX)\n",
    "predictedDf = pd.DataFrame({\"predicted\": predicted})\n",
    "\n",
    "# dataframe for X\n",
    "newXDf = pd.DataFrame(newX)\n",
    "newXDf.columns = feature_cols_selected\n",
    "# dataframe for y\n",
    "newYDf = pd.DataFrame(newY)\n",
    "newYDf.columns = ['MEDV']\n",
    "# join X, y and prediction\n",
    "mydata_with_pd = newXDf.join(newYDf).join(predictedDf)\n",
    "mydata_with_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Absolute Error: 3.467774\n",
      "Root Mean Squared Error: 4.936601\n",
      "Relative Absolute Error: 0.521689\n",
      "Relative Squared Error: 0.288678\n"
     ]
    }
   ],
   "source": [
    "# check performance metrics\n",
    "import numpy as np\n",
    "obs = mydata_with_pd.MEDV\n",
    "pred = mydata_with_pd.predicted\n",
    "\n",
    "mae = np.mean(abs(pred-obs))\n",
    "rmse = np.sqrt(np.mean((pred-obs)**2))\n",
    "rae = np.mean(abs(pred-obs))/np.mean(abs(obs-np.mean(obs)))\n",
    "rse = np.mean((pred-obs)**2)/np.mean((obs-np.mean(obs))**2)\n",
    "\n",
    "print(\"Mean Absolute Error: {0:0.6f}\".format(mae))\n",
    "print(\"Root Mean Squared Error: {0:0.6f}\".format(rmse))\n",
    "print(\"Relative Absolute Error: {0:0.6f}\".format(rae))\n",
    "print(\"Relative Squared Error: {0:0.6f}\".format(rse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 Web Service\n",
    "### 4.1 Set up a web service\n",
    "After developing the model, we can deploy it as a service so others can use it. The \"azureml\" package's services subpackage can be used for this purpose. The following lines of code are setting up a web service named \"demoservice\". Notice that you should enter your own workspace\\_id and authorization\\_token. Your returned API information should be different from the printout below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# extract workspace info\n",
    "from azureml import Workspace\n",
    "ws = Workspace()\n",
    "workdspace_id = ws.workspace_id\n",
    "authorization_token = ws.authorization_token\n",
    "\n",
    "# set up web service\n",
    "from azureml import services\n",
    "@services.publish(workdspace_id, authorization_token)\n",
    "@services.types(crim = float, nox=float, rm=float, dis=float, ptratio=float, lstat=float)\n",
    "@services.returns(float)\n",
    "def demoserviceVarSel(crim, nox, rm, dis, ptratio, lstat):\n",
    "    feature_vector = [crim, nox, rm, dis, ptratio,  lstat]\n",
    "    return lm2.predict(feature_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "service_url = demoserviceVarSel.service.url \n",
    "api_key = demoserviceVarSel.service.api_key\n",
    "help_url = demoserviceVarSel.service.help_url\n",
    "service_id = demoserviceVarSel.service.service_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Consume a web service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The purpose of the following code is to wait a few seconds so that the web service is successfully desployed and ready for consumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# wait \n",
    "import time\n",
    "time.sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below can be used to consume the web service. For illustration purpose the data for two records are provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"Results\":{\"output1\":{\"type\":\"table\",\"value\":{\"Values\":[[\"31.3651262005007\"],[\"26.018746711909\"]]}},\"output2\":{\"type\":\"table\",\"value\":{\"Values\":[[\"data:text/plain,\",\"data:text/plain,\",null]]}}}}\n"
     ]
    }
   ],
   "source": [
    "import urllib2\n",
    "# If you are using Python 3+, import urllib instead of urllib2\n",
    "\n",
    "import json \n",
    "\n",
    "\n",
    "data =  {\n",
    "\n",
    "        \"Inputs\": {\n",
    "\n",
    "                \"input1\":\n",
    "                {\n",
    "                    \"ColumnNames\": [\"crim\", \"lstat\", \"nox\", \"rm\", \"ptratio\", \"dis\"],\n",
    "                    \"Values\": [ [ \"0.00632\", \"4.98\", \"0.538\", \"6.575\", \"15.3\", \"4.0900\" ], \n",
    "                               [ \"0.02731\", \"9.14\", \"0.469\", \"6.421\", \"17.8\", \"4.9671\" ], ]\n",
    "                },        },\n",
    "            \"GlobalParameters\": {\n",
    "}\n",
    "    }\n",
    "\n",
    "body = str.encode(json.dumps(data))\n",
    "\n",
    "url = service_url\n",
    "api_key = api_key # Replace this with the API key for the web service\n",
    "\n",
    "headers = {'Content-Type':'application/json', 'Authorization':('Bearer '+ api_key)}\n",
    "\n",
    "req = urllib2.Request(url, body, headers) \n",
    "\n",
    "try:\n",
    "    response = urllib2.urlopen(req)\n",
    "\n",
    "    # If you are using Python 3+, replace urllib2 with urllib.request in the above code:\n",
    "    # req = urllib.request.Request(url, body, headers) \n",
    "    # response = urllib.request.urlopen(req)\n",
    "\n",
    "    result = response.read()\n",
    "    print(result) \n",
    "except urllib2.HTTPError, error:\n",
    "    print(\"The request failed with status code: \" + str(error.code))\n",
    "\n",
    "    # Print the headers - they include the requert ID and the timestamp, which are useful for debugging the failure\n",
    "    print(error.info())\n",
    "\n",
    "    print(json.loads(error.read()))                 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above service consumption code is based on those at the help\\_url, which contains, among others, sample code written in C#, Python, and R for consuming the web service. To check out the code for yourself, copy the help\\_url value to a new browser to open the web page. Scroll down the newly opened page till you see the section \"Sample Code.\" Click on the Python tab and you'll see the sample code. Two changes were made to the sample code:\n",
    "    1. Fill in the url and api_key with the values returned for the current web service, and \n",
    "    2. Enter the values for the first two records. Notice that the order of the columns here may be different from \n",
    "    that in your definition of the web service. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 5 Conclusion\n",
    "In this example we started by briefly reviewing the feature selection techniques available for Azure ML's experiments. Then we used the Boston housing dataset to illustrate how to use a technique that's not available for Azure ML's experiments: recursive feature selection. The sklearn package implements several other feature techniques and you can find more information from the [documentation][documentation link] site.\n",
    "\n",
    "[documentation link]: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_selection"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Anaconda2]",
   "language": "python",
   "name": "Python [Anaconda2]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}