Interactive_Norms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "## Subsurface Data Analytics \n",
    "\n",
    "## Interactive Demonstration of Machine Learning Norms\n",
    "\n",
    "#### Michael Pyrcz, Associate Professor, University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)\n",
    "\n",
    "\n",
    "### PGE 383 Exercise: Interactive Demonstration of Machine Learning Norms\n",
    "\n",
    "Here's a simple workflow, demonstration of predictive machine learning model norms. We use a:\n",
    "\n",
    "* linear regression model\n",
    "\n",
    "* 1 preditor feature and 1 response feature\n",
    "\n",
    "for an high interpretability model/ simple illustration.\n",
    "\n",
    "#### Norms\n",
    "\n",
    "Given a vector of error over the $n_{train}$ training data.\n",
    "\n",
    "\\begin{equation}\n",
    "\\Delta y \\rightarrow \\Delta y_i, \\forall i = 1,\\dots,n_{train}\n",
    "\\end{equation}\n",
    "\n",
    "We require a summarization of the error as a single value, this is a norm.\n",
    "\n",
    "A norm has the following properties:\n",
    "\n",
    "* norm of a vector maps the vaector values to a summary measures $\\rightarrow [0,\\infty)$\n",
    "\n",
    "Common norms include, Manhattan, Euclidean and the general p-norm.\n",
    "\n",
    "**Manhattan Norm** is defined as:\n",
    "\n",
    "\\begin{equation}\n",
    "||\\Delta y||_1 = \\sum_{i=1}^{n_{train}} |\\Delta y_i| \n",
    "\\end{equation}\n",
    "\n",
    "**Euclidean Norm** is defined as:\n",
    "\n",
    "\\begin{equation}\n",
    "||\\Delta y||_2 = \\sqrt{ \\sum_{i=1}^{n_{train}} \\left( \\Delta y_i \\right)^2 }\n",
    "\\end{equation}\n",
    "\n",
    "**p-Norm** is defined as:\n",
    "\n",
    "\\begin{equation}\n",
    "||\\Delta y||_p = \\left( \\sum_{i=1}^{n_{train}} \\left( \\Delta y_i \\right)^p \\right)^{\\frac{1}{p}}\n",
    "\\end{equation}\n",
    "\n",
    "#### Workflow Goals\n",
    "\n",
    "Learn the basics of machine learning training, tuning for model generalization while avoiding model overfit.\n",
    "\n",
    "This includes:\n",
    "\n",
    "* Demonstrate model training and tuning by hand with an interactive exercies\n",
    "\n",
    "* Demonstrate the role of data error in leading to model overfit with complicated models\n",
    "\n",
    "#### Import Required Packages\n",
    "\n",
    "We will also need some standard packages. These should have been installed with Anaconda 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import sys                                              # supress output to screen for interactive variogram modeling\n",
    "import io\n",
    "import numpy as np                                      # arrays and matrix math\n",
    "import pandas as pd                                     # DataFrames\n",
    "import matplotlib.pyplot as plt                         # plotting\n",
    "from scipy.optimize import minimize                     # linear regression training by-hand with variable norms\n",
    "from ipywidgets import interactive                      # widgets and interactivity\n",
    "from ipywidgets import widgets                            \n",
    "from ipywidgets import Layout\n",
    "from ipywidgets import Label\n",
    "from ipywidgets import VBox, HBox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Functions\n",
    "\n",
    "We have functions to perform linear regression for any norm. The code was modified from [N. Wouda](https://stackoverflow.com/questions/51883058/l1-norm-instead-of-l2-norm-for-cost-function-in-regression-model).\n",
    "* I modified the original functions for a general p-norm linear regression method\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def predict(X, params):                                 # linear prediction\n",
    "    return X.dot(params)\n",
    "\n",
    "def loss_function(params, X, y, p):                     # custom p-norm, linear regression cost function\n",
    "    return np.sum(np.power(np.abs(y - predict(X, params)),p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interactive Dashboard\n",
    "\n",
    "This code designed the interactive dashboard, prediction model and plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# widgets and dashboard\n",
    "l = widgets.Text(value='                                       Machine Learning Norms Demo, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))\n",
    "\n",
    "p_norm = widgets.FloatSlider(min=0.1, max = 10, value=1.0, step = 0.2, description = '$L^{p}$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)\n",
    "n = widgets.IntSlider(min=15, max = 80, value=30, step = 1, description = 'n',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)\n",
    "std = widgets.FloatSlider(min=0.0, max = .95, value=0.00, step = 0.05, description = 'Error (St.Dev.)',orientation='horizontal',style = {'description_width': 'initial'}, continuous_update=False)\n",
    "xn = widgets.FloatSlider(min=0, max = 1.0, value=0.5, step = 0.05, description = '$X_{n+1}$',orientation='horizontal',style = {'description_width': 'initial'}, continuous_update=False)\n",
    "yn = widgets.FloatSlider(min=0, max = 1.0, value=0.5, step = 0.05, description = '$Y_{n+1}$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)\n",
    "\n",
    "ui1 = widgets.HBox([p_norm,n,std],)\n",
    "ui2 = widgets.HBox([xn,yn],)\n",
    "ui = widgets.VBox([l,ui1,ui2],)\n",
    "\n",
    "def run_plot(p_norm,n,std,xn,yn):                       # make data, fit models and plot\n",
    "\n",
    "    np.random.seed(73073)                               # set random number seed for repeatable results\n",
    "\n",
    "    X_seq = np.linspace(0,100.0,1000)                   # make data and add noise\n",
    "    X_seq = np.asarray([np.ones((len(X_seq),)), X_seq]).T\n",
    "    X = np.random.rand(n)*0.5\n",
    "    y = X*X + 0.0 # fit a parabola\n",
    "    y = y + np.random.normal(loc = 0.0,scale=std,size=n) # add noise\n",
    "    X = np.asarray([np.ones((n,)), X]).T                 # concatenate a vector of 1's for the constant term\n",
    "    \n",
    "    X = np.vstack([X,[1,xn]]); y = np.append(y,yn)       # add the user specified data value to X and y\n",
    "    \n",
    "    x0 = [0.5,0.5]                                       # initial guess of model parameters\n",
    "    p = 2.0\n",
    "    output_l2 = minimize(loss_function, x0, args=(X, y, p)) # train the L2 norm linear regression model\n",
    "    p = 1.0\n",
    "    output_l1 = minimize(loss_function, x0, args=(X, y, p)) # train the L1 norm linear regression model\n",
    "    p = 3.0\n",
    "    output_l3 = minimize(loss_function, x0, args=(X, y, p)) # train the L3 norm linear regression model\n",
    "    \n",
    "    p = p_norm\n",
    "    output_lcust = minimize(loss_function, x0, args=(X, y, p)) # train the p-norm linear regression model\n",
    "\n",
    "    y_hat_l1 = predict(X_seq, output_l1.x)               # predict over the range of X for all models\n",
    "    y_hat_l2 = predict(X_seq, output_l2.x)\n",
    "    y_hat_l3 = predict(X_seq, output_l3.x)\n",
    "    y_hat_lcust = predict(X_seq, output_lcust.x)\n",
    "    \n",
    "    plt.subplot(111)                                     # plot the results\n",
    "    plt.scatter(X[:(n-1),1],y[:(n-1)],s=20,facecolor = 'yellow', edgecolor = 'black', alpha = 0.4)\n",
    "    plt.scatter(X[n,1],y[n],s=40,marker='^',facecolor = 'orange', edgecolor = 'black', alpha = 0.4)\n",
    "    plt.plot(X_seq[:,1],y_hat_l1,c = 'blue',alpha = 0.3,label = \"L1 Norm\")\n",
    "    plt.plot(X_seq[:,1],y_hat_l2,c = 'red',alpha = 0.3,label = \"L2 Norm\")\n",
    "    plt.plot(X_seq[:,1],y_hat_l3,c = 'green',alpha = 0.3,label = \"L3 Norm\")\n",
    "    plt.plot(X_seq[:,1],y_hat_lcust,c = 'black',alpha = 1.0,label = \"L\"+ str(p_norm) + \" Norm\")\n",
    "    plt.xlabel(r'Predictor Feature, $X_{1}$'); plt.ylabel(r'Response Feature, $y$'); plt.title('Linear Regression with Various Norms')\n",
    "    plt.xlim([0.0,1.0]); plt.ylim([0.0,1.0])\n",
    "    plt.legend(loc = 'lower right')\n",
    "    \n",
    "    plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.6, wspace=0.9, hspace=0.3)\n",
    "    plt.show()\n",
    "    \n",
    "# connect the function to make the samples and plot to the widgets    \n",
    "interactive_plot = widgets.interactive_output(run_plot, {'p_norm':p_norm,'n':n,'std':std,'xn':xn,'yn':yn})\n",
    "interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Machine Learning Norms Demonstation \n",
    "\n",
    "#### Michael Pyrcz, Associate Professor, The University of Texas at Austin \n",
    "\n",
    "Observe the impact of choice of norm with variable number of sample data, the data noise, and an outlier! \n",
    "\n",
    "### The Inputs\n",
    "\n",
    "* **p-norm** - 1 = Manhattan norm, 2 = Euclidean norm, etc., **n** - number of data, **Error** - random error in standard deviations\n",
    "* **$x_{n+1}$**, **$y_{n+1}$** - x and y location of an additional data value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "213df1f40e8e473fa262fc6ac6077d0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='                                       Machine Learning Norms Demo, Prof. Michael P…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13dbd90afbfc4cde94d4af9386254596",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '<Figure size 432x288 with 1 Axes>', 'i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ui, interactive_plot)                           # display the interactive plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comments\n",
    "\n",
    "This was a basic demonstration of machining learning norms. I have many other demonstrations and even basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy. \n",
    "  \n",
    "#### The Author:\n",
    "\n",
    "### Michael Pyrcz, Associate Professor, University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin\n",
    "\n",
    "#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)  \n",
    "  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}