preprocess_functions-8-19-17.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the following functions for preprocessing different categories of data in structured datasets. Each function takes as input pandas dataframe data sets and the name of column to be processed. Once defined the functions will be tested / demonstrated.\n",
    "\n",
    "process_numerical_class(mdf_train, mdf_test, column)\n",
    "\n",
    "process_binary_class(mdf, column, missing)\n",
    "\n",
    "process_text_class(mdf_train, mdf_test, column)\n",
    "\n",
    "These are meant to deal with some potential errors such as missing data, although other conceivable errors such as incompatible numeric vs string input are not addressed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas import Series\n",
    "from sklearn import preprocessing\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#process_numerical_class(mdf_train, mdf_test, column)\n",
    "#function to normalize data to mean of 0 and standard deviation of 1 from training distribution\n",
    "#takes as arguement pandas dataframe of training and test data (mdf_train), (mdf_test)\\\n",
    "#and the name of the column string ('column') \n",
    "#replaces missing or improperly formatted data with mean of remaining values\n",
    "#replaces original specified column in dataframe\n",
    "#returns transformed dataframe\n",
    "\n",
    "#expect this approach works better when the numerical distribution is thin tailed\n",
    "#if only have training but not test data handy, use same training data for both dataframe inputs\n",
    "\n",
    "\n",
    "\n",
    "def process_numerical_class(mdf_train, mdf_test, column):\n",
    "     \n",
    "    \n",
    "    #convert all values to either numeric or NaN\n",
    "    mdf_train[column] = pd.to_numeric(mdf_train[column], errors='coerce')\n",
    "    mdf_test[column] = pd.to_numeric(mdf_test[column], errors='coerce')\n",
    "\n",
    "    #get mean of training data\n",
    "    mean = mdf_train[column].mean()    \n",
    "    \n",
    "    #replace missing data with training set mean\n",
    "    mdf_train[column] = mdf_train[column].fillna(mean)\n",
    "    mdf_test[column] = mdf_test[column].fillna(mean)\n",
    "    \n",
    "    #subtract mean from column for both train and test\n",
    "    mdf_train[column] = mdf_train[column] - mean\n",
    "    mdf_test[column] = mdf_test[column] - mean\n",
    "    \n",
    "    #get standard deviation of training data\n",
    "    std = mdf_train[column].std()\n",
    "    \n",
    "    #divide column values by std for both training and test data\n",
    "    mdf_train[column] = mdf_train[column] / std\n",
    "    mdf_test[column] = mdf_test[column] / std\n",
    "\n",
    "    \n",
    "    return mdf_train, mdf_test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#process_binary_class(mdf, column, missing)\n",
    "#converts binary classification values to 0 or 1\n",
    "#takes as arguement a pandas dataframe (mdf), \\\n",
    "#the name of the column string ('column') \\\n",
    "#and the string classification to assign to missing data ('missing')\n",
    "#replaces original specified column in dataframe\n",
    "#returns transformed dataframe\n",
    "\n",
    "#missing category must be identical to one of the two existing categories\n",
    "#returns error message if more than two categories remain\n",
    "\n",
    "\n",
    "def process_binary_class(mdf, column, missing):\n",
    "    \n",
    "    #replace missing data with specified classification\n",
    "    mdf[column] = mdf[column].fillna(missing)\n",
    "    \n",
    "    #if more than two remaining classifications, return error message    \n",
    "    if len(mdf[column].unique()) > 2:\n",
    "        print('ERROR: number of categories in column for process_binary_class() call >2')\n",
    "        return mdf\n",
    "    \n",
    "    #convert column to binary 0/1 classification\n",
    "    lb = preprocessing.LabelBinarizer()\n",
    "    mdf[column] = lb.fit_transform(mdf[column])\n",
    "    \n",
    "    return mdf\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "#process_text_class(mdf_train, mdf_test, column)\n",
    "#preprocess column with text classifications\n",
    "#takes as arguement two pandas dataframe containing training and test data respectively \n",
    "#(mdf_train, mdf_test), and the name of the column string ('column')\n",
    "\n",
    "#note this trains both training and test data simultaneously due to unique treatment if any category\n",
    "#missing from training set but not from test set to ensure consistent formatting \n",
    "\n",
    "#deletes the original column from master dataframe and\n",
    "#replaces with onehot encodings\n",
    "#with columns named after column_ + text classifications\n",
    "#missing data replaced with category label 'missing'+column\n",
    "#any categories missing from the training set removed from test set\n",
    "#any category present in training but missing from test set given a column of zeros for consistent formatting\n",
    "#ensures order of all new columns consistent between both sets\n",
    "#returns two transformed dataframe (mdf_train, mdf_test)\n",
    "\n",
    "#if only have training but not test data handy, use same training data for both dataframe inputs\n",
    "\n",
    "\n",
    "def process_text_class(mdf_train, mdf_test, column):\n",
    "\n",
    "    #replace NA with a dummy variable\n",
    "    mdf_train[column] = mdf_train[column].fillna('_missing')\n",
    "    mdf_test[column] = mdf_test[column].fillna('_missing')\n",
    "\n",
    "    \n",
    "    #extract categories for column labels\n",
    "    #note that .unique() extracts the labels as a numpy array\n",
    "    labels_train = mdf_train[column].unique()\n",
    "    labels_train.sort(axis=0)\n",
    "    labels_test = mdf_test[column].unique()\n",
    "    labels_test.sort(axis=0)\n",
    "    \n",
    "    #transform text classifications to numerical id\n",
    "    encoder = LabelEncoder()\n",
    "    cat_train = mdf_train[column]\n",
    "    cat_train_encoded = encoder.fit_transform(cat_train)\n",
    "    \n",
    "    cat_test = mdf_test[column]\n",
    "    cat_test_encoded = encoder.fit_transform(cat_test)\n",
    "    \n",
    "    \n",
    "    #apply onehotencoding\n",
    "    onehotencoder = OneHotEncoder()\n",
    "    cat_train_1hot = onehotencoder.fit_transform(cat_train_encoded.reshape(-1,1))\n",
    "    cat_test_1hot = onehotencoder.fit_transform(cat_test_encoded.reshape(-1,1))\n",
    "    \n",
    "    #append column header name to each category listing\n",
    "    #note the iteration is over a numpy array hence the [...] approach\n",
    "    labels_train[...] = column + '_' + labels_train[...]\n",
    "    labels_test[...] = column + '_' + labels_test[...]\n",
    "    \n",
    "    \n",
    "    #convert sparse array to pandas dataframe with column labels\n",
    "    df_train_cat = pd.DataFrame(cat_train_1hot.toarray(), columns=labels_train)\n",
    "    df_test_cat = pd.DataFrame(cat_test_1hot.toarray(), columns=labels_test)\n",
    "\n",
    "    \n",
    "    #Get missing columns in test set that are present in training set\n",
    "    missing_cols = set( df_train_cat.columns ) - set( df_test_cat.columns )\n",
    "    #Add a missing column in test set with default value equal to 0\n",
    "    for c in missing_cols:\n",
    "        df_test_cat[c] = 0\n",
    "    #Ensure the order of column in the test set is in the same order than in train set\n",
    "    #Note this also removes categories in test set that aren't present in training set\n",
    "    df_test_cat = df_test_cat[df_train_cat.columns]\n",
    "    \n",
    "    \n",
    "    #concatinate the sparse set with the rest of our training data\n",
    "    mdf_train = pd.concat([df_train_cat, mdf_train], axis=1)\n",
    "    mdf_test = pd.concat([df_test_cat, mdf_test], axis=1)\n",
    "    \n",
    "\n",
    "    #delete original column from training data\n",
    "    del mdf_train[column]    \n",
    "    del mdf_test[column]\n",
    "    \n",
    "    \n",
    "    return mdf_train, mdf_test\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#create sample test and train data for demonstration purposes\n",
    "\n",
    "#train data set from list of dictionaries\n",
    "train = [{'number': 1, 'Y/N': 'Y', 'shape': 'circle', 'label': 'cat'}, \n",
    "         {'number': 2, 'Y/N': 'N', 'shape': 'square', 'label': 'dog'}, \n",
    "         {'number': None, 'Y/N': 'Y', 'shape': 'circle', 'label': 'cat'}, \n",
    "         {'number': 3.1, 'Y/N': None, 'shape': 'square', 'label': 'cat'}, \n",
    "         {'number': -1, 'Y/N': 'N', 'shape': None, 'label': 'dog'}, \n",
    "         {'number': 'Q', 'Y/N': 'N', 'shape': 'oval', 'label': 'dog'}]\n",
    "\n",
    "#convert train data to pandas dataframe\n",
    "train = pd.DataFrame(train)\n",
    "\n",
    "#test data set from list of dictionaries\n",
    "test = [{'number': 2.1, 'Y/N': 'N', 'shape': 'square'}, \n",
    "        {'number': -1, 'Y/N': 'N', 'shape': None},\n",
    "        {'number': 1, 'Y/N': 'Y', 'shape': 'circle'}, \n",
    "        {'number': None, 'Y/N': 'Y', 'shape': 'square'}, \n",
    "        {'number': 3, 'Y/N': None, 'shape': 'circle'}, \n",
    "        {'number': 0, 'Y/N': 'N', 'shape': 'octogon'}, \n",
    "        {'number': 'Q', 'Y/N': 'Y', 'shape': 'square'}]\n",
    "\n",
    "#convert test data to pandas dataframe\n",
    "test = pd.DataFrame(test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Y/N</th>\n",
       "      <th>number</th>\n",
       "      <th>shape</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Y</td>\n",
       "      <td>1</td>\n",
       "      <td>circle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>N</td>\n",
       "      <td>2</td>\n",
       "      <td>square</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Y</td>\n",
       "      <td>None</td>\n",
       "      <td>circle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>None</td>\n",
       "      <td>3.1</td>\n",
       "      <td>square</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>N</td>\n",
       "      <td>-1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>N</td>\n",
       "      <td>Q</td>\n",
       "      <td>oval</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Y/N number   shape\n",
       "0     Y      1  circle\n",
       "1     N      2  square\n",
       "2     Y   None  circle\n",
       "3  None    3.1  square\n",
       "4     N     -1    None\n",
       "5     N      Q    oval"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#seperate labels from train data\n",
    "labels = train['label']\n",
    "del train['label']\n",
    "\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Y/N</th>\n",
       "      <th>number</th>\n",
       "      <th>shape</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Y</td>\n",
       "      <td>-0.203760</td>\n",
       "      <td>circle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>N</td>\n",
       "      <td>0.537184</td>\n",
       "      <td>square</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Y</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>circle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>None</td>\n",
       "      <td>1.352223</td>\n",
       "      <td>square</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>N</td>\n",
       "      <td>-1.685648</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>N</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>oval</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Y/N    number   shape\n",
       "0     Y -0.203760  circle\n",
       "1     N  0.537184  square\n",
       "2     Y  0.000000  circle\n",
       "3  None  1.352223  square\n",
       "4     N -1.685648    None\n",
       "5     N  0.000000    oval"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#test process_numerical_class()\n",
    "train, test = process_numerical_class(train, test, 'number')\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Y/N</th>\n",
       "      <th>number</th>\n",
       "      <th>shape</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>-0.203760</td>\n",
       "      <td>circle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0.537184</td>\n",
       "      <td>square</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>circle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1.352223</td>\n",
       "      <td>square</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>-1.685648</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>oval</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Y/N    number   shape\n",
       "0    1 -0.203760  circle\n",
       "1    0  0.537184  square\n",
       "2    1  0.000000  circle\n",
       "3    1  1.352223  square\n",
       "4    0 -1.685648    None\n",
       "5    0  0.000000    oval"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#test process_binary_class()\n",
    "train = process_binary_class(train, 'Y/N', 'Y')\n",
    "test = process_binary_class(test, 'Y/N', 'Y')\n",
    "train\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shape__missing</th>\n",
       "      <th>shape_circle</th>\n",
       "      <th>shape_oval</th>\n",
       "      <th>shape_square</th>\n",
       "      <th>Y/N</th>\n",
       "      <th>number</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.203760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.537184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.352223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.685648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shape__missing  shape_circle  shape_oval  shape_square  Y/N    number\n",
       "0             0.0           1.0         0.0           0.0    1 -0.203760\n",
       "1             0.0           0.0         0.0           1.0    0  0.537184\n",
       "2             0.0           1.0         0.0           0.0    1  0.000000\n",
       "3             0.0           0.0         0.0           1.0    1  1.352223\n",
       "4             1.0           0.0         0.0           0.0    0 -1.685648\n",
       "5             0.0           0.0         1.0           0.0    0  0.000000"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#test process_text_class():\n",
    "train, test = process_text_class(train, test, 'shape')\n",
    "train\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shape__missing</th>\n",
       "      <th>shape_circle</th>\n",
       "      <th>shape_oval</th>\n",
       "      <th>shape_square</th>\n",
       "      <th>Y/N</th>\n",
       "      <th>number</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.611279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.685648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.203760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.278128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.944704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shape__missing  shape_circle  shape_oval  shape_square  Y/N    number\n",
       "0             0.0           0.0           0           1.0    0  0.611279\n",
       "1             1.0           0.0           0           0.0    0 -1.685648\n",
       "2             0.0           1.0           0           0.0    1 -0.203760\n",
       "3             0.0           0.0           0           1.0    1  0.000000\n",
       "4             0.0           1.0           0           0.0    1  1.278128\n",
       "5             0.0           0.0           0           0.0    0 -0.944704\n",
       "6             0.0           0.0           0           1.0    1  0.000000"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined our funcations for data preprocessing of structures data, we'll begin a new notebook incorporating these functions and applying them to the processing and training of the Kaggle house price regression data set. To be continued in next notebook."
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "12px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}