instacart-feature-engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instacart: Feature Engineering\n",
    "\n",
    "This notebook constructs matrices $\\{X_s\\}_{s \\in \\mathrm{DSets}}$, where $\\mathrm{DSets} = \\{\\mathrm{train, test, kaggle}\\}$. These matrices are inputs for the random forest classifier which we tune in [Instacart: Random Forest ParameterGrid Search](./instacart-random-forest-parametergrid-search/) and train in [Instacart: Top-N Random Forest Model](./instacart-top-n-random-forest-model/). Most features – columns of $X_s$ – are computed via aggregations and transformations of the raw data provided and studied in [Instacart: Exploratory Data Analysis](./instacart-exploratory-data-analysis.ipynb). In addition, some features are computed via an unsupervised learning technique, [Latent Dirichlet Allocation (LDA)](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation), introduced in \\cite{bleiLatentDirichletAllocation2003}. This is a probabilistic generative model that applied to the matrix of user-product purchase counts. Although this data is already fed into a random forest classifier as a column of $X_s$, the distributional assumptions of LDA can yield a bit more predictive power. We can view this as a simple [model-based collaborative filtering](https://en.wikipedia.org/wiki/Collaborative_filtering#Model-based) technique.\n",
    "\n",
    "The matrices $X_s$ are limited to roughly 50 columns on the Kaggle platform on which the collection of notebooks comprising this project were run. The limiting resource is memory though the limitation occurs at the `sklearn.ensemble.RandomForestClassifier` calls in the subsequent notebooks. Kaggle provides instances with 16GB of memory, although the instance provides no virtual memory (on disk) so this is a hard limit on memory availability.\n",
    "\n",
    "As well, there are additional sorts of complex features which require a modest increase in memory. For example, [non-negative matrix factorization (NMF)](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization) techniques have traditionally been used in recommendation systems. Such techniques may be appropriate for the user-user matrix of, say, counts of common product purchases. One application of NMF to this matrix is dimensionality reduction – to create a relatively small number of user \"topics\" based on the common purchase count matrix. While this matrix is sparse, experimentation suggests it is tractable at a size of perhaps roughly 10GB for $s=\\text{'train'}$ as indicated by constructing such matrices on subsets of the overall dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Dictionary\n",
    "\n",
    "There are a few groups of features –  Profiles – this notebook constructs. The User Profiles, for example, consists of operations and aggregations grouped by user, so that the index for the user profile is $U_s$, the list of users. The rows of $X_s$ are not merely users, but user-product pairs, which means that the User Profile is broadcast to the user-product index $I_s$ via a `.join()` operation. That is, the values of the User Profile are repeated across all products in the user-product index for any given user. An analogous statement holds for the Product Profile. Therefore, the User-Product profile will have the features with the greatest information content (and the Aisle and Department profiles the leat). The list of feature groups and features follows. \n",
    "\n",
    "| feature group prefix | name |\n",
    "|---|---|\n",
    "| `U` | User Profile |\n",
    "| `P` | Product Profile |\n",
    "| `UP` | User-Product Profile |\n",
    "| `AD` | Aisle and Department Profiles (ignored) |\n",
    "| `LDA` | Latent Dirichlet Allocation User Features |\n",
    "\n",
    "\n",
    "| feature | dtype | description |\n",
    "|---|---|---|\n",
    "| `U_ultimate_order_dow` | `                float16 ` | dow of user's ultimate order |\n",
    "| `U_ultimate_order_hour_of_day` | `        float16 ` | hour of user's ultimate order |\n",
    "| `U_ultimate_days_since_prior_order` | `   float16 ` | days since user's previous order (from ultimate)|\n",
    "| `U_orders_num` | `                        uint8 ` |   number of orders a given user has placed |\n",
    "| `U_items_total` | `                       uint16 ` |  number of total items a given user has purchased |\n",
    "| `U_order_size_mean` | `                   float16 ` |  mean basket size for a given user|\n",
    "| `U_order_size_std` | `                    float16 ` |  std basket size for a given user |\n",
    "| `U_unique_products` | `                   uint16 ` |   number of unique products a given user has purchased|\n",
    "| `U_reordered_num` | `                     uint16 `  |  number of total items a given user has purchased which are reorders    |\n",
    "| `U_reorder_size_mean` | `                 float16 ` |  mean reorders per basket    |\n",
    "| `U_reorder_size_std` | `                  float16 ` |  std reorders per basket    |\n",
    "| `U_reordered_ratio` | `                   float16 ` |  proportion of items a given user has purchased which are reorders    |\n",
    "| `U_order_dow_mean` | `                    float16 ` |  mean order_dow    |\n",
    "| `U_order_dow_var` | `                     float16 ` |  var order_dow    |\n",
    "| `U_order_dow_score` | `                   float16 ` |  ultimate score for order_dow using circstd = sqrt(-2ln(circvar))    |\n",
    "| `U_order_hour_of_day_mean` | `            float16 ` |  mean order_hour_of_day    |\n",
    "| `U_order_hour_of_day_var` | `             float16 ` |  var order_hour_of_day    |\n",
    "| `U_order_hour_of_day_score` | `           float16 ` |  ultimate score for order_hour_of_day using circstd = sqrt(-2ln(circvar))    |\n",
    "| `U_days_since_prior_order_mean` | `       float16 ` |  mean days since prior order (mean user order time interval)    |\n",
    "| `U_days_since_prior_order_std` | `        float16 ` |  std days since prior order (std user order time interval)    |\n",
    "| `P_orders_num` | `                        uint32 ` |  number of total purchases    |\n",
    "| `P_unique_users` | `                      uint16 ` |  number of purchasers    |\n",
    "| `P_reorder_ratio` | `                     float16 ` | reorder ratio     |\n",
    "| `P_order_hour_of_day_mean` | `            float16 ` | mean order_hour_of_day     |\n",
    "| `P_order_hour_of_day_var` | `             float16 ` | var order_hour_of_day     |\n",
    "| `P_order_dow_mean` | `                    float16 ` | mean order_dow     |\n",
    "| `P_order_dow_var` | `                     float16 ` | var order_dow     |\n",
    "| `UP_orders_num` | `                       uint8 ` |    number of times particular user has ordered particular product  |\n",
    "| `UP_orders_since_previous` | `            uint8 ` |    number of orders since previous purchase of product by user  |\n",
    "| `UP_days_since_prior_order` | `           uint16 ` |   days since user last ordered product   |\n",
    "| `UP_days_since_prior_order_score` | `     float16 ` |  normalize above by user's days_since_prior_order    |\n",
    "| `UP_reordered` | `                        bool ` |     boolean indicating whether the product was ever reordered by user |\n",
    "| `UP_order_ratio` | `                      float16 ` |  fraction of baskets in which a given product appears for a given user (count of orders in which product appears divided by total orders)    |\n",
    "| `UP_penultimate` | `                      bool ` |     products in user's penultimate (previous) order as `bool` (`train` and `test` sets contain ultimate order) |\n",
    "| `UP_antepenultimate` | `                  bool ` |     products in user's antepenultimate order as `bool` |\n",
    "| `UP_order_dow_score` | `                  float16 ` |  ultimate score for order_dow using (`U_ultimate` - `P_order_dow_mean`) / `P_order_dow_std` (intuitively, how 'far' is a user's ultimate order dow from the mean dow product is ordered)    |\n",
    "| `UP_order_hour_of_day_score` | `          float16 ` |  ultimate score for order_hour_of_day using (`U_ultimate` - `P_order_hour_of_day_mean`) / `P_order_hour_of_day_std` (intuitively, how 'far' is a user's ultimate order hour_of_day from the mean hour_of_day product is ordered)    |\n",
    "| `LDA_1` | `                               float16 ` |   Latent Dirichlet Allocation Feature 1     |\n",
    "| `LDA_2` | `                               float16 ` |   Latent Dirichlet Allocation Feature 2     |\n",
    "| `LDA_3` | `                               float16 ` |   Latent Dirichlet Allocation Feature 3     |\n",
    "| `LDA_4` | `                               float16 ` |   Latent Dirichlet Allocation Feature 4     |\n",
    "| `LDA_5` | `                               float16 ` |   Latent Dirichlet Allocation Feature 5     |\n",
    "| `LDA_6` | `                               float16 ` |   Latent Dirichlet Allocation Feature 6     |\n",
    "| `LDA_7` | `                               float16 ` |   Latent Dirichlet Allocation Feature 7     |\n",
    "| `LDA_8` | `                               float16 ` |   Latent Dirichlet Allocation Feature 8     |\n",
    "| `LDA_9` | `                               float16 ` |   Latent Dirichlet Allocation Feature 9     |\n",
    "| `LDA_10` | `                              float16 ` |   Latent Dirichlet Allocation Feature 10    |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aisles.csv\t order_products__prior.csv  orders.csv\r\n",
      "departments.csv  order_products__train.csv  products.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls instacart_data/all/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "pd.options.display.latex.repr=True\n",
    "\n",
    "file_path = 'instacart_data/all/'\n",
    "\n",
    "load_data_dtype = {\n",
    "    'order_id': np.uint32,\n",
    "    'user_id': np.uint32,\n",
    "    'eval_set': 'category',\n",
    "    'order_number': np.uint8,\n",
    "    'order_dow': np.uint8,\n",
    "    'order_hour_of_day': np.uint8,\n",
    "    # pandas 'gotcha'; leave as float:\n",
    "    'days_since_prior_order': np.float16,\n",
    "    'product_id': np.uint16,\n",
    "    'add_to_cart_order': np.uint8,\n",
    "    'reordered': np.bool\n",
    "}\n",
    "\n",
    "df_aisles = pd.read_csv(file_path + 'aisles.csv')\n",
    "df_departments = pd.read_csv(file_path + 'departments.csv')\n",
    "df_products = pd.read_csv(file_path + 'products.csv')\n",
    "\n",
    "# Specify dtype to reduce memory utilization\n",
    "df_order_products_prior = pd.read_csv(file_path + 'order_products__prior.csv',\n",
    "                                      dtype=load_data_dtype)\n",
    "df_order_products_train = pd.read_csv(file_path + 'order_products__train.csv',\n",
    "                                      dtype=load_data_dtype)\n",
    "df_orders = pd.read_csv(file_path + 'orders.csv', dtype=load_data_dtype)\n",
    "\n",
    "# df_prior = full products from all prior orders\n",
    "df_prior = pd.merge(df_orders[df_orders['eval_set'] == 'prior'],\n",
    "                    df_order_products_prior,\n",
    "                    on='order_id')\n",
    "\n",
    "# # Useful DataFrame for aisle and department feature construction\n",
    "# df_ad = pd.merge(df_prior, df_products, how='left',\n",
    "#                  on='product_id').drop('product_name', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train, Test, and Kaggle Sets\n",
    "\n",
    "As this dataset comes from a (completed) Kaggle competition, the set of users whose ultimate order matches `eval_set == 'test'` form the test set for the competition; the ultimate order for this set is held aside by Kaggle so that participants can submit a prediction which Kaggle scores against the withheld set.\n",
    "\n",
    "Partitions of the dataset by user are defined by\n",
    "* $U_\\text{train}$: 80% of the 131,209 users whose ultimate orders are available.\n",
    "* $U_\\text{test}$: 20% of the 131,209 users whose ultimate orders are available.\n",
    "* $U_\\text{kaggle}$: The 75,000 users whose ultimate orders are withheld by Kaggle. This project does not explicitly use this set; predictions on this set merely serve as a sanity check via submission to Kaggle.\n",
    "\n",
    "The [sklearn.model_selection.train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) utility defines the partition between $U_\\text{train}$ and $U_\\text{test}$. The list of strings of datasets, `dsets = ['train', 'test', 'kaggle']`, instantiates $\\mathrm{DSets}$. `users` is a dictionary of lists of `user_ids` keyed by `dsets`, initialized by \n",
    ">`users = dict.fromkeys(dsets)`,\n",
    "\n",
    "so that `users[ds]` for `ds in dsets` instantiates $U_s$ for $s \\in \\mathrm{DSets}$. Similarly, this notebook constructs matrices `X[ds]`, initialized by\n",
    "> `X = dict.fromkeys(dsets)`,\n",
    "\n",
    "to instantiate $X_s$. Aside from the analogy between dictionary keys and subscripts, the `dict` type offers a coherent way to partition the dataset $D_s$ of user orders and baskets in `orders.csv` and `order_products*.csv` into separate DataFrames `orders[ds]` and `prior[ds]` for `ds in dsets` at the outset so as to avoid potential data leaks. The following is a partial notational dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Names of dataset partitions\n",
    "dsets = ['train', 'test', 'kaggle']\n",
    "\n",
    "# Partition of users into dsets\n",
    "users = dict.fromkeys(dsets)\n",
    "\n",
    "# Use sklearn utility to partition project users into train and test user lists.\n",
    "users['train'], users['test'] = train_test_split(list(\n",
    "    df_orders[df_orders.eval_set == 'train']['user_id']),\n",
    "                                                 test_size=0.2,\n",
    "                                                 random_state=20190502)\n",
    "\n",
    "# Kaggle submissions test set\n",
    "users['kaggle'] = list(\n",
    "    df_orders[df_orders.eval_set == 'test']['user_id'])  #.to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split DataFrames we will use in feature construction into dicts of DataFrames\n",
    "prior = dict.fromkeys(dsets)\n",
    "orders = dict.fromkeys(dsets)\n",
    "orders_full = dict.fromkeys(dsets)\n",
    "\n",
    "# ad = dict.fromkeys(dsets)\n",
    "\n",
    "for ds in dsets:\n",
    "    prior[ds] = df_prior[df_prior['user_id'].isin(users[ds])]\n",
    "    orders[ds] = df_orders[df_orders['user_id'].isin(users[ds])\n",
    "                           & (df_orders.eval_set == 'prior')]\n",
    "    orders_full[ds] = df_orders[df_orders['user_id'].isin(users[ds])]\n",
    "#     ad[ds] = df_ad[df_ad['user_id'].isin(users[ds])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It will be useful to have an `Index` of users and a `MultiIndex` of user-product pairs.\n",
    "\n",
    "The \"full\" index could include a few dozen user-product pairs which appear in 'train' but not 'prior'. To consider products users have previously ordered, these are discluded. Further, the technical complication in reindexing and deciding good fillna values is unlikely worth the additional predictive ability of including these user-product pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.23.4'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Index of all users\n",
    "# for pandas 0.24:\n",
    "# u_index[ds], _ = pd.MultiIndex.from_frame(orders[ds]['user_id']).sortlevel()\n",
    "# for pandas 0.23.4:\n",
    "\n",
    "u_index = dict.fromkeys(dsets)\n",
    "\n",
    "for ds in dsets:\n",
    "    u_index[ds], _ = pd.Index(list(orders[ds]['user_id'].values),\n",
    "                              name='user_id').sortlevel()\n",
    "    u_index[ds] = u_index[ds].drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create MultiIndex of all (nonempty) (user, product) pairs\n",
    "# and empty DataFrame with that MultiIndex for joins with\n",
    "# features with user index or product index\n",
    "# for pandas 0.24:\n",
    "# up_index[ds], _ = pd.MultiIndex.from_frame(prior[ds][['user_id', 'product_id']]).sortlevel()\n",
    "# for pandas 0.23.4:\n",
    "\n",
    "up_index = dict.fromkeys(dsets)\n",
    "up_empty_df = dict.fromkeys(dsets)\n",
    "\n",
    "for ds in dsets:\n",
    "    up_index[ds], _ = pd.MultiIndex.from_tuples(\n",
    "        list(prior[ds][['user_id', 'product_id']].values),\n",
    "        names=prior[ds][['user_id', 'product_id']].columns).sortlevel()\n",
    "    up_index[ds] = up_index[ds].drop_duplicates()\n",
    "    up_empty_df[ds] = pd.DataFrame(index=up_index[ds])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $X_s$ Ultimate\n",
    "These DataFrames are helpful in building the features prefixed by `U_ultimate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The ultimate orders\n",
    "ultimate = dict.fromkeys(dsets)\n",
    "\n",
    "ultimate['train'] = df_orders[(df_orders['eval_set'] == 'train')\n",
    "                              & df_orders['user_id'].isin(users['train'])]\n",
    "# 'eval_set' == 'train' is correct here since that is *Kaggle's* train:\n",
    "ultimate['test'] = df_orders[(df_orders['eval_set'] == 'train')\n",
    "                             & df_orders['user_id'].isin(users['test'])]\n",
    "ultimate['kaggle'] = df_orders[(df_orders['eval_set'] == 'test')\n",
    "                               & df_orders['user_id'].isin(users['kaggle'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $y_\\text{train}$ and $y_\\text{test}$\n",
    "\n",
    "The true $y$-vectors are below. Kaggle witholds the data $y_\\text{kaggle}$; instead, Kaggle competitors may submit a prediction $\\hat{y}_\\text{kaggle}$ which Kaggle scores against $y_\\text{kaggle} $."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build y['train'] and y['test']\n",
    "# df_present = ultimate train and test orders\n",
    "df_y = pd.merge(df_orders[df_orders['eval_set'] == 'train'],\n",
    "                df_order_products_train,\n",
    "                on='order_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = dict.fromkeys(dsets)\n",
    "\n",
    "y['train'] = (\n",
    "    pd.DataFrame(\n",
    "        [[True]],\n",
    "        index=pd.MultiIndex.from_tuples(\n",
    "            list(\n",
    "                # (user, product) pairs of purchases in 'train' df -> list\n",
    "                df_y[df_y['user_id'].isin(users['train'])]\n",
    "                [['user_id', 'product_id']].values)))\n",
    "    # Fill unpurchased items in overall up_index as False\n",
    "    .reindex(up_index['train']).fillna(False))\n",
    "\n",
    "y['test'] = (\n",
    "    pd.DataFrame(\n",
    "        [[True]],\n",
    "        index=pd.MultiIndex.from_tuples(\n",
    "            list(\n",
    "                # (user, product) pairs of purchases in 'test' df -> list\n",
    "                df_y[df_y['user_id'].isin(users['test'])]\n",
    "                [['user_id', 'product_id']].values)))\n",
    "    # Fill unpurchased items in overall up_index as False\n",
    "    .reindex(up_index['test']).fillna(False))\n",
    "\n",
    "y['kaggle'] = pd.DataFrame(data=['foo'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save $y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('io.hdf.default_format', 'table')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "store = pd.HDFStore('io.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "for dset, dframe in y.items():\n",
    "    store['/y/' + str(dset)] = dframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.is_open"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup y\n",
    "del df_y, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dimensions\n",
    "users_num = df_orders['user_id'].max()\n",
    "products_num = df_products['product_id'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a dict to collect groups of features (e.g. profiles, clusterings, etc)\n",
    "groups_dict = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since both `order_dow` and `order_hour_of_day` are cyclic temporal features, it may help the model to encode them as such. To do so, tranform the cyclic features to angles in radians and use circular statistics as described in [Directional Statistics](https://en.wikipedia.org/wiki/Directional_statistics#Measures_of_location_and_spread) and the first sections of [NCSS Circular Data Analysis](https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Circular_Data_Analysis.pdf).\n",
    "\n",
    "In addition, [the implementation of the scipy circular variance calculation is suspect](https://stackoverflow.com/questions/52856232/scipy-circular-variance), while the [astropy.stats.circstats](http://docs.astropy.org/en/stable/stats/circ.html) calculation seems correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from astropy.stats import circmean, circvar\n",
    "\n",
    "def angle_transform(series, period):\n",
    "    return series.multiply(2 * np.pi / period).sub(np.pi).astype('float16')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ultimate User Features\n",
    "\n",
    "These are the (known) `order_dow`, `order_hour_of_day`, and `days_since_prior_order` of the ultimate order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "# dictionary to store given user features\n",
    "u_given_dict = defaultdict(dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute each feature separately for 'train', 'test,', and 'kaggle' in dsets\n",
    "for ds in dsets:\n",
    "\n",
    "    # ultimate order_dow\n",
    "    u_given_dict['U_ultimate_order_dow'][ds] = angle_transform(\n",
    "        ultimate[ds].set_index('user_id').order_dow, 7)\n",
    "\n",
    "    # ultimate order_hour_of_day\n",
    "    u_given_dict['U_ultimate_order_hour_of_day'][ds] = angle_transform(\n",
    "        ultimate[ds].set_index('user_id').order_hour_of_day, 24)\n",
    "\n",
    "    # ultimate days_since_prior_order\n",
    "    u_given_dict['U_ultimate_days_since_prior_order'][ds] = (\n",
    "        ultimate[ds].set_index('user_id').days_since_prior_order)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename feature columns/pandas Series object by u_given_dict key name pointing to it.\n",
    "\n",
    "for ds in dsets:\n",
    "    for k, v in u_given_dict.items():\n",
    "        v[ds].rename(k, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine given user features; store as key 'U_given'\n",
    "\n",
    "groups_dict['U_given'] = {\n",
    "    ds: pd.concat([u_given_dict[k][ds] for k in u_given_dict.keys()], axis=1)\n",
    "    for ds in dsets\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User Profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dictionary to store user features\n",
    "u_dict = defaultdict(dict)\n",
    "\n",
    "for ds in dsets:\n",
    "\n",
    "    # number of orders a given user has placed\n",
    "    u_dict['U_orders_num'][ds] = (\n",
    "        prior[ds]\n",
    "        .groupby(by='user_id')['order_number']\n",
    "        .max().apply(pd.to_numeric,\n",
    "                     downcast='unsigned'))\n",
    "\n",
    "    # number of total items a given user has purchased\n",
    "    u_dict['U_items_total'][ds] = (\n",
    "        prior[ds].groupby('user_id')['product_id'].count().apply(\n",
    "            pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # mean basket size for a given user\n",
    "    u_dict['U_order_size_mean'][ds] = (u_dict['U_items_total'][ds].div(\n",
    "        u_dict['U_orders_num'][ds]).astype('float16'))\n",
    "\n",
    "    # std basket size for a given user\n",
    "    u_dict['U_order_size_std'][ds] = (prior[ds].groupby([\n",
    "        'user_id', 'order_number'\n",
    "    ]).add_to_cart_order.max().groupby('user_id').std().astype('float16'))\n",
    "\n",
    "    # number of unique products a given user has purchased\n",
    "    u_dict['U_unique_products'][ds] = (\n",
    "        prior[ds].groupby('user_id')['product_id'].nunique().apply(\n",
    "            pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # number of total items a given user has purchased which are reorders\n",
    "    u_dict['U_reordered_num'][ds] = (\n",
    "        prior[ds].groupby('user_id')['reordered'].sum().apply(\n",
    "            pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # mean reorders per basket\n",
    "    u_dict['U_reorder_size_mean'][ds] = (u_dict['U_reordered_num'][ds].div(\n",
    "        u_dict['U_orders_num'][ds]).astype('float16'))\n",
    "\n",
    "    # std reorders per basket\n",
    "    u_dict['U_reorder_size_std'][ds] = (prior[ds].groupby([\n",
    "        'user_id', 'order_number'\n",
    "    ]).reordered.sum().groupby('user_id').std().astype('float16'))\n",
    "\n",
    "    # proportion of items a given user has purchased which are reorders\n",
    "    u_dict['U_reordered_ratio'][ds] = (u_dict['U_reordered_num'][ds].div(\n",
    "        u_dict['U_items_total'][ds]).astype('float16'))\n",
    "\n",
    "    # mean order_dow\n",
    "    u_dict['U_order_dow_mean'][ds] = pd.concat(\n",
    "        [\n",
    "            orders[ds]['user_id'],\n",
    "            angle_transform(\n",
    "                # load-bearing .rename(). Fix.\n",
    "                orders[ds]['order_dow'].rename('U_order_dow_mean'),\n",
    "                7)\n",
    "        ],\n",
    "        axis=1).groupby('user_id').aggregate(circmean).astype(\n",
    "            'float16').U_order_dow_mean\n",
    "\n",
    "    # var order_dow\n",
    "    u_dict['U_order_dow_var'][ds] = pd.concat(\n",
    "        [\n",
    "            orders[ds]['user_id'],\n",
    "            angle_transform(\n",
    "                # load-bearing .rename(). Fix.\n",
    "                orders[ds]['order_dow'].rename('U_order_dow_var'),\n",
    "                7)\n",
    "        ],\n",
    "        axis=1).groupby('user_id').aggregate(circvar).astype(\n",
    "            'float16').U_order_dow_var\n",
    "\n",
    "    # ultimate score for order_dow using circstd = sqrt(-2ln(circvar))\n",
    "    u_dict['U_order_dow_score'][ds] = (\n",
    "        u_given_dict['U_ultimate_order_dow'][ds]\n",
    "        .sub(u_dict['U_order_dow_mean'][ds])\n",
    "        .div(u_dict['U_order_dow_var'][ds]\n",
    "             .apply(lambda x: np.sqrt(-2 * np.log(x))))\n",
    "        .fillna(0)\n",
    "        .clip(-20, 20)\n",
    "        .astype('float16'))\n",
    "\n",
    "    # mean order_hour_of_day\n",
    "    u_dict['U_order_hour_of_day_mean'][ds] = (\n",
    "        pd.concat(\n",
    "            [orders[ds]['user_id'],\n",
    "                angle_transform(\n",
    "                    orders[ds]['order_hour_of_day']\n",
    "                    # load-bearing .rename(). Fix.\n",
    "                    .rename('U_order_hour_of_day_mean'),\n",
    "                    24)\n",
    "            ],\n",
    "            axis=1)\n",
    "        .groupby('user_id')\n",
    "        .aggregate(circmean)\n",
    "        .astype('float16')\n",
    "        .U_order_hour_of_day_mean)\n",
    "\n",
    "    # var order_hour_of_day\n",
    "    u_dict['U_order_hour_of_day_var'][ds] = (\n",
    "        pd.concat(\n",
    "            [\n",
    "                orders[ds]['user_id'],\n",
    "                angle_transform(\n",
    "                    orders[ds]['order_hour_of_day']\n",
    "                    # load-bearing .rename(). Fix.\n",
    "                    .rename('U_order_hour_of_day_var'),\n",
    "                    24)\n",
    "            ],\n",
    "            axis=1)\n",
    "        .groupby('user_id')\n",
    "        .aggregate(circvar)\n",
    "        .astype('float16')\n",
    "        .U_order_hour_of_day_var)\n",
    "\n",
    "    # ultimate score for order_hour_of_day using circstd = sqrt(-2ln(circvar))\n",
    "    u_dict['U_order_hour_of_day_score'][ds] = (\n",
    "        u_given_dict['U_ultimate_order_hour_of_day'][ds]\n",
    "        .sub(u_dict['U_order_hour_of_day_mean'][ds])\n",
    "        .div(u_dict['U_order_hour_of_day_var'][ds]\n",
    "             .apply(lambda x: np.sqrt(-2 * np.log(x))))\n",
    "        .fillna(0)\n",
    "        .clip(-20, 20)\n",
    "        .astype('float16')\n",
    "    )\n",
    "\n",
    "    # mean days since prior order (mean user order time interval)\n",
    "    u_dict['U_days_since_prior_order_mean'][ds] = (\n",
    "        orders_full[ds]\n",
    "        .groupby('user_id')\n",
    "        .days_since_prior_order\n",
    "        .mean()\n",
    "        .astype('float16')\n",
    "    )\n",
    "\n",
    "    # std days since prior order (std user order time interval)\n",
    "    u_dict['U_days_since_prior_order_std'][ds] = (\n",
    "        orders_full[ds]\n",
    "        .groupby('user_id')\n",
    "        .days_since_prior_order\n",
    "        .std()\n",
    "        .astype('float16')\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename feature columns/pandas Series object by u_dict key name pointing to it.\n",
    "\n",
    "for ds in dsets:\n",
    "    for k, v in u_dict.items():\n",
    "        v[ds].rename(k, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine user features; store as key 'U'\n",
    "\n",
    "groups_dict['U'] = {ds : pd.concat([u_dict[k][ds] for k in u_dict.keys()], axis=1) for ds in dsets}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Product Profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dictionary to store product features\n",
    "p_dict = defaultdict(dict)\n",
    "\n",
    "for ds in dsets:\n",
    "\n",
    "    # number of total purchases\n",
    "    p_dict['P_orders_num'][ds] = (\n",
    "        prior[ds]\n",
    "        .groupby('product_id')['order_id']\n",
    "        .count()\n",
    "        .apply(pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # number of purchasers\n",
    "    p_dict['P_unique_users'][ds] = (\n",
    "        prior[ds]\n",
    "        .groupby('product_id')['user_id']\n",
    "        .nunique()\n",
    "        .apply(pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # reorder ratio\n",
    "    p_dict['P_reorder_ratio'][ds] = (\n",
    "        prior[ds]\n",
    "        .groupby(['product_id'])['reordered']\n",
    "        .mean()\n",
    "        .astype('float16'))\n",
    "\n",
    "    # mean order_hour_of_day\n",
    "    p_dict['P_order_hour_of_day_mean'][ds] = angle_transform(\n",
    "        prior[ds]\n",
    "        .set_index('product_id')\n",
    "        .order_hour_of_day,\n",
    "        24).groupby('product_id').aggregate(circmean)\n",
    "\n",
    "    # var order_hour_of_day\n",
    "    p_dict['P_order_hour_of_day_var'][ds] = angle_transform(\n",
    "        prior[ds]\n",
    "        .set_index('product_id')\n",
    "        .order_hour_of_day,\n",
    "        24).groupby('product_id').aggregate(circvar)\n",
    "\n",
    "    # mean order_dow\n",
    "    p_dict['P_order_dow_mean'][ds] = angle_transform(\n",
    "        prior[ds]\n",
    "        .set_index('product_id')\n",
    "        .order_hour_of_day,\n",
    "        7).groupby('product_id').aggregate(circmean)\n",
    "\n",
    "    # var order_dow\n",
    "    p_dict['P_order_dow_var'][ds] = angle_transform(\n",
    "        prior[ds]\n",
    "        .set_index('product_id')\n",
    "        .order_hour_of_day,\n",
    "        7).groupby('product_id').aggregate(circvar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename feature columns/pandas Series objects by p_dict key name pointing to it.\n",
    "\n",
    "for ds in dsets:\n",
    "    for k, v in p_dict.items():\n",
    "        v[ds].rename(k, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine product features; store as key 'P'\n",
    "\n",
    "groups_dict['P'] = {\n",
    "    ds: pd.concat([p_dict[k][ds] for k in p_dict.keys()], axis=1)\n",
    "    for ds in dsets\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User-Product Profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dictionary to store user-product features\n",
    "up_dict = defaultdict(dict)\n",
    "\n",
    "for ds in dsets:\n",
    "\n",
    "    # number of times particular user has ordered particular product\n",
    "    up_dict['UP_orders_num'][ds] = (\n",
    "        prior[ds]\n",
    "        .groupby(['user_id', 'product_id'])['order_id']\n",
    "        .count()\n",
    "        .apply(pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # number of orders since previous purchase of product by user\n",
    "    # fill_value = infty?\n",
    "    up_dict['UP_orders_since_previous'][ds] = (\n",
    "        prior[ds].groupby(['user_id'])['order_number']\n",
    "        .max()\n",
    "        - prior[ds]\n",
    "        .groupby(['user_id', 'product_id'])['order_number']\n",
    "        .max()\n",
    "        .apply(pd.to_numeric, downcast='unsigned'))\n",
    "\n",
    "    # days since user last ordered product\n",
    "    # groups of days_since_prior_order by user_id\n",
    "    days_gpby_user = (\n",
    "        orders_full[ds]\n",
    "        .groupby('user_id')\n",
    "        .days_since_prior_order\n",
    "    )\n",
    "\n",
    "    # given 'order_number' is UP_orders_since_previous\n",
    "    # sum last orders_ago+1 days_since_prior_order\n",
    "    def days_ago(row):\n",
    "        orders_ago = int(row['order_number'])\n",
    "        user = row['user_id']\n",
    "        return (days_gpby_user\n",
    "                .get_group(user)\n",
    "                .iloc[-(orders_ago + 1):]\n",
    "                .sum())\n",
    "\n",
    "    # apply days_ago to UP_orders_since_previous\n",
    "    up_dict['UP_days_since_prior_order'][ds] = (pd.Series(\n",
    "        data=up_dict['UP_orders_since_previous'][ds]\n",
    "        .reset_index()\n",
    "        .apply(days_ago, axis=1)\n",
    "        .values,\n",
    "        index=up_dict['UP_orders_since_previous'][ds].index)\n",
    "    .astype('uint16'))\n",
    "\n",
    "    # clean-up\n",
    "    del days_gpby_user\n",
    "\n",
    "    # normalize above by user's days_since_prior_order\n",
    "    # maybe use t-score instead?\n",
    "    up_dict['UP_days_since_prior_order_score'][ds] = (\n",
    "        up_dict['UP_days_since_prior_order'][ds]\n",
    "        .sub(up_empty_df[ds].join(\n",
    "            u_dict['U_days_since_prior_order_mean'][ds]).iloc[:, 0])\n",
    "        .div(up_empty_df[ds].join(\n",
    "            u_dict['U_days_since_prior_order_std'][ds]).iloc[:, 0])\n",
    "        .fillna(0).clip(-20, 20).astype('float16'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ds in dsets:\n",
    "\n",
    "    # reordered as `bool`\n",
    "    up_dict['UP_reordered'][ds] = (\n",
    "        prior[ds]\n",
    "        .groupby(['user_id', 'product_id'])['reordered']\n",
    "        .any())\n",
    "\n",
    "    # fraction of baskets in which a given product appears for a given user,\n",
    "    # count of orders in which product appears divided by total orders\n",
    "    up_dict['UP_order_ratio'][ds] = (\n",
    "        prior[ds].groupby(['user_id', 'product_id'])['order_number']\n",
    "        .count()\n",
    "        .div(prior[ds].groupby(['user_id'])['order_number']\n",
    "             .max())\n",
    "        .astype('float16')\n",
    "    )\n",
    "\n",
    "    # products in user's penultimate (previous) order as `bool`\n",
    "    # (`train` and `test` sets contain ultimate order)\n",
    "\n",
    "    up_dict['UP_penultimate'][ds] = (\n",
    "        prior[ds].groupby(['user_id', 'product_id'])\n",
    "        .order_number\n",
    "        .max() \n",
    "        == prior[ds].groupby(['user_id'])\n",
    "        .order_number\n",
    "        .max()\n",
    "        .reindex(up_index[ds], level=0)\n",
    "    )\n",
    "\n",
    "    # products in user's antepenultimate order as `bool`\n",
    "    # index = UP pair (not distinct) with data = order_number\n",
    "    past_orders = (\n",
    "        prior[ds][['user_id', 'order_number', 'product_id']]\n",
    "        .set_index(['user_id', 'product_id'])\n",
    "    )\n",
    "    \n",
    "    # all UP pairs with max order_number - 1\n",
    "    max_order_number_sub1 = (\n",
    "        prior[ds].groupby(['user_id'])\n",
    "        .order_number\n",
    "        .max()\n",
    "        .sub(1)\n",
    "        .reindex(up_index[ds], level=0)\n",
    "        .to_frame()\n",
    "    )\n",
    "    \n",
    "    # intersection\n",
    "    up_dict['UP_antepenultimate'][ds] = (\n",
    "        pd.merge(\n",
    "            past_orders,\n",
    "            max_order_number_sub1,\n",
    "            on=['user_id', 'product_id', 'order_number'])\n",
    "        .reindex(up_index[ds], fill_value=False)\n",
    "        .astype('bool')\n",
    "        .iloc[:, 0]\n",
    "    )\n",
    "    \n",
    "    # cleanup\n",
    "    del past_orders, max_order_number_sub1\n",
    "\n",
    "    # ultimate score for order_dow using circstd = sqrt(-2ln(circvar))\n",
    "    # using (U_ultimate - P_order_dow_mean) / P_order_dow_std\n",
    "    # broadcast to up_index\n",
    "    # intuitively, how 'far' is a user's ultimate order dow from the mean dow product is ordered\n",
    "    up_dict['UP_order_dow_score'][ds] = (\n",
    "        pd.DataFrame(\n",
    "            data=(up_empty_df[ds]\n",
    "                      .join(u_given_dict['U_ultimate_order_dow'][ds])\n",
    "                      .iloc[:, 0]\n",
    "                  .sub(up_empty_df[ds]\n",
    "                       .join(p_dict['P_order_dow_mean'][ds])\n",
    "                       .iloc[:, 0])\n",
    "                  .div(up_empty_df[ds]\n",
    "                       .join(p_dict['P_order_dow_var'][ds]\n",
    "                             .apply(lambda x: \n",
    "                                    np.sqrt(-2 * np.log(x))))\n",
    "                       .iloc[:, 0])\n",
    "                  ),\n",
    "            index=up_index[ds])    \n",
    "        .fillna(0)\n",
    "        .clip(-20, 20)\n",
    "        .astype('float16')\n",
    "        .iloc[:, 0]\n",
    "    )\n",
    "        \n",
    "    # ultimate score for order_hour_of_day using circstd = sqrt(-2ln(circvar))\n",
    "    # using (U_ultimate - P_order_hour_of_day_mean) / P_order_hour_of_day_std\n",
    "    # broadcast to up_index\n",
    "    # intuitively, how 'far' is a user's ultimate order hour_of_day from the mean hour_of_day product is ordered\n",
    "    # ndarray instead of pandas; couldn't resolve an arithmetic issue\n",
    "    up_dict['UP_order_hour_of_day_score'][ds] = (\n",
    "        pd.DataFrame(\n",
    "            data=(up_empty_df[ds]\n",
    "                      .join(u_given_dict['U_ultimate_order_hour_of_day'][ds])\n",
    "                      .iloc[:, 0]\n",
    "                  .sub(up_empty_df[ds]\n",
    "                       .join(p_dict['P_order_hour_of_day_mean'][ds])\n",
    "                       .iloc[:, 0])\n",
    "                  .div(up_empty_df[ds]\n",
    "                       .join(p_dict['P_order_hour_of_day_var'][ds]\n",
    "                             .apply(lambda x: \n",
    "                                    np.sqrt(-2 * np.log(x))))\n",
    "                       .iloc[:, 0])\n",
    "                  ),\n",
    "            index=up_index[ds])    \n",
    "        .fillna(0)\n",
    "        .clip(-20, 20)\n",
    "        .astype('float16')\n",
    "        .iloc[:, 0]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename feature columns/pandas Series objects by up_dict key name pointing to it.\n",
    "\n",
    "for ds in dsets:\n",
    "    for k, v in up_dict.items():\n",
    "        v[ds].rename(k, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine user-product features; store as key 'UP'\n",
    "\n",
    "groups_dict['UP'] = {\n",
    "    ds: pd.concat([up_dict[k][ds] for k in up_dict.keys()], axis=1)\n",
    "    for ds in dsets\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latent Dirichlet Allocation Features\n",
    "\n",
    "The parameter values for [`sklearn.decomposition.LatentDirichletAllocation`](https://scikit-learn.org/stable/modules/decomposition.html#latent-dirichlet-allocation-lda) below are found and discussed in the notebooks:\n",
    "* [Instacart: LDA GridSearchCV (Course)](./instacart-lda-gridsearchcv-course)\n",
    "* [Instacart: LDA GridSearchCV (Fine)](./instacart-lda-gridsearchcv-fine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scipy sparse matrix of number of times particular user has ordered particular product\n",
    "UP_count_matrix = dict.fromkeys(dsets)\n",
    "\n",
    "for ds in dsets:\n",
    "    UP_count_matrix[ds], _, _ = (groups_dict['UP'][ds]['UP_orders_num'].apply(\n",
    "        pd.to_numeric, downcast='unsigned').to_sparse().to_coo())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "LDA_features = dict.fromkeys(dsets)\n",
    "\n",
    "for ds in dsets:\n",
    "    lda = LatentDirichletAllocation(n_components=10,\n",
    "                                    max_iter=10,\n",
    "                                    learning_decay=0.85,\n",
    "                                    n_jobs=1,\n",
    "                                    learning_method='online')\n",
    "\n",
    "    LDA_features[ds] = lda.fit_transform(UP_count_matrix[ds])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "groups_dict['LDA'] = {\n",
    "    ds: pd.DataFrame(data=LDA_features[ds],\n",
    "                     index=u_index[ds],\n",
    "                     columns=[\n",
    "                         'LDA_' + str(k + 1)\n",
    "                         for k in range(LDA_features[ds].shape[1])\n",
    "                     ]).astype('float16')\n",
    "    for ds in dsets\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aisle and Department Features\n",
    "\n",
    "The features below did not score well in previous versions of [Instacart: Top-N Random Forest Model](./instacart-top-n-random-forest-model/). User-aisle and user-department features defined in analogy to user-product features above, should perform considerably better than the aisle and department features below, which are defined in analogy to product features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # dictionary to store aisle features\n",
    "# a_dict = defaultdict(dict)\n",
    "\n",
    "# for ds in dsets:\n",
    "\n",
    "#     # mean order_hour_of_day\n",
    "#     a_dict['A_order_hour_of_day_mean'][ds] = angle_transform(ad[ds].set_index('aisle_id')\n",
    "#                                                 .order_hour_of_day,\n",
    "#                                                 24\n",
    "#                                                 ).groupby('aisle_id').aggregate(circmean)\n",
    "\n",
    "#     # std order_hour_of_day\n",
    "#     a_dict['A_order_hour_of_day_var'][ds] = angle_transform(ad[ds].set_index('aisle_id')\n",
    "#                                                .order_hour_of_day,\n",
    "#                                                24\n",
    "#                                                ).groupby('aisle_id').aggregate(circvar)\n",
    "\n",
    "#     # mean order_dow\n",
    "#     a_dict['A_order_dow_mean'][ds] = angle_transform(ad[ds].set_index('aisle_id')\n",
    "#                                         .order_dow,\n",
    "#                                         7\n",
    "#                                         ).groupby('aisle_id').aggregate(circmean)\n",
    "\n",
    "#     # var order_dow\n",
    "#     a_dict['A_order_dow_var'][ds] = angle_transform(ad[ds].set_index('aisle_id')\n",
    "#                                        .order_dow,\n",
    "#                                        7\n",
    "#                                        ).groupby('aisle_id').aggregate(circvar)\n",
    "\n",
    "#     # reorder ratio\n",
    "#     a_dict['A_reorder_ratio'][ds] = (ad[ds].groupby(['aisle_id'])['reordered']\n",
    "#                        .mean()\n",
    "#                        .astype('float16')\n",
    "#                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Rename feature columns/pandas Series objects by a_dict key name pointing to it.\n",
    "\n",
    "# for ds in dsets:\n",
    "#     for k, v in a_dict.items():\n",
    "#         v[ds].rename(k, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Combine aisle features into a_features\n",
    "# # Reindex to products index for join with up_index\n",
    "\n",
    "# #a_features = {ds : pd.DataFrame(index=groups_dict['P'][ds].index).join(\n",
    "# #    pd.concat([a_dict[k][ds] for k in a_dict.keys()], axis=1)) for ds in dsets}\n",
    "\n",
    "# # a_features = {ds : pd.concat([a_dict[k][ds] for k in a_dict.keys()], axis=1) for ds in dsets}\n",
    "\n",
    "# groups_dict['A'] = {ds : \n",
    "#     # \"dict\" from product_id -> aisle_id (index=product_id, col=aisle_id)\n",
    "#                     df_ad[['aisle_id', 'product_id']]\n",
    "#                     .drop_duplicates()\n",
    "#                     .set_index('product_id')\n",
    "#                     .sort_index()\n",
    "#     # join with aisle features with aisle_id as column\n",
    "#                     .join(\n",
    "#                         pd.concat([feature[ds] for feature in a_dict.values()], axis=1),\n",
    "#         on='aisle_id')\n",
    "#     .drop('aisle_id', axis=1)\n",
    "#                     for ds in dsets}\n",
    "\n",
    "# for ds in dsets:\n",
    "#     groups_dict['A'][ds].index.rename('product_id', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # dictionary to store department features\n",
    "# d_dict = defaultdict(dict)\n",
    "\n",
    "# for ds in dsets:\n",
    "    \n",
    "#     # mean order_hour_of_day\n",
    "#     d_dict['D_order_hour_of_day_mean'][ds] = angle_transform(ad[ds].set_index('department_id')\n",
    "#                                                 .order_hour_of_day,\n",
    "#                                                 24\n",
    "#                                                 ).groupby('department_id').aggregate(circmean)\n",
    "\n",
    "#     # std order_hour_of_day\n",
    "#     d_dict['D_order_hour_of_day_var'][ds] = angle_transform(ad[ds].set_index('department_id')\n",
    "#                                                .order_hour_of_day,\n",
    "#                                                24\n",
    "#                                                ).groupby('department_id').aggregate(circvar)\n",
    "\n",
    "#     # mean order_dow\n",
    "#     d_dict['D_order_dow_mean'][ds] = angle_transform(ad[ds].set_index('department_id')\n",
    "#                                         .order_dow,\n",
    "#                                         7\n",
    "#                                         ).groupby('department_id').aggregate(circmean)\n",
    "\n",
    "#     # var order_dow\n",
    "#     d_dict['D_order_dow_var'][ds] = angle_transform(ad[ds].set_index('department_id')\n",
    "#                                        .order_dow,\n",
    "#                                        7\n",
    "#                                        ).groupby('department_id').aggregate(circvar)\n",
    "\n",
    "#     # reorder ratio\n",
    "#     d_dict['D_reorder_ratio'][ds] = (ad[ds].groupby(['department_id'])['reordered']\n",
    "#                        .mean()\n",
    "#                        .astype('float16')\n",
    "#                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Rename feature columns/pandas Series objects by d_dict key name pointing to it.\n",
    "\n",
    "# for ds in dsets:\n",
    "#     for k, v in d_dict.items():\n",
    "#         v[ds].rename(k, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Combine department features into a_features\n",
    "# # Reindex to products index for join with up_index\n",
    "\n",
    "# #a_features = {ds : pd.DataFrame(index=groups_dict['P'][ds].index).join(\n",
    "# #    pd.concat([d_dict[k][ds] for k in d_dict.keys()], axis=1)) for ds in dsets}\n",
    "\n",
    "# # a_features = {ds : pd.concat([d_dict[k][ds] for k in d_dict.keys()], axis=1) for ds in dsets}\n",
    "\n",
    "# groups_dict['D'] = {ds : \n",
    "#     # \"dict\" from product_id -> department_id (index=product_id, col=department_id)\n",
    "#                     df_ad[['department_id', 'product_id']]\n",
    "#                     .drop_duplicates()\n",
    "#                     .set_index('product_id')\n",
    "#                     .sort_index()\n",
    "#     # join with department features with department_id as column\n",
    "#                     .join(\n",
    "#                         pd.concat([feature[ds] for feature in d_dict.values()], axis=1),\n",
    "#         on='department_id')\n",
    "#     .drop('department_id', axis=1)\n",
    "#                     for ds in dsets}\n",
    "\n",
    "# for ds in dsets:\n",
    "#     groups_dict['D'][ds].index.rename('product_id', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup intermediate dicts\n",
    "del (\n",
    "    u_given_dict,\n",
    "    u_dict,\n",
    "    p_dict,\n",
    "    up_dict,\n",
    "    #     a_dict,\n",
    "    #     d_dict\n",
    ")\n",
    "\n",
    "# Cleanup dataframes\n",
    "del (  #df_ad,\n",
    "    df_aisles, df_departments, df_order_products_prior,\n",
    "    df_order_products_train, df_orders, df_prior, df_products)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LDA_features\t LatentDirichletAllocation\t UP_count_matrix\t angle_transform\t circmean\t circvar\t days_ago\t defaultdict\t dframe\t \n",
      "ds\t dset\t dsets\t file_path\t groups_dict\t k\t lda\t load_data_dtype\t np\t \n",
      "orders\t orders_full\t pd\t prior\t products_num\t store\t train_test_split\t u_index\t ultimate\t \n",
      "up_empty_df\t up_index\t users\t users_num\t v\t \n"
     ]
    }
   ],
   "source": [
    "%who"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concatenate\n",
    "\n",
    "Combine the above constructed features into a `dset`-keyed `dict` `X[ds]` to instantiate $\\{X_s\\}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate list of elements of groups_dict for each dset\n",
    "X = {\n",
    "    ds: pd.concat([\n",
    "        pd.DataFrame(index=up_index[ds]).join(group[ds])\n",
    "        for group in groups_dict.values()\n",
    "    ],\n",
    "                  axis=1)\n",
    "    for ds in dsets\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[False, False, False]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Nulls make sklearn unhappy\n",
    "[X[ds].isnull().any().any() for ds in dsets]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above undid `uint` downcasts somewhere. For now, fix manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "MultiIndex: 6760791 entries, (1, 196) to (206209, 48742)\n",
      "Data columns (total 47 columns):\n",
      "U_ultimate_order_dow                 float16\n",
      "U_ultimate_order_hour_of_day         float16\n",
      "U_ultimate_days_since_prior_order    float16\n",
      "U_orders_num                         int64\n",
      "U_items_total                        int64\n",
      "U_order_size_mean                    float16\n",
      "U_order_size_std                     float16\n",
      "U_unique_products                    int64\n",
      "U_reordered_num                      float64\n",
      "U_reorder_size_mean                  float16\n",
      "U_reorder_size_std                   float16\n",
      "U_reordered_ratio                    float16\n",
      "U_order_dow_mean                     float16\n",
      "U_order_dow_var                      float16\n",
      "U_order_dow_score                    float16\n",
      "U_order_hour_of_day_mean             float16\n",
      "U_order_hour_of_day_var              float16\n",
      "U_order_hour_of_day_score            float16\n",
      "U_days_since_prior_order_mean        float16\n",
      "U_days_since_prior_order_std         float16\n",
      "P_orders_num                         int64\n",
      "P_unique_users                       int64\n",
      "P_reorder_ratio                      float16\n",
      "P_order_hour_of_day_mean             float16\n",
      "P_order_hour_of_day_var              float16\n",
      "P_order_dow_mean                     float16\n",
      "P_order_dow_var                      float16\n",
      "UP_orders_num                        int64\n",
      "UP_orders_since_previous             int64\n",
      "UP_days_since_prior_order            uint16\n",
      "UP_days_since_prior_order_score      float16\n",
      "UP_reordered                         bool\n",
      "UP_order_ratio                       float16\n",
      "UP_penultimate                       bool\n",
      "UP_antepenultimate                   bool\n",
      "UP_order_dow_score                   float16\n",
      "UP_order_hour_of_day_score           float16\n",
      "LDA_1                                float16\n",
      "LDA_2                                float16\n",
      "LDA_3                                float16\n",
      "LDA_4                                float16\n",
      "LDA_5                                float16\n",
      "LDA_6                                float16\n",
      "LDA_7                                float16\n",
      "LDA_8                                float16\n",
      "LDA_9                                float16\n",
      "LDA_10                               float16\n",
      "dtypes: bool(3), float16(35), float64(1), int64(7), uint16(1)\n",
      "memory usage: 1.2 GB\n"
     ]
    }
   ],
   "source": [
    "X['train'].info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = [\n",
    "    'U_orders_num', 'U_items_total', 'U_unique_products', 'U_reordered_num',\n",
    "    'P_orders_num', 'P_unique_users', 'UP_orders_num',\n",
    "    'UP_orders_since_previous'\n",
    "]\n",
    "\n",
    "for ds in dsets:\n",
    "    X[ds][cols] = X[ds][cols].apply(pd.to_numeric,\n",
    "                                    errors='coerce',\n",
    "                                    downcast='unsigned')\n",
    "\n",
    "# for ds in dsets:\n",
    "#     X[ds]['UP_order_dow_score'] = np.nan_to_num(X[ds]['UP_order_dow_score'])\n",
    "#     X[ds]['UP_order_hour_of_day_score'] = np.nan_to_num(X[ds]['UP_order_hour_of_day_score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "MultiIndex: 6760791 entries, (1, 196) to (206209, 48742)\n",
      "Data columns (total 47 columns):\n",
      "U_ultimate_order_dow                 float16\n",
      "U_ultimate_order_hour_of_day         float16\n",
      "U_ultimate_days_since_prior_order    float16\n",
      "U_orders_num                         uint8\n",
      "U_items_total                        uint16\n",
      "U_order_size_mean                    float16\n",
      "U_order_size_std                     float16\n",
      "U_unique_products                    uint16\n",
      "U_reordered_num                      uint16\n",
      "U_reorder_size_mean                  float16\n",
      "U_reorder_size_std                   float16\n",
      "U_reordered_ratio                    float16\n",
      "U_order_dow_mean                     float16\n",
      "U_order_dow_var                      float16\n",
      "U_order_dow_score                    float16\n",
      "U_order_hour_of_day_mean             float16\n",
      "U_order_hour_of_day_var              float16\n",
      "U_order_hour_of_day_score            float16\n",
      "U_days_since_prior_order_mean        float16\n",
      "U_days_since_prior_order_std         float16\n",
      "P_orders_num                         uint32\n",
      "P_unique_users                       uint16\n",
      "P_reorder_ratio                      float16\n",
      "P_order_hour_of_day_mean             float16\n",
      "P_order_hour_of_day_var              float16\n",
      "P_order_dow_mean                     float16\n",
      "P_order_dow_var                      float16\n",
      "UP_orders_num                        uint8\n",
      "UP_orders_since_previous             uint8\n",
      "UP_days_since_prior_order            uint16\n",
      "UP_days_since_prior_order_score      float16\n",
      "UP_reordered                         bool\n",
      "UP_order_ratio                       float16\n",
      "UP_penultimate                       bool\n",
      "UP_antepenultimate                   bool\n",
      "UP_order_dow_score                   float16\n",
      "UP_order_hour_of_day_score           float16\n",
      "LDA_1                                float16\n",
      "LDA_2                                float16\n",
      "LDA_3                                float16\n",
      "LDA_4                                float16\n",
      "LDA_5                                float16\n",
      "LDA_6                                float16\n",
      "LDA_7                                float16\n",
      "LDA_8                                float16\n",
      "LDA_9                                float16\n",
      "LDA_10                               float16\n",
      "dtypes: bool(3), float16(35), uint16(5), uint32(1), uint8(3)\n",
      "memory usage: 958.0 MB\n"
     ]
    }
   ],
   "source": [
    "X['train'].info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save $\\{X_s\\}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "store.open()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.is_open"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "for dset, dframe in X.items():\n",
    "    store['/X/' + str(dset)] = dframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/X/kaggle', '/X/test', '/X/train', '/y/kaggle', '/y/test', '/y/train']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "(<a id=\"cit-bleiLatentDirichletAllocation2003\" href=\"#call-bleiLatentDirichletAllocation2003\">Blei, Ng <em>et al.</em>, 2003</a>) Blei David M., Ng Andrew Y. and Jordan Michael I., ``_Latent Dirichlet Allocation_'', Journal of Machine Learning Research, vol. 3, number Jan, pp. 993-1022,  2003.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "ctrl-meta-e"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "496px",
    "width": "373px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}