2_Pandas_apply_optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing functions on pandas dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we compare the efficiency of several ways of applying a function to a Pandas DataFrame:\n",
    "1. Crude looping over df using indices\n",
    "2. Looping with iterrows\n",
    "3. `.apply()`\n",
    "4. Vectorization with Pandas series\n",
    "5. Vectorization with NumPy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using the `%%timeit` magic command to get execution times. This command runs the cell 7 times and returns mean and standard deviation of the run times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:29:05.790356Z",
     "start_time": "2020-06-06T08:29:03.827403Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:29:07.283880Z",
     "start_time": "2020-06-06T08:29:07.003852Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('./data/input_file.csv.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The files contains pairs of (lat.,long.).<br>\n",
    "We will use the Haversine distance function on this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:29:08.914537Z",
     "start_time": "2020-06-06T08:29:08.888006Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lat_a</th>\n",
       "      <th>long_a</th>\n",
       "      <th>lat_b</th>\n",
       "      <th>long_b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>id1_id1</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>id1_id2</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.487651</td>\n",
       "      <td>88.350805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>id1_id3</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.486251</td>\n",
       "      <td>88.352905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>id1_id4</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.569698</td>\n",
       "      <td>88.350871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>id1_id5</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.498853</td>\n",
       "      <td>88.372714</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id      lat_a     long_a      lat_b     long_b\n",
       "0  id1_id1  22.541167  88.326124  22.541167  88.326124\n",
       "1  id1_id2  22.541167  88.326124  22.487651  88.350805\n",
       "2  id1_id3  22.541167  88.326124  22.486251  88.352905\n",
       "3  id1_id4  22.541167  88.326124  22.569698  88.350871\n",
       "4  id1_id5  22.541167  88.326124  22.498853  88.372714"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file has <b>250K</b> records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:29:14.641590Z",
     "start_time": "2020-06-06T08:29:14.611533Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 250000 entries, 0 to 249999\n",
      "Data columns (total 5 columns):\n",
      " #   Column  Non-Null Count   Dtype  \n",
      "---  ------  --------------   -----  \n",
      " 0   id      250000 non-null  object \n",
      " 1   lat_a   250000 non-null  float64\n",
      " 2   long_a  250000 non-null  float64\n",
      " 3   lat_b   250000 non-null  float64\n",
      " 4   long_b  250000 non-null  float64\n",
      "dtypes: float64(4), object(1)\n",
      "memory usage: 9.5+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Haversine Function\n",
    "[More info](https://www.movable-type.co.uk/scripts/latlong.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:29:17.621699Z",
     "start_time": "2020-06-06T08:29:17.612174Z"
    }
   },
   "outputs": [],
   "source": [
    "def calculate_distance_haversine(lat_a, lng_a, lat_b, lng_b):\n",
    "    '''\n",
    "    Function to calculate distance between two sets of points defined by (latitude, longitude)\n",
    "    Parameters:\n",
    "     lat_a: latitude of point a\n",
    "     lng_a: longitude of point a\n",
    "     lat_b: latitude of point b\n",
    "     lng_b: longitude of point b     \n",
    "    Returns:\n",
    "     Haversine distance, unit: kms\n",
    "    '''\n",
    "    #   Convert lat lng in radians\n",
    "    lng_a, lat_a, lng_b, lat_b = map(np.deg2rad, [lng_a, lat_a, lng_b, lat_b])\n",
    "    d_lat, d_lng = lat_b - lat_a, lng_a - lng_b\n",
    "    #\n",
    "    temp = (\n",
    "            np.sin(d_lat / 2) ** 2\n",
    "            + np.cos(lat_a)\n",
    "            * np.cos(lat_b)\n",
    "            * np.sin(d_lng / 2) ** 2\n",
    "            )\n",
    "    distance = 6373.0 * 2 * np.arcsin(np.sqrt(temp))\n",
    "    return np.round(distance,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping over row indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:55:25.605381Z",
     "start_time": "2020-06-06T08:29:19.546629Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3min 9s ± 3.49 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "distance = []\n",
    "for i in range(df.shape[0]):\n",
    "    d = calculate_distance_haversine(df.loc[i]['lat_a'], df.loc[i]['long_a'], df.loc[i]['lat_b'], df.loc[i]['long_b'])\n",
    "    distance.append(d)\n",
    "df['distance'] = distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T08:55:57.440707Z",
     "start_time": "2020-06-06T08:55:57.428702Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lat_a</th>\n",
       "      <th>long_a</th>\n",
       "      <th>lat_b</th>\n",
       "      <th>long_b</th>\n",
       "      <th>distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>id1_id1</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>id1_id2</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.487651</td>\n",
       "      <td>88.350805</td>\n",
       "      <td>6.470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>id1_id3</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.486251</td>\n",
       "      <td>88.352905</td>\n",
       "      <td>6.700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>id1_id4</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.569698</td>\n",
       "      <td>88.350871</td>\n",
       "      <td>4.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>id1_id5</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.498853</td>\n",
       "      <td>88.372714</td>\n",
       "      <td>6.713</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id      lat_a     long_a      lat_b     long_b  distance\n",
       "0  id1_id1  22.541167  88.326124  22.541167  88.326124     0.000\n",
       "1  id1_id2  22.541167  88.326124  22.487651  88.350805     6.470\n",
       "2  id1_id3  22.541167  88.326124  22.486251  88.352905     6.700\n",
       "3  id1_id4  22.541167  88.326124  22.569698  88.350871     4.066\n",
       "4  id1_id5  22.541167  88.326124  22.498853  88.372714     6.713"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping with iterrows()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:01:14.488101Z",
     "start_time": "2020-06-06T08:56:09.176641Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "41.7 s ± 949 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "haversine_series = []\n",
    "for index, row in df.iterrows():\n",
    "    d = calculate_distance_haversine(row['lat_a'], row['long_a'], row['lat_b'], row['long_b'])\n",
    "    haversine_series.append(d)\n",
    "\n",
    "df['distance'] = haversine_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:07:18.071710Z",
     "start_time": "2020-06-06T09:07:18.058713Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lat_a</th>\n",
       "      <th>long_a</th>\n",
       "      <th>lat_b</th>\n",
       "      <th>long_b</th>\n",
       "      <th>distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>id1_id1</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>id1_id2</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.487651</td>\n",
       "      <td>88.350805</td>\n",
       "      <td>6.470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>id1_id3</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.486251</td>\n",
       "      <td>88.352905</td>\n",
       "      <td>6.700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>id1_id4</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.569698</td>\n",
       "      <td>88.350871</td>\n",
       "      <td>4.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>id1_id5</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.498853</td>\n",
       "      <td>88.372714</td>\n",
       "      <td>6.713</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id      lat_a     long_a      lat_b     long_b  distance\n",
       "0  id1_id1  22.541167  88.326124  22.541167  88.326124     0.000\n",
       "1  id1_id2  22.541167  88.326124  22.487651  88.350805     6.470\n",
       "2  id1_id3  22.541167  88.326124  22.486251  88.352905     6.700\n",
       "3  id1_id4  22.541167  88.326124  22.569698  88.350871     4.066\n",
       "4  id1_id5  22.541167  88.326124  22.498853  88.372714     6.713"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:09:44.698125Z",
     "start_time": "2020-06-06T09:07:22.509400Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.2 s ± 1.56 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "df['distance'] = df.apply(lambda row: calculate_distance_haversine(row['lat_a'],row['long_a'],row['lat_b'],row['long_b']),\n",
    "                          axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:10:40.045244Z",
     "start_time": "2020-06-06T09:10:40.035242Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lat_a</th>\n",
       "      <th>long_a</th>\n",
       "      <th>lat_b</th>\n",
       "      <th>long_b</th>\n",
       "      <th>distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>id1_id1</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>id1_id2</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.487651</td>\n",
       "      <td>88.350805</td>\n",
       "      <td>6.470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>id1_id3</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.486251</td>\n",
       "      <td>88.352905</td>\n",
       "      <td>6.700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>id1_id4</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.569698</td>\n",
       "      <td>88.350871</td>\n",
       "      <td>4.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>id1_id5</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.498853</td>\n",
       "      <td>88.372714</td>\n",
       "      <td>6.713</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id      lat_a     long_a      lat_b     long_b  distance\n",
       "0  id1_id1  22.541167  88.326124  22.541167  88.326124     0.000\n",
       "1  id1_id2  22.541167  88.326124  22.487651  88.350805     6.470\n",
       "2  id1_id3  22.541167  88.326124  22.486251  88.352905     6.700\n",
       "3  id1_id4  22.541167  88.326124  22.569698  88.350871     4.066\n",
       "4  id1_id5  22.541167  88.326124  22.498853  88.372714     6.713"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas series vectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:10:49.174233Z",
     "start_time": "2020-06-06T09:10:44.774236Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53.6 ms ± 3.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit \n",
    "\n",
    "df['distance'] = calculate_distance_haversine(df['lat_a'],df['long_a'],df['lat_b'],df['long_b'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:12:13.820963Z",
     "start_time": "2020-06-06T09:12:13.809963Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lat_a</th>\n",
       "      <th>long_a</th>\n",
       "      <th>lat_b</th>\n",
       "      <th>long_b</th>\n",
       "      <th>distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>id1_id1</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>id1_id2</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.487651</td>\n",
       "      <td>88.350805</td>\n",
       "      <td>6.470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>id1_id3</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.486251</td>\n",
       "      <td>88.352905</td>\n",
       "      <td>6.700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>id1_id4</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.569698</td>\n",
       "      <td>88.350871</td>\n",
       "      <td>4.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>id1_id5</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.498853</td>\n",
       "      <td>88.372714</td>\n",
       "      <td>6.713</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id      lat_a     long_a      lat_b     long_b  distance\n",
       "0  id1_id1  22.541167  88.326124  22.541167  88.326124     0.000\n",
       "1  id1_id2  22.541167  88.326124  22.487651  88.350805     6.470\n",
       "2  id1_id3  22.541167  88.326124  22.486251  88.352905     6.700\n",
       "3  id1_id4  22.541167  88.326124  22.569698  88.350871     4.066\n",
       "4  id1_id5  22.541167  88.326124  22.498853  88.372714     6.713"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NumPy arrays vectorization\n",
    "Please note that this requires your function to use numpy methods. Methods from other packages like math aren't supported"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:12:31.493058Z",
     "start_time": "2020-06-06T09:12:28.243845Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46 ms ± 1.14 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "df['distance'] = calculate_distance_haversine(df['lat_a'].values,\n",
    "                                              df['long_a'].values,\n",
    "                                              df['lat_b'].values,\n",
    "                                              df['long_b'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-06T09:13:05.463844Z",
     "start_time": "2020-06-06T09:13:05.453077Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lat_a</th>\n",
       "      <th>long_a</th>\n",
       "      <th>lat_b</th>\n",
       "      <th>long_b</th>\n",
       "      <th>distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>id1_id1</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>id1_id2</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.487651</td>\n",
       "      <td>88.350805</td>\n",
       "      <td>6.470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>id1_id3</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.486251</td>\n",
       "      <td>88.352905</td>\n",
       "      <td>6.700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>id1_id4</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.569698</td>\n",
       "      <td>88.350871</td>\n",
       "      <td>4.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>id1_id5</td>\n",
       "      <td>22.541167</td>\n",
       "      <td>88.326124</td>\n",
       "      <td>22.498853</td>\n",
       "      <td>88.372714</td>\n",
       "      <td>6.713</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id      lat_a     long_a      lat_b     long_b  distance\n",
       "0  id1_id1  22.541167  88.326124  22.541167  88.326124     0.000\n",
       "1  id1_id2  22.541167  88.326124  22.487651  88.350805     6.470\n",
       "2  id1_id3  22.541167  88.326124  22.486251  88.352905     6.700\n",
       "3  id1_id4  22.541167  88.326124  22.569698  88.350871     4.066\n",
       "4  id1_id5  22.541167  88.326124  22.498853  88.372714     6.713"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By avoiding looping over rows in a pandas dataframes, you can speed up your code snippets by more than 100x.\n",
    "- Looping over indices: ~3 min\n",
    "- Looping using iterrows: ~40 s\n",
    "- apply: ~20 s\n",
    "- Pandas Vectorization: ~60 ms\n",
    "- Numpy vectorization: ~40 ms"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}