Part_3_Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Part 3: Pandas\n",
    "\n",
    "[pandas](https://pandas.pydata.org/) is the primary data analysis library for Python, with DataFrames similar to those used in R.\n",
    "\n",
    "First import it and set the options to display all columns in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.options.display.max_columns = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Pandas basics\n",
    "### Series\n",
    "\n",
    "A series is basically an array with an index of labels. If an index isn't specified a default will be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "random.seed(2024)\n",
    "\n",
    "pd.Series(random.sample(range(1000), 6))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "A series _can_ contain multiple types but it's really best if it doesn't.\n",
    "\n",
    "Using dates for indices can be useful for time series calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "n = 100\n",
    "ts = pd.Series(\n",
    "    random.sample(range(1000), n), \n",
    "    index = pd.date_range(\"2024-01-01\", periods=n, freq=\"d\"), \n",
    ")\n",
    "ts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    "We can use slice notation on the index labels to get the data for the first week in February."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ts.loc[\"2024-02-01\":\"2024-02-07\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "Or we can use integer based positioning. Notice that the position based slicing does not include the final value but the label based slice does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ts.iloc[0:7]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10",
   "metadata": {},
   "source": [
    "We can use a boolean mask with `.loc`. For example, to find the days when more than 900 things happened"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ts.loc[ts > 900]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "We can assign new values using a `.loc` slice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ts.loc[ts > 900] = 10\n",
    "ts.loc[\"2024-02-01\":\"2024-02-07\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {},
   "source": [
    "We can resample to get the monthly sum of whatever it is we're counting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ts.resample(\"ME\").sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16",
   "metadata": {
    "tags": []
   },
   "source": [
    "We can use other aggregate functions too.\n",
    "\n",
    "#### Task \n",
    "\n",
    "Find the weekly mean. **Hint** Check the frequency strings [in this section](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects).\n",
    "\n",
    "Find the weeks it is above 600.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "  \n",
    "<pre><code>\n",
    "m = ts.resample(\"W\").mean()\n",
    "m[m > 600]\n",
    " </code></pre>\n",
    "</details>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {
    "tags": []
   },
   "source": [
    "We can use the normal mathematical operators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ts * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Dataframes\n",
    "\n",
    "A dataframe is, simplifying, a load of Series bundled together with both a row and column index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"A\": random.sample(range(1000), n),\n",
    "        \"B\": random.sample(range(1000), n),\n",
    "        \"C\": random.sample(range(1000), n),\n",
    "    },\n",
    "    index = pd.date_range(\"2024-01-01\", periods=n, freq=\"d\")\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {},
   "source": [
    "`.iloc` and `.loc` work in the same way but with an extra dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.iloc[:10, 1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.loc[\"2024-02-01\":\"2024-02-07\", [\"A\", \"C\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25",
   "metadata": {},
   "source": [
    "It's good practice to use `.loc` and `.iloc` notation as much as possible but it's often convenient to just use square brackets to select one or more columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df[\"A\"] = 1\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27",
   "metadata": {},
   "source": [
    "Define a second dataframe with an overlapping index to the first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame(\n",
    "    {\n",
    "        \"D\": random.sample(range(1000), n),\n",
    "        \"E\": random.sample(range(1000), n),\n",
    "        \"F\": random.sample(range(1000), n),\n",
    "    },\n",
    "    index = pd.date_range(\"2024-02-01\", periods=n, freq=\"d\")\n",
    ")\n",
    "df2.resample(\"ME\").sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29",
   "metadata": {},
   "source": [
    "As the indices overlap we can join on the index using `concat` which is far more efficient than using `merge`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.concat([df.resample(\"ME\").sum(), df2.resample(\"ME\").sum()], axis=1, join=\"inner\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31",
   "metadata": {},
   "source": [
    "Interestingly if we use an outer join the values are converted to floats. This is because the original values use `numpy`'s non-nullable int type and would need to be converted to the newer nullable `Int64` first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.concat([df.resample(\"ME\").sum(), df2.resample(\"ME\").sum()], axis=1, join=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33",
   "metadata": {},
   "source": [
    "Note that because we're using a common index we don't have to perform a join to make calculations on columns of two dataframes. For example, if we want to add the weekly totals for column `A` in `df` and column `D` in `df2` we can write the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df[\"A\"].resample(\"W\").sum() + df2[\"D\"].resample(\"W\").sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35",
   "metadata": {},
   "source": [
    "### Further reading\n",
    "\n",
    "Understanding the index and selection operators is key to getting the most out of pandas. The [Getting Started with pandas](https://wesmckinney.com/book/pandas-basics#pandas_summarize) chapter in Wes McKinney's [Python for Data Analysis](https://wesmckinney.com/book/) covers more than we have time for here, and at some point you should read the user guide on [Indexing and selecting data](https://pandas.pydata.org/docs/user_guide/indexing.html) which is very thorough.\n",
    "\n",
    "It should be noted that many people productively use pandas without touching the index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36",
   "metadata": {},
   "source": [
    "## Returning to the worked example\n",
    "### Data preparation\n",
    "\n",
    "We would normally use `pd.read_json` or `pd.read_csv` or `pd.read_parquet` etc to read the file but in this case it leaves us with embedded json columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "username = os.environ[\"AWS_ROLE_ARN\"].split(\"/\")[-1]\n",
    "filename = f\"s3://alpha-everyone/python_training/{username}/crime.json\"\n",
    "# If you're having access problems uncomment the following\n",
    "# filename = \"crime.json\"\n",
    "\n",
    "pd.read_json(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38",
   "metadata": {},
   "source": [
    "Instead use `smart_open` and `json` with `pd.json_normalize` to flatten the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import smart_open\n",
    "import json\n",
    "\n",
    "with smart_open.open(filename, \"r\") as f:\n",
    "    crimes = pd.json_normalize(json.loads(f.read()))\n",
    "    \n",
    "crimes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40",
   "metadata": {},
   "source": [
    "Note that the columns names have been augmented where the json has been unpacked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "crimes.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42",
   "metadata": {},
   "source": [
    "One way of accessing a dataframe's column is by putting its name in square brackets. The `unique` method can help us determine when a column has no data or would be better as a factor."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43",
   "metadata": {},
   "source": [
    "Drop the columns without data or we are not interested in.\n",
    "\n",
    "#### Task\n",
    "\n",
    "Drop any columns you feel are unnecessary for the analysis.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "  \n",
    "<pre><code>\n",
    "crimes = crimes.drop(\n",
    "    columns = [\n",
    "        \"context\", \"outcome_status\", \"persistent_id\", \"id\"\n",
    "    ]\n",
    ")\n",
    " </code></pre>\n",
    "</details>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44",
   "metadata": {},
   "outputs": [],
   "source": [
    "crimes = crimes.drop(\n",
    "    columns = [\n",
    "        # insert your columns here as a list\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "crimes.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46",
   "metadata": {
    "tags": []
   },
   "source": [
    "We can now start converting the columns to more appropriate types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "crimes[\"category\"] = crimes[\"category\"].astype(\"category\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48",
   "metadata": {},
   "source": [
    "It's easier to assign or reassign multiple columns using the `assign` method. While we're at it rename some of the unpacked columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "crimes = crimes.rename(\n",
    "    columns = {\n",
    "        \"location.latitude\": \"latitude\",\n",
    "        \"location.longitude\": \"longitude\",\n",
    "        \"location.street.name\": \"street_name\",\n",
    "        \"outcome_status.category\": \"outcome_category\",\n",
    "        \"outcome_status.date\": \"outcome_date\"\n",
    "    }\n",
    ")\n",
    "\n",
    "crimes = crimes.assign(\n",
    "    location_type = crimes[\"location_type\"].astype(\"category\"),\n",
    "    location_subtype = crimes[\"location_subtype\"].astype(\"category\"),\n",
    "    month = pd.to_datetime(crimes[\"month\"]),\n",
    "    latitude = crimes[\"latitude\"].astype(\"Float64\"),\n",
    "    longitude = crimes[\"longitude\"].astype(\"Float64\"),\n",
    "    street_name = crimes[\"street_name\"].astype(pd.StringDtype()),\n",
    "    outcome_category = crimes[\"outcome_category\"].astype(\"category\"),\n",
    "    outcome_date = pd.to_datetime(crimes[\"outcome_date\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50",
   "metadata": {},
   "source": [
    "### An aside on pydbtools\n",
    "\n",
    "Now we have edited our table and corrected the types we may want to create a database table on Athena. [pydbtools](https://github.com/moj-analytical-services/pydbtools) is a package developed by the data engineers to help use Athena databases. Keeping as much processing as possible as SQL queries on Athena will be faster and avoid memory problems in Jupyterlab. The [documentation](https://moj-analytical-services.github.io/pydbtools/) contains [examples](https://moj-analytical-services.github.io/pydbtools/examples/mojap_tools_demo/) of how to use the library.\n",
    "\n",
    "For now create a database if it doesn't exist and write the crimes dataframe to a table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pydbtools as pydb\n",
    "\n",
    "table_name = f\"crimes_{username}\"\n",
    "s3_location = f\"s3://alpha-everyone/python_training/db/{username}\"\n",
    "\n",
    "pydb.create_database(\"python_training\")\n",
    "pydb.dataframe_to_table(\n",
    "    df = crimes,\n",
    "    database = \"python_training\",\n",
    "    table = table_name,\n",
    "    location = s3_location\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52",
   "metadata": {},
   "source": [
    "This can then be queried as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pydb.read_sql_query(\n",
    "    f\"\"\"\n",
    "    select * from python_training.{table_name}\n",
    "    where category = 'violent-crime'\n",
    "    \"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54",
   "metadata": {},
   "source": [
    "Much of what follows would be better off in SQL but today we're learning pandas so \n",
    "\n",
    "### Back to the worked example\n",
    "\n",
    "Here we are recreating the analysis in Part 2 where we look at the most violent crimes per DLR station.\n",
    "\n",
    "Remember the `.loc` method is important as it defines the part of the dataframe you're working on. Here we're filtering the rows in the first field with a boolean mask, and selecting the columns in the second field. \n",
    "\n",
    "Note that the boolean operators in the mask are the Python bitwise operators, so `&` instead of `and`, `|` instead of `or`, and `~` instead of not. These have different operator precedence to the normal boolean operators so we need to keep the separate parts of the `&` expression in brackets.\n",
    "\n",
    "The brackets are escaped with a backslash as the `.str.contains` method expects a regular expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(\n",
    "    crimes\n",
    "    .loc[\n",
    "        (crimes[\"category\"] == \"violent-crime\") & (crimes[\"street_name\"].str.contains(\"\\(dlr\\)\")), \n",
    "        [\"street_name\", \"category\"]\n",
    "    ]\n",
    "    .groupby(\"street_name\")\n",
    "    .aggregate(\"count\")\n",
    "    .rename(columns = {\"category\": \"violent_crimes\"})\n",
    "    .sort_values(\"violent_crimes\", ascending=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56",
   "metadata": {},
   "source": [
    "Why is Heron Quays relatively rough? Are more people using the station? Let's have a look at station footfall from TFL's data, the latest of which I could find is from 2022."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with smart_open.open(\"https://crowding.data.tfl.gov.uk/Network%20Demand/StationFootfall_2022.csv\") as f:\n",
    "    footfall = pd.read_csv(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "footfall"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59",
   "metadata": {},
   "source": [
    "Add together the entries and exits so we can calculate "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "footfall[\"tap_count\"] = footfall[\"EntryTapCount\"] + footfall[\"ExitTapCount\"]\n",
    "footfall[\"Station\"] = footfall[\"Station\"].astype(pd.StringDtype())\n",
    "footfall_2022 = footfall.groupby(\"Station\")[\"tap_count\"].sum()\n",
    "footfall_2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61",
   "metadata": {},
   "source": [
    "Now looking at all crimes for 2022 only for any TFL station."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "crimes_2022 = (\n",
    "    crimes\n",
    "    .loc[crimes[\"street_name\"].str.contains(\"\\(lu|\\(dlr\") & (crimes[\"month\"].dt.year == 2022)]\n",
    "    .groupby(\"street_name\")[\"category\"]\n",
    "    .count()\n",
    ")\n",
    "crimes_2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63",
   "metadata": {},
   "source": [
    "We need to do a bit of processing to make sure the station names match TFL's, which only specify DLR when there is an alternative station with the same name, as with Canary Wharf."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64",
   "metadata": {},
   "outputs": [],
   "source": [
    "lookup = {\n",
    "    station: station.replace(\" (dlr)\", \"\")\n",
    "    for station in crimes_2022.index\n",
    "}\n",
    "lookup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "lookup[\"Canary Wharf (dlr)\"] = \"Canary Wharf DLR\"\n",
    "lookup[\"Canary Wharf (lu Station)\"] = \"Canary Wharf\"\n",
    "lookup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "crimes_2022.index = [lookup[i] for i in crimes_2022.index]\n",
    "crimes_2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67",
   "metadata": {
    "tags": []
   },
   "source": [
    "We can now calculate the ratio of crimes to station visits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68",
   "metadata": {},
   "outputs": [],
   "source": [
    "crimes_per_footfall = crimes_2022 / footfall_2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69",
   "metadata": {},
   "source": [
    "Convert these to percentages as thankfully they're small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pct_crimes_per_footfall = 100 * crimes_per_footfall.sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pct_crimes_per_footfall.loc[~pct_crimes_per_footfall.isna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72",
   "metadata": {},
   "source": [
    "This tells me that I should get the Jubilee line to Stratford rather than the DLR."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73",
   "metadata": {},
   "source": [
    "#### Final task\n",
    "\n",
    "Find out something interesting! Are people more lairy during office Christmas party season? What could a [map](https://plotly.com/python/maps/) of local crime look like?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74",
   "metadata": {},
   "source": [
    "### Further reading\n",
    "* [Python for Data Analysis](https://wesmckinney.com/book/) by Wes McKinney, the original author of pandas\n",
    "* The [Modern Pandas](https://tomaugspurger.net/posts/modern-1-intro/) series of blog posts\n",
    "\n",
    "#### Alternative dataframes\n",
    "pandas is not the only option for data processing.\n",
    "* [duckdb](https://duckdb.org/docs/api/python/overview.html) uses SQL and can be used in conjunction with pandas. Very useful for inequality joins and the like and [integrates nicely with Jupyter](https://duckdb.org/docs/guides/python/jupyter.html).\n",
    "* [dask](https://docs.dask.org/en/stable/) is designed for larger-than-memory datasets.\n",
    "* [polars](https://docs.pola.rs/) is a fast alternative to pandas. \n",
    "\n",
    "#### Recommended libraries\n",
    "##### MoJ libraries\n",
    "* [pydbtools](https://github.com/moj-analytical-services/pydbtools) for accessing Athena databases\n",
    "* [mojap-metadata](https://github.com/moj-analytical-services/mojap-metadata) for handling metadata\n",
    "* [arrow-pd-parser](https://github.com/moj-analytical-services/mojap-arrow-pd-parser) for reading and writing files while ensuring metadata conformance\n",
    "##### Others\n",
    "* [arrow](https://arrow.apache.org/docs/python/index.html) You may not need to use this directly but almost certainly will behind the scenes. It provides efficient data formats for cross-platform development, including `.parquet` files which are incredibly useful for preserving types and working across R and Python. Allows for chunkable datasets which don't have to be read into memory at once.\n",
    "* [awswrangler](https://aws-sdk-pandas.readthedocs.io/en/stable/index.html) is useful for dealing with AWS.\n",
    "* [numpy](https://numpy.org/) is compatible with pandas and has a huge amount of useful functionality for numerical computing.\n",
    "* [matplotlib](https://matplotlib.org/) is a powerful but complex data visualisation library.\n",
    "* [seaborn](https://seaborn.pydata.org/) is an easier to use data visualisation library built on matplotlib.\n",
    "* [plotly](https://plotly.com/python/) is an alternative data visualisation library, particularly useful for...\n",
    "* [dash](https://dash.plotly.com/) which can be used for webapps.\n",
    "* [streamlit](https://streamlit.io/) is a simpler webapp framework, useful for single page dashboards.\n",
    "* [scikit-learn](https://scikit-learn.org/stable/) The starting point for machine learning in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python training",
   "language": "python",
   "name": "intro-to-python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}