r/lecture1-eventlogs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "This notebook is part of a course on Applied Process Mining. The collection of notebooks is a *living document* and subject to change. \n",
    "\n",
    "# Lecture 1 - 'Event Logs and Process Visualization' (R / bupaR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "<img src=\"http://bupar.net/images/logo_text.PNG\" alt=\"bupaR\" style=\"width: 200px;\"/>\n",
    "\n",
    "In this notebook, we are going to need the `tidyverse` and the `bupaR` packages. If you run this notebook in the recommended Docker environment then there is no need to install any packages. Otherwise, you may need to install the requirements that are commented out below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Perform the commented out commands below in a separate R session\n",
    "# install.packages(\"tidyverse\")\n",
    "# install.packages(\"bupaR\")\n",
    "# install.packages(\"processmapR\")\n",
    "# install.packages(\"processanimateR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are setting up some convenicence options for the notebook and import dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for larger and readable plots\n",
    "options(jupyter.plot_scale=1.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the initial execution of these may give you warnings that we can safely ignore\n",
    "suppressPackageStartupMessages (library(tidyverse)) \n",
    "suppressPackageStartupMessages (library(bupaR))\n",
    "library(processmapR)\n",
    "library(processanimateR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Logs\n",
    "\n",
    "This part introduces event logs and their unique properties that provide the basis for any Process Mining method. Together with `bupaR` several event logs are distributed that can be loaded without further processing. \n",
    "In this lecture we are going to make use of the following datasets:\n",
    "\n",
    "* Patients, a synthetically generated example event log in a hospital setting.\n",
    "* Sepsis, a real-life event log taken from a Dutch hospital. The event log is publicly available here: https://doi.org/10.4121/uuid:915d2bfb-7e84-49ad-a286-dc35f063a460 and has been used in many Process Mining related publications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring Event Data\n",
    "\n",
    "Let us first explore the event data without any prior knowledge about event log structure or properties. We convert the `patients` event log below to a standard `tibble` (https://tibble.tidyverse.org/) and inspect the first rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%\n",
    "    as_tibble() %>%\n",
    "    head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most important ingredient of an event log is the timestamps column `time`. This allows us to establish a sequence of events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "  filter(time < '2017-01-31') %>% \n",
    "  ggplot(aes(time, \"Event\")) + \n",
    "  geom_point() + \n",
    "  theme_bw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to have information on the kind of actions or `activities` performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%\n",
    "    as_tibble() %>% \n",
    "    distinct(handling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at what other data is available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%\n",
    "    as_tibble() %>% \n",
    "    distinct(patient)  %>% \n",
    "    head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe the patient identifier could be a good candidate for defining a process `case` since this is an 'entity' that we would like to follow. When counting the events that occurred per individual patient it seems that there is a similar number of events for each patient, which is generally a good indicator for a process case identifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%\n",
    "    as_tibble() %>% \n",
    "    count(patient) %>% \n",
    "    head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let use decide that we want to look at the process by following the patient identifier as `case identifier`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "  filter(time < '2017-01-31') %>% \n",
    "  ggplot(aes(time, patient, color = handling)) + \n",
    "  geom_point() + \n",
    "  theme_bw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scatterplot above is known as `Dotted Chart` in the process mining community and provides an 'at a glance' overview on the events and their temporal relation when grouped by a case. It seems that each of the sequences of events (also known as `traces`) start with the `Registration` event. Let us have a look at the event data sorted by patient identifier and by time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "  as_tibble() %>% \n",
    "  arrange(patient, time) %>% \n",
    "  head(14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An individual process execution (e.g., for patient 1) consists of several activities that are done in a sequence. However, we have more information available than simply the sequence of events. For each occurrence of an activity we have two events: a `start` event and a `complete` event as captured in the column `registration_type`. These event refer to the lifecycle of an activity and allow us to capture the `duration` of an activity. Much more complex lifecycles of activities are possible, a general model is described here: http://bupar.net/creating_eventlogs.html#Transactional_life_cycle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [XES Standard](http://xes-standard.org/)\n",
    "* [Creating event logs from CSV files in bupaR](http://bupar.net/creating_eventlogs.html)\n",
    "* [Changing the case, activity notiions in bupaR](http://bupar.net/mapping.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reflection Questions\n",
    "\n",
    "* What could be the reason a column `.order` is included in this dataset?\n",
    "* How could the column `employee` be used?\n",
    "* What is the use of the column `handling_id` and in which situation is it required?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Process Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several generic visualizations that can be used to get a basic understanding of the process behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set of Traces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since a process, in our basic definition, is a set of event sequences or traces, we can simply visualize the set of distinct trace variants. Here we only consider the `trace variant` which means that we only consider the order of activities executed disregarding any other aspect (timing, lifecycles)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "  trace_explorer(coverage = 1.0, abbreviate = T, type = ) # abbreviated here due to poor Jupyter notebook output scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dotted Chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Dotted Chart` adds the timing aspect of the individual traces and visualized all of them at-a-glance. It can be configured in many different ways and provides a good insight into time-related aspects of the process behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%\n",
    "    filter(time < '2017-01-31') %>% \n",
    "    dotted_chart(add_end_events = T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%    \n",
    "    dotted_chart(\"relative\", add_end_events = T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use `plotly` to get an interactive visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>%    \n",
    "    dotted_chart(\"relative\", add_end_events = T, plotly = TRUE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis %>% \n",
    "    dotted_chart(\"relative_day\",\n",
    "                 sort = \"start_day\", \n",
    "                 units = \"hours\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out other process visualization options using bupaR:\n",
    "\n",
    "* [Further Dotted Charts](http://bupar.net/dotted_chart.html)\n",
    "* [Exploring Time, Resources, Structuredness](http://bupar.net/exploring.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process Map Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "    process_matrix() %>% \n",
    "    plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "    process_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients %>% \n",
    "    process_map(type = performance(units = \"hours\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Real-life Processes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis %>% \n",
    "  precedence_matrix() %>% \n",
    "  plot()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.2.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "28aff1567d8aae5536826c1be921f2ff2e204808293d43dc67bdcb73bd29110e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}