Event tracking and analysis

[EwoutH]

In ABM, we're often looking for rare events, emergent behavior and unexpected interactions. If we find these, we want to explain them. However, Mesa is currently lacking built-in event tracking and analysis functionality. I would like to open a discussion about adding that.

Defining events

First, you will want to define an event. It might be a state change, some condition being fulfilled, some spatial or temporal relation. They can be combined into complicated criteria (for example: two agents with both some state being at the some place within some timeframe under some environmental condition). We need to properly define this and the cases.

Defining collected data

If such an event happens, you want to collect some data. Maybe agent states, model states, the function that triggered the event, or spatial/temporal stuff. We also need to think about how to define that.

Tracking and logging events

Then, we need to detect in runtime when these events are happening, and log them somewhere in some data structure.

Analysis

Finally, after a model run you want to see what triggers such an event. So you want to analyze / aggregate the data that you collect it, and maybe compare it to the general population or environment. What states, function or temporal/spatial factors made the event happen?

Curious what everybody thinks about (rare) event tracking and analysis! (@quaquel probably has some ideas)

[quaquel]

In devs, events are well defined. Any state change is an event. So, in case of MESA, an event would be a 4-tuple (time_instant, object, name_of_state, new_value) Also, the formal understanding of an event implies that conditions and relations are not events but operate on the system state at a given time instant t_i. If you want to do some form of event mining, for example to find outliers, you need to store the full sequence of events (i.e., state transitions).

Being able to store the full sequence of events has other benefits as well. First, it is how many discrete event simulation libraries decouple model progression from model animation. In fact, given the full sequence of events, you can replay an animation without having to rerun the underlying model. Second, capturing events is how various simulation packages handle data collection. Data collectors subscribe to specific subsets of event changes and perform operations on those to produce outcomes of interest.

So, if you want to support the analysis of events, you first need a way to gather them and store them. One way that I have seen used in other libraries is to declare specific objects to be Observerable. Next, you add Observers that monitor one or more types of Observables. This monitoring can be done in a pull oriented way where every so often each observer queries their Observables for the current system state. This is effectively how MESA currently also works with the DataCollector.

However, in discrete event simulation, there is no natural or logical tilmestep for querying Observables. So, instead one can use a push oriented design. Here, Observables inform their observers whenever they undergo a state change. This can be implemented in various ways.

For example, in one Java library I have used extensively in the past, this was done using a publish and subscribe architecture. Observables published messages of their state changes. Observers subscribed to these messages. The nice thing of this architecture is that it enables fully asynchronous communication. I am not sure how performed, however, such a design would be in Python because you cannot rely on threading for the asynchronous processing of messages (unless you fully write this in C / Cython while releasing the gill).

It is, however, quite easy to implement a push oriented design using python Descriptors. Below is a rough sketch of what this would look like. You have to declare descriptors at the class level (state = Observable()). Next, whenever you assign to the state(self.state = 1), the __set__ method is called on the Observable. This set method not only sets a value on the underlying instance, but can also notify the Observer of the fact that the state is being assigned a new value.

class Observable:

    def __get__(self, instance, owner):
        return getattr(instance, self.private_name)

    def __set__(self, instance, value):
        instance._observers.notify(instance, self.public_name, value)
        setattr(instance, self.private_name, value)
        
    def __set_name__(self, owner, name):
        self.public_name = name
        self.private_name = f'_{name}'
        

class MyAgent(Agent):
    state = Observable()
    
    def __init__(self, unique_id, model)
        super().__init__(unique_id, model)
        self.state = 1

[EwoutH]

Thanks a lot for writing this out.

Already suspected this would be highly compatible with your idea for a push based datacollector, and that seems to be the case.

Some stuff to digest. We can discuss this further later this afternoon.

[quaquel]

An important conceptual question is whether you only want to know the state of the model at each tick, or whether you want to capture the full sequence of state transitions. The current datacollectod in MESA does the former. For proper event mining, however, I would argue you need the latter.

What you want for model visualization, I don't know. In principle, showing the full state for each tick is fine, but you might miss relevant and subtle within tick dynamics. However, if you show within tick state changes, you now have to carefully manage screen time. Either, each tick get the same on screen time, or each state transition get the same screen time. In the former case, you need to divide the per tick screen time by the number of state changes to get the per state change screen time. In the later, you are warping tick time were ticks with many state changes take longer to show then ticks were little happens.

[EwoutH]

@Corvince from your mesa-interactive experience, you also worked with some kind of state tracking / caching right? What do you think of this?

[wang-boyu]

Would this be useful: https://eventsourcing.readthedocs.io/en/stable/topics/introduction.html

A similar idea that is long overdue: #574

Making models serializable would be greatly useful. But I'm wondering whether it's better to have event based modeling as a separate extension, so that users don't need to install or learn the concepts of events if they don't need them.

[quaquel]

Thanks @wang-boyu , that is quite useful. The linked library is roughly what I had in mind and is inspired by the same underlying ideas. However, they rely on annotations rather than descriptors. This makes sense if you want to track individual method and function calls. However, in the case of an ABM, I would argue that state is captured in the agent attributes. Hence, I suggested descriptors instead. Second, this library is pure python which might cause performance bottle necks.

The idea in #574 is exactly in line with my thinking. In fact, it draws a need distinction between the models overall state and the collection of data. I had not yet made that distinction. My suggestion, however, would be to force users to be explicit about which subset of the full model state they want to capture. With descriptors, you can mix normal attributes that are not observed as part of the system state with Obverservabe attributes which are part of the system state.

There is also no need to make the system state at a given instance of time persistent. All you need is the sequence of state change events. Contra @Corvince, I would stay away from pickle. It is not a reliable cross platform cross machine storage format in my experience. Some very simple database solution might be all that is needed (if my 4 place tuple hunch is correct).

If MESA would have a queryable SystemState object, and a simple way to update this system state automatically (e.g., through descriptors). you would have something imminently useful for data collection, animation, model run persistence, and event mining. Apart from the Observerable descriptor, this entire machinery can be hidden from the new user. While advanced users can use it to great effect.

Ideally, you want to be able to easily query the system state. For example, you might want to know the wealth of all agents, or you might instead want to have for each agent all their observable attributes (Altair tooltips in animations?).

I have some rough code that I have been playing with over Christmas for data collection that I can quite easily rewrite to provide a proof of principle for this.