Are large grids a problem in mesa?

[CRG180]

Are large grids computationally burdensome? When building a 10000 x 10000 grid it takes about 1 min to build the grid. Is this normal? I suppose 10000^2 is a large number. My initial thought is that I would like to build the grid with 1 x 1-meter resolution. I'm I missing something? What are some best practices for sizing the grid?

`class CombatModel(mesa.Model):

def __init__(self, width = 10000, height = 10000):
    self.schedule = mesa.time.RandomActivation(self)
    self.grid =mesa.space.MultiGrid(height, width, False)
    self.step_num = 1 `

[rht]

You may use the recently merged new feature PropertyLayer, #1898, to scale operations on large grid. It's just that while the operations on the PropertyLayer may be fast, the operations on the 2D grid itself is still slow. If we could migrate the 2D grid implementation to be based on NumPy array, then we could scale to large size of the grid.

[rht]

Can't remember exactly. Maybe @Tortar still remembers the explanation.

[quaquel]

Just a hunch, but it might be that you lose some of the type-checking overhead by doing getitem in C instead of in python.

[Tortar]

Hi guys,

I don't remember exactly why, but you could try to inspect the annotate cython code to understand it I think

[Corvince]

Thanks for jumping in, but I don't find anything that convinces me a 18x speed up would be possible. There are no type annotations so I don't know how cython would do this. Makes me skeptical of the rest of the benchmark!

[Tortar]

I'm not sure you mean the same thing as me, by annotation I mean the ones generated when you compile the files with cython and you pass the necessary compiler flag. Also, I think the benchmark code is also on the repo so you could look at it and see if there are any flaws

[quaquel]

Regarding Cython. Is there a way to only cythonize single functions? From the performance benchmarks I think only get_cell_list_contents seems worth pursuing and it is also the most used function, so that would make sense in my opinion.

Yes it is quite easy to cythonize individual functions. In fact, that is how pandas slowly moved most of their code from python to cython in the days before pandas 1.0.

[rht]

I'm currently trying to integrate mesa-perf into projectmesa/mesa, via https://pypi.org/project/hatch-cython/0.0.1/.

[EwoutH]

An initial step would be creating a benchmarks.py script that you can run on your own computer. It will at least be able to see the large performance differences. Then everyone can run it themselves on the main branch and on their PR branch and compare the results.

[rht]

Regarding with the Cythonizing, I tried https://pypi.org/project/hatch-cython/0.0.1/#description and it seems to be buggy. The only way forward I can think of is to migrate from Hatch to either meson-python (used by pandas and NumPy), or Poetry. Which one do you guys prefer and why? I don't know of a major Cythonized project that uses Poetry. But I know several that still uses setuptools.

[quaquel]

I would follow pandas and numpy. It will give us good templates for actions etc. as well.

[Corvince]

@EwoutH Good heads-up on runner variability. But as long as the both runs happen on the same action it should be fine, right? I mean for example to compare without a PR and with a PR. The times won't be comparable between Action runs, but at least we would know how it affects performance. But I think it should be an optional Action, only activated when needed.

[EwoutH]

Main problem is modern hyperthreading and turboing. While great for optimizing overal performance, they are not good for consistency. On these kind of service, you won't be the only job running on that core.

If the core is busy on the first half and less busy on the second half, that can seriously affect you numbers.