Move to a monorepo

[EwoutH]

We've discussed this often enough. Linking dependencies is hell. Examples not being in the main repo is hell. Mesa-geo being compatible with some and then with some others is also hell.

Let's go to a monorepo.

Everything you need to know: https://monorepo.tools/

Todo:

• Move mesa-geo
• Move mesa-examples

Retain git history at all costs.

[EwoutH]

@wang-boyu what would you think of making Mesa-geo just a separate module of Mesa?

[quaquel]

I agree in principle with this. However, we want to delineate what is in and out of scope carefully. For example, while going through the examples and updating them to the experimental grid spaces, I noticed a range of issues: massive code duplication, outdated and unpythonic code, and unusual models included in the examples. A similar argument can be made regarding the various other mesa libraries. So which of those would we consider part of the core of MESA and thus want to actively maintain and keep in sync with the core mesa library?

[EwoutH]

For Mesa-examples I would propose just keeping them, but pinning major versions. We guarantee that examples work with a certain Mesa series (2.x, 3.x), but we don't actively update them to new major versions.

A small set of versatile, high-quality examples (5 to 8) is included in the main repo, which is continually tested. We can even fix seeds and start testing the consistency of output.

[Corvince]

I have been thinking about the required tooling and uvs workspaces might be useful https://docs.astral.sh/uv/concepts/workspaces/

[EwoutH]

PEP 735 – Dependency Groups in pyproject.toml just got accepted. Could make managing dependencies in a monorepo easier.

PyPI tracking issue here: pypa/pip#12963

Edit: A draft PR to add support to uv workspaces was also already opened: astral-sh/uv#8104

[EwoutH]

A monorepo with examples would allow me to simply click "open in GitHub Desktop":

and start reviewing a PR. Now I first have to do a pip install, switch to mesa-examples. It's just really cumbersome and deters me from reviewing PRs as quickly as I did like.

[EwoutH]

Okay, here's my supposed approach:

1. Move examples

• Select 5-7 classic and high-quality examples, that really show of ABM and its features (emergence, path dependency, etc.) and Mesa and its features. I would say:
  • Boltzmann wealth (basic and classic)
  • Boid Flockers (clear, visual emergent behavior, continuous space)
  • Conway's Game of Life (classic, visual emergent behavior)
  • Wolf Sheep (multi-agent, more complicated group dynamic, path dependency, stability, PropertyLayer)
• And maybe
  • Sugarscape (another great emergence model, potential for PropertyLayer)
  • Epstein Civil Violence Model (multi-agent interactions)
  • something using the NetworkGrid / social networks
• Move those examples to the main Mesa repo. Test them in CI (including batch_run and viz), run the benchmarks on those, and packages them with Mesa.
  • Keep full Git history!
• All examples use stable features. We might have one or two additional examples in the experimental module that use experimental features.
• Keep some other examples, but pin the major versions. An example uses either version 2, 3 of 4, but generally we won't update them to new major versions.

2. Integrate Mesa-Geo

• Integrate Mesa-Geo in the monorepo. Either as:
  • A separate package, like it currently is
  • A separate module, like mesa.geo (CC @wang-boyu)

[EwoutH]

For anyone interested this will be the procedure to update the examples:

1. Clone the mesa-examples repository:

   git clone https://github.com/projectmesa/mesa-examples.git
   cd mesa-examples

2. Create a new branch for filtering:

   git checkout -b move-examples-to-mesa

3. Filter the desired directories:
   Since you want to move multiple directories, you can retain their histories by filtering all of them:

   git filter-branch --subdirectory-filter examples/boltzmann_wealth_model -- --all
   git filter-branch --subdirectory-filter examples/boid_flockers -- --all
   git filter-branch --subdirectory-filter examples/conways_game_of_life -- --all
   git filter-branch --subdirectory-filter examples/wolf_sheep -- --all

4. Add the mesa repository as a remote:

   git remote add mesa https://github.com/projectmesa/mesa.git

5. Fetch the latest changes from the mesa repository:

   git fetch mesa

6. Create a new branch in the mesa repository:
   Switch to a new branch on the mesa repository where you will add the examples.

   git checkout -b add-examples-to-mesa

7. Merge the filtered branch into the new mesa branch:
   Since the two repositories have different histories, you need to allow unrelated histories:

   git merge move-examples-to-mesa --allow-unrelated-histories

8. Push the new branch to the mesa repository:

   git push mesa add-examples-to-mesa

9. Open a Pull Request (PR):

   • Go to the mesa repository on GitHub.
   • You should see an option to create a new pull request from the branch add-examples-to-mesa.
   • Open a PR and describe the changes you're making (moving examples with full history, etc.).

[EwoutH]

For historical reference, a lot of the discussion on the example movement was had in #2349, and they were moved in #2358.

[tpike3]

So I am a little slow on the uptake here, but I am still trying to internalize what happens across the stack if we go monorepo. I found this a pretty detailed eplanation .. https://medium.com/opendoor-labs/our-python-monorepo-d34028f2b6fa; but OpenDoor GitHub has 202 repos... does anyone have an exemplar on GitHub they can point me too?