Mesa RL

[harshmahesheka]

Mesa provides a series of amazing multi-agent environments with various objectives. A natural use case of this is in Multi-Agent Reinforcement Learning. This year's GSoC project is on a similar idea - visit here. This thread is open to discuss how to go forward with this.

For getting started, I have implemented a few examples of MARL with Mesa using Stable Baselines and RLlib here. Other than the environments I think the visualization tools that mesa offers can really be helpful for RL tasks. Building on this learning, I propose wrappers around the existing Mesa Framework, allowing it to work directly with popular MARL libraries like RLlib. This approach avoids modifying existing Mesa implementations and makes it easier to develop and maintain. This concept aligns with my experience from the previous GSoC project, where I worked on similar functionalities. We could even include various trained model examples to give users a good starting point. Also, I've been exploring Petting Zoo, where they convert environments to RLlib-compatible formats using function calls. We can do a similar implementation for our case.

I'd love to hear everyone's feedback and thoughts on the project and idea!

[EwoutH]

Thanks for the write up. I agree with the wrapper approach. Most important for me would be a clean user-facing API to use it, together with solid documentation.

[harshmahesheka]

I read through Petting Zoo's documentation. Petting Zoo offers two APIs, Parallel and AEC, which provide different approaches for MARL tasks. Additionally, it provides a plethora of functions to interact with its extensive collection of environments, which are compatible with all major MARL libraries. Leveraging Petting Zoo's functions and open-source base can greatly benefit our project. There is a series of third part environments based on the petting zoo mentioned here, and we can do something similar.

Here's a possible plan:

Integration with Petting Zoo: We'll create a generalized wrapper to convert our environments into Petting Zoo environments. Then we use this wrapper and write code on top of it to create specific environment files with proper reward function action space and observation space for the environment as examples to be used directly.

Enhanced Visualization with Mesa Tools: We can improve our training and inference processes by integrating Mesa visualisation tools. These tools will offer insights and aid in analyzing agent behaviour during training and evaluation. We also leverage Petting Zoo's functions to enhance our environments with additional features.

Training and Documentation: After converting and enhancing our environments, we'll train them using various algorithms supported by Petting Zoo. We'll also provide thorough documentation to ensure easy adoption by other users, making our trained policies accessible to a wider audience.

By following this approach, we'll be able to make the most of Petting Zoo's API and its open-source ecosystem while ensuring compatibility with various MARL libraries instead of just one. This will allow us to maximize the effectiveness of our environments. This is just a idea and your everyone's feedback on it will be great.

[EwoutH]

This AI-summary sounds nice on a high level. I would need more specifics to judge it further. I think minimal usage examples/demos are now the next steps.

[harshmahesheka]

Sure, I will work on training some more examples. This was to get started on a possible idea for a proposal. Do you have suggestions for any specific environment I should work next with?

[EwoutH]

Nothing specific, I would go for something small that shows the functionality well.

[rht]

What are the alternatives to Petting Zoo, and why did you choose Petting Zoo over them? For context, we did an exhaustive compare/contrast for choosing the next gen framework used for Mesa frontend (Streamlit, Voila, panel, ipywidgets, Solara, gradio.app, ... #1622), and for choosing the way to speed up Mesa (Numba, Cython, Rust, ... #1610)

[harshmahesheka]

There are a few other options, such as Open Speil, but Petting Zoo is the most standard and is widely used by the community. It has an easy-to-integrate API and the largest set of compatible libraries. Moreover, I have not decided which one to use, so I will make a detailed comparison soon.
I discussed above whether it would make sense to use libraries like RLlib with Mesa directly or convert Mesa environments to MARL environments like PettingZoo. I have already done 2 examples of the first case; I am working on an example of 2nd method. Once we decide approach, I can do a detailed analysis of which framework to choose.

[harshmahesheka]

I have added an example implementation of converting a mesa environment into a petting zoo environment and using RLlib with it. What do you think about which of the two approaches makes more sense? Directly using Mesa with RL frameworks or converting it to environments like Petting Zoo.

[rht]

The Petting Zoo environment looks simpler and more modular to me, compared to the RLlib one. But this is also because you have moved the helper functions in utils_function.py. What is the equivalent for the RLlib version's get_observation in the Petting Zoo version?

[harshmahesheka]

I think both implementations have the same get_observation function.

[EwoutH]

Thanks a lot for this, it's really exciting! I left some notes at harshmahesheka/mesa_rl#1.

@quaquel I can highly recommend you to check this out.