mesa 3, rl wolf-sheep

- wolfsheep updates to 3

.gitignore

@@ -23,6 +23,9 @@ var/
 .installed.cfg
 *.egg
 
+# ignore RL file - users download model on own
+rl
+
 # PyInstaller
 #  Usually these files are written by a python script from a template
 #  before PyInstaller builds the exe, so as to inject date/other infos into it.
@@ -58,6 +61,7 @@ target/
 
 # Jupyter and iPython notebook checkpoints
 *.ipynb_checkpoints
+*.virtual_documents
 
 # Spyder app workspace config file
 .spyderworkspace

rl/README.md

@@ -14,25 +14,28 @@ This repository demonstrates various applications of reinforcement learning (RL)
 1. **Install Mesa Models**
    Begin by installing the Mesa models:
 
+#TODO: Update this -- do release?
+
    ```bash
    pip install -U -e git+https://github.com/projectmesa/[email protected]#egg=mesa-models
    ```
 
-2. **Install RLlib for Multi-Agent Training**
+3. **Install RLlib for Multi-Agent Training**
    Next, install RLlib along with TensorFlow and PyTorch to support multi-agent training algorithms:
 
    ```bash
    pip install "ray[rllib]" tensorflow torch
    ```
+#TODO Update requirements to mesa[rec] >3.0
 
-3. **Install Additional Dependencies**
+4. **Install Additional Dependencies**
    Finally, install any remaining dependencies:
 
    ```bash
    pip install -r requirements.txt
    ```
 
-4. **Download Pre-Trained Weights**
+5. **Download Pre-Trained Weights**
    Download pre-trained weights from hugging face:
 
    ```bash

rl/Tutorials.ipynb

@@ -6,24 +6,50 @@
    "source": [
     "# Tutorial: Reinforcement Learning with Mesa Environments\n",
     "\n",
-    "# Welcome to this comprehensive guide on integrating reinforcement learning (RL) with Mesa environments. \n",
-    "# Mesa, an agent-based modeling framework, offers an excellent platform to experiment with RL algorithms. \n",
-    "# In this tutorial, we'll explore several examples of how RL can be applied to various Mesa environments, \n",
-    "# starting with the **Epstein Civil Violence model**.\n",
-    "\n",
-    "# ## Getting Started\n",
+    "Welcome to this comprehensive guide on integrating reinforcement learning (RL) with Mesa environments. \n",
+    "Mesa, an agent-based modeling framework, offers an excellent platform to experiment with RL algorithms. \n",
+    "In this tutorial, we'll explore several examples of how RL can be applied to various Mesa environments, \n",
+    "starting with the **Epstein Civil Violence model**."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Getting Started\n",
     "\n",
-    "# Before diving into the implementation, take a moment to familiarize yourself with the Epstein Civil Violence model.\n",
-    "# This will give you a solid understanding of the environment we’ll be working with.\n",
+    "Before diving into the implementation, take a moment to familiarize yourself with the Epstein Civil Violence model.\n",
+    "This will give you a solid understanding of the environment we’ll be working with.\n",
     "\n",
-    "# Next, ensure all dependencies are installed by following the instructions in the `README.md`.\n"
+    "Next, ensure all dependencies are installed by following the instructions in the `README.md`."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "WARNING:tensorflow:From C:\\Users\\thoma\\miniconda3\\envs\\mesa_dev\\Lib\\site-packages\\ray\\rllib\\utils\\framework.py:130: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
+      "\n"
+     ]
+    },
+    {
+     "ename": "ModuleNotFoundError",
+     "evalue": "No module named 'mesa_models.epstein_civil_violence'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[1], line 5\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# ### Step 1: Importing the Necessary Modules\u001b[39;00m\n\u001b[0;32m      2\u001b[0m \u001b[38;5;66;03m# To begin, let’s import the required modules for the Epstein Civil Violence model:\u001b[39;00m\n\u001b[0;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mepstein_civil_violence\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m EpsteinCivilViolenceRL\n\u001b[1;32m----> 5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mepstein_civil_violence\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mserver\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m run_model\n\u001b[0;32m      6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mepstein_civil_violence\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtrain_config\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m config\n\u001b[0;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtrain\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m train_model\n",
+      "File \u001b[1;32m~\\Documents\\GitHub\\dev\\mesa-examples\\rl\\epstein_civil_violence\\server.py:4\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[0;32m      3\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mray\u001b[39;00m\n\u001b[1;32m----> 4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mmesa_models\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mepstein_civil_violence\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mportrayal\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m citizen_cop_portrayal\n\u001b[0;32m      5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mray\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m tune\n\u001b[0;32m      6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mray\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mrllib\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01malgorithms\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01malgorithm\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Algorithm\n",
+      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'mesa_models.epstein_civil_violence'"
+     ]
+    }
+   ],
    "source": [
     "# ### Step 1: Importing the Necessary Modules\n",
     "# To begin, let’s import the required modules for the Epstein Civil Violence model:\n",
@@ -122,37 +148,42 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# ### Alternative Approach: Using Stable-Baselines with Mesa\n",
+    "# Alternative Approach: Using Stable-Baselines with Mesa\n",
     "\n",
-    "# In the example above, we utilized RLlib to integrate reinforcement learning algorithms with the Mesa environment, \n",
-    "# which is particularly useful when you want different policies for different agents. \n",
-    "# However, if your use case requires a simpler setup where all agents follow the same policy, \n",
-    "# you can opt for Stable-Baselines. An example of integrating Stable-Baselines with Mesa can be found in the Boltzmann Money model.\n",
+    "In the example above, we utilized RLlib to integrate reinforcement learning algorithms with the Mesa environment, which is particularly useful when you want different policies for different agents. \n",
+    "However, if your use case requires a simpler setup where all agents follow the same policy, you can opt for Stable-Baselines. An example of integrating Stable-Baselines with Mesa can be found in the Boltzmann Money model.\n",
     "\n",
-    "# You can explore more on how to use Stable-Baselines with Mesa by following the respective documentation.\n"
+    "You can explore more on how to use Stable-Baselines with Mesa by following the respective documentation.\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# ### Implementing Your Own Cases\n",
+    "# Implementing Your Own RL Models\n",
     "\n",
-    "# If you're ready to explore RL in different agent-based scenarios, you can start by experimenting with various examples we provide at Mesa-Examples:\n",
-    "# Link: https://github.com/projectmesa/mesa-examples\n",
+    "If you're ready to explore RL in different agent-based scenarios, you can start by experimenting with various examples we provide at Mesa-Examples:\n",
+    "Link: https://github.com/projectmesa/mesa-examples\n",
     "\n",
-    "# These examples cover a range of scenarios and offer a great starting point for understanding how to apply RL within Mesa environments.\n",
+    "These examples cover a range of scenarios and offer a great starting point for understanding how to apply RL within Mesa environments.\n",
     "\n",
-    "# If you have your own scenario in mind, you can create it as a Mesa model by following this series of Tutorials:\n",
-    "# Link: https://mesa.readthedocs.io/en/stable/tutorials/intro_tutorial.html\n",
+    "If you have your own scenario in mind, you can create it as a Mesa model by following this series of Tutorials:\n",
+    "Link: https://mesa.readthedocs.io/en/stable/tutorials/intro_tutorial.html\n",
     "\n",
-    "# Once your scenario is set up as a Mesa model, you can refer to the code in the provided implementations to see how the RL components are built on top of the respective Mesa models.\n"
+    "Once your scenario is set up as a Mesa model, you can refer to the code in the provided implementations to see how the RL components are built on top of the respective Mesa models.\n"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "test",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -166,9 +197,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.0"
+   "version": "3.12.5"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

rl/epstein_civil_violence/agent.py

@@ -1,4 +1,4 @@
-from mesa_models.epstein_civil_violence.agent import Citizen, Cop
+from mesa.examples.advanced.epstein_civil_violence.agents import Citizen, Cop
 
 from .utility import move
 

rl/epstein_civil_violence/model.py

@@ -1,7 +1,7 @@
 import gymnasium as gym
 import mesa
 import numpy as np
-from mesa_models.epstein_civil_violence.model import EpsteinCivilViolence
+from mesa.examples.advanced.epstein_civil_violence.model import EpsteinCivilViolence
 from ray.rllib.env import MultiAgentEnv
 
 from .agent import CitizenRL, CopRL

rl/wolf_sheep/agents.py

@@ -1,6 +1,5 @@
-from mesa_models.wolf_sheep.agents import GrassPatch, Sheep, Wolf
-
-from .utility import move
+from mesa.examples.advanced.wolf_sheep.agents import GrassPatch, Sheep, Wolf
+from utility import move
 
 
 class SheepRL(Sheep):
@@ -28,7 +27,7 @@ def step(self):
             # Death
             if self.energy < 0:
                 self.model.grid.remove_agent(self)
-                self.model.schedule.remove(self)
+                self.model.remove(self)
                 living = False
 
         if living and self.random.random() < self.model.sheep_reproduce:
@@ -38,7 +37,7 @@ def step(self):
             unique_id_str = f"sheep_{self.model.next_id()}"
             lamb = SheepRL(unique_id_str, self.pos, self.model, self.moore, self.energy)
             self.model.grid.place_agent(lamb, self.pos)
-            self.model.schedule.add(lamb)
+            self.model.add(lamb)
 
 
 class WolfRL(Wolf):
@@ -62,12 +61,12 @@ def step(self):
 
             # Kill the sheep
             self.model.grid.remove_agent(sheep_to_eat)
-            self.model.schedule.remove(sheep_to_eat)
+            self.model.remove(sheep_to_eat)
 
         # Death or reproduction
         if self.energy < 0:
             self.model.grid.remove_agent(self)
-            self.model.schedule.remove(self)
+            self.model.remove(self)
         else:
             if self.random.random() < self.model.wolf_reproduce:
                 # Create a new wolf cub
@@ -77,4 +76,4 @@ def step(self):
                     unique_id_str, self.pos, self.model, self.moore, self.energy
                 )
                 self.model.grid.place_agent(cub, cub.pos)
-                self.model.schedule.add(cub)
+                self.model.add(cub)

rl/wolf_sheep/app.py

@@ -0,0 +1,131 @@
+from mesa.examples.advanced.wolf_sheep.agents import GrassPatch, Sheep, Wolf
+from mesa.examples.advanced.wolf_sheep.model import WolfSheep
+from mesa.experimental.devs import ABMSimulator
+from mesa.visualization import (
+    Slider,
+    SolaraViz,
+    make_plot_component,
+    make_space_component,
+)
+from model import WolfSheepRL
+from ray import tune
+from ray.rllib.algorithms.algorithm import Algorithm
+
+model_params = {
+    "width": 20,
+    "height": 20,
+    "initial_sheep": 100,
+    "initial_wolves": 25,
+    "sheep_reproduce": 0.04,
+    "wolf_reproduce": 0.05,
+    "wolf_gain_from_food": 20,
+    "grass": True,
+    "grass_regrowth_time": 30,
+    "sheep_gain_from_food": 4,
+    "seed": 42,
+    "simulator": ABMSimulator(),
+    "vision": 4,
+    "model_path": None,
+}
+
+
+class WolfSheepServer(WolfSheepRL):
+    def __init__(self, **model_params):
+        super().__init__(**model_params)
+
+        def env_creator(_):
+            return WolfSheepRL(**model_params)
+
+        tune.register_env("WorldSheepModel-v0", env_creator)
+        self.iteration = 0
+        # Load the model from checkpoint
+        checkpoint_path = self.model_path
+        algo = Algorithm.from_checkpoint(checkpoint_path)
+        self.wolf_policy = algo.get_policy("policy_wolf")
+        self.sheep_policy = algo.get_policy("policy_sheep")
+
+
+def wolf_sheep_portrayal(agent):
+    if agent is None:
+        return
+
+    portrayal = {
+        "size": 25,
+    }
+
+    if isinstance(agent, Wolf):
+        portrayal["color"] = "tab:red"
+        portrayal["marker"] = "o"
+        portrayal["zorder"] = 2
+    elif isinstance(agent, Sheep):
+        portrayal["color"] = "tab:cyan"
+        portrayal["marker"] = "o"
+        portrayal["zorder"] = 2
+    elif isinstance(agent, GrassPatch):
+        if agent.fully_grown:
+            portrayal["color"] = "tab:green"
+        else:
+            portrayal["color"] = "tab:brown"
+        portrayal["marker"] = "s"
+        portrayal["size"] = 75
+
+    return portrayal
+
+
+model_params = {
+    "seed": {
+        "type": "InputText",
+        "value": 42,
+        "label": "Random Seed",
+    },
+    "grass": {
+        "type": "Select",
+        "value": True,
+        "values": [True, False],
+        "label": "grass regrowth enabled?",
+    },
+    "grass_regrowth_time": Slider("Grass Regrowth Time", 20, 1, 50),
+    "initial_sheep": Slider("Initial Sheep Population", 100, 10, 300),
+    "sheep_reproduce": Slider("Sheep Reproduction Rate", 0.04, 0.01, 1.0, 0.01),
+    "initial_wolves": Slider("Initial Wolf Population", 10, 5, 100),
+    "wolf_reproduce": Slider(
+        "Wolf Reproduction Rate",
+        0.05,
+        0.01,
+        1.0,
+        0.01,
+    ),
+    "wolf_gain_from_food": Slider("Wolf Gain From Food Rate", 20, 1, 50),
+    "sheep_gain_from_food": Slider("Sheep Gain From Food", 4, 1, 10),
+}
+
+
+def post_process_space(ax):
+    ax.set_aspect("equal")
+    ax.set_xticks([])
+    ax.set_yticks([])
+
+
+def post_process_lines(ax):
+    ax.legend(loc="center left", bbox_to_anchor=(1, 0.9))
+
+
+space_component = make_space_component(
+    wolf_sheep_portrayal, draw_grid=False, post_process=post_process_space
+)
+lineplot_component = make_plot_component(
+    {"Wolves": "tab:orange", "Sheep": "tab:cyan", "Grass": "tab:green"},
+    post_process=post_process_lines,
+)
+
+simulator = ABMSimulator()
+model = WolfSheep(simulator=simulator, grass=True)
+
+page = SolaraViz(
+    model,
+    components=[space_component, lineplot_component],
+    model_params=model_params,
+    name="Wolf Sheep",
+    simulator=simulator,
+)
+page  # noqa