AgentSet: Refactor shuffle (overhaul)

[EwoutH]

More radical approach then #2283 to approaching shuffle performance.

Small benchmark:

from mesa import Agent, Model

class MyAgent(Agent):
    def __init__(self, model):
        super().__init__(model)

    def step(self):
        pass

class MyModel1(Model):
    def __init__(self):
        super().__init__()
        # Create 10000
        for _ in range(1000000):
            MyAgent(self)

    def step(self):
        self.agents.shuffle().do("step")


class MyModel2(Model):
    def __init__(self):
        super().__init__()
        # Create 10000
        for _ in range(1000000):
            MyAgent(self)

    def step(self):
        self.agents.shuffle(inplace=True).do("step")

# Benchmark the two models
from time import time

models = [MyModel1(), MyModel2()]
times = []
print("Running models...")
for model in models:
    print(f"Running {model}")
    start = time()
    for _ in range(100):
        model.step()
    times.append(time() - start)

print(f"Results: {times}")

Configuration	shuffle().do()	shuffle(inplace=True).do()	shuffle_do()	new shuffle().do()	new shuffle(inplace=True).do()
10,000 agents, 1,000 steps	8.27 s	3.65 s	3.06 s	3.26 s	2.90 s
100,000 agents, 100 steps	13.71 s	6.31 s	3.71 s	4.89 s	3.94 s
1,000,000 agents, 10 steps	18.36 s	9.44 s	5.75 s	4.93 s	3.92 s

This does include overhauling the AgentSet internal structure, but it's a lot faster, and now works with all functions chained behind shuffle()(so not do().

Adding agents should be quite fast, but removing might be slower.

[Corvince]

Adding agents should be quite fast, but removing might be slower.

I am really worried about the removal performance. This used to be a huge bottleneck in the (very old) space module. I think we definitely need to test this. For some models removal is quite common

[quaquel]

If you want to improve performance on shuffle, why not keep the dict, actively maintain a list with weakreffed agents, set a flag if agent.remove is called, and rebuild the list on next shuffle? worst case performance would be equal to current but in models with many shuffles but no removals you get the fast performance of shuffle?

Only thing I would have to think through and check the code for is how this would work with weakrefs, but I think it should be fine.

[EwoutH]

Closing in favor of #2283.