Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use wave scenario in performance tests #1574

Closed
xsebek opened this issue Oct 7, 2023 · 1 comment · Fixed by #1576
Closed

Use wave scenario in performance tests #1574

xsebek opened this issue Oct 7, 2023 · 1 comment · Fixed by #1576
Labels
C-Low Hanging Fruit Ideal issue for new contributors. S-Nice to have The bug fix or feature would be nice but doesn't currently have much negative impact. T-Challenges Involves the challenge scenarios - shorter games with objectives. Z-Developer Experience This issue seeks to make life easier for developers writing Scenarios or other Swarm code. Z-Performance This issue concerns memory or time efficiency of the Swarm game.

Comments

@xsebek
Copy link
Member

xsebek commented Oct 7, 2023

Is your feature request related to a problem? Please describe.

Our performance tests are out of date, we should measure realistic play scenarios.

Describe the solution you'd like

Let's use the Wave scenario (#1556) that has shown useful for measuring.

To truly stress the system and see how the number of running robots affects performance,
we should generate longer versions of the map. (e.g. 100, 300, 1000 robots).

Reusing the scenario description and just making longer maps should prove simpler
and more useful then writing the benchmarks by hand.

The advantage of using a scenario is that we could actually play it and see the difference.

Describe alternatives you've considered

We could write a new benchmark purely in Haskell, but that seems hard. (Terms can use TemplateHaskell though.)

We could also reuse the scenario except for the robots and place those dynamically like in other benchmarks.

Additional context

Ideally we would measure the overhead of drawing the map (terrain, entities and moving robots) as the base robot moves.

This could be achieved by serialising the visible area to Text in each tick, which could make it easier to inspect the benchmark.

@xsebek xsebek added C-Low Hanging Fruit Ideal issue for new contributors. S-Nice to have The bug fix or feature would be nice but doesn't currently have much negative impact. Z-Performance This issue concerns memory or time efficiency of the Swarm game. T-Challenges Involves the challenge scenarios - shorter games with objectives. Z-Developer Experience This issue seeks to make life easier for developers writing Scenarios or other Swarm code. labels Oct 7, 2023
@xsebek
Copy link
Member Author

xsebek commented Oct 8, 2023

After reading the benchmarking code, I realised that the alternative is easier and makes it clearer what is tested.

Benchmarking map drawing and scenario condition evaluation should be separate benchmarks.

@mergify mergify bot closed this as completed in #1576 Oct 9, 2023
mergify bot pushed a commit that referenced this issue Oct 9, 2023
* add wave program and parametrise it to compare inlined/generic version
* use [`tasty-bench`](https://hackage.haskell.org/package/tasty-bench) library to show comparison
* move benchmarks to test folder as they can now share tasty code
* closes #1574 

Using the recursive definition with ifs leads to a 3x slowdown:
```
wavesInlined
  10: OK
    361  ms ±  29 ms
  20: OK
    718  ms ±  35 ms
  30: OK
    1.066 s ±  28 ms
  40: OK
    1.437 s ±  37 ms
wavesWithDef
  10: OK
    1.052 s ±  51 ms, 2.92x
  20: OK
    2.117 s ±  34 ms, 2.95x
  30: OK
    3.144 s ±  80 ms, 2.95x
  40: OK
    4.191 s ±  91 ms, 2.92x
```
But if we just inline and simplify the code, we can remove the runtime overhead completely.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-Low Hanging Fruit Ideal issue for new contributors. S-Nice to have The bug fix or feature would be nice but doesn't currently have much negative impact. T-Challenges Involves the challenge scenarios - shorter games with objectives. Z-Developer Experience This issue seeks to make life easier for developers writing Scenarios or other Swarm code. Z-Performance This issue concerns memory or time efficiency of the Swarm game.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant