Automated benchmarks and profiling

[chippmann]

Note: The target branch is set to feature/improve-ci-cd-workflow as it depends on it and so you can see the changes related to this branch. Once the target branch is merged, the base branch of this PR will be updated automatically

This adds support for automated benchmarking and profiling to our benchmarks project.

Data

Benchmark data

Can be found here: https://docs.google.com/spreadsheets/d/1aIclio2oYs_ua_4tENVjfn-OrwKkgVSCRfo50NTMve0/edit?usp=sharing
Everything in this spread sheet is autogenerated from a google app script which takes the benchmark data json we send it.

These benchmarks are automatically executed and the data updated on every push to master and periodically once a week on a dedicated self hosted physical windows runner (see the corresponding readme).

Profiling data

Can be found here (public read only link): https://monitoring.hippmann.ch/public-dashboards/22a47dbec7e04eed8fdad1135e22ef09
The profiling data is taken from the benchmarks project using pyroscope and grafana.

It is gathered on a linux runner as pyroscope currently only supports linux and macos.
As i do currently not yet have a physical self hosted linux runner setup, the data is gathered using a regular github action runner. Hence the data will be inconsistent atm.
The data is also automatically collected and updated on every push to master and periodically once a week.

Future plans

• Move the spreadsheet data to an influx db so it can also be queried and displayed using grafana which is way more powerful than a spreadsheeet based solution
• Setup a physical linux runner for consistent data

Note: the maintainers received a user for grafana to query data at will and create/edit dashboards. The two links above are read only and cannot be edited.

This data also contains code for a task which compares new benchmark data with existing benchmark data to detect if new code has a worse performance. This task is unused atm as the data is too inconsistent yet to make save assumptions on code performance and deviations. I left it here as the task might me useful none the less. But i can also remove it again if preferred.

[chippmann]

Closing this for now. We first need to integrate better benchmarks for such a use case.