Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup CI to run benchmarks and access results #2235

Open
ghostdogpr opened this issue May 17, 2024 · 3 comments
Open

Setup CI to run benchmarks and access results #2235

ghostdogpr opened this issue May 17, 2024 · 3 comments
Labels
build Build and CI tasks good first issue Good for newcomers

Comments

@ghostdogpr
Copy link
Owner

It would be nice to run benchmarks:

  • on demand (manual trigger)
  • on each release

Then, the job should gather results and upload them somewhere (gist?) so that we could visualize them like this: https://jmh.morethan.io/?source=https://gist.githubusercontent.com/fwbrasil/27c8abec86e947e9719d41a859deb5d2/raw/814ca4ebb3a1294f8dd7bbec9b54ea0957b92434/jmh-result.json

We can probably copy Kyo's CI: https://github.com/getkyo/kyo/blob/main/.github/workflows/bench.yml

@ghostdogpr ghostdogpr added the build Build and CI tasks label May 17, 2024
@kyri-petrou
Copy link
Collaborator

Just to mention that we should probably be a bit selective with which benchmarks to run. e.g., I don't think we need to run the benchmarks for other libraries, and probably just the jsoniter JSON benchmark will do.

Regarding the Nested benchmarks, I think running just the 100 par / batched variants should suffice.

We should probably also reduce warmup and measurement time to 1 second (3 seconds seems like an overkill) to reduce execution time of benchmarks. At the same time, we might want to use @Fork(2) or @Fork(4) as I've noticed that in some cases benchmarks will be significantly slower or faster. I think this has to do with infamous JDK-8180450 issue, and their 12% occurance seems consistent with my observations

Finally, we probably want to check how we're doing comparitively when running multi-threaded vs single-threaded, so we we might want to run

@ghostdogpr
Copy link
Owner Author

We could start with just running GraphQLBenchmarks and then we fine-tune it from there once the CI is in place.

@ghostdogpr
Copy link
Owner Author

ghostdogpr commented May 19, 2024

Here's an example of the end result (I ran the test and uploaded to gist manually): https://jmh.morethan.io/?source=https://gist.githubusercontent.com/ghostdogpr/589d379ba2d096afcbf5a27d4298fb16/raw/e98d61fc0a3d87ad7d31b9f17eeb6596a014a19d/jmh-result.json

File obtained from running Jmh / run SimpleQueryBenchmark ComplexQueryBenchmark FragmentsQueryBenchmark ParserBenchmark -rf json

@ghostdogpr ghostdogpr added the good first issue Good for newcomers label May 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Build and CI tasks good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants