Skip to content

Benchmarking tips

HyperSoop edited this page Oct 7, 2023 · 4 revisions

If for any reason you want to compare two instances of Minecraft in performance, be it comparing optimization modpacks or seeing if a mod or any tweak is worth adding, here's some advice to help you keep the comparasion fair.

The settings

It should go without saying that across your tests any settings that affect visuals should remain the same. Reducing the render distance isn't really optimization, and for the same reasons neither is disabling entity shadows, disabling vingette, lowering the FOV, changing particle visiblity, etc.

The rule of thumb is anything that in any meaningful way affects how the game looks should not differ across tests. Do not just use the same options.txt - doing it won't catch things in mods' own config files like More Culling's leaves culling mode (which needs to be standardized too) and might disable some potentially beneficial non-visual changes (SO has one or two of those in the options.txt).

This includes things like tutorial toasts ("Move with W, A, S and D; Jump with Space"). SO disables those out of the box, some other packs also do, and some don't. Make sure to disable them in all of the ones you're testing by setting tutorialStep:none in the options.txt, since rendering toasts (like any UI elements) has a performance impact and these only show up on new instances anyway, so there's no need to take them into account.

The scenario

Make sure the scenario you're testing in is exactly the same across all of your tests. That, down to the camera angle, position and time of day. Not to mention things like the world seed and the weather. You can't really compare FPS measures taken in different circumstances with different levels and kinds of load, even if the difference is really small.

Don't forget: /gamerule doDaylightCycle false /gamerule doWeatherCycle false

(You can also access gamerules during world creation)

A good thing to do is to generate/open a world in one of your instances and copy it to all of your other ones before starting to test. This ensures your camera angle and position is exactly the same everywhere, and prevents features like trees generating differently.

Another thing to note, especially in singleplayer tests: before starting recording performance metrics, wait until all of the chunks that can load, load. Not only does chunks still loading have a performance impact, less chunks having loaded puts less load on the GPU, so once again, wait until all of the chunks have been generated, loaded and built before taking any measures.

Fun fact: you can use F3+Esc to pause the game without showing the pause menu. This allows you to factor out the performance impact of any block updates (and all ticking whatsoever) if you so wish in local worlds. You probably shouldn't use it unless you need to, though.

Probably don't test in game server lobbies - the amount and the kinds of entities rendered can change drastically with time and between hubs. (Worth mentioning, though, that Hypixel has the /swaplobby command, allowing you to take measures in the same lobby every time you do it, making the results more accurate, especially if the lobby you're warping to is an empty one)

Recording

If you want reliable results, taking a screenshot of F3 isn't the best thing you could do. F3 takes a toll on performance (just like any overlay, it's another thing that needs to be rendered when enabled) and can't give you an accurate idea of the average or median framerate in cases where performance fluctuates. Ideally, you should be using software like CapFrameX.

Sharing

When sharing the results, make sure to mention your OS and hardware (the bare minimum is CPU + GPU, but the more you specify, the better). Also note the resolution you were running the game at, the map or world type + seed you've been using, your location and camera angle if it's not otherwise obvious and of course the graphical options you've settled on.

(If you want to submit a benchmark for the Simply Optimized Modrinth page, you should either organize the information into an image with pretty graphs and whatnot or make a sheet or a Gist for it; then just reach out to me.)

Clone this wiki locally