Benchmarks results on different projects

[goriunov]

Hi there, i have not looked in to the details or the code used for the benchmark as of yet, but was hoping whoever already checked it might give a bit of details, before I dig in to this.

Primarily, I am curious in the benchmark details on the gws repository itself (README):

VS the results provided in the:

https://github.com/lesismal/go-websocket-benchmark

(image is just example, including all other cases as well)

If I interpret, the results correctly, in the GWS benchmark the GWS beats the gorilla websocket (note i really only care in comprising with gorilla websocket as we're considering migration off it) quite substantially especially in the larger messages sizes. But on the lesismal benchmark it is very questionable and in some cases the gorilla websocket is beating the gws in temrs of latency and CPU.

I know that we should actually test and benchmark for our own use cases, but I am just curious if there are any out of the box improvements that gws gives us in terms of performance without taking in account our work load as yet. And if it is worth of migration.

(just heads up our system is very latency dependent and passes large amounts of data in between, even small improvements on latency are important to us)

[lxzan]

lesismal/go-websocket-benchmark is optimized for running press tests on gorilla/websocket, using the stream api and a statically allocated buffer. In reality, buffers are basically impossible to allocate statically.

You can run the benchmark yourself to see the difference between ReadMessage and NextReader.

[lxzan]

Test code is here.

https://github.com/lxzan/go-websocket-testing/blob/main/cmd/gorilla/main.go

https://github.com/lesismal/go-websocket-benchmark/blob/main/frameworks/gorilla/server.go

[lxzan]

On the other hand, gws uses the open-source compression library https://github.com/klauspost/compress, which has a much higher compression performance than the standard library.

[lxzan]

Even though gorilla is optimized for benchmarking, gws runs lessismal slightly better than gorilla, and you should look at gws_std rather than gws when comparing to gorilla.

Using my test method, the result is more stable than lesismal, I use runtime.GOMAXPROCS() to limit the number of CPUs, while lesismal uses the taskset command.

My test tool is wsbench, 1000 connection test, lesismal uses 10000 connections, differences in the number of connections also have an impact on the results.

wsbench echo -c 1000 -n 1000000 -p 1000 -u 'ws://127.0.0.1:8000/connect' --latency

[goriunov]

Thanks @lxzan this is very useful information, I will give it a try and see how it also runs for our particular work load.

[goriunov]

By any chance @lxzan have you had a benchmark run on go 1.22 vs go 1.21? Noticed a bit of the regression on latency, i think it is not related to the project but thought i would just drop it here:

Our latency increased after upgrade (still looking in to what might be the reason, may be new semantics of the for loops)

[lxzan]

Performance does vary between compiler versions. A while ago I found that windows cross-compiling to linux/amd64 was more performant than the native platform compiler, by about 10%, I can't remember if it was 1.21 or 1.22.