Use Leshan library to simulate several clients

[sbernard31]

This issue aims to centralize information about using Leshan to simulate several clients.

Here is known issue, generally this is about using too many thread:

Problem	Github issue	Possible clean solution	Known workaround
UDPConnector of californium need at least 2 thread by client	eclipse-californium/californium#1203	Implement connector based on Netty ?
DTLSConnector of californium need at least 1 thread by client	eclipse-californium/californium#1203	Implement connector based on Netty ?	TBD
Not clear DefaultRegistrationEngine blocking issue	#1574 (comment)		TBD

Possible solution/workaround about using too many thread:

• Trying new Virtual Thread of Java 21. (not tested, feedback is welcome)
• Trying to increase your OS Thread limit.

Number of port limitation:
Using several clients on the same machine may reach to the limit of available ports. (see : #1406 (comment))

A solution could be to implement queue mode and so sleeping client could free IP port and so you will be able to simulate more client by machine but the maximum of client talking at the same time is still limited by the maximum number of port (~35k).

Here is old issue related to that:

• Use Leshan Client to launch thousands LWM2M clients in the same VM. #491
• Scaling Leshan v2.x to 10,000 Clients, General Performance Query.  #1406

Here is a tool which aims to to that : https://github.com/sbernard31/benchmark-clients

(Content of this issue will be updated when we get new information)

[jvermillard]

About thread limitation, using https://github.com/golioth/xk6-coap, which is Go-based (a similar solution to Java 21 virtual threads), will overcome this (with some extra benefits of the k6 performance framework)

Another solution for the number of ports is to run multiple interfaces on the client machine because the 16-bit port limitation is a per-interface limitation. I wonder if you can do this by running one docker container for the server and multiple docker containers for the client and keeping the traffic in the docker network.

Also, you can get more ports per interface by changing this kernel parameter: /proc/sys/net/ipv4/ip_local_port_range
Example: echo 10000 64000 > /proc/sys/net/ipv4/ip_local_port_range

[sbernard31]

About thread limitation, using https://github.com/golioth/xk6-coap, which is Go-based (a similar solution to Java 21 virtual threads), will overcome this (with some extra benefits of the k6 performance framework)

Tell, me If I misunderstood : this is an alternative of "Using Leshan library to simulate several clients", if you use it then you don't use Leshan client at all ?

[jvermillard]

Yes you are right. When I need to do these kinds of tests I often need to simulate the CoAP part (for example the registration) and also use some HTTP API to verify things on the server or trigger more commands like LWM2M reads. So that's why I'm exploring this way of load testing CoAP/LWM2M servers.

[Nick-The-Uncharted]

Tried virtual thread, it is helping, (client count per machine(16u32g) increase from 20000 to 60000).

The things I do based on M15:

1. Change org.eclipse.californium.scandium.DTLSConnector.Worker to Runnable and submit them using ExecutorService dtlsReceiverExecutor = Executors.newThreadPerTaskExecutor(Thread.ofVirtual().name("DTLS-Receiver-", 0).factory());
2. Change timer and executorService: private static ScheduledExecutorService DEFAULT_TIMER_EXECUTOR = Executors.newScheduledThreadPool(30000, Thread.ofVirtual().name("DTLS-Timer-", 0).factory()); timer = DEFAULT_TIMER_EXECUTOR
   and executorService = Executors.newThreadPerTaskExecutor(Thread.ofVirtual().name("DTLS-Worker-" + lastBindAddress + "#", 0).factory());
3. Change all synchronized in DefaultRegistrationEngine to lock
4. Set a timeout for udp socket (https://stackoverflow.com/questions/78837170/java-22-0-2-virtualthread-unparker-seems-parked-forever) at DTLSConnector#init.
5. Cache DmServerInfo for all client (it is a hack to reduce gc pressure, all my client use the same cert)
6. Use generation zgc and use following parameter -Xmn4096m -Xms24576m -Xmx24576m -XX:-ZUncommit -XX:+UseZGC -XX:+ZGenerational -XX:ConcGCThreads=12 -XX:ZAllocationSpikeTolerance=8

Sorry for unable to submit a mr because of the security policy of my company.

[sbernard31]

@Nick-The-Uncharted Thx for taking time to share this 🙏

[Ozame]

@Nick-The-Uncharted Thank you so much, these points will be helpful. I've tried a similar setup, but faced an issue where the communication sooner or later hungs with large number of devices. Will try again with these in mind.

Could you clarify couple points:

Change all synchronized in DefaultRegistrationEngine to lock

What do you mean by this, can you give an example? Isn't synchronization already doing the locking?

Set a timeout for udp socket

I assume this is with this.socket.setSoTimeout(xxx)? What kind of number you found to be working nicely?

One last one - how many receiverthreads are you using on the DTLSConnector? The default config for this is 1.

[jvermillard]

What do you mean by this, can you give an example? Isn't synchronization already doing the locking?

It's a problem with virtual thread implementation: if the code use synchronized, the thread will be pinned, with lock it will not:
https://mikemybytes.com/2024/02/28/curiosities-of-java-virtual-threads-pinning-with-synchronized/

[Nick-The-Uncharted]

@Ozame I set soTimeout to 50000, actually any number less than Long.MAX_VALUE / 2 is good, see my question at https://stackoverflow.com/questions/78837170/java-22-0-2-virtualthread-unparker-seems-parked-forever.

how many receiverthreads are you using on the DTLSConnector? The default config for this is 1.

I didn't change that, it's still 1.

If all your client stucks, maybe all your virtualthread is pinned, try using -Djdk.tracePinnedThreads to find out why. (Or the bug mention in my stackoverflow question occurred)

[Ozame]

Ok, will check all of these things. Thanks for the pointers @jvermillard @Nick-The-Uncharted !