Scaling Ditto

[dkc6gh]

I'm working on scaling up Ditto on Kubernetes to handle a massive load. This means it needs to effectively manage 0.5 - 1 million things with 500-1000 updates per second.
I found a helpful guide on https://eclipse.dev/ditto/2023-10-09-ditto-benchmark.html, but unfortunately, it lacks some details on resource allocation (CPU and memory) for each service and potential tuning parameters.
I'd greatly appreciate any advice on configuring Ditto for optimal performance. Currently, I'm testing on a server-grade PC with 24 CPU cores and 128GB of RAM, while simulating a load of 100 updates/second. While the services appear to be functional, I'm encountering recurring warning and error messages from the connectivity services. Additionally, all the pods are restarting every few hours, suggesting instability.

{"message":"Failure updating Ditto pub/sub subscription - trying again next clock tick", ...
{"message":"heartbeat interval is growing too large for address pekko://[email protected]:2551: ...
{"message":"Exception thrown during connection pool background maintenance task", ...

[thjaeckle]

@vvasilevbosch could you provide additional information about which configuration of e.g. memory/cpu requests/limits you used for the benchmark?

[thjaeckle]

@dkc6gh with which memory/cpu requests and limits do you run the Ditto services with?
How many instances of them do you run?
Does the MongoDB run on the same hardware? How is the MongoDB tuned?

[vvasilevbosch]

Hi @dkc6gh @thjaeckle, I have documented the configuration in warmup and scenarios-run sections:

Heap memory needed according to number of things:
10 000 hot things ~1.8GB for things service and ~1GB for policies service.

100 000 hot things - ~3.5GB for things service and ~3GB policies

300 000 hot things - ~5GB for things service and ~5GB for policies service

1 000 000 hot things - ~16GB for things service and ~15GB for policies service

The size varies, depending on the size of the things and policies.

The scenarios are run with 100 000 warmed up things.
.....
All services use the following java vm options:

-XX:InitialHeapSize=5g

-XX:MaxHeapSize=5g

-XX:MaxRAMPercentage=75

-XX:ActiveProcessorCount=16

[dkc6gh]

Here are my settings on Azure AKS using L series VMs (8 vCPU/64GB RAM):
connectivity (3 pods each): 4 vCPU/48GB RAM
thing (3 pods each): 4 vCPU/48GB RAM
things search. (3 pods each): 4 vCPU/32GB RAM
policy (3 pods each): 1 vCPU/12GB RAM
MongoDB (3+1 pods HA on the same cluster): 6 vCPU/32GB RAM
-XX:InitialHeapSize=5g
-XX:ActiveProcessorCount=16
Six MQTT clients are currently publishing modify thing commands to Ditto at a rate of 100 updates per second each. While all the pods appear to be operational, I'm seeing warning and error messages, suggesting the system might require some tuning to handle this load effectively. From Connection Status, Metrics and Logs from Ditto UI, there is no failure, so this is little bit confusing.
@vvasilevbosch Would you be able to share CPU/Memory allocation and some other details or the Helm chart used?

[vvasilevbosch]

The cpu/memory allocation is documented in the blogpost. Helm chart version is 3.0.0, but it is not uploaded in the repo. You can use latest version, I believe the default values for jvm are unchanged.

[dkc6gh]

Does "-XX:ActiveProcessorCount=16" mean 8 CPU per service? The default values from Helm chart is very low, like 0.5-1 CPU and 1GB RAM. The blog post says "1 000 000 hot things - ~16GB for things service and ~15GB for policies service". Does this suggest total 32GB or more for 1 million hot things?

[vvasilevbosch]

-XX:ActiveProcessorCount=16 is 16 CPU per service. Memory for things depends on thing size, but 16GB should be enough