[Kubernetes] Controlling resources for stderr, stdin, heartbeat...

[monik-a]

We have the newest Airbyte deployed (actually, we have two: 0.50.3 and 0.50.34) on Kubernetes.
We incorporated resource requests/limits for all components and jobs too, but I couldn't find anything on how I can change the requests/limits values for relay-stdout, relay-stderr, call-heartbeat-server containers.
It'd be great if we could control them arbitrarily, but I want to at least even them out (request=limit).
I know it's managed somewhere here, but didn't find coresponding env variables or anything in the doc.
Could you please point me in the right direction ? 🙏

[marcosmarxm]

@gosusnp do you have any info about if it's possible to control these containers?

[gosusnp]

Sorry I missed this. Those configuration are not directly exposed through environment variables at the moment, mostly to reduce the explosion of configurations.

@monik-a , if you don't mind sharing the concern or intent behind adjusting the resources for those containers, that could help us think about how to expose those configs or maybe figure out better default.

[binhvcom]

Hello mate, have you found the answer for this ?, i am looking at this too, but unable to find how to update

[tautvydas-v]

Hey @monik-a , @binhvcom , I think I found a way to control these if it's still an issue for you.

There are environment variables defined in airbyte-commons , namely - SIDECAR_KUBE_CPU_LIMIT which seems to be responsible for call-heartbeat-server resources. I managed to use this variable inside globals.env_vars and it worked. For socat relay-stdout, there is SOCAT_KUBE_CPU_LIMIT variable, also can be defined in globals.env_vars Values.yaml file. Will update this message if I will find out a way to fix relay-stderr, but I'm assuming it should be coming from the same SOCAT_KUBE_CPU_LIMIT, but maybe it's not picked up due to different TCP port / or it's running in a different container. I'm still learning Kubernetes, so I might be mistaken with the last statement.

@marcosmarxm FYI if it's interesting, might be useful for future documentation.

[monik-a]

Thank you @tautvydas-v
I saw these somewhere yes, but are there some that control memory? That's more important to us to be honest 😅

[tautvydas-v]

@monik-a In the same link that I posted under airbyte-commons, you can click on it and find more variables. I suppose what you're searching for is SIDECAR_MEMORY_REQUEST and SIDECAR_KUBE_MEMORY_LIMIT, or try setting JOB_MAIN_CONTAINER_CPU_LIMIT,
JOB_MAIN_CONTAINER_CPU_REQUEST,
JOB_MAIN_CONTAINER_MEMORY_LIMIT,
JOB_MAIN_CONTAINER_MEMORY_REQUEST

under the same env_vars field. At least in our case, with the sidecar containers, they don't really use up any memory, pretty much up to 25Mi. Do you have an issue with discovery / check jobs or syncing jobs?

[RaymondvdW-AB]

@tautvydas-v I am trying to modify the CPU request of the stdin and stdout containers in the source & destination pods. Adding the SOCAT__KUBE_CPU_REQUEST environment variable did not work, but maybe that is because I am using an older Airbyte version. I will see if I can upgrade.

[tautvydas-v]

@RaymondvdW-AB Did you try playing around with the configs described here? https://github.com/airbytehq/airbyte-platform/blob/fab6227c7d02921105b66ddfe5af9cc91a6d32f3/airbyte-server/src/main/resources/application.yml#L168 I suppose you could try to modify them in the worker.kube-job-configs if I'm not mistaken (not entirely sure if it's possible, since it seems they're hardcoded and only a few variables can be changed).

Do you have an issue with source / destination checks though? I am seeing limit of 2 CPUs for relay-stderr with the screenshot below:

[RaymondvdW-AB]

@tautvydas-v thanks for your reply. I am not sure how to modify the config file you sent the link for. And I think that the only file I need to change is the values.yaml when using helm charts.

I do not have issues with the limits, only with the requests. In my case I can only run one connection at time because of the high requests of the stdout of the source pod and the stdin of the destination pod.

[tautvydas-v]

Hmm, looking again at the code, did you try using these variables
JOB_MAIN_CONTAINER_CPU_LIMIT,
JOB_MAIN_CONTAINER_CPU_REQUEST,
JOB_MAIN_CONTAINER_MEMORY_LIMIT,
JOB_MAIN_CONTAINER_MEMORY_REQUEST

inside global.env_vars, right? To be honest, I didn't encounter this issue yet since I'm still working on the issue I mentioned as source / destination checks are constantly made with each connection trigger, thus creating high unnecessary resource usage

[RaymondvdW-AB]

@tautvydas-v the JOB_MAIN_... variables only affect the cpu and memory of the main containers in the source and destination, not of the other containers (stderr, stdin, stdout, ...). I did not encounter any issues with the check pods. In my case they run very shortly at the beginning of the sync and then they are gone.

[xurtasun]

hey @tautvydas-v we are facing issues also with this. We want to increase the resources assigned to our sync destination and source pods, specially relay-stdout in source container. We have now a bottleneck with 1 cores assigned by default.