Memory leak

[timdittler]

Hey, thank you very much for your extremely interesting project. I would like to try it out with Istio 1.18.5.

This is what my config looks like

---
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: istio-coraza-waf
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  url: oci://ghcr.io/corazawaf/coraza-proxy-wasm:0.4.0
  imagePullPolicy: IfNotPresent
  phase: AUTHN
  pluginConfig:
    default_directives: default
    directives_map:
      default:
        - Include @recommended-conf
        - Include @crs-setup-conf
        - Include @owasp_crs/*.conf
        - SecRuleEngine DetectionOnly

However, it looks like there might be a memory leak in the proxy. The memory consumption of the ingressgateway is stable before enabling the coraza-proxy-wasm, but grows continues after enabling it. Memory growth when disabling coraza-proxy-wasm.

Have you seen this behavior before? Can I help debug the problem?

[M4tteoP]

Hey, thanks for the report. @anuraaga has carried out a lot of work around memory management and https://github.com/wasilibs/nottinygc, but maybe something has to be still taken care of. How is your traffic? Are you experiencing the same behavior even disabling the body analysis (SecRequestBodyAccess off and SecResponseBodyAccess off)?

Thanks for your help

[timdittler]

I did some new tests today.

• 11:05 activate plugin with SecRequestBodyAccess off & SecResponseBodyAccess off
• 13:35 remove SecRequestBodyAccess off

So it looks kind of stable without Request Body Access, but than again continually growing with it, but not as bad as with Response Body Access in the pictures above.

What do you mean by "How is your traffic?"

[M4tteoP]

Thanks for the additional information. I meant how the traffic that your ingress gateway is receiving, such as mostly get requests, JSON payloads, or files uploaded with a multipart/form-data content type. Mostly I was trying to grasp how the waf was behaving in terms of the body processor used

[timdittler]

The WAF is attached to the ingress gateway of a kubernetes cluster in the public internet. Therefore, it's really hard to determine what kind of traffic it gets. There is a lot of normal HTTP and REST traffic. There's probably also a good amount of file uploads. Some request looks weird as they have complex parameters. Grafana queries are usally a good example of that. Additionally, we get a lot of scanner and potential exploit traffic.

[jcchavezs]

Maybe related? wasilibs/nottinygc#46

[kamelj]

Hello @timdittler @M4tteoP
we have almost the exact behavior of a memory leak,
but how do things change when you add SecResponseBodyAccess off and SecRequestBodyAccess off
Since the default value is off, the behavior shouldn't be changed if you add it with off or remove it.

[timdittler]

They slowed the leakage down, but not enough too be usable in our scenario

[jcchavezs]

While I will spend some time chasing this down I would say the next milestone is to have Go Gc on board as per https://github.com/wasilibs/nottinygc?tab=readme-ov-file#nottinygc.

…

On Mon, May 13, 2024 at 9:25 AM Tim Dittler ***@***.***> wrote: They slowed the leakage down, but not enough too be usable in our scenario — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAR76YTDYMA73BAH6SLZCBTIDAVCNFSM6AAAAABAC5AQYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBWHA2DANZQHE> . You are receiving this because you commented.Message ID: ***@***.***>

[kamelj]

Hello @jcchavezs
So, to solve the root cause of this issue "nottinygc" should be replaced by "Go Gc"?
If so, will this milestone be with the upcoming release let's say 0.51 😄

Thank you guys for your efforts 🙏🏻