Memory leak in long-running applications

[mx781]

Consider this example:

import idom
import time
from threading import Thread
from functools import partial


class SomeLayout:
    def __init__(self):
        self.ticks = 0
        self.set_ticks = None
        self.thread = Thread(target=self.ticker)
        self.thread.start()

    def ticker(self):
        while True:
            if self.set_ticks is not None:
                self.set_ticks(self.ticks + 1)
            time.sleep(0.01)

    @staticmethod
    @idom.component
    def SomeComponent(layout):
        layout.ticks, layout.set_ticks = idom.hooks.use_state(0)

        def decrement_ticker(e):
            layout.set_ticks(layout.ticks - 100)

        return idom.html.div(
            idom.html.div(idom.html.button(
                {"onClick": decrement_ticker},   # remove this line to fix leak
                f"Decrement ticker ({layout.ticks})"
            ))
        )


layout = SomeLayout()
idom.run(partial(SomeLayout.SomeComponent, layout))

This is a dumbed-down version of my application. It is a dashboard-like application (i.e. it's long-running), and the issue I'm facing is that over time it leaks memory to the point it gets reaped by the OOM killer.

The example above is exaggerated as the actual ticker I have is much slower (1s), but it still occurs then, just at a slower pace. If you run this example and monitor that process' memory, you should see it grow rapidly.

The issue at hand seems to be the passing of the onClick handler - if you remove that, the problem goes away completely. I suspect binding components to a class like this might be an anti-pattern (or perhaps calling a state update function from another thread is one?), so my question is - am I violating some design rules with this example, or is this a bug?

idom==0.33.1
Python 3.9.1
Ubuntu 20.04 (Linux fossa 5.4.0-57-generic)

---

edit:

In fact, it can be reproduced without the threading and class bit. I realise this effectively triggers an infinite loop and isn't what I'd want to do in a real application, but I imagine it shouldn't be leaking memory either way?

import idom

@idom.component
def SomeComponent():
    ticks, set_ticks = idom.hooks.use_state(0)
    set_ticks(ticks + 1)

    def decrement_ticker(e):
        set_ticks(ticks - 100)

    return idom.html.div(
        idom.html.div(idom.html.button(
            {"onClick": decrement_ticker},  # removing this fixes leak
            f"Decrement ticker ({ticks})"
        ))
    )


idom.run(SomeComponent)

[rmorshea]

This has been fixed now. See #510

The fix is included on the 0.33.2 release.

[mx781]

Hi Ryan, thanks a lot for the speedy fix! That seems to have done the trick, but I just wanted to check if it didn't introduce any unwanted side-effects - if you run the original example I posted, the "decrement ticker" button does not work and the console is full of logs like 2021-09-09T15:12:07+0300 | INFO | Ignored event - handler 'a59c12a03d384262a8341b944c0eb1dd' does not exist or its component unmounted - guess that's due to the ref cleanup you added. Is this expected?

[rmorshea]

I think that can be resolved by setting IDOM_FEATURE_INDEX_AS_DEFAULT_KEY=1 as an environment variable when you invoke your program. This will become the default behavior in the near future and, eventually be more thoroughly documented. For some resources to help you understand what layout "keys" are, refer to these resources from React which apply similarly to IDOM:

• Usages of keys - https://reactjs.org/docs/lists-and-keys.html
• Why keys are needed at all - https://reactjs.org/docs/reconciliation.html#keys

Side note: I'd also recommend setting Thread(daemon=True) so the thread will exit at the same time your program does.

[mx781]

Thanks for the pointer, I will try out the env flag. In reality my application ticker is much slower so it's not a huge issue in practice.

However, I'm afraid I've stumbled upon another memory leak, this time in SharedClientStateServer. Try this:

@idom.component
def SomeComponent():
    ticks, set_ticks = idom.hooks.use_state(0)
    set_ticks(ticks + 1)
    return idom.html.div()


server = SharedClientStateServer(SomeComponent)
server.run("127.0.0.1", 8000)

If you open the browser window, and then exit it, the process memory runs amok. Again, this is primarily driven by frequent state changes - if you use my original example and reduce the ticker interval, it does not occur. However, if you create a lot of elements dependent on that state (e.g. 1000 buttons but a 0.1s timer), it happens again. The issue I have in practice is that I believe the leak still occurs in a normal application if you exit the browser window at an inconvenient time - and this is the closest example I've been able to come up with.

Happy to move this to a new discussion/issue if you prefer that.

[rmorshea]

I'll open up an issue for this. We can investigate further there.