Leave task containers running for real time results?

[ChrisBarefoot]

Hey Flyte community.

We're currently playing with Flyte for a proof of concept.

One thing we noticed is that the overhead for small requests of starting and stopping each task's container is significant. Our use case targets real-time responses, if possible.

Is there a concept in flight of leaving those containers up and running and responding to new requests as they come in?

Thanks so much!

[kumare3]

Hi @ChrisBarefoot great question. Firstly welcome to the community and thank you for the questions.

So at the moment Flyte restarts a new pod for every task execution. This is to allow multiple versions to co-exist. And as you experienced this can be slow. This is also because Flyte is optimized for longer running tasks where the overhead can be ammortized.

There are a few things we will soon start working on

• Task coalescing, fuse multiple tasks onto the same container. [Feature] [Optimization] Coalesce steps into a single container execution #649
• KNative integration - support light weight functions with low
  Spin up overhead.

But these are slated to be released closer to the end of the year.

As you can imagine there are a few teams interested in the knative plugin

[ChrisBarefoot]

Thanks so much @kumare3 for the quick response. We'll take a look at both of those.

[georgesnelling]

Hi Chris, Just curious, what is your high-level use case? If that's confidential I completely understand.

…

-George

On Wed, Sep 15, 2021 at 1:27 PM ChrisBarefoot ***@***.***> wrote: Thanks so much @kumare3 <https://github.com/kumare3> for the quick response. We'll take a look at both of those. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1481 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADC72DDLISCL3DSG5XLUUDUCD6SHANCNFSM5ECUNE5Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[kumare3]

@ChrisBarefoot I completely understand the usecase and we would love to discuss more if your are open to it and talk about it on the slack channel or over vc?

[ChrisBarefoot]

Thanks @kumare3. We're still exploring, but will certainly reach out as we have questions. Appreciate it!

[kumare3]

@ChrisBarefoot There is a lot of interest for this in the community - and we are looking into options.
cc @JakeNeyer, Donald MacDonald (github handle missing), @EngHabu so we would love to collaborate if your team is interested as well

[AdrianoKF]

Commenting so I'll get notifications (and as a stand-in for Donald MacDonald, since he's my manager). Yes, we'd be quite interested in this use case and possible solutions as well (the Knative and task coalescing approaches mentioned in the other thread sound promising)

[georgesnelling]

• @hamersaw since he's already hip-deep in propeller optimization.