FlytePropeller sometimes fails with Pods dont exist error!

[kumare3]

FlytePropeller will fail to mark a task as complete with Pods do not exist error.
On looking into Kubelet logs

I0722 21:48:56.709630 2364 kubelet.go:1901] SyncLoop (PLEG): "fry3cswi-n0-0_pineapple-development(fa355eb2-e457-4298-b9fc-1e000ce82b94)", event: &pleg.PodLifecycleEvent{ID:"fa355eb2-e457-4298-b9fc-1e000ce82b94", Type:"ContainerDied", Data:"28ba188898d20f9cc937d41843a22e45983198aea054c663020035bd36f7b6fc"}
``
It seems that the pleg was `ContainerDied`.

But after retried Flyte was successful? What should I do!

Question by - @jeevb (on slack)

[kumare3]

Pods vanishing is usually related to Pods being garbage collected by K8s. Pods are garbage collected randomly and at high number of pods, the garbage collection becomes more obvious with these failures in Flyte.

solution:
Use inject-finalize = true for FlytePropeller.k8s config.

Why does this happen?
This happens because K8s will garbage collect pods randomly. And FlytePropeller runs in sync loops. It is possible that a pod vanishes and FlytePropeller will not notice it. this is because the loop may lag and propeller faces an issue like motion-blur. The pod was visible a moment ago as running and then it vanished.
Propeller does not know how to handle this, one option could be to optimistically look for the output-file. (we will create an issue for this). This will assume that the distributed storage is consistent. This is now mostly true for all major stores.

How does the proposed solution fix it?
Injecting a finalizer informs k8s that the object cannot be reclaimed, as long as the finalizer exists. Once the finalizer is removed K8s is free to remove it. Propeller can thus rely on the object being there till the finalizer exists. Flytepropeller will thus ensure that the finalizer is not removed unless it has safely checked and bookmarked the state.

Alternative solution (needs work), look if the output was generated.