Cache recreation for interlink

[Surax98]

I was thinking about how to implement a cache restoring system on interlink's side, but before asking for opinions, let's quickly break through how things are working right now.

To better follow the workflow, you can find schemas at the following link:
https://excalidraw.com/#json=wwvC3eA1fjRhDBrCD9P3Y,nTrhSA-H_k0doBesYAsxDw
I also added simple state diagrams for VK/InterLink/Sidecar, very simple and functional for our case to be better understood.

Diagram A) VK already restores its pods by querying InterLink at startup (1), so that it can provide cached pods (2) and let the VK restore everything by comparing cached pods with the ones registered to the cluster (3). It doesn't require any disk space at all and relies on the fact InterLink keeps a coherent cache. Needs to be improved, but it works for now.

However, it is not possible to guarantee that InterLink will never go down, so a cache restoring system must me implemented.

Diagram B) My idea was to use a similar approach to the one used by the VK, allowing InterLink to query the below Sidecar (1) to provide all running (and ended/scheduled as well) jobs (2) and then rebuild the cache based on this (3). This mechanism is similar to the one used by the VK (diagram A) and relies on the sidecar's caching system as well, which must be kept coherent too.

At this point, the problem is quite obvious: what happens if the sidecar goes down? And that's why I am here to ask how to behave. I have thought about 2 different approaches:

• Diagram D and E) same schema as VK for InterLink, relying on disk caching (basically dumping useful structs to disk) to restore sidecar's cache, so it would be up to the sidecar developer to assure cache reconstruction in case of a sidecar failure;
• Diagram C and F) disk caching for both interlink and sidecar, as suggested by @dciangot, so at least the interlink cache is always recreated, without asking the sidecar to kick in.

Any useful feedback would be much appreciated, since it's in the interested of everyone knowing how the caching mechanism for Interlink will be implemented (especially if it will require effort by sidecar's dev's side)

[dciangot]

at @Surax98 , please expand the context a little bit. Possibly with some schema of the current workflows in the main outage scenarios.

Then we can discuss more carefully about details, otherwise we risk to go in the wrong way.

E.g. which info are needed to restore a cache, where can be recovered, and which is the impact in terms of work to be done and user/dev experience

[Surax98]

Added schemas and references

[dciangot]

Alright @Surax98 , if I read the schema correctly, I think we should go for the scenario 2, where interlink does know where to look to restore the status of submitted jobs/containers.

The plugin should be as much stateless as possible in my vision.

@spigad this is what we disussed last time, if you confirm, I think @Surax98 should go into the details of the scenario 2 here in the discussion:

• cache on disk? is this a simple yaml with the pod description to be dumped in the same place we are putting job.sh sbatch file? We need anything else?
• where are the changes in the core codebase expected?

[Surax98]

I was talking to @ttedeschi yesterday and we were discussing about separating the InterLink API and sidecars as much as possible, so they can be considered nearly as standalone as possible, so I'd suggested to avoid using the same directories used by sidecars and instead thinking of a different directory, maybe located in the same parent dir, but slightly different, so things are easily split.
For example, we could have a $INTERLINK_DIR/CachedPods/ directory in which we should put all the InterLink cached pods in the form of podUID.yaml and, when needed, restoring the whole cache based on them. To finish answering the first point, I think the whole PodStatus struct should be enough to be dumped to the file, so InterLink can easily restore everything at startup (if anything in the directory is available).
About the changes, they should be very easy to implement within the InterLink API code, it's just about writing, reading and deleting files when caching and starting up, not anything particularly hard to achieve.

But now I have one more question:
since we were thinking about keeping each part as standalone as possible, wouldn't be a good idea to apply the same schema to the Virtual Kubelet instead of letting it query InterLink for pods?

[dciangot]

The last proposal is equivalent to the one I proposed about storing information IN K8s pod resources right? We do already have that kind of cache, and it is the k/v store of k8s itself.

If we store the jobID in the pod resource after the creation, there is no point in storing it anywhere else. VK is capable of retrieving all the information from there.

Do you agree on that?

That would let VK rebuilding cache for everyone, querying interlink for all the pods known to k8s

[Surax98]

I think storing the JobID in the Pod struct would be ok until there is a one-to-one correspondence between VK, InterLink and Sidecar.
Let me elaborate further. VK gets N Pods registered, it submits to InterLink the Jobs and InterLink submits to the Sidecar. The Sidecar then provides backward the JobID to the VK and the VK stores the ID in the struct. Now the VK goes down and when going up, it doesn't know which Pods had before without asking to interlink, because it would be able to scan all pods registered to cluster THAT HAVE THE JOBID STORED, but how would it know which interlink "owns" them? So it MUST query InterLink in case of multiple InterLink instances, which would mean the VK is not standalone.
At least, this would happen if the use case wouldn't be a one to one correspondence, but a one to many (one vk, multiple interlinks).
Probably I haven't been much clear in my explanation

[dciangot]

There is no way in which we might support one VK with multiple interlinks... Only the other way around.

Does this solve your doubts?

[dciangot]

1 VK -> 1 interlink.. I have no valid use cases so far for supporting 1vk 2 interlinks.

While we do have reasons for 2 VK to 1 interlink

[Surax98]

Ok, recap:

• We need to slightly rework the actual InterLink API to allow the Sidecar to return the JobID so it can be stored inside the Pod struct in the VK (issue Slight API rework #206). This involves a slight rework even sidecar side.
• VK and InterLink will be both stateless
• • VK will be able to rebuild its Pods pool based on the Pods having the JobID within their metadata (issue Set jobID metadata on VK #205)
• • InterLink will be able to rebuild its cache by using some space on physical drive: restoring cache from the disk at startup and then updating by querying the below sidecar (issue InterLink disk caching #204)

[dciangot]

"they are both stateless" looks like in contraddiction with the last bullet. So, just to clarify, the restart from disk cache is NOT a requirement, it is only there for performance reason.

If the interlink cache is lost, the VK should be able to query interlink and rebuild the cache of the latter on the fly. At the cost of potentially thousands of request to the plugin.