How to handle 100's of CRDs? Can we process them concurrently

[ewassef]

I would have swore that at some point, you would get multiple calls to ReconcileAsync at the same time. However, it now seems that this is not the case. Is there a way to run say X entities in parallel? our reconcile loops are taking an hour now because of the sheer number of CRDs we are creating

[ewassef]

@buehler what do you think?

[buehler]

Hey @ewassef
The current mechanism is serial. There is (sadly) no way around it. I'm currently on vacation, I'll have a look at v8 when I am home :-)
If I recall it correctly, v8 could theoretically run in parallel.

[ewassef]

I was curious about your plans for that. It looks like the reactive extensions are being replaced with a concurrent dictionary. However, I think we may be able to process multiple entities concurrently (safely also) by modifying this line to this:

.Merge(maxConcurrent: _operatorSettings.ConcurrentExecutionLimit ?? 2)

how do you mean theoretically run in parallel in v8?

[buehler]

Hey @ewassef
I just checked, in v8 the logic is:

private async Task ReconcileModification(TEntity entity)
{
    var latestGeneration = _entityCache.GetOrAdd(entity.Uid(), 0);
    if (entity.Generation() <= latestGeneration)
    {
        _logger.LogDebug(
            """Entity "{kind}/{name}" modification did not modify generation. Skip event.""",
            entity.Kind,
            entity.Name());
        return;
    }

    _entityCache.TryUpdate(entity.Uid(), entity.Generation() ?? 1, latestGeneration);
    await using var scope = _provider.CreateAsyncScope();
    var controller = scope.ServiceProvider.GetRequiredService<IEntityController<TEntity>>();
    await controller.ReconcileAsync(entity);
}

And this method is called on an event base of the watcher. This means, that 100's of CRDs could theoretically run in parallel. There is no semaphore lock anymore.

[jaccus]

Hi @buehler! Would it be possible to enable running reconciliation for multiple entities of the same type in parallel without modifying the source code of operator sdk library? Or with "could theoretically" were you just referring to a potential feature improvement?

I am struggling with a similar problem as author. Multiple independent instances of the same entity running their reconciliations serially result in drastically increased deployment times. This unfortunately makes it hard to justify using the library in our productions scenarios.

[buehler]

Hmm, fair enough. I thought that using async await here can help to reconcile the entities in parallel. However, if that's not the case we need to investigate a threading mechanism. This would introduce a lot of complexity in the system.

[slacki123]

Hey, I'm working with something similar now and I'm noticing some strange behaviour. I'm wondering if this is related or there's something wrong with the requeue mechanism

When using the same custom resource definition, and then creating something like 10 custom resources based on it with varying requeue time varying from 5 seconds to 60 seconds per resource.

I'm noticing that on the startup of the operator, all 10 resources get updated and processed, but then after a while I only see 2 or 3 resources getting processed while the remaining resources are ignored despite having been added to the requeue.

The way I set up my requeue is this:

  public class ClusterController : IEntityController<MyEntity>
{
    private readonly EntityRequeue<MyEntity> _requeue;

    public ClusterController(
        EntityRequeue<MyEntity> requeue)
    {
        _requeue = requeue;
    }

    public async Task ReconcileAsync(MyEntity myEntity)
    {
        if (!myEntity.Spec.Enabled)
        {
            return; // will not trigger a requeue
        }
        
        // Update status here
  
        var pollingRate = TimeSpan.FromSeconds(myEntity.Spec.ReconciliationLoopPollSeconds);
        _requeue(myEntity, pollingRate);
    }

Has anyone encountered this?

EDIT:
now related to #707

[slacki123]

Building on the question of processing 100s of CRDS further. What is the expected behaviour when dealing with long running Reconciliation processes?

For example say that:

1. we add a custom resource - it gets reconciled
2. the process runs for some time
3. In the meantime, we modify the custom resource, which should lead to triggering another reconciliation loop

now the question is, will the process wait for the existing process to finish, or will it start running another one in parallel? (which could be problematic if conflicts are an issue)

I have some ideas how to deal with this issue on the implementation side, as well as the KubeOps resource watcher side, but would like to understand how deeply has this already been thought through first

[buehler]

My two cents: right now, in v8, it will trigger another reconciliation. I don't know the best practices of Kubernetes about long running reconcile loops.

Maybe we should add back some sort of queue, or even better: use cancellation tokens and if the current resource gets modified while the reconcile loop is running, we cancel the token and re-reconcile with the new entity.

[slacki123]

hm it's an interesting problem then I guess. I'd need to understand more use cases, but I wonder if it would be useful to have a few configurable processing options.

1. for each event or requeue, wait for each ReconcileAsync process for each unique resource UID to finish processing until it returns before reprocessing (for those who always need the process to finish per unique resource). After that process the resource just once.
2. For each unique resource UID, run a new ReconcileAsync process in parallel for every new event (it is how it works now)
3. cancel an existing process if a new event is triggered (as you suggested above)

Whichever one is the option would be configurable somewhere from the DI or operator settings, or a value on the CR.
Our use case for example requires option 1

[buehler]

Yeah, it's a complicated topic.

Making it configurable seems to be a great idea. When I built a similar logic in v7, a lot of problems surfaced. For example: if the reconciliation loop takes more time than the requested requeue, you create an infinite backlog.

Then I started to use time durations to "memorize" the reconcile loop and if something is happening right now, ignore the current reconcile and requeue after.

Then another implementation took place which I don't really remember anymore.

The queue always caused problems in some cases. That is one of the reasons I rewrote the whole thing for v8. Right now, it is plain and simple. We can do such a queue mechanism, but it should be well thought through.

Is there someone who knows (more) about best practices in terms of Kubernetes Operators? Should it be possible to have long running reconcile loops? Or should you make multiple operators with multiple resources?

As a sidenote: maybe we can make this caching / queueing mechanism with an Interface, then, one can add their own strategy without having to add it directly to the package.

[slacki123]

I see the infinite loop backlog - although could be solved if you don't build the backlog and only ever process the last event grouped by event type? Although I agree that a highly experienced veteran's opinion would be ideal on this. The interface is a nice idea, but depending on how useful that would be. As for right now I can also write my own task manager that tracks the ongoing reconciliations, and if the current one is still processing then skip, and if not then reconcile for example. Either way, I'd end up managing my own code for this