Terminate & purge all running orchestrations

[cargilltay]

Hey @cgillum we're also encountering a use case for full termination of all orchestrations on a project I'm working on. The scenario is the following:

A HTTP trigger hit.
A config is pulled which contains N number of routes.
An orchestration is started for each route. Each route restarts itself upon completion via ContinueAsNew.

What we're trying to do:
Hit the same HTTP endpoint and teardown all executions, clear all history, clear all queues, and then start fresh with a new config.

What we're running into:
Doing a looped approach for termination takes too long for us. Local it seems to work after many minutes. Deployed we both hit the HTTP timeout limit on HTTP Triggers & other timeout limits on queues when we tried to offload this work to a queue.

Is it possible to accomplish the scenario I just described in an easy manner? Turning off the function app and deleting the storage tables/queues takes no time at all so I'd imagine a complete stop & clear must be feasible in code.

[cgillum]

I'm interested to understand why it takes so long to do the cleanup work in the HTTP function. Would you be able to share some code that you're using to do this? I'm wondering if it can be optimized somehow to run faster.

A couple questions in the meantime:

• Roughly how big is N (the number of routes)?
• Can the cleanup of each route/orchestration be done in parallel rather than serially?
• Is it okay to clear all state or do you need to clean up just those N routes?

[cargilltay]

@cgillum Thanks for the speedy reply. It does seem a little odd that i'd have to do all of this from another function app. What is the feasibility of adding functionality to IDurableOrchestrationClient to accomplish this behavior. I could see this as a great improvement for workflows where a complete reset is needed.

[cgillum]

You don't have to do it from another function app, but it would potentially save you from a bunch of errors in your logs if you could shut down the existing triggers before blowing up the storage.

It also just occurred to me that we have an API for this already that you can use. It's not part of IDurableOrchestrationClient, but it would be relatively easy to code up in an HTTP-triggered function. Here's an example that I quickly whipped up and tested locally. This will be much safer than manually enumerating and deleting storage resources.

class ResetDurableStateHelper
{
    readonly INameResolver nameResolver;

    // INameResolver is a service of the Functions host that can be used
    // to look up app settings.
    public ResetDurableStateHelper(INameResolver nameResolver)
    {
        this.nameResolver = nameResolver;
    }

    [FunctionName("ResetDurableState")]
    public async Task<HttpResponseMessage> ResetDurableState(
        [HttpTrigger(AuthorizationLevel.Function, methods: "post")] HttpRequestMessage req,
        [DurableClient] IDurableClient client,
        ILogger log)
    {
        // Get the connection string from app settings
        string storageConnectionString = this.nameResolver.Resolve("AzureWebJobsStorage");
        var storageServiceSettings = new AzureStorageOrchestrationServiceSettings
        {
            StorageConnectionString = storageConnectionString,
            TaskHubName = client.TaskHubName,
        };

        // AzureStorageOrchestrationService is defined in Microsoft.Azure.DurableTask.AzureStorage, which
        // is an implicit dependency of the Durable Functions extension.
        var storageService = new AzureStorageOrchestrationService(storageServiceSettings);

        // This will delete all Azure Storage resources associated with this task hub
        log.LogInformation("Deleting all storage resources for task hub {taskHub}...", storageServiceSettings.TaskHubName);
        await storageService.DeleteAsync();

        // Wait for a minute since Azure Storage won't let us immediately recreate resources
        // with the same names as before.
        log.LogInformation("The delete operation completed. Giving Azure Storage a minute to internally process the deletes...");
        await Task.Delay(TimeSpan.FromMinutes(1));

        // Optional: Recreate all the Azure Storage resources for this task hub. This is done
        // automatically whenever the function app restarts, so it's not a required step.
        log.LogInformation("Recreating storage resources for task hub {taskHub}...", storageServiceSettings.TaskHubName);
        await storageService.CreateIfNotExistsAsync();

        return req.CreateResponse(System.Net.HttpStatusCode.OK);
    }
}

Note that I made this into a class with a non-static member so that I could take advantage of dependency injection and get the connection string information. There might be other ways to do this as well.

[cargilltay]

@cgillum Amazing! Spent a load of time trying to figure out a similar implementation of queue/table deletion. As you mentioned I saw a slew of errors with that approach.

I'll give this a shot. Thanks as always!

[olitomlinson]

@cgillum I guess the caveat to this approach is that Terminated life-cycle events will not get published, right?

[cgillum]

@olitomlinson, correct, this is very much a non-graceful purge of all orchestration data.