Error Handling in Fan-Out / Fan-In Pattern

[PaulKruger]

I've been implementing a Fan Out / Fan In pattern, and I'm having difficulty appropriately catching and handling the errors. Is there some documentation or recommendation on how to properly handle errors when one of the activities in this scenario failes? I've tried three separate approaches, and in all cases, error logs still seem to show an uncaught exception. Here are simplified example attempts:

Situation A: Try Catch surrounding entire block:

    List<Task> parallelTasks = new List<Task>();  
    try
    {  
        for (int i = 0; i < numberOfTasks; i++)
        {
          Task task = context.CallActivityAsync("ActivityWhoFailes", i);
          parallelTasks.Add(task);
        }
        await Task.WhenAll(parallelTasks);
    }
    catch (Exception exception)
    {
       log.LogWarning(exception, exception.ToString());
    }

Situation B: Try Catch Individually

    List<Task> parallelTasks = new List<Task>();  
    for (int i = 0; i < numberOfTasks; i++)
    {
        try
	    {  
	        Task task = context.CallActivityAsync("ActivityWhoFailes", i);
	        parallelTasks.Add(task);
        }
           catch (Exception exception)
        {
           log.LogWarning(exception, exception.ToString());
        }
    }
       
    try
    {
       await Task.WhenAll(parallelTasks);
    }
    catch (Exception exception)
    {
	    log.LogWarning(exception, exception.ToString());
    }

Try Catch Embedded into a function

    List<Task> parallelTasks = new List<Task>();  
    for (int i = 0; i < numberOfTasks; i++)
    {
        Func<Task> run = async () =>
        {
	        try
		    {  
		        return await context.CallActivityAsync("ActivityWhoFailes", i);
	        }
	           catch (Exception exception)
	        {
	           log.LogWarning(exception, exception.ToString());
	        }
	    }
	    Task<ActivityFunctionStatus> task = run(); 
	    parallelTasks.Add(task);
	    
    }
       
    try
    {
       await Task.WhenAll(parallelTasks);
    }
    catch (Exception exception)
    {
	    log.LogWarning(exception, exception.ToString());
    }

I have also tried these situations with a RetryOptions option in conjunction with CallActivityWithRetryAsync.

Update based on more research:
Is it the case that an error is still output into the logs when it is thrown in an activity function yet marshalled in an orchestration function? I have noticed that when I write the code above other activity functions still execute; however, if I remove the try/catch, the issue reports to the logs and all execution seems to halt.

[cgillum]

The error handling experience for fan-out/fan-in in Durable Functions is the same as it would be for a normal C# app since Durable Functions are just using C# tasks for orchestration.

Is it the case that an error is still output into the logs when it is thrown in an activity function yet marshalled in an orchestration function?

Yes, we log activity function errors immediately after an activity fails, regardless of whether the orchestration attempts to handle the exception. If the orchestration doesn't handle the exception, then you'll see a second error message in the logs, this time for the orchestrator function itself.

if I remove the try/catch, the issue reports to the logs and all execution seems to halt.

Removing the try/catch would cause your orchestrator function to fail as part of the await Task.WhenAll(parallelTasks). My understanding of Task.WhenAll is that it doesn't throw until all the tasks complete or fail, (source) so I don't know why you'd see any difference between the case with try/catch and the case without it (unless you have some other tasks running in the background).

[PaulKruger]

Thank you. In my actual code (not the simplified version exposed here), there was another problem using a string to integer conversion in the for loop (Int32.Parse), which must hijack and dispose of my iterator variable (i) and compounded my troubleshooting.

That said, it would be nice then if the documentation here ([https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=csharp]) called that out and referenced the c# task error handling documentation. Even better if it contained an example.