Would it be a good idea to create an Activity function just for API based logging?

[a3y3]

I have a custom logging implementation that sends a log to an external service via an HTTP POST. I understand that orchestration functions should never do any kind of IO, but I'm a little confused as to how I should log stuff out to my service in that case.

Would it be overkill to create an activity function that just sends a message to the logging API via an HTTP client? Note that I will essentially be calling this activity whenever I want to log debug/info/error messages from my orchestrator function. Is the overhead of starting an Activity high compared to how lightweight (asynchronous, and non blocking) logging should be?

[cgillum]

This feels like a bit of overkill. We typically see most users either use Application Insights for logging or use dependency injection to plug in their own loggers which intercept the normal log statements from functions and publish it somewhere else. This approach tends to scale better since logging statements are non-blocking and logs can be pushed out in batches, outside the main orchestrator function invocation.

What language are you using to develop your apps? Have you considered using the built-in logging infrastructure of Azure Functions?

https://docs.microsoft.com/en-us/azure/azure-functions/functions-monitoring

[a3y3]

Well - our team uses the external logging API for all our other services so switching away from that isn't an option. My concern with respect to the "overkill" was purely from a performance standpoint - I had this idea that Activity functions take a considerable time to spin up, and I wasn't sure if it was the right approach for something as lightweight as logging.

I do notice however, that the docs don't mention how long activity functions take to start. is there some data on this? If they spin up quickly, I think I'll be fine with implementing our API logger this way.

[fiddlerpianist]

So here's our situation in a little more detail (a3y3 and I are on the same team).

We are kind of the pioneers here when it comes to using Functions at our company, and our current deployment environment is highly regulated and very k8s-centric. We aren't given access to Azure Portal for production resource groups at this time, so almost all of the visibility we do get comes in through whatever we log in Kibana. Could we potentially hook up Application Insights to an Event Hub to be consumed by a component which could feed it into the index? Absolutely, but we aren't there yet. That kind of work would have to be taken up with the platform infrastructure group, and it's not even on their roadmap. The only viable alternative in the short term is to HTTP POST using the exposed API from the stack.

Now, couple this with the fact that some of these API calls are being made from with an orchestration Functions, where performing I/O is strongly discouraged and can cause issues with scheduling orchestrators if done from within that Function (even if you async them). In fact, we're seeing very odd, sporadic behavior where an orchestration will be scheduled but then will never be started. Last week, we waited 6 hours for an orchestration to start, before which we were seeing scheduling delays of 45+ seconds. Restarting the host immediately fixed the problem. While we don't really have any idea why that happened, it does seem conceptually possible at least that our current unawaited-but-async logging API calls could have caused scheduling backups, particularly if the downstream API were to have become temporarily unresponsive. We would like to reduce the risk of that happening in our current situation.

We are using C# .NET 6 on Azure Functions 4.

[cgillum]

You certainly could use activity functions. The overhead associated with them is the time it takes to enqueue a message plus the time it takes to dequeue that message before executing the code, not to mention the resources required to serialize and deserialize the activity function inputs. Activity functions can execute quite quickly (less than 100ms) but the actual time it takes may depend on whether the work-item queues which trigger activity functions start getting backed up.

For your situation, I certainly wouldn't recommend going through Application Insights. Rather, you may want to consider a custom logging service that gets injected into the Functions host. This is an option for you if you're using .NET to implement your apps. More information on working with Azure Functions logging abstractions can be found here. You still need to be careful about not doing I/O in the orchestrator function directly, but this at least allows you to have a common abstraction for logging that can be used anywhere (see here for how you can safely use an ILogger in an orchestrator function).

It's hard to know exactly what you're seeing sporadic behavior in your orchestrations. Doing I/O (especially async I/O) is one potential cause. To narrow down the issue, I suggest starting by moving the API-based logging code into activity functions first to see if that resolves the issue, then look further into the longer-term strategy for logging.

[fiddlerpianist]

This is really helpful. Thank you!

Can you briefly explain why it is save to use any ILogger implementation (from an I/O perspective)? I get how it can be made replay safe, but I'm not sure how it can be made I/O safe from an orchestration function...or how that would be different than doing async I/O inline like we are doing today.

[cgillum]

@fiddlerpianist sorry, you are correct - the link I shared shows how to have a replay-safe logger but not necessarily an I/O safe logger. In my head what I envisioned was that the custom ILogger you had simply writes to an in-memory queue of log messages (non-blocking). Separately, some background thread running as a service you register in the Functions host dequeues these log messages and publishes them to your HTTP endpoint.