Feature request: managed task hubs #1757
Replies: 11 comments 7 replies
-
|
Hi @vigouredelaruse. Thanks for this feedback. As you rightly point out, the developer has the burden of ensuring that there is no PII leak into the storage account since that account is owned by the developer. You mention that you'd prefer Durable Functions be more "opaque" such that you don't have to worry about such concerns. Is it fair to say that having the Azure Functions manage the underlying storage account for you (making this our burden instead of yours) would satisfy your concern? |
Beta Was this translation helpful? Give feedback.
-
|
hi @cgillum thanks for looking into it based on the documentation i referenced (sorry for being all legal about it here - one does have to careful what one asks for) and for my particular implementation, the functions runtime's use of 'opaque infrastructure' storage account resources would resolve my concerns, and the concerns of users of a docker appliance i'm implementing using durable functions and entities in this case, the appliance will enumerate storage accounts, retrieve keys and enumerate tables and entities all with the caller's token i would prefer the token to disappear from the user's browser session into an opaque infrastructure for delivery to the function endpoint. the persistence of such tokens would serve as an impediment to this implementation's survival of a rigorous risk vector review. obviously for the implementation i'm concerned about, all the above applies to the leakage of the storage account keys and other table entity meta-data via orchestration history at the very minimum it seems we're discussing something akin to managed disk vs unmanaged disk. and there's the pricing rub. one may need to be careful here not to produce a class system, where people pay for the security they can afford, and everybody still has to code defensively it's relevant to consider the situation over at azure devops, where public repositories are apparently under abuse, and have to be requested, whereupon the abuse shifted to private repositories. it's impossible to make anything foolproof because of course, fools like me are so ingenious |
Beta Was this translation helpful? Give feedback.
-
|
@cgillum couple of thoughts
I’m thinking similar to my idea of passing references to large object graphs, what if we pass around references to key vault secrets instead?(is this actually a KeyVault input/output binding?!?!) - Onus is on the customer to ensure all secrets are in KV, and customer manages the lifecycle of those secrets in KV. Imo this provides a stronger security story than opaque TaskHubs. Not to say that both ideas couldn’t be combined for even more depth of security. |
Beta Was this translation helpful? Give feedback.
-
|
@olitomlinson yeah, this is something I'd love to do (TaskHub RP), I just need to figure out how to make the business case for it to my management. :) Secrets is interesting. I can't really think of a scenario where it would make sense to pass secrets around an orchestration though. It feels like that scenario would be better served using input/output bindings for KeyVault on individual activity functions. |
Beta Was this translation helpful? Give feedback.
-
|
@cgillum FYI I've just seen that Dapr has an input binding for retrieving secrets, and one of the implementations that that the Secret Component has is a KeyVault implementation. This is exactly what I was calling for, but of course without the dependency on Dapr. namespace dotnet_azurefunction
{
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Dapr.AzureFunctions.Extension;
using System.Collections.Generic;
public static class RetrieveSecret
{
/// <summary>
/// Example to use Dapr Service Invocation Trigger and Dapr Secret input binding to retrieve a saved state from statestore
/// </summary>
[FunctionName("RetrieveSecret")]
public static void Run(
[DaprServiceInvocationTrigger] object args,
[DaprSecret("kubernetes", "my-secret", Metadata = "metadata.namespace=default")] IDictionary<string, string> secret,
ILogger log)
{
log.LogInformation("C# function processed a RetrieveSecret request from the Dapr Runtime.");
// print the fetched secret value
// this is only for demo purpose
// please do not log any real secret in your production code
foreach (var kvp in secret)
{
log.LogInformation("Stored secret: Key = {0}, Value = {1}", kvp.Key, kvp.Value);
}
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
A somewhat lightweight way to get secrets in Azure Functions is KeyVault References. Then you could just read it from an environment variable in activity function. It has it's own downsides, like I believe that the app will be recycled when a secret updates to make sure it gets the latest value, but I think that is the closest thing we have for now. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @ConnorMcMahon There is a world of difference between accessing a handful of static keyfault settings via AppSettings/Configuration that are declared at the infrastructure level (ARM) and then accessing potentially thousands of KeyVault secrets at runtime dynamically via the potential of a KeyVault input binding. In my use-case I had a multi-tenant orchestration that needed to make HTTP calls to customers own 3rd party endpoints beyond the bounds of my system. I had to store their auth credentials along with the rest of the data that flowed around the orchestration think of a model like this used as {
"customer" : "11111",
"endpoint" : "http://contoso.com/api/v1/handler",
"bearer" : "29fi203i0293i23i9fi2093if"
}It would be far safer for me to include a KV reference in the model, instead of a plain text bearer token (which will then get serialised into the TaskHub eventually...) {
"customer" : "11111",
"endpoint" : "http://contoso.com/api/v1/handler",
"BearerKeyVaultReference" : "https://myvault.vault.azure.net/secrets/customer-11111/"
}Then in my DF Activity I could use something like public static class CallCustomerEndpointActivity
{
[FunctionName("ActivityA")]
public static void Run(
[ActivityTrigger] CustomerEndpointData data,
[KeyVaultSecret(Reference = "data.BearerKeyVaultReference")] string secret,
ILogger log)
{
using (var requestMessage = new HttpRequestMessage(HttpMethod.Get, data.endpoint))
{
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", secret);
await httpClient.SendAsync(requestMessage);
...
}
}
}Now imagine doing the above but for thousands of different customers with different endpoints in a multi-tenant system. Hope that helps clarify :) |
Beta Was this translation helpful? Give feedback.
-
|
on the proactive risk management, regarding the proposed 'managed task hub' sku please don't forget disaster recovery and the need to promote security by default on the configuration blade. by this i mean
|
Beta Was this translation helpful? Give feedback.
-
Few ideas for the business case :
I would actually fully get behind a KeyVault input/output binding for Functions, so that I can resolve secrets dynamically at runtime per function call, and create new secrets in the same vein. I don’t think there is a KV binding? Where is the best place to raise this as a feature request? |
Beta Was this translation helpful? Give feedback.
-
i welcome your thoughts on the following. consider a potentially long-running situation where you have to mutate an unknowable number of table entities in an arbitrary user's storage account. for easy deployment you want to avoid having to keyvault creds for every single storage account users want to operate against. also dynamic environment discovery, starting with the user's auth token prevents the app from having hardcoded god permissions, albeit backed by keyvault. so the user's permissions enable activity. because of auth method impedance mismatch you decide to leverage impersonation to read the service keys of storage accounts within the user's rbac scope. you decide to embed this key retrieval in a function app orchestration, next to where they'll be used you don't want to post the keys to the orchestration for good reasons. but now you have locally scoped variables in an orchestration referring to secrets like table names and entity lookup keys. and then you pay closer attention to task hubs and realize you have a problem and now thanks to you we're discussing this as a feature request |
Beta Was this translation helpful? Give feedback.
-
|
I totally understand using RBAC / Managed Identity to discover storage accounts attached to an AAD User and subscription and list out the keys as secrets. I agree this can be done in an Activity. in the Activity, if there was such thing as a KeyVault output binding available at the point of listing out the keys, we could easily create KeyVault secrets and then pass references back to the orchestration. Subsequently, when it’s time to use those secrets we would use the KeyVault input binding in the Activity also. — Functions support for KeyVault comes from a different world where secrets are configuration driven constants accessible from app settings etc I think the problem here is that dynamically creating and reading secrets in Functions is not a well supported story, so people are put off completely from using it. When IMO, if KV was much easier to utilise from a Function it would get more adoption. |
Beta Was this translation helpful? Give feedback.
-
|
a v0.5 slightly contrived use case, a microservice supporting a subscriber signup website (hosted anywhere) that
for v1 obviously we go for data insertion/retrieval on-prem proposed solution
the walkthrough
must mention implementation details the wins
division of realms
gimmie some sausages widdit alternatively the implementer gets to manage the deployment of sdk packs. not their own personal ones of course, but those sourced from the private vetted repo oh all right maybe you're a .gov or a .mil or .edu and your secrets are associated with a main frame where you keep your vector output dataset. and uh, you don't trust the interconnect code in the sidecar, so you want to own the datapath with your own en plus, this also suggests on-demand builds of a tailored sidecar, driven by a ci/cd pipeline that must support gosh, who knows how many instances hmm. this starts to look like the spec for |
Beta Was this translation helpful? Give feedback.
-
|
associated work shows this is a nascent market segment dominated by ambassador pattern solutions https://www.infoq.com/news/2021/04/cncf-radar-secrets-management/ |
Beta Was this translation helpful? Give feedback.
-
|
perhaps you're talking about a different product. consider: if the ambassador pattern (that what it is?) is embedded in the keyvault mythos (and indeed the pricing/transaction suggests lowish expected transaction rates) and instead you want arbitrary rates of secret production and consumption then maybe hosting your own keyvault might seem like good horse sense when you're fine with sitting grim faced at kubectl>. your idea starts to sound more like a 'opaque infrastructure' azure/kerberos key distribution center or something like it, where secret providers have endpoints (and transaction rates are prorated in the zillions) that can be surfaced by queue to a [binding] or something else like it that sounds like it involves a multitude of cross-cutting concerns. so- those concerns that i can see are pairing/paring the cost of the queue and the connector mechanism for all the secret repositories supported, and making this a default tls required choice for getting at your client secrets and certificate thumbprints everywhere on the internet uhhh - you could start the chain of trust with an auth token associated with a suitably armoured public/confidential client app. or is that firewall penetrating broker client. um and zone certificate authorities i'm not so sure this is just theoretical computer science. if i'm not mistaken there are doors slamming shut on the internet and security is a big deal |
Beta Was this translation helpful? Give feedback.
-
|
I am very intrigued by this conversation. Since this feature request is still pretty hazy and needs some more fleshing out, I'm going to move it to the new discussion format. |
Beta Was this translation helpful? Give feedback.
-
|
@cgillum ditto - let's say you're supervising a team of workflow developers building out functions. as it is one would have to account for human error when working with sensitive data, to a different degree than a normal webapp deployment. sure it's a skill to have but if one could factor the human error part out of the equation i'm certain it would harden deployments against secret loss. so at this point in the discussion i can still see the need. it would be interesting to know if people other than the two of us did |
Beta Was this translation helpful? Give feedback.
-
|
congrats to the functions team for noticeable plaudits from forrester here https://visualstudiomagazine.com/articles/2021/03/31/forrester-faas.aspx as per |
Beta Was this translation helpful? Give feedback.
-
|
as well, the following really needs to be called out
the argument trends to suggesting infrastructure is not just under attack, but creaking like crazy |
Beta Was this translation helpful? Give feedback.
-
a durable function orchestration places an onus on the developer to avoid pii leakage - see the additional context below from here https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-serialization-and-persistence?tabs=csharp
the summary is that function inputs may be persisted to a storage account visible to the tenant where a function app is deployed
Describe the solution you'd like
i'd prefer not to have to censor the various kinds of IDs and/or command channel data i send to my durable function orchestrations. other kinds of web apps deployed to azure app service don't bring quite such a visceral risk of implementation detail exposure to outsiders.
Describe alternatives you've considered
the mitigations are already described at the azure docs i reference here. what i'm suggesting is that the perhaps the need for mitigations ought to be factored out, given the increasingly grim security situation on the information highways of the internet
Additional context
the durable orchestration framework is not quite as opaque as other azure offerings. for example, i cannot directly examine the storage subsystem of azure ad without microsoft employee credentials. perhaps it should be so with durable functions
note that except perhaps for .mil and .gov type customers i do not believe these comments should be used as a justification for the eradication of all but the consumption plan, which is suitably opaque, but risky fiscally in the case of a ddos attack on a consumption plan endpoint for example
FROM THE DOCUMENTATION
The following is a list of the different types of data that will be serialized and persisted when using features of Durable Functions:
All inputs and outputs of orchestrator, activity, and entity functions, including any IDs and unhandled exceptions
Orchestrator, activity, and entity function names
External event names and payloads
Custom orchestration status payloads
Orchestration termination messages
Durable timer payloads
Durable HTTP request and response URLs, headers, and payloads
Entity call and signal payloads
Entity state payloads
Working with sensitive data
When using Azure Storage, all data is automatically encrypted at rest. However, anyone with access to the storage account can read the data in its unencrypted form.
Beta Was this translation helpful? Give feedback.
All reactions