merge queue: embarking main (1d078fb) and #6614 together

docs/contributing/999-rfcs/2024-06-02-ts-client.md

@@ -0,0 +1,336 @@
+---
+title: "Exported TypeScript Clients Across Wing Applications"
+description: Exported TypeScript Clients Across Wing Applications
+---
+
+# Exported TypeScript Clients Across Wing Applications
+
+- **Author(s)**: @MarkMcCulloh
+- **Submission Date**: 2024-06-02
+
+## Use Case
+
+Given some infrastructure defined in wing by platform engineers, we want to provide a way for application developers in the same organization to interact with that infrastructure in a type-safe way from their own codebases without dealing with wing or infrastructure themselves.
+
+As a platform engineer I'd like to use wing to create the preflight infrastructure for developers in my organization, and then produce client libraries in TypeScript for accessing that infrastructure within other pieces of code.
+Imagine for example that the application developers write TypeScript running in containers in their own repo, and that repo's code is already being automatically deployed with GitOps via ArgoCD or other means.
+
+As an application developer I'd like to consume the TypeScript package published by the platform team and write application code that interacts with these resources (using whichever methods or permissions have been granted by the platform team). If I'm running my TypeScript code / container locally I expect the wing clients to connect to the wing simulator and interact with the simulated resources, and when my code is deployed to the AWS I expect the clients to connect to the real cloud resources.
+
+## Proposal
+
+The `typescript` module has a `Client` class used to create a client. When this is instantiated, `wing compile` will produce a publishable JavaScript npm package with TypeScript type declarations.
+
+```wing
+bring clients;
+
+let client = new typescript.Client(
+  name: "@acme/infra-client",
+  // version: "0.0.0",
+  // outdir: join(@dirname, "dist"),
+);
+```
+
+To lift a resource into the client to make it accessible to consumers, add a `@lift` expression:
+
+```wing
+bring cloud;
+
+let bucket = new cloud.Bucket();
+
+let client = new typescript.Client(
+  name: "@acme/infra-client",
+  lifts: @lift({ bucket: [put] }),
+);
+```
+
+In order to access the lifted objects at runtime, the TypeScript client needs to know how it should obtain the initialization data for instantiating the inflight clients (e.g. resource ARNs and other preflight state). This information is stored in `client.context` in preflight.
+When the client is used later (inflight), the environment variable `WING_CONTEXT` is expected to the same value previously stored in `client.context`.
+
+The context is just a string value (base64 encoded JSON), so you can store it in any way that is convenient for your deployment process.
+
+For example, it could be stored in a bucket:
+
+```wing
+bring cloud;
+bring typescript;
+bring fs;
+
+let client = new typescript.Client(
+  name: "@acme/infra-client",
+);
+
+// The deployer of the client host will need to retrieve/link this data later
+let clientContext = new cloud.Bucket();
+clientContext.addObject("ctx", client.context);
+```
+
+When compiling to non-simulator targets, the client will store the expected permissions to be used elsewhere.
+For example, in `tf-aws`, the client will generate a policy to be attached to a role or user. The name of the policy is required to be set in the `wing.toml` file.
+
+```toml
+[tf-aws]
+client_iam_policy.MyClient = "AcmeClientPolicy"
+```
+
+After the client is generated and published, it can be used in any TypeScript project:
+
+```ts
+import { Client } from "@acme/infra-client";
+
+// Will use the environment variable WING_CONTEXT to resolve the context
+const client = await Client.new();
+
+// Optionally, you can pass the context directly to bypass the environment variable
+// const client = new Client({ context: "..." });
+
+await client.bucket.put("key", "value");
+```
+
+The client code is the same as the generated javascript as if it were used within wing itself!
+
+## Implementation Details
+
+The section below discusses the implementation details of the proposed feature. They are concepts that are intentionally not exposed directly to users.
+
+### Client context data
+
+The client context data is a JSON object that contains all the necessary information to connect to the resources.
+It's read through the environment variable `WING_CONTEXT` and is expected to be a base64 encoded.
+
+For now, the only information that needs to be stored are environment variables.
+This corresponds to the only API available on `IInflightHost`, which is `addEnvironment`, which is how these values are added:
+
+```json
+{
+  "version": "1",
+  "env": {
+    "BUCKET_HANDLE_a107cec5": "abcxyz"
+  }
+}
+```
+
+The client itself is effectively a proxy of an unseen/unavailable `IInflightHost`, so any expressivity that interface has must be handled by the client.
+
+### Permissions for the simulator
+
+The client is effectively an `IInflightHost`, so it any resource lifted into it registers the permissions it needs.
+This is implemented on a per-target basis to expose the permissions in a useful way. In the simulator, the client resource itself is given permission to access other resources so it can be handled automatically.
+Any usage of the client itself (with the proper context) calls into the simulator as if it is the client resource.
+
+### Type generation with `@lift`
+
+Wing's compiler has type information available for use, but only before preflight execution occurs.
+This makes an API like the following not feasible to implement using logic similar to the existing `extern` or `@inflight`:
+
+```wing
+client.lift("bucket", bucket, ["put"]);
+```
+
+While executing preflight, the knowledge about the type of the `bucket` instance is lost, so generating the actual client code will have no way to reference it.
+
+This RFC proposes changing the existing `lift` syntax as a way to capture the necessary type information during preflight.
+An important feature of this syntax is that it's entirely statically analyzable, so anything we need to do with it can be done before-preflight.
+The existing `lift` syntax is currently intended to be used only inside inflights to explicitly lift preflight data into the defined scope:
+
+```wing
+let b1 = new cloud.Bucket();
+inflight () => {
+  let var b = b1;
+  lift { b: [put] } {
+    b.put("key", "value");
+  }
+}
+```
+
+With some changes, with syntax could also be used as a way to create a preflight container for lifted data.
+
+```wing
+let b = new cloud.Bucket();
+let lifts = @lift({ b: [put] });
+//  ^^^^^ is a std.LiftQualifications
+```
+
+This is very similar to the `lift` mechanism in `@wingcloud/framework`
+
+```ts
+const b = new cloud.Bucket(app, "Bucket");
+const lifts = lift({ bucket }).grant({ bucket: ["put"] });
+```
+
+In the [RFC for explicit lift qualifications](https://github.com/winglang/wing/blob/main/docs/contributing/999-rfcs/2024-03-14-explicit-lift-qualification.md), there is a mention of a `std.LiftQualifications`.
+
+If this data structure is augmented to also include type information, it could be used to generate the client during preflight.
+The resulting data would look very similar to the existing _liftMap, with the addition of the link to the `.d.ts` generated by the compiler for the aggregate type information.
+
+### Client Contents
+
+The client package will contain the following:
+
+- package.json
+
+Minimal package.json with the name and version. For MVP, the package will not have any dependencies because the client code is self-contained.
+
+- index.js
+
+Exports the `Client` class. This class has a static async method called `init` that returns the lifted objects (`_toInflight`) using the data in environment variable `WING_CONTEXT` (or function argument).
+
+- index.d.ts
+
+Exports the `Client` class, with fields for each lifted object. Each field will reference a type from `lifts.d.ts`.
+
+- lifts.d.ts
+
+The resulting types from `@lift`
+
+## Example Usage
+
+### EKS
+
+This walks through the setup of a client by a central platform team for an EKS cluster in AWS.
+
+> **Note:**
+> In this example, wing is also used to manage some of the kubernetes manifests. This is not a requirement, but greatly shortens the example
+> and is a recommended setup. If not, imagine all the data from wing was simply exported (e.g. `cdktf.Output`) and managed in kubernetes manifests manually.
+
+We start by creating the client and lifting a bucket:
+
+```wing
+bring cloud;
+
+let bucket = new cloud.Bucket();
+
+let client = new typescript.Client(
+  name: "@acme/infra-client",
+  lifts: @lift({ bucket: [put] }),
+);
+```
+
+To extract the necessary permissions, we set up the wing.toml file to create a new policy:
+
+```toml
+[tf-aws]
+client_iam_policy.MyClient = "AcmeClientPolicy"
+```
+
+After this policy is created, a role and associated service account must be created for application pods to use it.
+This is only needed for `tf-aws` deployment so it can either be inside a check to the `WING_TARGET` environment variable or in a separate class/file:
+
+```hcl
+data "aws_iam_policy" "client_policy" {
+  arn = "arn:aws:iam::123456789012:policy/AcmeClientPolicy"
+}
+
+resource "aws_iam_role" "client_role" {
+  name = "client_role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Action = "sts:AssumeRole"
+        Effect = "Allow"
+        Principal = {
+          Service = "eks.amazonaws.com"
+        }
+      },
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "client-attach" {
+  role       = aws_iam_role.client_role.name
+  policy_arn = aws_iam_policy.client_policy.arn
+}
+```
+
+In the cluster, and ServiceAccount and ConfigMaps can be used to make all the relevant configuration and permissions available to any Pods.
+Upon deployment of the centralized infrastructure, the role ARN and context can be outputs available for the manifests to use:
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: acme-client
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:policy/AcmeClientPolicy
+
+---
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: client-ctx
+  data:
+    ctx: ZGF0YSAiYXdzX2lhbV9yb2xlIiAiZXhhbXBsZSIgewogIG5hbWUgPSAiYW5fZXhhbXBsZV9yb2xlX25hbWUiCn0=
+
+```
+
+The data can also be mapped with dynamic methods like SSM or buckets, but that would incur a networking cost.
+
+This finishes the setup of the client and the responsibilities of the platform team, so we now switch over to the application developers.
+They can define a pod with the client context and service account in the namespace they've been alloted:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: app
+  namespace: app
+  spec:
+    serviceAccountName: acme-client
+    containers:
+      - name: app
+        image: app-image
+        env:
+          - name: WING_CONTEXT
+            valueFrom:
+              configMapKeyRef:
+                name: client-ctx
+                key: ctx
+```
+
+And now `app-image` can use the client to interact with the bucket!
+
+To run locally, the platform team can provide a script to run the core infrastructure (`wing it`) run a 
+local cluster with all the same expected config maps and service accounts to just work.
+
+### AWS Lambda
+
+This walks through the setup of a client by a central platform team for application teams deploying to AWS Lambda.
+
+```wing
+bring cloud;
+bring clients;
+bring util;
+bring "@cdktf/provider-aws" as aws;
+bring "cdktf" as cdktf;
+
+let bucket = new cloud.Bucket() as "shared";
+
+let client = new typescript.Client(
+  name: "@acme/infra-client",
+  lifts: @lift({ bucket: [put] }),
+);
+
+// Set up an SSM parameter to store the context
+new aws.ssmParameter.SsmParameter(
+  name: "/acme/infra-client/context",
+  type: "String",
+  value: client.context,
+);
+```
+
+To extract the necessary permissions, we set up the wing.toml file to create a new policy:
+
+```toml
+[tf-aws]
+client_iam_policy.MyClient = "AcmeClientPolicy"
+```
+
+Now the application developers need to setup the lambda with two steps:
+
+1. Map the SSM parameter as an environment variable, which is a directly supported feature in AWS
+2. Attach the created policy to the lambda role
+
+For local development, the owners of the central infrastructure can provide a script to lookup the SSM parameter and add the environment variable to the lambda either via an env or whatever deployment process being used.