Feature Programmatic Inspection of Diff

[moofish32]

The Diff operation is very useful. Users may want to control or conditionally apply a ChangeSet based on the Diff information. However, we need to expose that information to an API.

For example, if I have a rolling update type of ec2 deployment a diff that replaces a Launch Configuration is expected and the deployment should proceed. However, a diff that replaces a subnet is likely not safe and should require a human review. There are many use cases like this that become available with programmatic access to the diff results.

[eladb]

That's an awesome idea. At the minimum, perhaps we can just expose a --json flag for cdk diff that will return a result in JSON format so that downstream tools can inspect.

[moofish32]

At the minimum, perhaps we can just expose a --json flag for cdk diff that will return a result in JSON format so that downstream tools can inspect.

Great idea and quick to get some feedback with jq etc.

[hlascelles]

We would very much like this.

Actually, just in general, we would like to do cdk diff --json or similar. Sometimes we see a diff that shows us a resource is to be created, but not "what is in it". Is it possible to see the full raw stack JSON diff?

[awsiv]

Would greatly benefit from this as well. Terraform has a terraform show command which is super useful. Eg _terraform show -json tfplan.binary > tfplan.json _

I used the output to detect if there were any changes in RDS for example and create a snapshot. Might also help integrate with other tools such as OPA

[akash1810]

I've been wanting this too and have devised two potential solutions (barring first class support from CDK):

1. Performing a cdk deploy setting --no-execute and then using the AWS CLI to describe the change set aws cloudformation describe-change-set. This requires fairly detailed knowledge of the change set json schema.
2. Calling diffTemplate and processing the TemplateDiff response. This function is ultimately what's driving the cdk diff command.

I favour 1 as it provides a more complete answer. In 2, we are comparing templates. What this doesn't tell you is if any resources will be replaced - there are some scenarios where resource replacement is undesired. The change set includes this information, in addition to the cause (some of which is bizarre, such as security group description changes yielding replacement!).

To add a bit more context, my use case is wanting to convert a YAML defined stack into a CDK stack. The best process for doing this that I've seen is described here:

1. Create an empty CDK stack
2. Use a CfnInclude to add the YAML
3. Start moving things from YAML to CDK

I'm wanting to add a step between 2 and 3: perform a diff of the now CDK defined stack and the running infrastructure. If the diff is 0, then it is safe to deploy and continue to step 3.

Would love to hear if the AWS team have more thoughts on this.

[skinny85]

Hey @akash1810 ,

a few minor points:

1. Make sure to use the CfnInclude class from this module, and not the one from the core module (which is deprecated).
2. I think you can achieve that gate you mentioned by using the --fail flag to cdk diff.

Hope this helps!

Thanks,
Adam

[akash1810]

Thanks for the response.

1. Make sure to use the CfnInclude class from this module, and not the one from the core module (which is deprecated).

Ah yes! I am using CfnInclude from @aws-cdk/cloudformation-include - the link I shared was just the first returned in search. Oops!

2. I think you can achieve that gate you mentioned by using the --fail flag to cdk diff.

From what I understand, the --fail flag for the diff command only changes the exit code.

The questions I'm looking to answer during a YAML to CDK migration are:

• Will this change result in resource replacement?
• If so, would the replacement cause service interruption?

Additionally, I'd ask these questions on each iteration within step 3.

AFAICT only change sets can provide complete answers here, so adding the --fail flag to diff doesn't quite work as diff does template comparison.

[skinny85]

But isn't the idea to include the template, run the diff, see if there are any differences - if there are, fix and repeat, if not, you are done?

Migration should be a one-time thing, not something that needs to be done constantly.

[akash1810]

But isn't the idea to include the template, run the diff, see if there are any differences - if there are, fix and repeat, if not, you are done?

Not quite in our use-case. The ambition is to stop using YAML entirely, meaning "done" is achieved when each YAML resource has been replaced by a CDK construct/pattern and the YAML file has been deleted.

CDK is opinionated. The way it, for example, grants access to a bucket doesn't necessarily match the way access was granted in the YAML template. Whilst we could create a custom CDK construct that matches the YAML version, using the CDK provided policy is easier to maintain (read less code!).

Consequently, we have found it simplest, safest and easiest to follow and review changes, when a single resource is moved at a time. This means our cycle is:

• Import YAML
• Perform diff - if 0, deploy and continue
• Migrate a resource from YAML into CDK
• Create a change set - confirm any replacements are safe (i.e. won't cause service interruption), when happy, deploy and continue
• Migrate another resource from YAML into CDK and repeat until YAML template is empty

As described above, comparing templates provides a limited amount of confidence - only at runtime do you know for certain if a particular change would result in a running resource being updated or replaced. (You don't want stateful resources such as databases, load balancers or DNS being unintentionally replaced).

More context
To add (a lot) more context, at the Guardian the vast majority of our stacks today are defined in YAML. This has served us pretty well, but YAML is only so good. For example, YAML templates have:

• A long feedback loop as errors are (largely) first seen at runtime (template validation doesn't catch all errors)
• A tendency to be copy pasted as YAML templates are finicky to write (see above). The template that is copied from isn't necessarily following the latest best practices either.

CDK directly solves these two problems (and more!). We're building a library of components and patterns that encode our best practices. When best practice evolves, we publish a new version of the library and update downstream repository's package.json files (sometimes automatically via Dependabot, depending on the change). We think this is the best and simplest way to ensure all infrastructure across the estate is adhering to latest best practices (it's not uncommon for our YAML templates to be written, deployed and forgotten).

Summary
In short, if one wants to replace a YAML template entirely with CDK, we've found it easiest to move resources one at a time. Using change sets each time to be sure we're happy with the CDK defined resource as it's unlikely to be 100% equivalent to the YAML defined resource but still yields the same result.

[skinny85]

@akash1810 I wonder if you realize that cdk diff also tells you if replacement will happen?

Here's some example output:

[~] AWS::Lambda::Permission RestAPi/Default/{proxy+}/ANY/ApiPermission.HackathonExampleStackRestAPi5AD17B30.ANY..{proxy+} RestAPiproxyANYApiPermissionHackathonExampleStackRestAPi5AD17B30ANYproxy43EDB9E2 replace

So maybe you can simplify the cycle?

(I assume you've already completed step 1, and have the template included in your CDK code, and deployed with the include)

1. Move a resource from YAML to CDK.
2. Run cdk diff.
3. See if there are replacements in the output.
4. If no, commit, push, and get it deployed through your pipeline.

Does this make sense?

[akash1810]

Oh gosh, you're totally right! I've just performed a YAML to CDK migration using totally vanilla AWS CDK and diff works exactly as you describe. I still have something awry somewhere, but it is not within the diff command. Please ignore most of what I previously said, and apologies for any noise it's created!

JSON output from diff for consumption by downstream tools would still be a nice feature, however I'm not blocked by it.

Thank you for your patience @skinny85 !

[automartin5000]

This would really help with building approval workflows. +1 for --json

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[meleksomai]

Hi team,

This will be an important feature to include. We are currently leveraging CDK Deploy through GHA, and we like to include custom checks based on the results of the DIFF. With the current logs of the CDK Diff, it is very hard to parse. JSON, YAML, or any machine-readable format will be fine.

[omriman12]

How can it be that this capability hasn't progressed in 5 years

[evgenyka]

We will handle it as part of aws/aws-cdk-rfcs#300

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.