CloudFormation Rollback Failed Exercise

This codebase uses the AWS CDK to deploy resources to your AWS account and set up an exercise where you experience a CloudFormation UPDATE_ROLLBACK_FAILED error and need to resolve it.

Motivation

This exercise was created because I too frequently see people deleting CloudFormation stacks to resolve these errors, instead of resolving them and continuing the rollback. In non-production environments these actions are often frustrating, but aren't considered an incident. In production environments deleting a Stack often has far more impact.

Cost

This exercise should be within the monthly free-tier usage limits for the Lambda service. There should be no charges associated with the CloudFormation service.

References:

• AWS Lambda Pricing
• AWS CloudFormation Pricing

Setup

You will need Node.js v20 and Git installed. You will also need access to an AWS account, to be safe you should have administrator access. You can probably complete this exercise with lower access levels, but due to the number of potential configurations I can't say that for certain.

Codebase Setup

1. Using git, clone this respository to your machine. The command below will only clone the most recent code, since you don't need the code's entire history for this exercise.

   git clone --depth 1 https://github.com/jbutz/aws-cfn-rollback-exercise.git

2. Install the node modules using NPM

   npm install

Exercise Setup

1. Configure your command line to have access to your AWS account.
2. If it is not already set, set the AWS_REGION environment variable to the AWS Region you want to use, it will make things easier

   # On Linux and Mac you can use the command below to set the region to US East 2 (Ohio)
   export AWS_REGION=us-east-2

3. Bootstrap the AWS CDK into your account and region

   npm run bootstrap

4. Deploy the application into your AWS account using the CDK. The deployment will ask your permission to make IAM changes. You must allow these to deploy the exercise.

   npm run deploy

5. Without making any code changes, deploy the application again. This will cause the UPDATE_ROLLBACK_FAILED error and enable you to resolve the issue.

   npm run deploy

The Exercise

You should use the AWS Management Console for the exercise and navigate to the CloudFormation service to view your CloudFormation stack, it should be called CfnRollbackExercise. Click the Stack actions button, notice that there is an option there labeled Continue update rollback. Your goal is to be able to use that option and successfully rollback the stack and have the stack in the UPDATE_ROLLBACK_COMPLETE status. Once you have done that, delete the stack from the console. If the delete fails, discovery the issue and resolve it.

To view the notes about the solutions, go to SOLUTION.md.