Skip to content

Commit

Permalink
fix(cli): report errors from resource failures in nested stacks (#27318)
Browse files Browse the repository at this point in the history 
fix(cli): report errors from resource failures in nested stacks.

## Description
Currently `StackActivityMonitor` uses `readNewEvents()` method to constantly poll CFN to get the latest deployment updates. However it only does it for the root stack and the resources in the root stack. If one of the resource in the root stack is another nested stack and one of the resource in that nested stack fails, CFN does not propagate or copy the error message in the nested stack failure rather it's a generic `Embedded stack <stackArn> was not successfully updated`

This PR updates the `readNewEvents()` to recursively poll for events from the nested stack deployments as well. If errors are detected in the nested stack events, they are added to both `StackActivityMonitor:errors` as well as added in the `Printer`.

Following is a before/after this change. We are deploying RootStack -> Nested Stack -> AppSync Resolver and the AppSync resolver fails with the error `Only one resolver is allowed per field` in CFN.

### Before this change
```
✨  Synthesis time: 3.8s

amplify-sample-samsara-app-pravgupt-sandbox: deploying... [1/1]
amplify-sample-samsara-app-pravgupt-sandbox: creating CloudFormation changeset...
7:59:45 PM | UPDATE_FAILED        | AWS::CloudFormation::Stack | data7552DF31
Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason:
The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF].


 ❌  amplify-sample-samsara-app-pravgupt-sandbox failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

 ❌ Deployment failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
```

### After this change 
```
✨  Synthesis time: 4.17s

amplify-sample-samsara-app-pravgupt-sandbox: deploying... [1/1]
amplify-sample-samsara-app-pravgupt-sandbox: creating CloudFormation changeset...
12:57:07 PM | CREATE_FAILED        | AWS::AppSync::Resolver     | amplifyDataL2Graph...teresolver2355E3CF
Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy:
null)

12:57:10 PM | UPDATE_FAILED        | AWS::CloudFormation::Stack | data7552DF31
Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8ef
d5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicater
esolver2355E3CF].


 ❌  amplify-sample-samsara-app-pravgupt-sandbox failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

 ❌ Deployment failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
```

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
@Amplifiyer
Amplifiyer authored Sep 29, 2023
1 parent a1f2ec2 commit 1f639c7
Show file tree
Hide file tree
Showing 2 changed files with 194 additions and 92 deletions.
19 changes: 13 additions & 6 deletions packages/aws-cdk/lib/api/util/cloudformation/stack-activity-monitor.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -221,14 +221,15 @@ export class StackActivityMonitor {
* see a next page and the last event in the page is new to us (and within the time window).
* haven't seen the final event
*/
private async readNewEvents(): Promise<void> {
private async readNewEvents(stackName?: string): Promise<void> {
const stackToPollForEvents = stackName ?? this.stackName;
const events: StackActivity[] = [];

const CFN_SUCCESS_STATUS = ['UPDATE_COMPLETE', 'CREATE_COMPLETE', 'DELETE_COMPLETE', 'DELETE_SKIPPED'];
try {
let nextToken: string | undefined;
let finished = false;
while (!finished) {
const response = await this.cfn.describeStackEvents({ StackName: this.stackName, NextToken: nextToken }).promise();
const response = await this.cfn.describeStackEvents({ StackName: stackToPollForEvents, NextToken: nextToken }).promise();
const eventPage = response?.StackEvents ?? [];

for (const event of eventPage) {
Expand All @@ -249,6 +250,13 @@ export class StackActivityMonitor {
event: event,
metadata: this.findMetadataFor(event.LogicalResourceId),
});

if (event.ResourceType === 'AWS::CloudFormation::Stack' && !CFN_SUCCESS_STATUS.includes(event.ResourceStatus ?? '')) {
// If the event is not for `this` stack, recursively call for events in the nested stack
if (event.PhysicalResourceId !== stackToPollForEvents) {
await this.readNewEvents(event.PhysicalResourceId);
}
}
}

// We're also done if there's nothing left to read
Expand All @@ -258,7 +266,7 @@ export class StackActivityMonitor {
}
}
} catch (e: any) {
if (e.code === 'ValidationError' && e.message === `Stack [${this.stackName}] does not exist`) {
if (e.code === 'ValidationError' && e.message === `Stack [${stackToPollForEvents}] does not exist`) {
return;
}
throw e;
Expand Down Expand Up @@ -475,7 +483,7 @@ abstract class ActivityPrinterBase implements IActivityPrinter {
this.resourcesPrevCompleteState[activity.event.LogicalResourceId] = status;
}

if (hookStatus!== undefined && hookStatus.endsWith('_COMPLETE_FAILED') && activity.event.LogicalResourceId !== undefined && hookType !== undefined) {
if (hookStatus !== undefined && hookStatus.endsWith('_COMPLETE_FAILED') && activity.event.LogicalResourceId !== undefined && hookType !== undefined) {

if (this.hookFailureMap.has(activity.event.LogicalResourceId)) {
this.hookFailureMap.get(activity.event.LogicalResourceId)?.set(hookType, activity.event.HookStatusReason ?? '');
Expand Down Expand Up @@ -803,4 +811,3 @@ function shorten(maxWidth: number, p: string) {

const TIMESTAMP_WIDTH = 12;
const STATUS_WIDTH = 20;

267 changes: 181 additions & 86 deletions packages/aws-cdk/test/util/stack-monitor.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,95 +10,171 @@ beforeEach(() => {
printer = new FakePrinter();
});

test('continue to the next page if it exists', async () => {
await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(102)],
NextToken: 'some-token',
};
},
(request) => {
expect(request.NextToken).toBe('some-token');
return {
StackEvents: [event(101)],
};
},
]);

// Printer sees them in chronological order
expect(printer.eventIds).toEqual(['101', '102']);
});
describe('stack monitor event ordering and pagination', () => {
test('continue to the next page if it exists', async () => {
await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(102)],
NextToken: 'some-token',
};
},
(request) => {
expect(request.NextToken).toBe('some-token');
return {
StackEvents: [event(101)],
};
},
]);

test('do not page further if we already saw the last event', async () => {
await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(101)],
};
},
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(102), event(101)],
NextToken: 'some-token',
};
},
(request) => {
// Did not use the token
expect(request.NextToken).toBeUndefined();
return {};
},
]);

// Seen in chronological order
expect(printer.eventIds).toEqual(['101', '102']);
});
// Printer sees them in chronological order
expect(printer.eventIds).toEqual(['101', '102']);
});

test('do not page further if we already saw the last event', async () => {
await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(101)],
};
},
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(102), event(101)],
NextToken: 'some-token',
};
},
(request) => {
// Did not use the token
expect(request.NextToken).toBeUndefined();
return {};
},
]);

// Seen in chronological order
expect(printer.eventIds).toEqual(['101', '102']);
});

test('do not page further if the last event is too old', async () => {
await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(101), event(95)],
NextToken: 'some-token',
};
},
(request) => {
// Start again from the top
expect(request.NextToken).toBeUndefined();
return {};
},
]);

// Seen only the new one
expect(printer.eventIds).toEqual(['101']);
});

test('do a final request after the monitor is stopped', async () => {
await testMonitorWithEventCalls([
// Before stop
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(101)],
};
},
],
// After stop
[
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(102), event(101)],
};
},
]);

test('do not page further if the last event is too old', async () => {
await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(101), event(95)],
NextToken: 'some-token',
};
},
(request) => {
// Start again from the top
expect(request.NextToken).toBeUndefined();
return {};
},
]);

// Seen only the new one
expect(printer.eventIds).toEqual(['101']);
// Seen both
expect(printer.eventIds).toEqual(['101', '102']);
});
});

test('do a final request after the monitor is stopped', async () => {
await testMonitorWithEventCalls([
// Before stop
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(101)],
};
},
],
// After stop
[
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [event(102), event(101)],
};
},
]);

// Seen both
expect(printer.eventIds).toEqual(['101', '102']);
describe('stack monitor, collecting errors from events', () => {
test('return errors from the root stack', async () => {
const monitor = await testMonitorWithEventCalls([
(request) => {
expect(request.NextToken).toBeUndefined();
return {
StackEvents: [addErrorToStackEvent(event(100))],
};
},
]);

expect(monitor.errors).toStrictEqual(['Test Error']);
});

test('return errors from the nested stack', async () => {
const monitor = await testMonitorWithEventCalls([
(request) => {
expect(request.StackName).toStrictEqual('StackName');
return {
StackEvents: [
addErrorToStackEvent(
event(100), {
logicalResourceId: 'nestedStackLogicalResourceId',
physicalResourceId: 'nestedStackPhysicalResourceId',
resourceType: 'AWS::CloudFormation::Stack',
resourceStatusReason: 'nested stack failed',
},
),
],
};
},
(request) => {
expect(request.StackName).toStrictEqual('nestedStackPhysicalResourceId');
return {
StackEvents: [
addErrorToStackEvent(
event(101), {
logicalResourceId: 'nestedResource',
resourceType: 'Some::Nested::Resource',
resourceStatusReason: 'actual failure error message',
},
),
],
};
},
]);

expect(monitor.errors).toStrictEqual(['actual failure error message', 'nested stack failed']);
});

test('does not check for nested stacks that have already completed successfully', async () => {
const monitor = await testMonitorWithEventCalls([
(request) => {
expect(request.StackName).toStrictEqual('StackName');
return {
StackEvents: [
addErrorToStackEvent(
event(100), {
logicalResourceId: 'nestedStackLogicalResourceId',
physicalResourceId: 'nestedStackPhysicalResourceId',
resourceType: 'AWS::CloudFormation::Stack',
resourceStatusReason: 'nested stack status reason',
resourceStatus: 'CREATE_COMPLETE',
},
),
],
};
},
]);

expect(monitor.errors).toStrictEqual([]);
});
});

const T0 = 1597837230504;
Expand All @@ -115,10 +191,28 @@ function event(nr: number): AWS.CloudFormation.StackEvent {
};
}

function addErrorToStackEvent(
eventToUpdate: AWS.CloudFormation.StackEvent,
props: {
resourceStatus?: string,
resourceType?: string,
resourceStatusReason?: string,
logicalResourceId?: string,
physicalResourceId?: string,
} = {},
): AWS.CloudFormation.StackEvent {
eventToUpdate.ResourceStatus = props.resourceStatus ?? 'UPDATE_FAILED';
eventToUpdate.ResourceType = props.resourceType ?? 'Test::Resource::Type';
eventToUpdate.ResourceStatusReason = props.resourceStatusReason ?? 'Test Error';
eventToUpdate.LogicalResourceId = props.logicalResourceId ?? 'testLogicalId';
eventToUpdate.PhysicalResourceId = props.physicalResourceId ?? 'testPhysicalResourceId';
return eventToUpdate;
}

async function testMonitorWithEventCalls(
beforeStopInvocations: Array<(x: AWS.CloudFormation.DescribeStackEventsInput) => AWS.CloudFormation.DescribeStackEventsOutput>,
afterStopInvocations: Array<(x: AWS.CloudFormation.DescribeStackEventsInput) => AWS.CloudFormation.DescribeStackEventsOutput> = [],
) {
): Promise<StackActivityMonitor> {
let describeStackEvents = (jest.fn() as jest.Mock<AWS.CloudFormation.DescribeStackEventsOutput, [AWS.CloudFormation.DescribeStackEventsInput]>);

let finished = false;
Expand All @@ -144,6 +238,7 @@ async function testMonitorWithEventCalls(
const monitor = new StackActivityMonitor(sdk.cloudFormation(), 'StackName', printer, undefined, new Date(T100)).start();
await waitForCondition(() => finished);
await monitor.stop();
return monitor;
}

class FakePrinter implements IActivityPrinter {
Expand Down

0 comments on commit 1f639c7

Please sign in to comment.