Dead letter queues

[eladb]

Use Case

I'd like to be able to specify dead letter queues in Wing for cloud.Functions and cloud.Queues.

Proposed Solution

bring cloud;

let deadLetters = new cloud.Queue();

new cloud.Function(inflight () => {}, dlq: deadLetters);
new cloud.Queue(dlq: deadLetters);

Implementation Notes

For AWS:

• AWS Lambda
• SQS

Component

SDK

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.
• If this issue is labeled needs-discussion, it means the spec has not been finalized yet. Please reach out on the #dev channel in the Wing Slack.

[staycoolcall911]

@Chriscbr - can you please share some guidelines?

[marciocadev]

I can pick up this one

[staycoolcall911]

Amazing @marciocadev ! All yours

[ekeren]

Oh great @marciocadev , you just made my request to make sure it also works on awscdk much simpler

[garysassano]

Using a Lambda onFailure destination for implementing a DLQ in a Lambda function is a more advanced and recent approach, offering a significant improvement. This method provides a more detailed payload when an error occurs, simplifying the process of diagnosing issues. A concise video that explains the advantages of this technique can be found here.

[marciocadev]

no problem @garysassano

I can set the onFailure destination to send messages to dlq

[marciocadev]

• dlq (with retries) for queue (tf-aws / awscdk)
• dlq (with retries) for queue (sim)
• dlq for function (tf-aws / awscdk)
• dlq for function (sim)

[marciocadev]

Guys, it's only possible to add a dead-letter queue (DLQ) to a lambda if the event producer is an asynchronous resource (such as Topic, DynamoDB and Bucket events, EventBridge events, etc).

The lambda that consumes an event from a Queue, for example, cannot have a DLQ because SQS is a synchronous resource; it waits for a response from the function and handles accordingly. The API Gateway also faces the same issue.

I'm saying this because I don't think the example new cloud.Function(inflight () => {}, dlq: deadLetters); can work

In the case of Wing, I think the best example would be to add a DLQ to a lambda that consumes asynchronous resources such as Topic, Schedule, Bucket events, etc.

bring cloud;

let deadLetters = new cloud.Queue();
let topic = new cloud.Topic();

topic.onMessage(inflight(msg: str) => {...}, dlq: deadLetters);

[Chriscbr]

That's a good callout @marciocadev - yeah, I'm not sure if the suggested API for adding DLQs on cloud.Function would be appropriate for the Wing SDK if it only works in combination with specific producers and if it only works on AWS.

Dead letter queues for cloud.Queue should be possible to support across other clouds (see https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues and https://cloud.google.com/pubsub/docs/handling-failures) so adding it to the Wing SDK surface area sounds like a more safe bet to me. @ekeren @eladb Do you know if this would serve the use cases you had in mind?

If we want to support AWS specific features / aren't sure if the design work to support this across other clouds is needed right now, it's possible we could first incubate some of these capabilities through AWS classes and methods, e.g.

let topic = new cloud.Topic();
let deadLetters = new cloud.Queue();
if let topic = aws.Topic.from(topic) {
  topic.onMessage(inflight(msg: str) => {...}, dlq: deadLetters);
}

[marciocadev]

Maybe we can add support to function dlqs in a winglib, that way we can define as specific for aws

[monadabot]

Congrats! 🚀 This was released in Wing 0.71.2.