The AWS SQS Sink is a custom user-defined sink for Numaflow that enables the integration of Amazon Simple Queue Service (SQS) as a sink within your Numaflow pipelines.
This quick start guide will walk you through setting up an AWS SQS sink in a Numaflow pipeline.
Using AWS CLI or the AWS Management Console, create a new SQS queue
- Save the following Kubernetes manifest to a file (e.g.,
sqs-sink-pipeline.yaml
) - Modifying the AWS region, queue name accordingly
- Specify the AWS credentials using any supported approach https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/#specifying-credentials
apiVersion: numaflow.numaproj.io/v1alpha1
kind: Pipeline
metadata:
name: simple-pipeline
spec:
vertices:
- name: in
source:
generator:
# How many messages to generate in the duration.
rpu: 1
duration: 5s
# Optional, size of each generated message, defaults to 10.
msgSize: 10
- name: aws-sink
scale:
min: 1
sink:
udsink:
container:
image: quay.io/numaio/numaproj-contrib/aws-sqs-sink-go:v0.0.1
env:
- name: AWS_SQS_QUEUE_NAME
value: "test-queue"
- name: AWS_REGION
value: "us-east-1"
- name: AWS_ACCESS_KEY_ID
value: "testing" ## This can be passed as k8s secret as well
- name: AWS_SECRET_ACCESS_KEY
value: "testing" ## This can be passed as k8s secret as well
edges:
- from: in
to: aws-sink
Then apply it to your cluster:
kubectl apply -f sqs-sink-pipeline.yaml
Verify the message using aws cli to poll message, refer: https://docs.aws.amazon.com/cli/latest/reference/sqs/receive-message.html
To delete the Numaflow pipeline:
kubectl delete -f sqs-sink-pipeline.yaml
To delete the SQS queue:
aws sqs delete-queue --queue-url <YourQueueUrl>
Congratulations!!! You have successfully set up an AWS SQS sink in a Numaflow pipeline.
For more information on Numaflow and how to use it to process data in a Kubernetes-native way, visit the Numaflow Documentation. For AWS SQS specific configuration, refer to the AWS SQS Documentation.