How to subscribe several lambda functions to the same SQS queue

amazon-snsamazon-web-services

What I want to achieve is to scale the number of consumers of a defined queue using lambda functions, instead of "self-hosted" subscribers.

For a given event-queue I want several consumers processing unique messages from this same queue, just like I would simply do with a cluster of processes, but I would like to use the cloud-hosted lambda functions. So far I just found how to subscribe with a single function, without scaling option.

Best Answer

I found this very helpful article from cloudonaut.io (by Andreas Wittig), that exemplify one approach to multiple AWS Lambdas consumers:

Integrate SQS and Lambda: serverless architecture for asynchronous workloads

To scale Andreas' solution you can use AWS CloudWatch Alarms from your AWS SQS queue and tigger AWS CloudFormation (and/or AWS Serverless Application Repository) to spin up more AWS Lambda consumers when needed.

Another approach is to use Amazon MQ.

If you use Amazon Simple Notification Service (SNS) or AWS Kinesis, the same message will trigger different lambdas and you'll probably get duplicated actions between your lambdas.

[update]

Now it is possible to use Amazon Simple Queue Service as a trigger for AWS Lambda:

More info here: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/

Related Topic