Which AWS messaging and queuing service to use?
AWS offers a number of messaging and queuing services, each one with its own pros and cons. This post describes each service and when to use it.
AWS SQS stands for Simple Queue Service. It is as simple as it gets -- you can send a message to a queue, you can receive it from a queue, and you can acknowledge the message by deleting it from the queue. Only one consumer can process the same message at a time. It is a native AWS service that requires the use of AWS REST API or an AWS SDK for your programming language.
It is a genuinely server-less managed service. The developers don't need to worry about scalability, storage, or performance. SQS queues scale as required. You only pay for what you use which turns out to be pennies per million messages.
SQS integrates well with other AWS services such as AWS Lambda. For example, SQS will trigger a lambda function for each message in the queue. While SQS is very simple in that it only supports "
SQS scales horizontally. Roughly speaking, if you have a single consumer able to process, say, 1000 messages per second, two instances of the consumer will process 2000 messages per second. If you use Lambda triggers, AWS will scale both the queues and the consumers for you.
SQS is durable and supports Dead Letter Queues and configurable re-delivery policy. If for some reason your consumer took a message off the queue but failed to correctly process it, SQS will re-attempt delivery a few times (configurable) before eventually delivering the failed message to the Dead Letter Queue.
You can find SQS documentation here.
Also see: Top ten differences between ActiveMQ and AWS SQS
AWS SNS is a Simple Notification Service. It delivers messages published to topics to one or more destinations. Destinations can be SQS queues, Lambda functions, HTTP POST endpoints, SMS text messages, mobile device push notifications, and more.
SNS offers some degree of durability. Messages are saved before producers get a confirmation. There is a configurable re-delivery policy depending on the destination. Applications requiring assurances that messages will eventually get delivered to their destinations should use SNS to route to SQS and then consume from SQS.
AWS Kinesis is a managed data streaming service. Although it does support video and multi-media streams, it is beyond the scope of this article.
Producers put data on a stream using Kinesis client library. Multiple different Kinesis data stream consumers can then process data from the stream concurrently. This is an important distinction from queues where only one kind of a consumer can take messages off the same queue.
For example, one consumer can archive product order data while another analyzes data for fraud in real-time, and yet another one uses the data to dynamically update pricing and inventory data.
It is possible to also arrange streams into direct acyclic graphs (DAG) such that messages are forwarded to other streams to create complex stream processing. This is another distinction from queues.
Kinesis integrates well with other AWS services, including AWS Lambda. Kinesis can trigger Lambda functions in response to messages. Additionally, AWS services such as S3 or Dynamo can publish data to Kinesis streams.
AWS MSK stands for "AWS Managed Streaming for Kafka." Conceptually, Kafka is similar to Kinesis: producers publish messages on Kafka topics (streams), while multiple different consumers can process messages concurrently.
Kafka is famous but can be "Kafkaesque" to maintain in production. MSK takes a lot of the operational difficulties out of running a Kafka cluster. MSK takes care of various maintenance tasks such as backups and routine upgrades.
MSK works similarly to AWS RDS (managed database service). MSK provisions and manages the underlying compute infrastructure on your behalf. You pay for this compute capacity regardless of the workload on your Kafka cluster. In other words, MSK is a managed service, but it is not server-less.
AWS MQ supports various broker topologies, including mesh networks. By being ActiveMQ-based, it supports the full range of services including queues, topics, and multiple configurations of durability, re-delivery, and replication.
Also see: Top ten differences between ActiveMQ and AWS SQS (note: I wrote that post in 2015, long before Amazon had a managed ActiveMQ service)
Liked this post? In September 2020 I will ride my bicycle for 275 miles from Boston to NYC to help end AIDS. Please consider contributing to my fundraising goals.
AWS SQS: managed durable queues
AWS SQS stands for Simple Queue Service. It is as simple as it gets -- you can send a message to a queue, you can receive it from a queue, and you can acknowledge the message by deleting it from the queue. Only one consumer can process the same message at a time. It is a native AWS service that requires the use of AWS REST API or an AWS SDK for your programming language.
It is a genuinely server-less managed service. The developers don't need to worry about scalability, storage, or performance. SQS queues scale as required. You only pay for what you use which turns out to be pennies per million messages.
SQS integrates well with other AWS services such as AWS Lambda. For example, SQS will trigger a lambda function for each message in the queue. While SQS is very simple in that it only supports "
send," "receive," and "delete" operations, I recommend abstracting these routines at a bit of a higher level.SQS scales horizontally. Roughly speaking, if you have a single consumer able to process, say, 1000 messages per second, two instances of the consumer will process 2000 messages per second. If you use Lambda triggers, AWS will scale both the queues and the consumers for you.
SQS is durable and supports Dead Letter Queues and configurable re-delivery policy. If for some reason your consumer took a message off the queue but failed to correctly process it, SQS will re-attempt delivery a few times (configurable) before eventually delivering the failed message to the Dead Letter Queue.
You can find SQS documentation here.
Also see: Top ten differences between ActiveMQ and AWS SQS
When to use SQS?
- Your architecture requires asynchronous processing with guarantees that all tasks get processed
- You need a durable and reliable queuing solution that will scale at low cost and require no management on your part
- You are building a brand new AWS-native application
- You are porting an existing application to server-less architecture
- You are adding queueing functionality to your architecture in AWS
When not to use SQS?
- Your application is sensitive to the latency associated with using AWS APIs via HTTP protocol.
AWS SNS
AWS SNS is a Simple Notification Service. It delivers messages published to topics to one or more destinations. Destinations can be SQS queues, Lambda functions, HTTP POST endpoints, SMS text messages, mobile device push notifications, and more.
SNS offers some degree of durability. Messages are saved before producers get a confirmation. There is a configurable re-delivery policy depending on the destination. Applications requiring assurances that messages will eventually get delivered to their destinations should use SNS to route to SQS and then consume from SQS.
When to use SNS?
- Your architecture requires delivering messages asynchronously to multiple consumers at a time but does not require journaling or strong delivery guarantees (if you need delivery guarantees you should use SQS queues)
- You need to trigger asynchronous events across different parts of your application
- You need to send mobile push notifications in a cross-platform manner
- You need to send SMS text messages
- You are building a brand new AWS-native application
- You are porting an existing application to server-less architecture
When not to use SNS?
- Your application sensitive to the latency associated with using AWS APIs via HTTP protocol and you have no need for specialized functions like SMS or mobile push
AWS Kinesis
AWS Kinesis is a managed data streaming service. Although it does support video and multi-media streams, it is beyond the scope of this article.
Producers put data on a stream using Kinesis client library. Multiple different Kinesis data stream consumers can then process data from the stream concurrently. This is an important distinction from queues where only one kind of a consumer can take messages off the same queue.
For example, one consumer can archive product order data while another analyzes data for fraud in real-time, and yet another one uses the data to dynamically update pricing and inventory data.
It is possible to also arrange streams into direct acyclic graphs (DAG) such that messages are forwarded to other streams to create complex stream processing. This is another distinction from queues.
Kinesis integrates well with other AWS services, including AWS Lambda. Kinesis can trigger Lambda functions in response to messages. Additionally, AWS services such as S3 or Dynamo can publish data to Kinesis streams.
When to use Kinesis?
- Your architecture requires complex real-time processing of data streams
- You have complex real-time analytics requirements
- You need to trigger asynchronous events across different parts of your application
- You are building a brand new AWS-native application
- You are adding stream-processing functionality to your AWS-native application
- You are porting an existing application to AWS-native architecture
When not to use Kinesis?
- You only need simple queues
- You just need simple topics
- Your architecture is sensitive to latencies. If that is the case consider AWS MSK (see below).
AWS MSK (managed Kafka)
AWS MSK stands for "AWS Managed Streaming for Kafka." Conceptually, Kafka is similar to Kinesis: producers publish messages on Kafka topics (streams), while multiple different consumers can process messages concurrently.
Kafka is famous but can be "Kafkaesque" to maintain in production. MSK takes a lot of the operational difficulties out of running a Kafka cluster. MSK takes care of various maintenance tasks such as backups and routine upgrades.
MSK works similarly to AWS RDS (managed database service). MSK provisions and manages the underlying compute infrastructure on your behalf. You pay for this compute capacity regardless of the workload on your Kafka cluster. In other words, MSK is a managed service, but it is not server-less.
When to use AWS MSK?
- You are migrating an existing Kafka-based application to AWS
- You already use Kafka and have a significant investment in Kafka-dependent codebase
- You want to reduce operational costs and difficulties out of running a Kafka cluster in AWS
- Your application is sensitive to the latency associated with using AWS APIs via HTTP protocol.
When not to use AWS MSK?
There is little reason to introduce Kafka to an application that is not already using it and does not have requirements to use Kafka specifically (for example due to latency and performance considerations). Consider using Kinesis instead.- You are building a brand new AWS-native application
- You are adding streaming functionality to an existing AWS-native application
AWS MQ (managed Apache ActiveMQ)
AWS MQ is a managed ActiveMQ service. Similar to MSK for Kafka, it takes operational complexity out of running an ActiveMQ cluster. It supports JMS, NMS, AMQP, STOMP, MQTT and other industry standard messaging protocols. Most legacy applications do not require significant changes to work in AWS.AWS MQ supports various broker topologies, including mesh networks. By being ActiveMQ-based, it supports the full range of services including queues, topics, and multiple configurations of durability, re-delivery, and replication.
Also see: Top ten differences between ActiveMQ and AWS SQS (note: I wrote that post in 2015, long before Amazon had a managed ActiveMQ service)
When to use AWS MQ (Apache ActiveMQ)?
- You are migrating an existing JMS or NMS-based application to AWS
- You already use ActiveMQ or a compatible product and have a significant investment in the codebase
- You want to reduce operational costs and difficulties out of running an ActiveMQ cluster in AWS
- Your application is sensitive to the latency associated with using AWS APIs via HTTP protocol.
When not to use AWS MQ?
- You are adding message queues to your AWS-native architecture and need something simple. Consider using SQS instead.
- Your application is not sensitive to the latency associated with AWS SQS API.
Liked this post? In September 2020 I will ride my bicycle for 275 miles from Boston to NYC to help end AIDS. Please consider contributing to my fundraising goals.