What is SQS?
AWS SQS is a key service used in many architectures. SQS is short for "Simple Queue Service".
SQS is a managed and highly available (within a region) AWS service.
Types Of Queues
There are two types of message queues which SQS supports. "First-in, First-out" or FIFO for short meaning messages are processed in order. FIFO queues support exactly-once-delivery. The other type of message queue is the standard MQ which is "best-effort" regarding the order of messages meaning it can happen that messages are not processed in order and delivered at-least-once.
The standard MQ type support virtually unlimited throughput but your application needs to handle messages in an idempotent way.
With message queues there are message publishers on one side and queue consumers on the other side polling a queue for new messages.
How To Process SQS Messages
SQS queue messages need to be explicitly deleted after successful processing. Otherwise, if processing failed or took too long for what ever reason, messages are requeued i.e. made visible again after a hidden visibility timeout.
If messages need longer for processing your application needs to be increasing the visibility timeout until the message is processed.
What To Do With Messages That Cannot Be Processed?
Sometimes messages can cause problems for the consumers i.e. cannot be processed successfully even after x amount of retries. In this case SQS let's us create "dead letter queues". SQS let's us create "dead letter queues" (DLQ) which can be used by other queues to push messages to that cannot be processed. This let's the processing of messages continue and also saves the problematic messages for later inspection and processing.
Example Elastic SQS Scenario
- User uploads a video to the web app
- Web app stores the video in a S3 bucket
- Web app publishes a message to an SQS queue with all the necessary information
- Workers in an auto scaling group (scales based on messages in the queue) are polling the SQS queue for messages
- After consuming a new message a worker fetches the video from the S3 bucket based on information contained in the message
- The worker transcodes the video and stores the transcoded videos back to the S3 bucket
There are many other architectures where SQS can be used.
Fanout With SNS & SQS
To make the above example more efficient we can add SNS to the picture and leverage a fanout architecture.
Other SQS Details
- Messages can be up to 256KB in size
- Data is stored over multiple servers for durability
- Standard queues support a nearly unlimited number of transactions per seconds
- FIFO queues support up to 300 messages per second (send, receive, delete operations) without batching
- Message timers allow for delaying messages by making them "invisible" for a certain amount of time
