## Table of Contents 1. [Introduction](#Intro) 2. [Standard vs FIFO Queue](#vs) 3. [Queue Configuration](#config) 4. [Create a SQS queue](#sqs) 5. [Create a Cloud9 instance](#cloud9) 6. [Challenge](#challenge) 7. [Summary](#summary) 8. [Referrals](#referrals) ### Introduction In this blog we are going to create a SQS queue using the AWS Console and then interact with it using `AWS CLI`, `bash` and `jq`. ### What is SQS? Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Some of the use cases of SQS - Increase application reliability and scale. - Decouple microservices and process event-driven applications. - Ensure work is completed cost-effectively and on time. - Maintain message ordering with deduplication --- #### Standard vs FIFO Queue SQS offers two types of queues – Standard & FIFO queues Some of the major differences between the standard and the FIFO queues are: https://dev.to/aws-builders/sagemaker-and-amazon-stack-2jaa ##### Message Order - Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. Occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order. - FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved. ##### Delivery - Standard queues guarantees that a message is delivered at least once and duplicates can be introduced into the queue. - FIFO queues ensure a message is delivered exactly once and remains available until a consumer processes and deletes it. The duplicates are not introduced into the queue. ##### Transactions Per Second (TPS) - Standard queues allow nearly-unlimited number of transactions per second. - FIFO queues are limited to 300 transactions per second per API action. However It can be increased to 3000 by using batching. --- #### Queue Configuration > Note: The following is from AWS Console **_Visibility timeout:_** Should be between 0 seconds and 12 hours (default: 30 seconds). - Visibility timeout sets the length of time that a message received from a queue (by one consumer) will not be visible to the other message consumers. - The visibility timeout begins when Amazon SQS returns a message. If the consumer fails to process and delete the message before the visibility timeout expires, the message becomes visible to other consumers. If a message must be received only once, your consumer must delete it within the duration of the visibility timeout. - The default visibility timeout setting is 30 seconds. This setting applies to all messages in the queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue. **_Message retention period:_** Should be between 1 minute and 14 days (default: 4 days). - The message retention period is the amount of time that Amazon SQS retains a message that does not get deleted. Amazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period. The default retention period is 4 days. The retention period has a range of 60 seconds to 1,209,600 seconds (14 days). - The expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp remains unchanged. For example, if a message spends 1 day in the original queue before being moved to a dead-letter queue, and the retention period of the dead-letter queue is set to 4 days, the message is deleted from the dead-letter queue after 3 days. For this reason, we recommend that you always set the retention period of a dead-letter queue to be longer than the retention period of the original queue. **_Delivery delay:_** Should be between 0 seconds and 15 minutes (default: 0 seconds). - If your consumers need additional time to process messages, you can delay each new message coming to the queue. The delivery delay is the amount of time to delay the first delivery of each message added to the queue. Any messages that you send to the queue remain invisible to consumers for the duration of the delay period. The default (minimum) delay for a queue is 0 seconds. The maximum is 15 minutes. - For standard queues, the per-queue delay setting is not retroactive; changing the setting doesn't affect the delay of messages already in the queue. - For FIFO queues, the per-queue delay setting is retroactive; changing the setting affects the delay of messages already in the queue. **_Maximum message size:_** Should be between 1 KB and 256 KB (default: 256 KB). - You can set the maximum message size for your queue. The smallest supported message size is 1 byte (1 character). The largest size is 262,144 bytes (256 KB). To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java (https://github.com/awslabs/amazon-sqs-java-extended-client-lib). This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB. **_Receive message wait time:_** Should be between 0 and 20 seconds (default: 0 seconds). - The receive message wait time is the maximum amount of time that polling will wait for messages to become available to receive. The minimum value is zero seconds and the maximum value is 20 seconds. - Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response). If a receive request collects the maximum number of messages, it returns immediately. It does not wait for the polling to time out. - If you set the receive message wait time to zero, the receive requests use short polling. --- ## Demo Let's get started with the demo. ### Create a SQS queue 1. Navigate to **Simple Queue Service** and click **Create queue** 2. Enter **Name** as `MyMessages`, leave the rest as defaults and click **Create queue** ![> SQS_4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2840jonlzd6rcvmtlhjr.png) > SQS_1 ![SQS_2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f0hc7eqa4g80wskip1i8.png) > SQS_2 ![SQS_3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fsrlwh25smnutkqh6nw4.png) > SQS_3 ![SQS_4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9cxuzg9jeq12ab6vi42x.png) > SQS_4 --- ### Cloud9 1. Navigate to **Cloud9** > **Create Environment** 2. Enter **Name** as `SQS` and click **Create** 3. Click **Open** under **AWS Cloud9** > **Environments** Let's check aws cli version. ```bash Sri:~/environment $ aws --version aws-cli/1.19.112 Python/2.7.18 Linux/4.14.301-224.520.amzn2.x86_64 botocore/1.20.112 ``` jq is not installed by default on Cloud9. ```bash Sri:~/environment $ jq bash: jq: command not found ``` Install jq by executing the following command. ```bash sudo yum install jq -y ``` How to check if jq is installed. ```bash Sri:~/environment $ rpm -qa|grep jq jq-1.5-1.amzn2.0.2.x86_64 ``` jq -help ```bash Sri:~/environment $ jq jq - commandline JSON processor [version 1.5] Usage: jq [options] [file...] jq is a tool for processing JSON inputs, applying the given filter to its JSON text inputs and producing the filter's results as JSON on standard output. The simplest filter is ., which is the identity filter, copying jq's input to its output unmodified (except for formatting). For more advanced filters see the jq(1) manpage ("man jq") and/or https://stedolan.github.io/jq Some of the options include: -c compact instead of pretty-printed output; -n use `null` as the single input value; -e set the exit status code based on the output; -s read (slurp) all inputs into an array; apply filter to it; -r output raw strings, not JSON texts; -R read raw strings, not JSON texts; -C colorize JSON; -M monochrome (don't colorize JSON); -S sort keys of objects on output; --tab use tabs for indentation; --arg a v set variable $a to value ; --argjson a v set variable $a to JSON value ; --slurpfile a f set variable $a to an array of JSON texts read from ; See the manpage for more options. ``` Send a SQS message using AWS CLI In order to send a message to SQS, we need `queue-url`. We could either hard code the `queue-url` or retreive the `queue-url` and then pass it to `send-message`, `receive-message` or `delete-message` ```bash # queueurl is required to send or receive messages from SQS # Note: AWS CLI and boto3 need to use legacy endpoint # If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html. # Otherwise, you will get the following error # botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint. export queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g') echo "queueurl:" $queueurl ``` ```bash Sri:~/environment $ aws sqs send-message --queue-url $queueurl --message-body Message --region us-east-1 { "MD5OfMessageBody": "4c2a8fe7eaf24721cc7a9f0175115bd4", "MessageId": "614ec45a-82b8-4788-aed6-6f3d80051780" } ``` ```bash Sri:~/environment $ aws sqs receive-message --queue-url $queueurl { "Messages": [ { "Body": "Message", "ReceiptHandle": "AQEBxxUZvU5S1OP2VFBIY5bpg267ZQvSkX3n7jfV505VumHa2JCHo5BR/trRSCKOlhcLSyvJzBUBhq9TPHlQ4F4nQu839HruurRMMYCaaZBiiEdFnADm3EwEi8+Hj9YsmmGf0YL4ftlTyMHnJFjL9l06e7qZZyV8lWKv+q/mnEU8v69AumtvcS7OhmgAtUzQUqG+4/enJgt3WrN1eJiJRlmiUbDpV3VCGr7wQbU5C+GAmYjVNs+vNrMSzOceKFs3XQH6vwhNnnp1d3dJZzn+JEjpfx8g6INrq1LYShgg6cTUVygc2jXuQNBoSHP4Jx6szscdLm2P4gUD515UJ948nX/Zlsu90GZtfyGY5S4wzgl9oYpJcPLRV7z6ISnte0WKuK9cHPCEzmyGeERS1yC7YKXRNA==", "MD5OfBody": "4c2a8fe7eaf24721cc7a9f0175115bd4", "MessageId": "614ec45a-82b8-4788-aed6-6f3d80051780" } ] } ``` :point_right: If you run the command again with in the Visibility timeout of 30 secs, SQS will not send the message again. However if the message is not processed and deleted by the consumer, the message becomes visible to other consumers. If a message must be received only once, the consumer must delete it within the duration of the visibility timeout. ```bash Sri:~/environment $ aws sqs receive-message --queue-url $queueurl Sri:~/environment $ ``` So in order to delete the message, we need to retreive the `receipt-handle` and then pass the `--receipt-handle` to `sqs delete-message` Since the output message is a JSON message and we only need `receipt-handle`, we can use **jq**, which is like sed for JSON data - you can use it to slice, filter, map and transform structured data in the same way that you would with sed, awk, grep and other linux tools. What does `get-queue-url` do? It retreives `QueueUrl` and then replaces the endpoint with a legacy endpoint and also removes double quotes from the `QueueUrl`. The following code snippet retrieves and deletes the message. ```bash export queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g') echo "queueurl:" $queueurl # receiptHandle is required to delete the message from SQS after the message is retreived export receiptHandle=$(aws sqs receive-message --queue-url $queueurl|jq .Messages[0].ReceiptHandle|sed 's/"//g') echo "receiptHandle:" $receiptHandle #echo -e "aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1" aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1; ``` Let's send a 100 messages to the SQS queue, retrieve them and then delete the messages using bash scripts. Open up two terminal windows on Cloud9 and use one to execute `./sendMessages.sh` and the other for `./receiveMessages.sh`. _sendMessages.sh_ ```bash #!/bin/bash # queueurl is required to send or receive messages from SQS # Note: AWS CLI and boto3 need to use legacy endpoint # If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html. # Otherwise, you will get the following error # botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint. queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g') echo "queueurl:" $queueurl echo "aws sqs send-message --queue-url $queueurl --message-body Message --region us-east-1" for i in {1..100} ; do aws sqs send-message --queue-url $queueurl --message-body "Message-$i" --region us-east-1 ; done; ``` _receiveMessages.sh_ ```bash #!/bin/bash # queueurl is required to send or receive messages from SQS # Note: AWS CLI and boto3 need to use legacy endpoint # If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html. # Otherwise, you will get the following error # botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint. queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g') echo "queueurl:" $queueurl for i in {1..100}; do # receiptHandle is required to delete the message from SQS after the message is retreived receiptHandle=$(aws sqs receive-message --queue-url $queueurl|jq .Messages[0].ReceiptHandle|sed 's/"//g') #echo "receiptHandle:" $receiptHandle #echo -e "aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1" aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1; echo "Sleep for 1 second..." sleep 1 done; ``` --- ### Challenge There are some improvements which we can make to the above. I will leave this as a challenge for now. Please let me know in the comments if you have managed to solve it. I will provide an update in a comment at some point. 1. We can use `--max-number-of-messages (integer)`. > The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1. 2. jq query in the command also need to be updated to retrieve an array of `--receipt-handle`. 3. bash script also need to store the returned array and loop through the array to delete all the messages returned by `--max-number-of-messages`. --- #### Clean Up 1. Delete the `MyMessages` Queue. 2. Terminate `Cloud9` EC2 instance. --- #### Summary - We learned about `SQS`. - We also learned about `bash scripting` and `jq`. See you next time 👋 --- ### Referrals - [Amazon SQS API Reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sqs/index.html) - [botot3 with sqs: the address 'QueueUrl' is not valid for this endpoint](https://stackoverflow.com/questions/74926354/botot3-with-sqs-the-address-queueurl-is-not-valid-for-this-endpoint) - [Amazon Simple Queue Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/sqs-service.html)

awcli

AWS Simple Queue Service (SQS) 101 with a demo using AWS CLI

Join Our Newsletter. No Spam, Only the good stuff.