ServerlessQ's Blog

ServerlessQ's Blog

Aws SQS CLI

Aws SQS CLI

Let's see what you can do with the AWS CLI for SQS

Sandro Volpicella
ยทApr 19, 2022ยท

8 min read

Table of contents

This article gives you an overview of the AWS CLI and which operations you can do with it for AWS's awesome Simple Queue Service (SQS in the following).

We start with some basic introductions to SQS, CLIs, and the AWS CLI. Then we'll focus on operations specifically for SQS.

In case you want to use the full capabilities of SQS without the overhead make sure to try out serverlessq.com after reading the article ๐Ÿ™‚

Let's go ๐Ÿš€

What is SQS?

Amazon SQS is short for AWS's Simple Queue Service. SQS is a message queue that can be polled from different types of consumers. Consumers typically are:

  • Lambda functions
  • EC2 instances
  • ECS cluster

SQS is a poll system but acts like a push system. Consumers are constantly polling for new messages, working on these messages, and deleting them afterward.

Fun Fact: SQS is the oldest service of AWS ๐Ÿ‘†๐Ÿฝ

What is a CLI?

A CLI is short for Command Line Interface. If you are a developer you surely have used CLIs everywhere. Normally, they are normal programs that take arguments via the command line.

CLIs can be created with different programming languages, nowadays many CLIs are written in either Node.js or Python.

CLIs are extremely powerful tools, especially in combination with existing bash programs like grep, sed, or jq.

While there are tons of CLIs out there this blog post handles only the AWS CLI.

AWS CLI

The AWS CLI is very popular among AWS users. Chance is that if you have any program on your PC that is interacting with AWS it uses the CLI under the hood.

The AWS CLI is also very powerful. You can interact with almost all services and do everything you can do in the console also in the CLI.

Popular examples are:

  • Creating S3 buckets
  • Creating Lambda Functions
  • Executing lambda functions
  • Launching an ECS Cluster

And much more. The AWS CLI is also often used in CI/CD clusters like Jenkins or GitHub Actions.

Configure AWS CLI

It is recommended to configure the AWS CLI before using it. There are different ways of configuring the CLI. The main goal is to have an access key and secret access key saved somewhere securely. Here is an official article by AWS on how to configure the CLI. I recommend doing it with names profiles and I show you quickly how you can do that.

Install AWS CLI

For installing the AWS CLI please follow AWS's official guidelines.

Configure Named Profiles for AWS CLI

Named profiles is a collection of different AWS configurations for typically several accounts. However, you can also just use it for one account.

Two files need to be created for your profile (all on Linux or Mac OS):

  1. ~/.aws/config -> All Configuration
  2. ~/.aws/credentials -> All Secret Keys

Config File

~/.aws/config

[default]
region=us-west-2
output=json

[profile user1]
region=us-east-1
output=json

Credentials File

~/.aws/credentials

[default]
aws_access_key_id=XXX
aws_secret_access_key=XXX

[user1]
aws_access_key_id=ACCESS_KEY
aws_secret_access_key=SECRET_ACCESS_KEY

I recommend adding XXX as the default profile key to not accidentally push something on the wrong account.

Now you can use the profile by appending --profile to your commands or by exporting the variable AWS_PROFILE before every call. I won't include --profile in the examples so keep that in mind!

If you want to export the environment variable for one session you can simply do the following:

export AWS_PROFILE=user1
aws s3 ls 

# Same effect as aws s3 ls --profile user1

You can test it for example with the command aws s3 ls --profile user1. You should now see all buckets in the account of user1.

CLI Autocomplete

One quick tip. Install AWS CLI autocomplete feature. With that, you can just hit TAB and you'll get recommendations for all services and operations you can do.

You can see the installation guide here.

Then you can hit the tab key after typing aws or after typing aws <SERVICE> to see all operations.

For example type aws sqs, hit tab and you see all operations you can do.

image.png

Even after hitting TAB after an operation, you see all available parameters. Believe me, this is an amazing help!

image.png

SQS Operations

Now we will look into SQS operations you can do with the CLI. I collected the most important commands and will show you what happens when you execute your commands.

Create a queue

aws sqs create-queue --queue-name myQueue1

It will return the following object (depending on the region):

{
    "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1"
}

If you check the AWS management console you will find the queue with its default parameter.

image.png

Important Parameters

--queue-name: The name of the queue

--attributes: Attributes like MessageRetentionPeriod, RedrivePolicy, Policy, FifoQueue, etc.

--tags: How to tag this resource

If you want to create a queue with different attributes you can do the following for example:

aws sqs create-queue --queue-name myCustomizedQueue.fifo --attributes VisibilityTimeout=100,FifoQueue=true

The input for the attributes can also be a .json file but here I am using the shorthand syntax.

{
    "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myCustomizedQueue.fifo"
}

This command created a FIFO queue with a visibility timeout of 100 seconds.

Listing Queues

You can list all queues in your account with the list-queues command.

aws sqs list-queues

It will return a JSON object with a QueueUrls array:

{
    "QueueUrls": [
        "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myCustomizedQueue.fifo",
        "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1"
    ]
}

Important Parameters

--queue-name-prefix: In SQS you always search by prefix.

--max-items: Total number of items to return.

--next-token: Used for pagination

--starting-token: Where to start querying from. Also used for pagination.

Get Queue Attributes

You can query queue attributes with the get-queue-attributes command.

aws sqs get-queue-attributes --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1 --attribute-names All

Output:

{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:REGION:ACCOUNT_NUMBER:myQueue1",
        "ApproximateNumberOfMessages": "0",
        "ApproximateNumberOfMessagesNotVisible": "0",
        "ApproximateNumberOfMessagesDelayed": "0",
        "CreatedTimestamp": "1650118025",
        "LastModifiedTimestamp": "1650118025",
        "VisibilityTimeout": "30",
        "MaximumMessageSize": "262144",
        "MessageRetentionPeriod": "345600",
        "DelaySeconds": "0",
        "ReceiveMessageWaitTimeSeconds": "0",
        "SqsManagedSseEnabled": "false"
    }
}

SQS works almost every time from its URL. Unlike other services that use the ARN (Amazon Resource Name) more often.

Getting the queue attributes is important to see how you configured your queue.

Important Parameters

--queue-url: Identify from which queue to get attributes from

--attribute-names: To get all attributes to define All

Get Queue URL

You will often need the queue URL. The command for that is get-queue-url. You can get the URL by its queue name.

aws sqs get-queue-url --queue-name myQueue1

This will return a JSON object with the URL

{
    "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1"
}

Send a message to the queue

Now it gets more interesting. Sending a message to a queue is one of the core operations in SQS.

Sending a message means the message will be queued and a consumer can pick it up. It depends on several parameters and the configuration of the consumers on how fast it will be picked up.

You can send a message with the send-message command.

aws sqs send-message --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1 --message-body test

It will return the MD5 hash of the message body (of the body, not attributes) and a message ID.

{
    "MD5OfMessageBody": "0096e740711f375cbdc1a47ac6864b49",
    "MessageId": "a104a85f-7ca4-483e-9f10-9e82a7ce7e1f"
}

Your message body can be:

  • XML
  • JSON
  • Plain text

Important Parameters

--queue-url: URL to send the message to

--message-body: The actual message to send

--delay-seconds: How long to delay till the message is sent. Between 0 and 900 seconds

If you have more messages to send you can also send them in batches with the send-message-batch command.

FIFO Queue

If you send messages to a FIFO (First-in, First-out) queue remember to include the following attributes:

--message-deduplication-id: What is a duplicate and what is not (necessary if ContentBasedDeduplication is false)

--message-group-id: Which message group does the message belong to

Receiving Messages from a Queue

For consumers, it is needed to poll messages. With AWS many services can be used as an event source mapping (e.g. lambda) and you don't need to take care of it.

In case you have a custom system you need to poll the messages.

With the receive-message command this is easily possible.

If we want to get our message from the send-message command we simply need to call the receive-message command with the queue URL.

aws sqs receive-message --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1

This will give us the following result:

{
    "Messages": [
        {
            "MessageId": "a104a85f-7ca4-483e-9f10-9e82a7ce7e1f",
            "ReceiptHandle": "AQEBmPvjM/hX33ri2SP8sxf8ZqmEAmuE1BgQgHi6eQ+RZn1qm0DjrimFJgg1GuMyGLKruSFuGRY9BS6u5F83yr/OujIcttjV8ywuTzhNYvVJf2TJ0Y2tmjEY3pGk14eIgjDI22cqSuj080hy7aLdx3BR8XdoEJyE2RTs+JPNcvmd6iO03CS0A8ElpZxIqIusdsP4+1DA5PHFvNgM42b2x+WoF7Ue/irr7Vol0Ut0FPMtIq3osmVlrD4iFRjzwBMaPWZuQA/sQyE9AOKfg9bcnptQeSThdraOfI+Qdfozu/vjGzKGgJjxUU0kdr6PvQg4cc0UmbYgkECJu+OL5EIKu2j9ptqkX2ezq7EA1+5tEi3HYHtohKGeeBsG3VBpEQ0BVbUzqJXAfmbDX3Qug9S9ewT1zg==",
            "MD5OfBody": "0096e740711f375cbdc1a47ac6864b49",
            "Body": "test"
        }
    ]
}

Import Parameter

--queue-url: URL to get the messages from

--attribute-names: Which attributes to get additional (e.g. All)

--message-attribute-names: The maximum number of messages to get

Receive messages in a loop

If you're building a system for SQS you don't want to manually execute the receive-message command all the time. You want to have a loop to constantly get messages.

For that you can use the following command:

while sleep 1; do aws sqs receive-message --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1; done

With that, you will get the messages constantly and can start working on them.

Delete a message

After working on a message you need to delete the message. If you don't remove it it will be picked up multiple times from another consumer.

You can do that by using the delete-message command.

aws sqs delete-message --queue-url https://sqs.eu-central-1.amazonaws.com/661734112387/myQueue1 --receipt-handle AQEBmPvjM/hX33ri2SP8sxf8ZqmEAmuE1BgQgHi6eQ+RZn1qm0DjrimFJgg1GuMyGLKruSFuGRY9BS6u5F83yr/OujIcttjV8ywuTzhNYvVJf2TJ0Y2tmjEY3pGk14eIgjDI22cqSuj080hy7aLdx3BR8XdoEJyE2RTs+JPNcvmd6iO03CS0A8ElpZxIqIusdsP4+1DA5PHFvNgM42b2x+WoF7Ue/irr7Vol0Ut0FPMtIq3osmVlrD4iFRjzwBMaPWZuQA/sQyE9AOKfg9bcnptQeSThdraOfI+Qdfozu/vjGzKGgJjxUU0kdr6PvQg4cc0UmbYgkECJu+OL5EIKu2j9ptqkX2ezq7EA1+5tEi3HYHtohKGeeBsG3VBpEQ0BVbUzqJXAfmbDX3Qug9S9ewT1zg==

To remove a message you need the queue-url and the receipt-handle. You get the handle together with the message.

Important Parameters

--queue-URL: URL of the queue

--receipt-handle: Identifier of the message to delete

This will delete our sent message from earlier.

Purge a Queue

In case you want to purge a whole queue, this is also possible via the CLI. You can use the purge-queue command.

aws sqs purge-queue https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1

This will remove all messages from the queue.

Important Parameters

--queue-url: URL of the queue

Delete a Queue

Finally, we can also remove queues with the delete-queue command.

aws sqs remove-queue --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1

aws sqs remove-queue --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myCustomizedQueue.fifo

After executing both of these we will see that all queues are gone.

image.png

ServerlessQ

With these CLI commands you can use SQS in its full capability just from the CLI. You don't need to access the Web UI once.

You see that SQS is an amazing service but there is a lot to understand and definitely some overhead involved.

If you want to use the full capabilities of SQS without the overhead you can try out serverlessq.com. We offer the same capabilities but give you an easier entry-point to do it.

  • No configuration
  • No AWS Account
  • You won't need to read more about SQS!

Give it a try at serverlessq.com ๐Ÿš€

Thanks for reading ๐Ÿ˜Ž

ย 
Share this