TheLazy.Dev
Portfolio GitHub

Blog

iam |aws |tutorial |security

AWS CLI - do it right

pawelpiwosz

Paweł Piwosz

Posted on February 20, 2022

AWS CLI - do it right

This time something easier. This time I show how to install AWS CLI 2, add autocompletion and possibility to use MFA from command line.

Simple doesn't mean short, however :) Also, the complexity of what will be done is growing through the tutorial.

We will go through the whole process of preparing our CLI to work with reasonably secure way. All these steps might be found on AWS documentation, but in multiple documents. I used AWS documentation to check myself during work on this tutorial :)

What it is about then?

  • First, we will create an user which we will use in the tutorial
  • We will install AWS CLI
  • And verify it with gpg
  • We will add auto-completion to make our lives easier
  • Then we will enable MFA for our user
  • And on the end we change almost everything to make it right.

So, let's get started!

IAM

First, we need to create a user with MFA enabled.

I created standard user, I called him clitest (very creative) and I enabled programmatic access only. To do it, go to IAM, click Users menu and Create user.

Create user

For this tutorial I attached the policy directly to the newly created user, but DO NOT DO IT IN REAL WORLD! This user here is a prerequisite and we do not use it in real work. Maybe I'll write tutorial with best practices about users :)

Attach policy

On the end of the user create process, I copied the credentials.

Ok, user is created. Navigate to his configuration, click Security credentials tab and now it is time to enable MFA.

Security credentials

The process is quite simple. Click Manage next to the setting called Assigned MFA device (the red oval on the screen above).

Enable MFA, step 1

With Virtual MFA device the process is almost instant. It is required that you have some MFA app installed (on your phone, not your laptop, where you usually login to the service; this should be a Multifactor Authentication!). There are many applications which can be used. Google Authenticator, Duo, to name two of them.

So, select Virtual MFA device, and click Continue.

Enable MFA, step 2

Here we can see what is needed. Installation of compatible app (and list of these apps is provided to you by AWS), then open your app, scan the code, and put two number sequences into proper fields. Done!

User fully created and secured

So, we have created IAM User with MFA enabled.

It is time to...

Install AWS CLI

In this tutorial I show how to install AWS CLI on Linux. In fact, CLI can be installed on Mac (with brew or from command line) or Windows (with Chocolatey or from Powershell), or as a Python package (with pip) as well.

First, we need to download the package. Please remember, we work on the latest version here. Commands might be different when specific version is in scope.



wget "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"
Enter fullscreen mode Exit fullscreen mode

Before we unzip the file, we will do...

Integrity verification

We do it to make sure that this is valid and proper package. We will verify the PGP signature of downloaded file with the one provided by AWS.

I assume you have gpg installed, if not, please install it. First, we need to create public key file using the snippet provided by AWS. So, we simply create a text file and paste there this code:



-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBF2Cr7UBEADJZHcgusOJl7ENSyumXh85z0TRV0xJorM2B/JL0kHOyigQluUG
ZMLhENaG0bYatdrKP+3H91lvK050pXwnO/R7fB/FSTouki4ciIx5OuLlnJZIxSzx
PqGl0mkxImLNbGWoi6Lto0LYxqHN2iQtzlwTVmq9733zd3XfcXrZ3+LblHAgEt5G
TfNxEKJ8soPLyWmwDH6HWCnjZ/aIQRBTIQ05uVeEoYxSh6wOai7ss/KveoSNBbYz
gbdzoqI2Y8cgH2nbfgp3DSasaLZEdCSsIsK1u05CinE7k2qZ7KgKAUIcT/cR/grk
C6VwsnDU0OUCideXcQ8WeHutqvgZH1JgKDbznoIzeQHJD238GEu+eKhRHcz8/jeG
94zkcgJOz3KbZGYMiTh277Fvj9zzvZsbMBCedV1BTg3TqgvdX4bdkhf5cH+7NtWO
lrFj6UwAsGukBTAOxC0l/dnSmZhJ7Z1KmEWilro/gOrjtOxqRQutlIqG22TaqoPG
fYVN+en3Zwbt97kcgZDwqbuykNt64oZWc4XKCa3mprEGC3IbJTBFqglXmZ7l9ywG
EEUJYOlb2XrSuPWml39beWdKM8kzr1OjnlOm6+lpTRCBfo0wa9F8YZRhHPAkwKkX
XDeOGpWRj4ohOx0d2GWkyV5xyN14p2tQOCdOODmz80yUTgRpPVQUtOEhXQARAQAB
tCFBV1MgQ0xJIFRlYW0gPGF3cy1jbGlAYW1hem9uLmNvbT6JAlQEEwEIAD4WIQT7
Xbd/1cEYuAURraimMQrMRnJHXAUCXYKvtQIbAwUJB4TOAAULCQgHAgYVCgkICwIE
FgIDAQIeAQIXgAAKCRCmMQrMRnJHXJIXEAChLUIkg80uPUkGjE3jejvQSA1aWuAM
yzy6fdpdlRUz6M6nmsUhOExjVIvibEJpzK5mhuSZ4lb0vJ2ZUPgCv4zs2nBd7BGJ
MxKiWgBReGvTdqZ0SzyYH4PYCJSE732x/Fw9hfnh1dMTXNcrQXzwOmmFNNegG0Ox
au+VnpcR5Kz3smiTrIwZbRudo1ijhCYPQ7t5CMp9kjC6bObvy1hSIg2xNbMAN/Do
ikebAl36uA6Y/Uczjj3GxZW4ZWeFirMidKbtqvUz2y0UFszobjiBSqZZHCreC34B
hw9bFNpuWC/0SrXgohdsc6vK50pDGdV5kM2qo9tMQ/izsAwTh/d/GzZv8H4lV9eO
tEis+EpR497PaxKKh9tJf0N6Q1YLRHof5xePZtOIlS3gfvsH5hXA3HJ9yIxb8T0H
QYmVr3aIUes20i6meI3fuV36VFupwfrTKaL7VXnsrK2fq5cRvyJLNzXucg0WAjPF
RrAGLzY7nP1xeg1a0aeP+pdsqjqlPJom8OCWc1+6DWbg0jsC74WoesAqgBItODMB
rsal1y/q+bPzpsnWjzHV8+1/EtZmSc8ZUGSJOPkfC7hObnfkl18h+1QtKTjZme4d
H17gsBJr+opwJw/Zio2LMjQBOqlm3K1A4zFTh7wBC7He6KPQea1p2XAMgtvATtNe
YLZATHZKTJyiqA==
=vYOk
-----END PGP PUBLIC KEY BLOCK-----
Enter fullscreen mode Exit fullscreen mode

(this is available in public documentation on AWS pages. Please verify if nothing changed.)

Import the signature (as I am very creative, I named my file... signature)



gpg --import signature
Enter fullscreen mode Exit fullscreen mode

The output should be like this:



gpg: keybox '/home/vagrant/.gnupg/pubring.kbx' created
gpg: /home/vagrant/.gnupg/trustdb.gpg: trustdb created
gpg: key A6310ACC4672475C: public key "AWS CLI Team <aws-cli@amazon.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
Enter fullscreen mode Exit fullscreen mode

Now it is time to download the signature from remote:



wget https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip.sig
Enter fullscreen mode Exit fullscreen mode

We are ready to verify the package



gpg --verify awscli-exe-linux-x86_64.zip.sig awscli-exe-linux-x86_64.zip
Enter fullscreen mode Exit fullscreen mode

The output should be similar to



gpg: Signature made Thu Feb 17 23:19:34 2022 UTC
gpg:                using RSA key FB5DB77FD5C118B80511ADA8A6310ACC4672475C
gpg: Good signature from "AWS CLI Team <aws-cli@amazon.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: FB5D B77F D5C1 18B8 0511  ADA8 A631 0ACC 4672 475C
Enter fullscreen mode Exit fullscreen mode

Now it is the time to unzip the file.



unzip awscli-exe-linux-x86_64.zip
Enter fullscreen mode Exit fullscreen mode

All files are unpacked to aws directory. We can remove not needed files now.



rm awscli-exe-linux-x86_64.zip*
Enter fullscreen mode Exit fullscreen mode

It is time to do the installation. 



sudo aws/install
Enter fullscreen mode Exit fullscreen mode

In my case (Ubuntu 20.04), CLI was installed in /usr/local/bin/aws. As I have path configured, I can easily use



aws --version
Enter fullscreen mode Exit fullscreen mode

So, this step is done too!

Auto-complete

Well, AWS CLI is cool, but typing all these long commands in the terminal... no, thank you.

Example:



aws cloudformation batch-describe-type-configurations .......
Enter fullscreen mode Exit fullscreen mode

Well, it is not comfortable. But we can have auto-completion! Let's configure it now (for bash on Linux).

First, AWS recommends to check if we have one specific piece of software and if this piece is in proper directory. Let's check



which aws_completer
Enter fullscreen mode Exit fullscreen mode

In my case all is ok:



/usr/local/bin/aws_completer
Enter fullscreen mode Exit fullscreen mode

If it isn't for you, you have to find the executable, and make sure it is in your PATH.

In case of bash, add to your .bashrc (or .bash_profile) this line



complete -C '/usr/local/bin/aws_completer' aws
Enter fullscreen mode Exit fullscreen mode

And reload it



source ./.bashrc
Enter fullscreen mode Exit fullscreen mode

And that is it!

Let's type 



aws clo
Enter fullscreen mode Exit fullscreen mode

And and press < TAB > twice. First, it will complete to cloud. Hit < TAB > once more:



aws cloud
cloud9             clouddirectory     cloudfront         cloudhsmv2         cloudsearchdomain  cloudwatch
cloudcontrol       cloudformation     cloudhsm           cloudsearch        cloudtrail
Enter fullscreen mode Exit fullscreen mode

Perfect.

CLI configuration

It is time to configure our credentials. Type



aws configure
Enter fullscreen mode Exit fullscreen mode

Pass Access Key for the user we created in first step, next Secret Access Key, for Region use your default one (for me it is eu-central-1), and the output can be left empty.

And... that's it!

Let's test it!



aws s3 ls
Enter fullscreen mode Exit fullscreen mode

Hint: This is the fastest way to check if your credentials work :)

And... they work! We saw bucket list (or emplty list).

But hey! One moment please! What about... MFA??? Where is it?

Well... Good catch! MFA works, but only for interactive login (and during the login process).

Now you think "so... why? Why we did it?"

My answer is: Surprise! I promised to write the tutorial for the IAM Users, so... now it will come! :)

The easy way to work with MFA from CLI

IAM Policy with conditions

The point here is that our policy which we used does not expect to control if session is or not using MFA. As you remember, during the assignment of the device, we saw the message, which explained it will be used during the login (in fact, interactive login to the GUI console). But we have a way to force IAM to control it for CLI as well.

First, we need to create new policy. Go to IAM service, Policies, click Create policy button.

Let's use visual editor for this case. Select S3 for Service. All actions for Actions, and All resources for Resources, like on picture below.

Create new policy, step 1

This is the way, how standard policy looks like. Now, we will add condition for MFA.

Open the Request conditions field and check MFA required. And that's it!

Create new policy, step 2

Finish the policy, and save it. I named mine s3mfa.

The final policy looks like this:



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*",
            "Condition": {
                "BoolIfExists": {
                    "aws:MultiFactorAuthPresent": "true"
                }
            }
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Now in User groups create a new group, and attach our clitest user and s3mfa policy.

Create user group

Last step is to remove previously attached policy (which we attached to the user). This way the user will use only these policies which are attached to the group.

What we will see when we try to run our previous command in CLI?



aws s3 ls
Enter fullscreen mode Exit fullscreen mode

Yes, we cannot list the buckets.



An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
Enter fullscreen mode Exit fullscreen mode

Access from CLI with MFA

First, we need to get the information about ARN of our MFA device.

Go to IAM user, Security credentials and find the Assigned MFA device. Previously it was empty, now you should see the ARN.

Assigned MFA device

Copy this ARN.

In CLI type



aws sts get-session-token --serial-number arn:aws:iam::<ACCOUNT>:mfa/clitest --token-code <CODE>
Enter fullscreen mode Exit fullscreen mode

Where:

  • ACCOUNT is the account id (don't worry, you have it in your ARN copied from your user)
  • CODE is the code displayed in your MFA device (Duo, Authenticator, etc)

You should receive something like this:



{
    "Credentials": {
        "AccessKeyId": "ACCESKEY",
        "SecretAccessKey": "<SECRETKEY>",
        "SessionToken": "aVeryLongStringHere",
        "Expiration": "2022-02-20T12:27:08+00:00"
    }
}
Enter fullscreen mode Exit fullscreen mode

Ok, you even see when you access will expire!

Now try 



aws s3 ls
Enter fullscreen mode Exit fullscreen mode

If you configured everything correctly, it will show the buckets list.

The recommended way to work with MFA from CLI

While it works, and does the job, it is not the best practice. It works for small scale, but in properly configured environment we should use IAM Roles.

When you executed aws configure, something happened. .aws directory was created (if you did it for the first time) and the default profile was configured.

Now we will do a lot of modifications.

First, please remove some stuff we did earlier:

  • The IAM User Group created
  • The IAM Policy where we configured MFA

We don't need them anymore.

Now we will create an IAM Role, and we name it mfarole (creativity, remember!)

Create role

As you can see, I simply added S3FullAccess policy.

We need to modify Trust relationship and pass there our user.

Modify Trust relationship

Now we are ready do this:

  • As user clitest
  • Assume role mfarole
  • With MFA enabled
  • and do what we want to do.

It is time to modify our .aws/credentials



[default]
aws_access_key_id = <ACCESS KEY>
aws_secret_access_key = <SECRET KEY>

[mfarole]
role_arn = arn:aws:iam::<ACCOUNT>:role/mfarole
source_profile = default
mfa_serial = arn:aws:iam::<ACCOUNT>:mfa/clitest
Enter fullscreen mode Exit fullscreen mode

in mfarole section we defined what role we want to assume (role_arn), as which user (source_profile) and also we added mfa_serial, which do great job. Let's see it on this example:



 aws sts get-caller-identity

{
    "UserId": "<USERID>",
    "Account": "<ACCOUNT>",
    "Arn": "arn:aws:iam::<ACCOUNT>:user/clitest"
}
Enter fullscreen mode Exit fullscreen mode

we get the info about our default user.



aws sts get-caller-identity --profile mfarole
Enter MFA code for arn:aws:iam::<ACCOUNT>:mfa/clitest:
{
    "UserId": "<USERID>:botocore-session-1645321182",
    "Account": "<ACCOUNT>",
    "Arn": "arn:aws:sts::<ACCOUNT>:assumed-role/mfarole/botocore-session-1645321182"
}
Enter fullscreen mode Exit fullscreen mode

And here about the assumed role by the user. But! We had to provide the token! Exactly what we expected.



aws s3 ls
aws s3 ls --profile mfarole
Enter fullscreen mode Exit fullscreen mode

Your pure user is not able to list buckets, but your profile is. It means that the user has to assume the role in order to operate.

Is this better practice?

Yes, it is. For multiple reasons.

  • It should be default, that your user assumes the role. This is another layer of security. Treat it as authentication & authorization, separated by function (user & role)
  • Anyway, with Organizations or SSO, etc it works (almost) this way, so why to not use it?
  • Once enabled, it is enabled for all actions taken through CLI. Not selected and configured in IAM Policices, where it is very easy to overlook.
  • It brings some more complexity, but it is worth it.
  • From security perspective - the user itself is "blocked". He cannot do anything. Any activites are possible after assuming the role. Therefore you need to know two elements of the puzzle.

Summary

On the end, we should have:

  • AWS CLI installed and configured
  • IAM User credenrtials created and stored in .aws/credentials
  • IAM Role configured in .aws/credentials
  • This IAM Role is configured with MFA enabled and with Trust Relationship for our IAM User
  • The Role has attached standard policies (In our case full access to S3)
  • The User doesn't have any policy attached (default deny everything)

With this setup, I repeat it, but this is imperative to understand, the IAM User is not able to perform any action. It is used as a "carier" for the IAM Role we created.

Why we created an IAM Policy then

I am sure you ask yourself about it. Why we created the Policy if we removed it very quickly. Well, I did it on purpose. It is learning by doing. When you created this policy by yourself, attached to user, you probably had the thought in your mind "but this will not work very well! I can easily workaround it and do not use MFA at all!". If you had this thought, than I achieved my goal, and you better understand the approach to programmatically access the AWS services with IAM Role.

💖 💪 🙅 🚩
pawelpiwosz
Paweł Piwosz

Posted on February 20, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

AWS CLI - do it right
iam AWS CLI - do it right

February 20, 2022