TheLazy.Dev
Portfolio GitHub

Blog

s3 |wasabi |aws |docker

AWS S3 CLI docker for the impatient (also wasabi)

jaeyson

Jaeyson Anthony Y.

Posted on October 26, 2020

AWS S3 CLI docker for the impatient (also wasabi)

What's the problem with AWS cli guide (it has good docs anyway)? well, nothing really complex unless you really want to use it asap. This is more of a basic setup and quick run using docker.

0. Things needed: Docker and AWS S3 creds (access key id/secret access key)

There's a guide for linux distro, and a post-installation steps to run Docker as non-root

1. Pull Docker image 

docker image pull amazon/aws-cli
Enter fullscreen mode Exit fullscreen mode

2. Make credentials

Create ~/.aws dir if not exists, then inside that dir create a file credentials and config:

~/.aws/credentials

[default]
aws_access_key_id=fffff3333ff55555GGGG
aws_secret_access_key=kkkkkkkkkkkkkkkeeeeeeeeeeeeeyyyyyyyyyyyy
Enter fullscreen mode Exit fullscreen mode

~/.aws/config

[default]
region=us-west-2
output=json
Enter fullscreen mode Exit fullscreen mode

Or use an interactive setup

# interactive setup for creds
docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure
Enter fullscreen mode Exit fullscreen mode

Check creds:

docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure

# or view
cat ~/.aws/credentials
Enter fullscreen mode Exit fullscreen mode

3. cp copy, sync , mv move and ls list files

3.a. cp copy command

$ aws s3 cp <source> <target> [--options]

source could mean your files from your computer or from S3

Add --recursive option if dir:

# upload a local file hyena.png (current dir) to mammals dir
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  cp hyena.png \
  s3://animal-bucket/mammals/
Enter fullscreen mode Exit fullscreen mode 
# download lynx.png from s3 to current dir
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp s3://animal-bucket/mammals/lynx.png ./
Enter fullscreen mode Exit fullscreen mode 
# upload all contents from reptiles dir (local) to s3
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp reptiles/ s3://animal-bucket/reptiles/ \
  --recursive
Enter fullscreen mode Exit fullscreen mode

3.b ls command 

# list all files in reptiles dir
docker container run --rm \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli s3 \
  ls s3://animal-bucket/reptiles/
Enter fullscreen mode Exit fullscreen mode

e.g. get the size of a bucket

docker container run --rm \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli s3 \
  ls s3://animal-bucket/ \
  --summarize \
  --human-readable \
  --recursive \
  --profile profile-name \
  --endpoint-url=https://s3.<region>.<provider>.com
Enter fullscreen mode Exit fullscreen mode

3.c sync command

aws s3 sync <source> <target> [--options]

Using cp will require the --recursive option to copy multiple files in a dir, while sync command will, by default, copy a whole directory. It will only copy new/modified files.

Add --delete option to remove <target> files that aren't in <source> dir.

# using new-reptiles folder as basis
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  sync new-reptiles/ s3://animal-bucket/old-reptiles/
Enter fullscreen mode Exit fullscreen mode

3.c mv move command

aws s3 mv <source> <target> [--options]

# Only markdown files are moved from new-reptiles
# to animal-bucket/old-reptiles. The rest aren't included.

docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  mv s3://new-reptiles/ s3://animal-bucket/old-reptiles/ \
  --recursive --exclude "*" --include "*.md" \
  --profile wasabi \
  --endpoint-url=https://s3.<region>.<provider>.com
Enter fullscreen mode Exit fullscreen mode

The rest of the commands can be found in their docs.

4. Using another cloud storage (ie Wasabi)

Same thing with step 2:

~/.aws/credentials

[default]
aws_access_key_id=fffff3333ff55555GGGG
aws_secret_access_key=kkkkkkkkkkkkkkkeeeeeeeeeeeeeyyyyyyyyyyyy
[wasabi]
aws_access_key_id=wwwaaasssaaabbbiiiii
aws_secret_access_key=wwwwwwwaaaaaassssssaaaaaabbbbbbbiiiiiiii
Enter fullscreen mode Exit fullscreen mode

~/.aws/config

[default]
region=us-west-2
output=json
[wasabi]
region=us-east-1
output=table
Enter fullscreen mode Exit fullscreen mode

Or use an interactive setup. Notice I added --profile <PROFILE_NAME> option:

# interactive setup for creds
docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure --profile wasabi
Enter fullscreen mode Exit fullscreen mode

Check creds:

docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure --profile wasabi
Enter fullscreen mode Exit fullscreen mode

Doing some commands (with --profile and --endpoint-url):

docker container run --rm \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli s3 \
  ls s3://wasabi-bucket/ \
  --profile wasabi \
  --endpoint-url=https://s3.us-west-2.wasabisys.com
Enter fullscreen mode Exit fullscreen mode

Shorten the command with alias:

alias aws=$(docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli)
Enter fullscreen mode Exit fullscreen mode

Errors 

An error occurred (NoSuchTagSetError) when calling the GetObjectTagging operation: There is no tag set associated with the bucket.
Enter fullscreen mode Exit fullscreen mode

If you came across this error while using sync, a fix could be either:

  • Grant the s3:GetObjectTagging permission to that user, or
  • adding this option --copy-props metadata-directive (it tells the cli not to get source object tags).

source can be found here.

💖 💪 🙅 🚩
jaeyson
Jaeyson Anthony Y.

Posted on October 26, 2020

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

AWS S3 CLI docker for the impatient (also wasabi)
s3 AWS S3 CLI docker for the impatient (also wasabi)

October 26, 2020