Syncing files from AWS S3 to local storage on Rails

I usually use Amazon S3 for storing images with Active Storage. There are some issues when pulling my database to work with it on localhost:

I don't want to accidentally delete something on S3
I don't want my localhost without images 🥲

I added this rake task to download files from S3 and format them in a way that works with local storage:

# lib/tasks/s3_to_local.rake 
desc "Sync files from S3 bucket to local storage folder"

task :s3_to_local do

  s3_bucket = "YOUR_S3_BUCKET"
  access_key_id = Rails.application.credentials.dig(:aws, :access_key_id)
  secret_access_key = Rails.application.credentials.dig(:aws, :secret_access_key)

  storage_folder = Rails.root.join('storage')
  storage_folder.mkpath

  system("AWS_ACCESS_KEY_ID=#{access_key_id} AWS_SECRET_ACCESS_KEY=#{secret_access_key} aws s3 sync s3://#{s3_bucket} #{storage_folder}")

  # Ignores sub_folders already created and .keep files
  images = storage_folder.children.select { |file| file.file? && !file.empty? }

  # Formats the file path of each image so ActiveStorage understands them using :local storage
  images.each do |path_name|
    dir, basename = path_name.split
    file_name = basename.to_s
    sub_folders = dir.join(file_name[0..1], file_name[2..3])
    sub_folders.mkpath # Create the subfolder used by active_record
    path_name.rename(dir + sub_folders + basename) # Renames file to be moved into subfolder
  end

end

What makes this tricky is that S3 storage has everything in the root of the bucket, and when using local storage with ActiveStorage, everything should be under a subfolder.

Also, you need AWS CLI for this to work:

brew install awscli

Basically copy this task into lib/tasks/s3_to_local.rake, add your bucket name, your AWS credentials, run rake s3_to_local and you're good to go

Blog

Syncing files from AWS S3 to local storage on Rails

Lucas Kuhn

Join Our Newsletter. No Spam, Only the good stuff.

Related