Migrate Data into S3 using AWS Transfer Family
Robina
Posted on July 19, 2022
What is AWS Transfer Family?
AWS Transfer Family is a fully managed AWS service that you can use to transfer files into and out of Amazon Simple Storage Service (Amazon S3) storage or Amazon Elastic File System (Amazon EFS) file systems over the following protocols:
- Secure Shell (SSH)
- File Transfer Protocol (SFTP)
- File Transfer Protocol (SFTP)
- File Transfer Protocol Secure (FTPS)
- File Transfer Protocol (FTP)
Read more: AWS Transfer Family
Benefits Of AWS Transfer Family?
- It is a fully managed, serverless File Transfer Workflow service that makes it simple to set up, execute, automate, and monitor file processing utilizing AWS Transfer Family.
- With your data safely stored in Amazon S3, you can utilize native AWS services for processing, analytics, reporting, auditing, and archiving.
- There are no upfront costs, and you pay only for the use of the service.
- You don't need to modify your applications or run any file transfer protocol infrastructure.
How it works?
AWS Transfer Family Architecture
AWS Transfer Family (SFTP, FTP, FTPS) – Step by Step Implementation Process
Setup the prerequisites for AWS transfer for SFTP
- S3 bucket
- EC2 instances (Linux and Windows)
Create an AWS IAM role and policy
- Edit the policy to provide the S3 access
Create the SFTP server
- Create users
- Create and assign public keys
Test the file transfer from SFTP clients
- For Linux
- For Windows
What is S3 Bucket?
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.
Read more: Amazon Simple Storage Service
Create a S3 Bucket
- Open AWS Console Click on Services
- In the Storage Section, Choose S3
- Click Create Bucket and Create Your Bucket
- Give Bucket Name (Should be unique Name use
small alphabets Without Space. hyphens and Periods}
- Choose your Region. (Where user wants to store their data)
rest settings choose default.
Create a VPC
- Open AWS Console Click on services
- In the Networking & Content Delivery Section, Click on VPC
Create Subnets
Create Two EC2 Instances (Linux & Windows)
- Open AWS Console Click on services
- In the Compute section, click on EC2
- Click Launch instance
Security Group rules for Linux Instance
Security Group rules for Windows Instance
Create an IAM Role
- Open AWS Console Click on services
- In the Security, Identity & Compliance section, click on IAM
Create IAM roles and policies from the IAM console (Choose AWS service as Transfer)
The policy to access the S3 bucket
Policy to access CloudWatch Logs
Create two roles and attached the policies with each role (Choose AWS service as Transfer)
Create the SFTP server, Users, Public Keys and Assign the keys
- Open AWS Console Click on services
- In the Migration & Transfer section, click on AWS Transfer Family
Click on create Server
Choose Protocols
Choose an Identity Provider
Choose an Endpoint
Choose a Domain
Add User to the Server
Generate the public key from the Linux machine?
Log in to your EC2 machine connect via Putty
Generate the key with the below command
ssh-keygen -P "" -f key
Transfer the files using WinSCP for Linux
Test the file transfer from the Linux SFTP client
Connect to the server
- sftp -i key rm-transfer-family-user@
s-c0ca408efe3d4c3e8.server.transfer.ap-southeast-1.amazonaws.com
- put AWS-Transfer-Family-Linux.txt rm-transfer-family-2022
AWS-Transfer-Family-Linux.txt
to/rm-transfer-family-2022/rm-transfer-family-user/AWS-Transfer-Family-Linux.txt
Successfully transfer File from Server to Bucket
Now do the same using the Windows Machine
Generate Public key for Windows user
Two Public keys are added
Now transfer the files using WinSCP for Windows
Connection established using WinSCP for Windows
Test the file transfer from SFTP client for Windows
Successfully Tested the file transfer from SFTP client and added to the Bucket
💖 💪 🙅 🚩
Robina
Posted on July 19, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.