Modernise data transfer with AWS Transfer Family

Is your organisation looking to modernise its legacy data transfer services? If so, you’re probably investigating ways to deploy a highly available and scalable solution that can manage in-flight data transmission even during an outage or maintenance.       

This article is a guide to the new features available with AWS Transfer Family, specifically, around the SFTP service with which you can easily deploy a highly scalable, robust solution at a competitive cost.    

Amazon Web Services (AWS) offers multiple services to modernise file transfer architectures. But when it comes to data in transit between different organisations, it’s paramount to ensure that the data is secure and resilient so that no transmission loss is experienced.    

For this scenario, we’ll assume that we’re attempting migration from an existing SFTP server and that the requirements for success are maximum security and availability to ensure no loss of information via malfunction or malicious activity.    

Step 1: assessment    

Versent begins each migration by using discovery workshops and tools to define our customers’ technical and business requirements. This process analyses how the current SFTP service works and how we can maintain or improve it after migration.    

During the discovery process, Versent works closely with your Subject Matter Experts (SMEs) to identify any data that’s sensitive or critical. This helps us understand which data and connections are most vital, which will factor into the deployment of Transfer Family Services and migration utilities.    

It’s important to note that AWS Transfer Family supports multiple types of backend storage services. The selection of the appropriate storage pool should align with the approach you plan for your storage environments from a security point of view. The storage pool types available on AWS are Elastic File System (EFS) and S3.    

During discovery, Versent analyses the following:       

• Frequency of data transmission    
• Concurrency threshold    
• User account structure    
• User account quantity    
• User account security and authentication    
• User account Groups and Management    
• Data attrition    
• Downstream connections    

There are two key elements to consider in planning a smooth migration for SFTP services:     

• data preservation and replication, &    
• connectivity to the new environment (networking & authentication).    

Taking care of these two objectives eliminates data loss and preserves strong, stable, active service for users and clients.    

Step 2: deployment    

When creating AWS Transfer Family services, it’s vital to select options that are in alignment with your business objectives. Additionally, the configuration must follow cloud architecture best practices.   

Best practices during deployment are:    

• Data encryption in transit and at rest    
• Data transmission is authenticated access only    
• Authentication is via User/Pass, PSK, Certificate Material, MFA or a mix    
• Data archival retention period    
• User management authority and change control    
• Service availability and access from Public or Private endpoints    

Versent always recommends that the creation of services and environments is programmatic and templated to ensure that a code-driven state can be aligned with the AWS service deployed.    

Step 3: data migration    

The data migration process has two main parts:    

• Exporting existing and historical data and populating this data into the new Transfer Family Service    
• Regular, ongoing synchronisation & replication between legacy and target environments prior to the migration cutover    

To complete a data migration, Versent uses AWS DataSync, which allows fast, efficient, secure data migration between a source and target environment.   

It’s important to note that a private access point – private IP to private IP – and authentication are core access requirements of AWS DataSync.  

The diagram below represents data migration from on-premises to AWS Transfer Family for SFTP, illustrating the private connectivity requirements.    

Step 4: go live & testing   

It’s essential to ensure that data transmission processes capture all the data sets from a legacy environment. Versent always recommends performing multiple data transfer dry run exercises to reduce risk and build confidence.  Most potential problems that can occur during data transfer can be mitigated with this methodology: issues such as corrupted data on the legacy environment. It’s best to fix these types of issues on the source legacy environment before transfer to avoid disruptions during migration verification.       

Step 5: verification       

After performing a migration, the final essential step is to verify the data has transferred accurately. This can be achieved with file counts, sample checking of file sets, and verification of their corresponding size and security configuration. These checks will confirm that the migration was successful.   

When cutting over, take care around enabling access to the new AWS Transfer Family environment for your customer and clients, and be sure to disable the legacy environment. It’s crucial that connectivity to the legacy environment is discontinued after the new AWS Transfer Family environment has gone live. If the old FTP system is left online, it may create data inconsistencies, such as legacy data sets being further ahead than that of your new transfer environment.    

Takeaways       

So, what are the core learnings we’ve discovered through this discussion of AWS Transfer Family migration?    

• SFTP services are often not maintained or regularly reviewed as solutions for improvement due to their simple and critical nature.    
• Versent has seen that legacy SFTP environments are used once a day by a handful of clients, but those files are so valuable to business operations that many teams and business units would not function without them.    
• Dry run testing can’t be overused. At Versent, we test rigorously prior to deployment of go-live activities, so our customers have confidence in the migration process.     

Want to learn more about the benefits of modernising data transfer with SFTP serverless & AWS?    
Get in touch with a Versent expert.