Migrating Servers Serverlessly — part 1

A two-part guide to migrating servers on AWS

Lift-and-shift migrations are a common pattern for the rapid movement of hosts into a cloud environment. Formally known as a rehost migration, this involves the creation of a like-for-like replica of the host in a targeted environment using an agent-based, block-level replication technique. Services such as CloudEndure and the AWS Application Migration Service facilitate this, providing the on-host replication agent and a dashboard from which the replications can be monitored.

As AWS cloud has become a bedrock technology, opportunities to lift-and-shift data centres of much larger scale and complexity have become available. Further, there is a growing demand for performing increasingly complex changes to the hosts in the target environment. There are several challenges that exist in dealing with this additional scale and complexity:

• Migrations require the attention of delivery managers and engineers, and this scales linearly as the number of servers grows, increasing the amount of process and organisational structure dedicated to the project.
• The migration process incurs resource costs, including the cost of running orchestration resources. By making the process cleaner and cheaper to operate, there is less attention required by engineers, and the costs can be put towards more cloud resources.
• Lift-and-shift migrations are known for their focus on velocity to the production cloud. If a rehost could be automated to adhere to AWS Well-Architected Framework, we could get the best of both worlds: a host moved to the cloud aligning to gold standard practices.
• With the entire server landscape of some data centres moving into the cloud, cyber security teams will no doubt be concerned that security guardrails and policies are aligned. Automating processes makes following guardrails and policies easy as they are baked into the solution.

This has meant more robust approaches to orchestrating server cutovers have had to be devised to deal with the potential complications. This includes state machines to structure the rehosting process to better orchestrate the actions needed when cutting over a host and adding more robust host bootstrapping procedures to handle the modifications in a structured, parallel manner.

If this challenge can be met, migrating more hosts that contain more complexity with a smaller team would open the door to enterprises realising the benefits of cloud practices earlier, allowing businesses to focus on what differentiates them from their competitors and maintain relevancy.

Serverlessly Orchestrating Rehosts

At Versent, we have an automation-first mindset and a pedigree of building solutions that make the best use of the services available through the AWS cloud. Serverless has grown in maturity and adoption and is seen as the gold standard method for hosting applications in the cloud and managing automated processes. AWS has adopted serverless patterns as a first-class citizen across most of its core services. For this reason, we see orchestrating migrations using AWS native serverless services as the best way forward for teams to manage rehosts, augmented with automated serverless tools.

With this in mind, this article will cover the following in two parts:

• part one will focus on some foundational rehosting tools you can use to accelerate a migration into the cloud, including application migration service, application discovery service and the AWS cloud migration factory.
• with part one laying the foundation, part two will focus on using the latest serverless AWS orchestration technologies to extend further on these solutions to manage greater scale and better align to AWS’s well-architected guidelines.

To begin, let’s look at the foundational tool for managing rehosts, the Application Migration Service (MGN).

Application Migration Service

MGN provides a method for replicating a server into the AWS networking environment using a host-based agent to securely transmit volume block-level data of the on-premises host into a replication server waiting in a temporary landing subnet. Once the server’s contents are fully transferred, the replication server can be imaged, and the host can be moved to its own network tenancy with desired rules to operate in the same manner as the original, except now it’s in the Amazon cloud.

MGN manages the block-level replication of servers into the cloud, providing a machine image snapshot of the host at a point in time

This functionality was originally provided by the service CloudEndure, which has now gone end-of-life, and as such, AWS Application Migration Service has taken its place. CloudEndure was an excellent replication tool with a strong API to automate against and a valuable service dashboard that provided easy-to-understand details about the host’s existing and target state configuration and replication status. CloudEndure has since been bought by Amazon and has been integrated into the AWS service family of offerings.

This transition between services has provided some uplift to rehosts that’s worth being aware of:

• Licensing: AWS has taken a different approach to licensing; CloudEndure licenses would expire after 90 days, while MGN licenses do not expire after 90 days; however, there is an associated service charge.
• Bootstrapping: The CloudEndure replication agent installed on a migrated host executes a bootstrap script from a specific folder. In MGN, an SSM agent is installed on hosts, allowing bootstrap to occur via SSM documents, providing a clearer view of what is executing and better feedback if issues occur.
• Host Templating: CloudEndure blueprints, which dictated the state of the host in the target AWS environment, have been replaced with the AWS native Launch Template.
• Encryption: The machine images created by CloudEndure could be created in an unencrypted state. In MGN, encryption is enforced via the default account KMS key or via a custom-created KMS key.

While CloudEndure is no longer available to most regions (it is still limitedly available until Nov 2024), MGN provides all the same functionality and includes many great security and best-practice improvements. Versent definitely endorses the use of the AWS Application Migration Service for the management of application rehosts.

Given that MGN is an AWS native service, it now aligns with the Amazon API-first mandate, and it can be easily integrated into server-less automation with a minimal code footprint. This ability fits well with the next service I will cover, the AWS Application Discovery Service, or ADS, which you can use to get a more detailed view of hosts.

Application Discovery Service

In large organisations, it’s often the case that there’s some server that provides a service, but the original owner has since left the building and didn’t provide any documentation. Customers are generally unwilling to perform a scream test, so running a discovery tool on the service can provide critical details that can mean turning anything off is no longer necessary. This is where AWS Application Discovery Service comes in great use; ADS can assist in building network maps to identify clusters of host interactions and get a view of application dependencies, uncovering network relationships that could stop an application from migrating and informing what traffic might be necessary.

An example network diagram generated by ADS. The orange node is the source server, and bold nodes are hosts known to ADS; the question mark nodes are unknown hosts being communicated to from the source or another known host.

ADS provides this service with an agent (or agentless) based method that can be used to collect host details. The agent-based approach is particularly useful when talking to security teams about collecting information from production environments with business-critical or sensitive data.

Types of Host Data

• Hostname, including changes to the name in the time since the ADS agent was first installed on the host
• Operating system and version
• Inbound and outbound communications, including target IP addresses and ports
• CPU load, available RAM, disk volume space
• Getting detailed visibility of the host function and the interactions it has with other hosts is a difficult specialist problem.

As with all AWS services, an API for the solution is exposed to query collected host information. This can be used to collect information for ETL tasks, collect details relevant to migration waves and can be provided to the customer in report form, providing automated documentation generation of the source state and giving customers a snapshot of how their applications are truly behaving.

Summary of Benefits of Application Discovery Service

• ADS agents are rolled out to migrated hosts and provide continuous recording of host behaviour. This data can be analysed to provide a complete picture of the server estate within a given timeframe.
• ADS eliminates risk in migration by showing up inbound/outbound communications that may not be obvious and processes running that might be critical to operation.
• Service can be queried using the AWS API to run your own analytics scripts to help inform host right-sizing prioritisation in the migration schedule.
• Enriches migration project with more metadata about applications as a whole.

Using both ADS and MGN provides the fine-grained “how” tools for a migration rehost; an engineer could manage a handful of lift-and-shifts using just these solutions. Adding anything more than 5–10 hosts at any given time with this approach becomes difficult to manage, considering configuration, discovery work testing, etc.

When you consider managing target configuration state, contacts for known stakeholders and discovered host metadata, a tool to help scale and take away some of the administrative burden worn by a migration team would be quite desirable. This is where a governance tool that tracks migration state between teams and provides a dashboard to stakeholders would be valuable. In the next section, we’ll review Cloud Migration Factory, which can do this for us.

AWS Cloud Migration Factory

The Cloud Migration Factory created by AWS provides a governance tool by which mass migrations can be managed behind a single pane of glass. When discussing this tool, I’m specifically referring to the optional migration tracker functionality, which provides methods to handle the bulk of server workload details that need to be tracked, analytics dashboards to provide visibility as to where workload migrations are up to when you are juggling multiple waves of 20–30 servers at a time.

The tracker feature is backed by DynamoDB so that migration teams can easily collect current and target state host information using an authenticated user through cognito and an API gateway fronted form tool.

This is ideal for independent teams contributing to a large-scale migration. This is because one team can be dedicated to discovery upstream, preparing the customer for change and business readiness. The downstream migration engineering team can handle the hard requirements of migrating the hosts. Another great benefit of the state tracker tables is being able to track tags for hosts. Tags being applied to hosts can be reviewed before migration to confirm consistency with customer policy.

Cloud Migration Factory provides dashboards and data entry functionality that helps independent migration teams keep a single source of truth and are in a format that can be used directly on the migrated hosts.

Another great feature of the solution is the migration analytics provided via QuickSight. As migration executive stakeholders are keen for feedback on the progress of what could be a significant engagement, providing metrics that show progress could give the evidence that could release additional funding or drive adoption from other departments, further extending the scope of the migration and furthering company cloud adoption.

The ability to add customer-specific custom metrics is also available to make sure you are tracking attributes relevant to the client and project.

Dashboarding for migrations is available through Quicksight and Cloud Migration Factory.

Cloud Migration Factory Benefits

• Risk Reduction: Making use of the AWS Cloud Migration Factory reduces the risk associated with the testing and production cutovers of workloads by providing a way to show enriched metadata for hosts being migrated.
• Stakeholder management and project visibility: Getting everyone informed and engaged in what the project is delivering.
• Organises horizontal and vertical migration teams: By having a tool for tracking the state of migrations and the metadata of hosts being migrated, it’s easier to organise separated migration teams with different functions, business readiness for engaging parts of a company having applications migrated and preparing them for the migration and doing discovery and parallel teams managing the technical aspects of the migration of an application that could have 20–30 hosts performing different roles.
• Reduction of double handling: Downstream migrations can directly use the information the upstream business readiness teams use.

For a walk-through of the capabilities of the cloud migration factory consider reviewing the following workshop to get familiar with the solution features.

Using MGN, ADS and the AWS Cloud Migration Factory will cover a lot of the heavy lifting in managing a migration at scale. As discussed, Cloud Migration Factory provides a lot of functionality that assists with the governance of a large-scale migration.

Final Thoughts

AWS provides many solutions and patterns to augment an at-scale cloud migration, and in this article, we have reviewed those relevant to rehosts. If you are considering moving your server infrastructure into the cloud, I recommend reviewing the AWS Cloud Adoption Framework for general considerations of the journey to the cloud and the provided lenses, then the AWS Migration Acceleration Program for a guide on how best to structure a migration into cloud-based infrastructure.

In an upcoming article, I will propose an architecture for the migration factory that uses the latest serverless AWS orchestration technologies to handle rehosts at scale over multiple accounts and aligns with AWS’s well-architected best practices. The Versent Serverless Migration Factory has matured from its humble beginnings to achieve fully automated and scale migrations with capability across an evergreen and complex AWS Landing Zone platform.

When migrations get complex, it is a core Versent principle to ensure automation takes away the effort so we focus on the right problems at the right time. We are pleased with how the Versent Serverless Migration Factory completes large and long automatic migrations and that it unlocks our engineers to spend time and add value to our customers and their workload challenges.

References

• AWS Migration Acceleration Program (MAP)
• AWS Cloud Adoption Framework
• AWS Well-Architected Framework
• Prioritization and Migration Strategy — AWS Prescriptive Guidance
• Cloud Application Migration Tool — AWS Application Migration Service — AWS
• Migration Planning — AWS Application Discovery Service — AWS
• Cloud Migration Factory on AWS | AWS Solutions | AWS Solutions Library