American Airlines gains speed and boosts reliability by flying real-time data to the cloud with Azure

Taking flight with Azure cloud services

When travelers receive notifications, such as a gate change, check-in reminders, or seat change, they don’t often pause to consider all the technology and team effort that delivers the most up-to-the-minute details to their devices. The information is just there.

To keep up with more than 16 million real-time messages and 17.4 million service calls per day—about 32,000 database transactions per second—Customer Hub at American Airlines recently shifted its database infrastructure from on-premises systems, which it installed in the 2000s, to Microsoft Azure SQL Managed Instance servers in multiple regions to provide redundancy, minimize downtime, and fit with the American Airlines culture of modernization.

The solution supports the heartbeat of the airline. “We use our Azure-based Customer Hub for various purposes like customer notifications, real-time flight decisions, automated refunds for some ancillary transactions, and frequent flyer mileage accrual,” says Melissa Blankenbaker, Technical Delivery Manager at American Airlines. “And we provide capabilities for some customer self-service activities through aa.com. We’re responsible for identifying customers and being the source of truth for profile and preferences.”

The only route possible

When the modernization journey began in December 2021, the Customer Hub team explored multiple database solutions within Azure before deciding that Azure SQL Managed Instance could deliver what the airline needed. ”Azure SQL Managed Instance was the only option that met all our requirements,” says Jayanta Saha, Principal Data Architect at American Airlines. “We had 10 terabytes of data, a massive transaction load, and the need for geo-replication supporting an active-standby architecture. And we can’t be down for even a minute.”

Adds American Airlines Customer Hub Senior Manager Minh Duong, “Basically, Azure SQL Managed Instance was the only data store that could hold our data and process the number of transactions that we have.”

The Customer Hub team started prototyping with the Azure SQL Managed Instance General Purpose tier but soon realized that the Business Critical tier had additional features the airline needed, including geo-replication. The team provided requirements that helped Microsoft upgrade SQL Managed Instance Business Critical with additional storage to deliver all the features needed in one product.

Compartmentalized approach

The database itself—in addition to its size and the transaction volume it supports—is relational and uses third normal form schema to help ensure integrity and reduce duplication. “It’s a very complex system,” says Saha. The team spent the first six months developing and refining a proof of concept, followed by product evaluation and testing.

A vendor partner helped the team by validating their Azure architecture plan, optimizing key use cases, developing a playbook for specific types of apps to reduce duplication of effort, and providing Azure expertise to accelerate the migration.

To ensure a smooth final cutover, the team used a migration plan in phases to provide redundancy and confirm functionality through careful testing, including many cycles of complex data comparison between the old and the new databases using the Database Compare tool from Microsoft.

• Phase 1: About 30 web services are migrated to Azure App Service but continue to access the on-premises database while Azure SQL Managed Instance is being set up and tested. 
• Phase 2: One-way data replication is enabled from on-premises servers to Azure SQL Managed Instance using Azure Data Factory together with another third-party tool. The Azure apps from Phase 1 are tested with Azure SQL Managed Instance in groups defined by whether they only read data or need write permissions as well, and by the specific tables needed.
• Phase 3: About 60 back-end on-premises applications are tested with Azure SQL Managed Instance using a similar group methodology.
• Phase 4: Back-end applications are fully migrated to Azure and connect only to Azure SQL Managed Instance. A few additional non-business-critical apps are migrated after the solution launches.

These phases weren’t always distinct. Due to the volume of services and apps being migrated, Phases 2 and 3 overlapped significantly. The team spent time up front evaluating each app, merging some apps that had similar functions and transferring others to the right product groups to support the company’s product-based model. A few apps were no longer needed, so those were sunsetted prior to the migration phase.

During the project, the team took advantage of the Microsoft Enterprise Skills Initiative for related training and certifications. After the team worked through several technical hurdles related to real-time syncing of data first in the pre-production environment and later between the primary and secondary Azure regions, American Airlines officially launched its Azure-based Customer Hub on October 15, 2022. Although the team prepared a rollback plan, it was not needed due to the successful launch.

“This was a huge accomplishment. We wanted to be certain that when we did the full cutover to Azure for Customer Hub, we were ready for it,” says Ravi Tellakula, Principal Architect at American Airlines. “There was no turbulence. There were no issues. We just had a very smooth takeoff.”

With its Azure-based Customer Hub in place, American Airlines has greater flexibility. The team can spin up new infrastructure whenever it’s needed—all while maintaining high standards of security and reliability. IT team members can shift their focus from managing an on-premises environment to instead delivering business features to enhance customer experience.

Another benefit of the modernization is that products previously connected by a shared infrastructure can now be independent and self-contained. “The shared infrastructure for our products made some efforts challenging,” says Blankenbaker. “Now, we’ve been able to manage apps separately and make ourselves more independent of each other. This helps us deliver faster while still working together as one team.”

“Customer Hub’s cloud migration is part of a bigger American Airlines Delivery Transformation initiative,” Duong says. “The goal is to transform the way we deliver value through modern technologies and engineering practices. Since the transformation is continuous, we needed to change our culture to one of collaboration, empowerment, and continuous learning. This helps us operate as a passionate, selfless, and accountable team.”

Find out more about American Airlines on Twitter, Facebook, YouTube, and LinkedIn.