Forza Horizon 5 crosses the finish line fueled by Azure Kubernetes Service (AKS)

A high-speed, low-latency race to the finish

Developed by Turn 10 Studios and Playground Games and published by Xbox Game Studios, the Forza Horizon series has grown in popularity with each new title. As the series has exploded in size, so too has the infrastructure needed to support it. At first, the Windows-based development team hosted the game on-premises, buying and borrowing servers as needed.

“Whenever we launched the next title, we'd use servers from the previous titles, basically playing that hardware game,” reports Tyler Hennessy, Turn 10 Studios Principal Software Engineering Lead with ForzaTech, the team responsible for the technology behind the Forza game series. A move to Azure changed all that. At first, ForzaTech moved its code, based on .NET Framework and Internet Information Services (IIS), to Azure Cloud Services (classic) and virtual machines (VMs). With control over its virtual infrastructure, the team had “much easier scalability,” Hennessy notes. The question was whether a different approach might provide even greater gains.

Like most in the gaming industry, the Forza Horizon code base is completely built in Windows. According to Turn 10 Program Manager Madden Osei, “We wanted to deliver more agility and speed to scale for millions of players while abstracting the lower layers of VM management. But our preferred solution needed to support Windows-based images.”

Going from 600,000 to 3 million concurrent users

The answer came from a technology that ForzaTech didn’t expect—Kubernetes, the popular container orchestration platform. “We had never used container technology whatsoever. There was never any reason to even look into Kubernetes,” Hennessy recalls. But the team leaned into the curve. “Within a month, basically we had converted our services over to AKS and had everything running.”

Hennessy reports that the transition from Azure Cloud Services to AKS was “good and easy.” Rather than rewrite the entire platform—a daunting option at one point—ForzaTech took advantage of the support in AKS for running native Windows container applications, such as those built on the .NET Framework, like Forza Horizon 5. The combination of .NET and IIS “lent itself perfectly to AKS Windows images. From the very beginning, it's a great story of going from on-premises and owning hardware to AKS,” according to Hennessy.

It was also a better story than a platform rewrite would have been. “We didn’t know that Kubernetes and Windows could go together,” Hennessy admits. Fortunately, the team tapped into the enthusiastic Kubernetes community and learned how AKS supports Windows containers. “There is a wealth of knowledge there. That helped us jump-start things and get going quickly,” he says.

AKS also allowed the team to iterate much faster during stress testing, a mission-critical step in preparing for the big launch day. Before moving to AKS, it could take ForzaTech developers half an hour to swap out old images on their VMs and prepare them for load. In AKS, this process takes seconds. The extra speed enables the team to instantly make repairs, overcome roadblocks, and scale as needed. In addition, the team can script cluster updates ahead of time, adding flexibility and agility to the demanding stress tests.

Well ahead of the big launch, the stress tests proved that ForzaTech’s services could easily scale from 600,000 to 3 million concurrent users. The team was more than a little happy. “We were thrilled!” Osei says. “Every reasonable estimate for players and concurrent users was broken.”

Turbocharging a gaming architecture

The Forza Horizon 5 architecture includes 17 core services that run in AKS. Among them, the leaderboard service tells users where they rank, while the user-generated content (UGC) service supports the ability for players to upload photos and other content. The auction house service enables players to sell cars to each other, and the analytics service collects game and player telemetry used in reports.

To simplify the migration to Windows containers, ForzaTech created a playbook based on the experience of moving the UGC service. After about two weeks of testing, the team applied the playbook to the other services. “It then took about one day per service for the other 16 services,” Hennessy reports. “There was very little code change at all. It was almost all configuration and deployment changes.”

The front door for all client requests is the aggregator service, a single autoscaling deployment that runs multiple pods in AKS. All the pods interact with the same AKS configurations and secrets. The aggregator service interacts with the other microservices running in containers. Their similar architectures made the playbook easy to follow. They are instrumented with Kubernetes liveness and readiness health probes, so the team can quickly identify any issues and make changes, such as adjusting the scale.

For business continuity, two instances of every service run on just four VMs. Before the move to AKS, ForzaTech ran a minimum of 40 VMs at all times. “Most of them were running very cold,” Hennessy admits. “This configuration reduced the cost of our test environment by 90 percent.”

To boost speed even more, VMs in the AKS clusters enable accelerated networking, an ultra-low latency, high-performance path that reduces jitter and decreases CPU utilization. “We saw a 50 percent latency reduction on some back-end network calls,” Hennessy recalls.

The storage layer also plays a role in meeting the scale and performance targets. The aggregator service interacts with Azure Cache for Redis, a highly scalable caching solution. When throughput needs a boost, data can be sharded across multiple Azure Storage accounts—directly impacting the team’s ability to scale up.

Revving up for the big launch

To ensure that the new container-based architecture could handle the anticipated loads, ForzaTech built a stress test harness on Azure and set a goal to exceed the launch volumes seen when the previous version of the game debuted. However, Forza Horizon 5 outpaced that launch with five times the number of players. Fortunately, the team worked with partners at Azure PlayFab, a service for building and operating games on a live platform with dedicated, global, multiplayer servers.

“The numbers were great, but some locations were running up to 70 percent of their previously allocated capacity,” Osei points out. “Thanks to the support of Azure PlayFab, the team was able to dynamically scale up and dramatically increase server capacity in all regions.”

The launch occurred during the winter holidays, when traffic was expected to be higher than usual. “At launch, when we had over 1 million concurrent users, all 17 services autoscaled and end-to-end latency never went above 100 milliseconds,” Hennessy observes. He also notes that the team intentionally overbuilt the infrastructure to handle the load.

He tells what happens when the team returned from the holiday break. “We knew, ‘Here's our stable concurrency throughout the day’—not just for AKS but all of our services.” After reassessing the game’s needs, the team scaled the infrastructure down, saving more than $100,000 the following month. Hennessy notes that it was as simple as turning a dial one way to scale up for the launch and expected holiday usage and then turning the dial the other way to scale down and reduce costs.

Looking for the next challenge

The numbers speak for themselves. The Forza Horizon 5 launch was so successful that Microsoft Chief Executive Officer (CEO) Satya Nadella posted about it on his Twitter feed, noting the accessibility features, and Metacritic rated the game 92 out of 100.

Not bad for a team new to Windows containers and Kubernetes.

“It was an easy migration to get on to AKS and stay on Windows,” Hennessy concludes. “Now we have the scalability and reliability of Azure and AKS without the time investment of changing our code base.” That means more time for Turn 10 Studios to thrill players with stunning new racing experiences.