Belgian railway operator, Infrabel, drives up data management with Azure Data Explorer

Infrabel is a government-owned transport company responsible for building, maintaining and upgrading the Belgian railway network. Ensuring the effective distribution of capacity on Belgian railways is one of the company’s primary responsibilities, with a continuous focus on safety and sustainability.

Managing large, fluctuating datasets

Having relied on on-premises log management infrastructure for many years, Infrabel needed a more effective solution for the future of its operations. It aimed to bring together many different telemetry points from different asset types across its vast rail network infrastructure. “We were seeking a platform able to ingest very large volumes of data that can vary considerably,” explains Jamal Menassar, Enterprise Architect at Infrabel.

“The platform had to be flexible enough to handle new ingestion requirements and data sources quickly, without changing our infrastructure,” notes Menassar. The railway operator also needed a way to simplify data visualization, specifically, to swiftly export data to Power BI to easily view the results of any analysis.

“Our final requirement was high availability,” highlights Menassar. “We must be able to capture every network log. We can’t afford to miss any, as this significantly affects our analysis of the network as a whole.”

Infrabel chose Microsoft Azure as its trusted cloud platform. “Thanks to the flexibility of Azure, we knew we could easily build and deploy any platform-as-a-service solutions we needed,” reveals Stéphane Samper, Team Lead for Microsoft Solutions at Infrabel. “We chose Azure Data Explorer (ADX) as our log management solution because it offered us real-time analysis of our network data. It could help us achieve that, even with an exceptionally large number of network components and data logs from multiple sources, including our firewall, Network Performance Monitor, load balancer, and so on.”

Infrabel’s legacy solutions led the company to choose a hybrid cloud deployment. “This approach enabled us to achieve total harmonization of all our data,” explains Samper.

A proof-of-concept solution was very rapidly created, accelerated by ADX’s compatibility with the open-source technology Infrabel was already using, including Logstash and GoFlow. “A key requirement for us was the ability to reuse all code deployed on Azure, on-premises as well. We found we could do that easily, which sped up the whole journey.”

With the ability to flexibly ingest 800 million lines of data in 24 hours and 9,300 logs per second, Infrabel immediately saw results.

Enhanced flexibility and time to delivery

“With ADX we experienced new levels of flexibility in terms of deployment,” says Baptiste Piron, Network Data Engineer at Infrabel. “Suddenly, it was very easy to create a new database and to ingest new data.” Another major benefit was the universality of the platform, offering a single storage place for all the company’s data, and centralized access via multiple endpoints depending on the use case. “If data is needed for visualization, we can easily shift it where it’s needed. If we required data in another application, there is a simple API. If we wanted to achieve the same thing using on-premises technology, we would have to manage servers, install middleware, and only then start the development process. Once that was complete, we would have to repeat the process for the production environment. With Azure we can focus only on developing and extract ourselves from those other tasks, saving time and energy.”

This has proven a big improvement, positively impacting time to delivery. “If you are a system or a database administrator, you can easily configure and create what you need when you need it,” stresses Menassar. “When you need an analysis, you cannot wait six months or a year to receive the results. You need the result as soon as possible. And with ADX, you have the result directly.”

Seamless visualization

Infrabel’s ability to visualize data has been simplified dramatically with Power BI. “Power BI provides great visualization capabilities. And it’s seamless,” reveals Piron. “ADX is a native data source for Power BI, so it isn’t necessary to have experience in other programming languages. That means that if you have to import a new data source, you simply enter the credentials and you’re all set.”

ADX’s programming language, KQL, has proven simple to learn. “I had zero experience with KQL before this project, so it was all new for me,” says Piron. “I would definitely say that KQL comes with only a small learning curve. The documentation is also very good, which helps you move forward quickly. Once you get used to the language, it’s all very logical.”

“Thanks to the solution, we can now quickly identify patterns, anomalies, and trends across all our network traffic,” shares Samper. “At a macro level, we have considerably gained in flexibility and agility using Microsoft Azure. More and more, we intend to look to the hybrid cloud environment for solutions.”