Dairy company centralizes its data in Azure, gains flexibility and productivity

Arla Foods is the fifth-largest dairy company in the world, with history dating back to the 1800s when the first cooperative dairy was established in Sweden. Because Arla is farmer-owned, all earnings go back to the farm owners.

Arla Foods puts a large emphasis on the journey that its milk products take from cows to customer. Every step of the way is recorded and traced. Arla is committed to high standards of animal welfare, product quality, and safety. Arla combines traditional craftmanship and technology to ensure products remain as close to nature as possible.

Kasper Pors Hansen is a Senior Manager at Arla Foods who has recently joined the company to help with modern digital transformation within the area of data and analytics. After Arla Foods faced some challenges in regard to disparate data, Kasper and his team, internally branded as the Arla Analytics Powerhouse, were given the task to help lead an initiative to centralize data across the company, making information more accessible and easy to understand.

Problem overview

Arla Foods strives to utilize modern technologies to track every step of its supply chain from farm to fridge.

About two years ago, Arla began to in-source software development to strategically take advantage of cutting-edge technologies and streamline the way Arla works with data. Previously, one data driven solution would be built and completed when a unit in a different country would then need a solution of their own. The next solution would use the output from the first solution plus the same source data from the original database. This pattern would repeat itself, resulting in an unmanageable spaghetti-like structure. The environment became flooded with duplication and inconsistencies that made getting reliable answers from data very difficult.

Each individual system had unique upkeep and management challenges. The cost of sustaining and maintaining each system began to grow and weigh on the company. Employees across Arla were spending time collecting knowledge on how to manage and use each system instead of gaining insights from the data within them.

Each application developed produced a new system that solved only one problem—and one problem only. The ecosystem contained thousands of different applications, but nothing was transferrable, making it impossible to extend an application to solve any other part of the business processes. This created a highly inefficient set of siloed solutions.

Many of the solutions used Microsoft Power BI as the intended visualization tool, but this was often done without having knowledge of the proper data architecture for maximum utilization. Data models were put directly into Power BI with no use of data marts anywhere in the data stack. All compiling was being performed directly in Power BI with no central location to pull data from except SharePoint. Power BI quickly became overloaded and reached capacity, resulting in Power BI applications beginning to fail.

As nothing else was available, Power BI developers in the business would often need specific data that was not easily accessible. To overcome this issue, they would dump data directly from SAP Business Warehouse (BW) into Excel spreadsheets, then upload the manually created datasets to build reports in Power BI, a costly and timely process. Employees were spending hours manually compiling information to use in Power BI, only to realize this process often resulted in different conclusions for each report and a significant workload on the Power BI developers in Arla to manually refresh the reports.

With thousands of separate applications, falling service levels toward the users, and maintenance expenses increasing exponentially, Arla Foods was forced to recognize that the current process was not working. The company was determined to create a cohesive data environment that would allow it to use its data to innovate and solve business problems efficiently. Arla Foods set out to find a solution that would both save it money and move its business forward.

Arla Foods’s journey

Arla Foods needed to create one source for all its data. Kasper and the Analytics Powerhouse were given the task and, in cooperation with the rest of the IT organization, they are leading the way forward for creating the data foundation that allows all data to be funneled into and extracted from one place for Power BI consumption, while at the same time enabling other teams across Arla to build new data driven applications and exploratory data science, all on the same data foundation.

Arla Foods is using SAP BW on HANA for financial data and analysis. SAP BW gives Arla Foods the services it needed for financial processing. In order to get the most value from SAP BW, Arla Foods decided to avoid duplication, latency, and added costs by keeping the system.  

For all the other data not in SAP BW, a solution was needed to combine all the data from all the sources. After being left with so many different systems, Arla Foods felt the monetary burden of maintaining on-premises systems and opted to move all non-SAP data to the cloud, a more affordable option. Arla wanted one vendor for the data foundation services, and Azure provided every tool necessary for a complete solution, eliminating the need for any third-party tools. Kasper’s team also found that no cloud service worked quite as well with Power BI integrations as Azure did. The team determined that Azure gave them the biggest flexibility for not only data visualization but potential for forecasting and future technologies. Using Azure would not only help them solve the immediate problem but open doors for future processes and innovation.  

The Analytics Powerhouse planned to develop a data foundation on Azure that would create one centralized location for data anywhere in the company to be ingested and consumed. The centralized data would then have the ability to be used for a variety of business needs, from self-service reporting in Power BI to exploratory analytics and for powering data driven applications.

Solution details

Within the data foundation architecture lives two main sources of data. SAP BW on HANA is used for all financial needs while all other data is combined in Azure. This solution helped solve the problem of combining all data in one location, but the Analytics Powerhouse needed to figure out how to get the data that lived in the SAP world to marry the data within Azure in order to produce relevant business information.

In order to create an architecture where SAP data and Azure data can be understood together, all raw data from the majority of the data generated in the SAP ERP Central Component (ECC) system is extracted via standard BW data sources directly into a SAP BW on HANA data warehouse. Once ingested into SAP BW, the data is transformed and modeled and stored in persistent storage. The data is made available for reporting via SAP BW queries and via SAP HANA views. The views and queries handle runtime aggregations, along with providing for slicing and dicing and other OLAP capabilities.

In parallel, data is extracted from other on-premises sources (in addition to a small number of SAP sources) via Azure Data Factory and stored raw (untransformed) in Azure Data Lake Store Gen2. In the same Azure Data Factory pipeline that extracted and loaded the raw data, an Azure Databricks notebook is executed to cleanse and transform the raw data into a curated data layer, also stored in Azure Data Lake Store Gen2.

For reporting, the curated data layer is loaded into specific data marts built per use case (which could be a simple file format stored on Azure Data Lake Store Gen2 or Azure SQL Database).

 For enabling advanced analytics or data science workloads, both the curated and the raw layer of Azure Data Lake Store Gen2 are made available to the user to interact with directly with R or Python or, more likely, via interactive Azure Databricks notebooks.

Once a model is built and trained, it is containerized and deployed in Azure Kubernetes Service (AKS) for consumption by other applications and APIs.

Data is also pulled from the data lake into Azure Synapse Analytics to manage and scale data, along with SQL Server on Azure. Power BI is used to connect to data within SQL Server on Azure, Azure Synapse Analytics, and SAP BW to produce interactive data visuals.

With the new architecture in place, Power BI can be used to its full extent, capacity can be managed, and data models can be significantly simplified to avoid performance issues that were previously persistent. Power BI datasets will be used as rich semantic models spanning multiple subject areas, empowering business users with limited technical skills to interactively create beautiful reports based on intuitive, standardized business definitions. Easy access to curated data through semantic models promotes consistent decision making in alignment with Arla Foods’s strategies and policies.

Utilizing waste management

Using Azure to fix the siloed systems created by multiple consulting company projects opened doors for transformation throughout the entire company.

One of the first areas for which Arla Foods took advantage of the new architecture is waste management. Before Azure, understanding the specifics of waste management produced throughout the company was unachievable. Because of the difficulties of combining all the data from SAP BW and the multiple individual sources, identifying detailed information about waste across production sites and identifying exact issues could not be done. SAP BW contained information about products moving through the supply chain, but there was no way to combine that data with information about the production sites and machinery from the other sources for comparison and understanding.

With the new data foundation, all sources combine data in Azure and can be pulled into Power BI along with data from SAP BW, allowing the whole story to be told. The cause of waste can be identified through Power BI drill-throughs and visualizations.

This allows the company to have a more effective production line, lower wastage, and the ability to make informed business decisions, saving the company money and time while reducing food waste. The new solution allows Arla Foods to use its time fixing problems instead of just finding them.

New architecture opens other doors

Once Arla Foods used the power of the newly established data foundation combined with Power BI in waste management, it used the same concept to elevate other areas of its business.

Examples include predicting the milk intake for better production planning, improving financial processes by creating an overview across regions, providing an innovation space for non-IT users to develop new KPIs, and enabling a cost allocation engine for creating transparency and improving logistics costs.

Then to now

Arla Foods is transitioning from a company having multiple siloed solutions scattered across the business to an enterprise leveraging data centralization to perform multiple processes across its business by utilizing detailed data. Arla Foods now transforms data in Azure and makes it available through semantic models in Power BI. Business users are empowered by creating data visualizations on curated data, allowing them to act. The organization is saving money and time with cloud services by not having to manage and maintain on-prem solutions. It is also reducing its bottom-line costs by using the Azure cloud platform and Power BI to determine where to adjust costs and prices.

The pattern of organized data throughout the company can be managed and extended to continue to solve deeper business issues and build new solutions. The new architecture makes it simpler to manage costs of delivery and maintenance and optimize business spending. No matter the problem, users now have one simple place to get answers from. Arla Foods now has flexibility in using its data, opening doors for endless innovation.

What’s next?

Having all the data available and set up properly isn’t worth much if people don’t know how and where to get what they need. Because of this, one of Arla Foods’s next plans is to implement a data catalog for understanding how to get information out of Azure. Data is also strategically being standardized across the data foundation to ensure the architecture never gets to the previous state it was in. In addition, skills and business processes within the organization need to adapt to new ways of working with data, and the Analytics Powerhouse is beginning an internal training program to upskill non-IT users in using the data foundation correctly, within a governance framework that has been launched that gives Arla employees flexibility in terms of being able to develop functionality on its own while at the same time maintaining control from a central point of view.

Arla Foods plans to innovate by using IoT devices to retrieve data from the farm and deliver it to Power BI. It plans to utilize Azure IoT Hub and Azure Event Hubs to accomplish this vision. The current architecture supports making these plans a reality, and the organization has plans in place to make this a reality in the upcoming year.