Since 1844, the Co-op’s mission has been to help its communities thrive. The group does this today by providing food, insurance, legal services, funeral care, and more to members and customers across the United Kingdom. When the Co-op’s siloed, on-premises data solution could no longer keep up with its needs, the team evaluated leading cloud solutions and chose Microsoft Azure and Azure Synapse Analytics for its modern data warehouse solution. Now, the Co-op is unlocking insights and new use cases to empower communities today—and far into the future.
“The partnership with Microsoft has been incredible. They wanted to engage. They were invested in helping us find answers and solve problems.”
Rob Wadsworth, Head of Data Engineering, the Co-op
A history of co-operation
In 1844, the Rochdale pioneers of Lancashire, England, created an idea that changed the world: the co-operative. The idea was revolutionary: a business that would be owned by its members and work for the common good. This idea—that principles are more valuable than profits—resulted in the formation of The Co-operative Wholesale Society in 1863. Today, the group’s more than 60,000 employees serve consumers through five major business groups: Food, Insurance, Legal Services, Health, and Funeralcare.
The limits of on-premises data
The Co-op’s on-premises data solutions were built in pieces over many years, resulting in data silos across the company. As a result, transformative insights were hard to come by, and the group often relied on guesswork or third-party studies to answer basic questions. For example, how much hot food should an individual store make in order to satisfy customer demand, without creating waste?
Rob Wadsworth, Head of Data Engineering at the Co-op, underscores the problem. “With over 2,000 properties, the data is massive: security, electricity, maintenance, stock, etc. And it all lived across different systems that often required manual manipulation in Microsoft Excel to provide a joined-up view. If a store manager wanted to know something about their store, there was no way to access the data. They’d email somebody who’d email somebody else who would get them a screenshot. And anything more advanced, such as real-time basket analysis, wasn’t possible due to the volume of data and the constraints of on-premises technology.”
Without cloud scale and a powerful analytics engine to effectively join datasets, the Co-op couldn’t make crucial data-driven decisions.
The search for a cloud provider
The Co-op knew that a cloud solution was the answer to breaking down barriers and becoming a data-centric company, so it embarked on a six-month evaluation of the leading cloud providers. It collected RFP responses, analyzed case studies, and carefully reviewed every product in the space.
The Co-op ultimately chose Microsoft, intrigued by the speed and scale that Microsoft Azure could lend to its data warehousing solution. Microsoft’s commitment to building a working relationship was an added bonus. “Their proactiveness, flexibility and active participation was certainly a benefit,” says Wadsworth. “When you want to talk about something, you don’t have to request a meeting in two weeks. There’s just somebody there to help.”
The team at Microsoft built a proof of concept to show how it could meet the Co-op’s needs.
“In my experience, suppliers don’t usually do a real proof of concept as part of an RFP process,” says Bob Marchant, Head of Supplier & Business Management at the Co-op. “We were blown away by the fact that Microsoft did.”
Together, the two companies rolled up their sleeves and got to work.
Unique migration challenge
The Co-op’s migration to the cloud faced an immediate hurdle: many of its internal clients used legacy solutions built on old data warehouses. These clients needed their solutions to continue working seamlessly, with no changes in service. So how would the Co-op get the data out of the old system and into the new one?
Microsoft had a solution in mind, from a Microsoft Partner Network member, to tackle this challenge. “And that’s where Datometry came in,” says Wadsworth. With Datometry technology, applications originally developed for Teradata transparently run on Azure. The Co-op built the Datometry solution into the architecture from day one so that the group could quickly and easily migrate on-premises retail data to Azure but keep existing scripts unchanged. Legacy users would gain the speed and scale of Azure without any interruption in services.
“Microsoft has a great partnership with Datometry. The team enabled us to quickly take data from our system, load it through one of the pipelines, and see how Datometry served that data up.”
Rob Wadsworth, Head of Data Engineering, the Co-op
Advanced analytics in a modern data warehouse
The Co-op’s new solution (Figure 1) uses Azure Data Factory to pull data from across the organization into Azure Data Lake Storage. Storing the data in an Apache Parquet format makes it easy to manipulate and read into the warehouse, so the Co-op can build and move quicker. And the scale of the data is huge. “We’ve got 18 billion rows of data across 22,000 files from 74 different data sources—and that’s just the insurance and membership part of the warehouse,” says Wadsworth.
With this powerful data warehousing solution that Azure Synapse Analytics provides, the Co-op is able to deliver enriched data to its data science and BI teams. And for legacy systems, the existing Teradata extract, transform, and load (ETL) scripts are seamlessly handled by the Datometry and Azure Synapse Analytics solution to give users the reports and models they’re used to.
Deeper insights, new use cases
The new solution is already reaping benefits, with lower costs, better performance, and faster time-to-insight enabling a new era of co-operation. “The Insurance team never had access to retail data before. Now, they’re sitting with the Food Data Science team and just experimenting with the data. Instead of six months of bureaucracy and third-party studies to answer a single question, they do it in real time,” says Wadsworth. “Processing time is down from days to minutes.”
“We can now run queries against data from over 2,000 stores in a matter of minutes.”
Rob Wadsworth, Head of Data Engineering, the Co-op
More importantly, the solution is unlocking new use cases that the Co-op had never even imagined. From eliminating food waste to optimizing funeral services, the Co-op is uncovering insights and cost efficiencies that make a real difference in the lives of its members and customers.
The future of co-operation
The Co-op is already looking ahead to the future. Recently, it built a proof of concept for a Delta Lake solution. The Co-op will be able to pull deltas from a data lake and rebuild its data vault more efficiently, setting the company up for later innovations around streaming data and APIs.
No matter what the future brings, the Co-op will be there to help communities thrive. And Microsoft is by its side every step of the way.
Find out more about the Co-op on Twitter, YouTube, and LinkedIn.
“In 18 months, we’ve gone from multiple siloed business units to leveraging the benefits of One Co-op.”
Bob Marchant, Head of Supplier & Business Management, the Co-op
Follow Microsoft