Co-op cuts 2 years off its modernization plan with Datometry Hyper-Q and Microsoft Azure Synapse

Two roads diverged

Co-op had used an on-premises datacenter to serve the needs of its retail business for decades, amassing 10 terabytes of live data in more than 10,000 tables fed by 885 scheduled processes. These, in turn, fed 6,000 reports to 1,000 office and field-based users. It was a monster. Co-op decided it needed replacing, but the company's data community was uncertain about the path forward. On one hand, the cloud, with its vast array of data-processing facilities, held enormous appeal, especially for Co-op's data scientists, who yearned for the speed and agility it promised. On the other hand, the prospect of rewriting, retesting, and redeploying reams of legacy SQL code – all without service interruption – was daunting. Payback would be long-term, and there was high risk of service disruption. 

Co-op's technology and data leaders decided the ecosystem was not portable. They assessed and affirmed that extract, transform, load (ETL) operations could not simply and easily be repointed. Co-op also was in the process of overhauling its data strategy, and the director didn't want to rush into a rewrite of the same business logic. Still, the legacy data warehouse servers were nearing end of life, with more frequent node failures and service outages, so standing still was not an option either.

A valuable introduction

Charlotte Lock, Director of Data, Digital & Loyalty at Co-op, said, "We started the journey thinking we had two options: We could reinvest in our on-premises data warehouse or we could migrate our retail data warehouse SQL to the cloud, with a number of vendor options. Microsoft offered us an introduction to a third way: Datometry.”

Having worked with Microsoft before, Co-op was receptive to Microsoft's suggestion.

Headquartered in San Francisco, California, Datometry brings deep technical expertise in data warehousing, database technology, and query processing. Its Datometry Hyper-Q virtualization platform, available in the Azure Marketplace, lets enterprises move from Teradata to Azure Synapse with minimal effort.

Datometry could enable a migration from Teradata to Azure Synapse before Co-op’s Teradata support expired. Datometry also would de-risk the project and enable Co-op to retain its investment in existing apps and ETL.

Rima Mutreja, Datometry’s VP of Customer Success, said, "Co-op was a well-suited client because they were driving a cloud-first datacenter exit strategy, so they were fully committed to the Azure journey. Co-op already had Azure experience with an existing platform, and Teradata did not align to Co-op's needs."

Executing the project

Before Datometry, Co-op’s modernization plan had been expected to take more than three years, but Datometry Hyper-Q virtualization capabilities allowed Co-op to emulate legacy data warehouse processes in the cloud without needing to rewrite them. This met Co-op’s strategic and operational goals and sped up the timeline. Azure Synapse, supported by Datometry, emulated the legacy logic, retaining service without disruption, while Co-op’s team designed a new analytics layer in parallel.

One year later, after user testing and successful parallel operation, Co-op cut over to Azure Synapse.

“Shifting from a legacy on-premises data warehouse to Azure Synapse, supported by Datometry, has allowed us to virtualize the vast majority of our code without needing to repoint it,” Lock said. “We have gained speed, performance, and agility while reducing costs and taken a big step forward in modernizing our enterprise data storage and management."

Through the process of emulation, Co-op’s team learned more about the company’s legacy data, and engineers took the opportunity to rewrite about 15 percent of the code.

Co-op has realized benefits in processing speed, with many processes taking hours less than service-level agreement standards and using less capacity than anticipated. Co-op has also saved multimillion-pound sums annually compared to the cost of license and service of its previous data warehouse.

“Between Microsoft, the wonderful data engineering community at the Co-op, and Datometry, we’ve achieved far more than we set out to at our first steering committee meeting, which was just two months before we were consumed by a global pandemic,” Lock said.