Vodafone uses Azure DevOps to power agile transformation

“Your ability to respond, react, to stay relevant, to innovate and to compete comes down to your ability to release value to your customers as frequently as possible. That’s why Azure DevOps is such an important underpinning for business success and for cultural change,” says Ben Connolly, Global Head of Cloud Engineering & Head of Digital Engineering UK at Vodafone. “It really has revolutionised how we work.”

Powering agile transformation at Vodafone

“I joined Vodafone in 2018 when we launched our agile transformation journey,” states Ben Connolly. “Traditionally, in a telecommunications business, IT is secondary to the network. Today, software is becoming the business – or, at least, one with the business. As a result, we knew we had to make changes.”

Change would mean transforming IT in terms of who the business recruited, how they worked together, how they motivated each other and the purpose of the journey.

Ben Connolly explains, “We knew the principles we were establishing: becoming more limber as an organisation, being able to release much more frequently; being far less siloed; and having fewer handoffs between writing a line of code and launching it into production and seeing the impact that you’ve had.”

To enable this change, Vodafone needed to rethink the disparate systems underpinning software engineering work across the company. It had different source repositories, change control systems that relied on Word documents and a stasis-inducing attitude to risk and compliance. Vodafone chose to standardise on Azure DevOps and GitHub to help it to drive consistency and performance.

“At that time, we were a long way off being able to automatically roll back a deployment and see specifically which line of code that caused a problem, link it back to the developer or see why we were doing it,” says Ben Connolly. “That’s been revolutionised by the power of Azure DevOps. Now, it’s all pipelines and it’s way better audited than anything we used to have.”

A single global backlog

One of the early quick wins which cemented enthusiasm for the transformation programme was moving to a global backlog to manage and prioritise Vodafone’s software development.

“It sounds really obvious, but switching to a global backlog was a big win for us,” explains Ben Connolly. “I remember when I first joined, asking our chief product owner if I could see the backlog and he had to dig through his emails to find the most recent version of an Excel spreadsheet. People were using local instances of Jira for their backlogs and he was compiling them in Excel. Because it wasn’t in the cloud, there was no way our engineers around the world could share the same instance of a global backlog.”

“Using Azure DevOps to push it into the cloud suddenly means that everyone is looking at the same instance of the same data; there is only one backlog and every single person is connected to it and able to play their part.”

An end-to-end process

Vodafone’s choice of Azure DevOps was firmly rooted in its agile transformation vision. Different engineers had preferences for particular tools, but the priority for Vodafone was to create an uninterrupted Dev-Sec-Ops operation with a seamless process from start to finish.

“Being able to connect the whole lifecycle process together – to have a single, global view of a line of code, to be able to look at the release, to see with which bug it was associated, to attach it to user story and to the epic – that was much more powerful and far more interesting and compelling than having a patchwork of tools that worked best for one particular part of that journey,” says Ben Connolly.

“Choosing tooling can quickly become like religious warfare: everybody has their favourites,” he continues. “But for me, it wasn’t about getting a preferred tool for a particular use case. I was more interested in that end-to-end perspective. That’s what Azure DevOps gives us.”

“As well as being a great tool with a great feature set, it’s actually going to help me change what I want in terms of behaviour and outcomes – specifically, for engineers to be responsible for what they do.”

Changing the way IT thinks about change

Fundamental to Vodafone’s five years of transformation has been changing the way software changes are released to the business.

“At first, when I would say ‘I’m going to deploy 100 times a day’ it was very foreign to us,” recalls Ben Connolly. “It seemed a ridiculous goal when it took six days to log a change request.”

Back in 2018, Vodafone deployed change across the entire IT estate, including the website and back-office systems, in big releases that were scheduled five times per year.

“Deploying was scary and expensive, so we did it very infrequently. All IT was changed in one humungous release – and we were celebrating that: this is our biggest ever release! Then we’d spend the week after trying to unpick what had gone wrong with it,” recalls Ben Connolly. “When you’ve only got five opportunities per year to make a change, you don’t experiment – you spend months de-risking.”

From five releases per year to 500 releases per month

The shift to a Dev-Ops approach underpinned by Azure DevOps tooling has transformed the way software changes are released to the business. By November 2022, Vodafone’s global engineering team were deploying more than 500 changes per month.

“The success factors which I’m most excited about are our release frequency and release quality,” states Ben Connolly. “This is the most fundamental manifestation of what all companies are trying to do – to accelerate and become a technology business. Now, making change is practically free and we can do it all the time – and this is fully down to our choice of Azure DevOps and its capabilities.”

Furthermore, the end-to-end visibility engineers now enjoy means that, as the number of releases has risen, so has the quality of those releases. Vodafone reports a 50 percent reduction in code defects and rework.

Develop once, deploy many: My Vodafone App

Developing these new capabilities has gone hand in hand with a cultural shift. Ben Connolly explains, “We’re one of the biggest telcos in the world, yet we were not leveraging our scale to benefit our teams locally.”

This missed opportunity was evident, for example, in the development of the My Vodafone App. Across Vodafone’s twelve European markets, the business deployed twelve different apps.

Ben Connolly continues, “The twelve markets have their own priorities and their own local contexts. If the UK has a proposition it wants to launch, for example, it doesn’t want to queue up to do so.”

However, building the same app twelve times is not an efficient use of resources. Traditionally, the answer would have been to try to rationalise and consolidate. Now, a new approach is possible using Azure DevOps in combination with GitHub.

Azure DevOps & GitHub

“After consulting with our teams, we decided GitHub would be the right code repository platform for us,” explains Ahmed El Sayed, UK CIO and Europe Digital Engineering Director at Vodafone. “It has all the features required, can scale for an enterprise like ours, and has a community of software developers and enterprises with which we can connect and share learning. As we progress, we realised GitHub Enterprise cloud is the right solution for us.”

“Now, with GitHub and other Azure DevOps capabilities, we can very simply say two things: if you’re going to build it and it already exists, don’t build it again. And if it doesn’t already exist and you need it, then build it yourself but build it so that everybody can use it,” states Ben Connolly. “By connecting us all globally like this, we can get the balance right between leveraging the scale of the company without slowing any of the markets down. Each market can address local needs in a way that everybody benefits.”

“Given the current skills shortage, it’s even more of a waste if we’re doing the same thing multiple times,” Ahmed El Sayed concurs. “Our team is super talented and we don’t want to waste their time.”

Code sharing drives efficiencies

“One of our most important assets is engineering hours. With GitHub and Azure DevOps we are now in a position to streamline and automate environments, coding, testing so that our engineers focus more of their time on building new customer features,” says Ahmed El Sayed. “By building global products, we make 40 percent time savings and we can reinvest this time in building future innovation.”

Through its global code reuse and automation, Vodafone has achieved a 40 percent time saving in the global web engineering team with the reuse of more than 200 web components. More than 50 percent of time has been saved in the global app engineering team.

Overall, Vodafone estimates its use of Azure DevOps and GitHub has resulted in a four times increase in developer productivity measured in terms of the increases in the number of releases and the quality of output without a commensurate increase in headcount.

“The fact that Azure DevOps and Git Hub are evolving so closely together, that’s just an added bonus for me,” enthuses Ben Connolly. “It’s been a real catalyst and a real enabler of so much cultural change – resulting in the heightened pace of development and delivery for Vodafone.”

More opportunity to create meaningful digital experiences for customers

Most excitingly, the potential impact of these new ways of working is also expanding because the ways in which customers interact with Vodafone is changing. Digital channels accounted for around 13% of customer interactions in 2017. This topped 40% during the pandemic. It currently stands at around 38% now that retail stores have reopened – and the trajectory towards digital CX is once again rising.

“This creates so much opportunity to engage people in a meaningful experience online – it’s incredible,” says Ben Connolly. Consequently, Vodafone’s ability to respond rapidly to customer requirements on digital channels is increasingly important. Experimentation and iteration enabled by the company’s use of Azure DevOps and the new ways of working are key to this.

The exponential pace of change

Success breeds success at Vodafone. Ben Connolly enthuses, “It’s awesome. We’ve gone from five releases per year to 500 per month and it’s not going to stop. It’s going faster and faster, the more we build into our pipelines in Azure DevOps.”

As Vodafone integrates new Azure tools, so developer productivity and deployment quality rises. “The switch to YAML pipelines for example, was a gamechanger for us,” he continues. “Suddenly, our testing was fully automated, the feedback loops to our engineers were automated, our rollbacks started to happen automatically. You can plot the capabilities of Azure DevOps and when we adopted them onto the hockey stick curve of our deployment chart.”

The new way of work becomes the new normal

Vodafone used to plan its release cycles in person in big plenary sessions. During the COVID-19 lockdowns, this became impossible. Vodafone leant into Azure DevOps to move the process to being completely remote without any loss of productivity. It was so successful, that remote planning has now been adopted as the new way of working.

“About two years ago, we suddenly felt everybody was behind us,” says Ben Connolly. “We had a lot of support at the beginning. To be able to recruit and train people in a completely different model to what went before was amazing. And we’ve come up against loads of challenges – how to reward people differently, how to work with partners differently; the whole company needs to be changed. But the things we’ve tried have been demonstrably successful. Now, it’s about really leveraging that power and becoming a true end-to-end software business. That’s really the exciting future we’ve got – the snowball is now unstoppable and its going to be a fun ride for the next few years.”

Changing the way the business thinks about change

“The next step for us is to take this power, this capability we’ve built, and change the way the business operates,” states Ben Connolly. “We might be able to deliver 500 times per month, but often we’re asked to deliver a small number of large programmes. We need to treat our capability to build and release products as much of an asset as the proposition we are launching itself. We need to leverage that power we’ve got – to be able to iterate and experiment – and do all those things we can now do because it’s now cheap and easy and very low risk to make a change.”

The business is beginning to embrace this functionality and, in many cases, is funding teams instead of funding specific projects or deliverables.

“Rather than a big project that last nine months and has a particular business case that’s likely shifted by the time it’s finished, it’s now an exploratory and experimentation approach which is delivering amazing results because we’re able to experiment,” explains Ben Connolly. “They are looking for optimisations, experiments, trials, and then quickly iterating on top of that – so if it works they do more, if it doesn’t they do less. It’s making a huge impact.”

Real business impact

One such team has focused on improving the checkout experience for customers. They are tasked with achieving particular business goals; for example, trying to get net conversion rate from X to Y. They do this through iteration and experimentation.

The zero-touch launch of a major project

In the summer of 2022, Vodafone deployed an insanely huge programme that would fundamentally change the way it interacts with customers. In the past a deployment of the size, would have required tens of people to be up all night.

“It would have taken us 12 to 16 hours of deployment and then loads of days trying to fix it,” explains Ben Connolly. “Instead we deployed a feature flag through the Azure DevOps pipeline and that was it. It was all done zero touch – literally flicking a virtual switch through a pipeline – it was incredible.”

Towards Dev-Sec-Ops

The unified vision of end-to-end management of the software development lifecycle using Microsoft tooling has become a reality for Vodafone, right through to the involvement of the security team.

“We would deploy stuff up to a point, then we’d have to hand it over to the security team for pen testing,” explains Ben Connolly. “Instead of doing that they now own the part of our pipeline associated with that work and the handover of that work is automatic; there’s no human activity involved. They’ve even allowed us to fully automate the WAF in an Azure DevOps pipeline.”

“Everybody is using Azure Dev Ops now. Even our business folks are using it for their demand management.”

More about this diagram

A boost to developer morale

The new ways of working have had a dramatic impact on employee engagement at Vodafone, says Ben Connolly.

“Today, we are setting up around disciplines rather than projects and organising people into a community of similar disciplines. This enables us to give them really great career paths and democratise them out into the business. Rather than having a self-contained IT factory, we’re now a fully distributed team. Today, IT and the business is much less siloed and much more connected.”

“Working like this is way more empowering. It’s now about problem solving: here’s a net conversion rate, I want you to get it from here to here – how are you going to do it? A developer can get excited about the challenge and its impact,” he explains. “Ultimately, it’s about being proud and engaged in the work that you do.”

“That’s the type of business we want to be; where our engineers are really proud and excited about what they’re doing – because then we’ve cracked it. Largely because of our work with Microsoft and the Azure tools – that’s now our culture – and it’ll take care of itself – our engineers will innovate, they will experiment.”

Cultural change

“We’re now a community of diverse talent, bringing teams together around a shared manifesto and shared tooling. Our team has seven different paths to progress and grow and be part of our corporate strategy,” states Ahmed El Sayed.

The team at Vodafone are excited about the future road map for Microsoft Azure and Azure DevOps.

“Some of these capabilities weren’t available when we first adopted Azure DevOps but Azure DevOps has evolved with us and we’ve been able to connect and grow together. In fact, one of the best ways I found to extoll the benefits of our new approach was to use Azure DevOps itself as a case study,” states Ben Connolly.

“We have watched Azure DevOps develop, advance and get better, faster and more robust as a feature set in front of our eyes without us doing anything. There were no service packs, no release notes – it was just ‘hey, have you seen this new visualisation for the flow of work through your backlog?’. It was a brilliant example of what we’re trying to achieve ourselves: of the types of experiences we want to give our customers; constant deployment, constant evolution.”

“There are a growing number of initiatives and stakeholders who really want to tap into this new way of working,” explains Ben Connolly. “For example, in broadband, we have set up a value stream to deliver a constant stream of value to the customer. As soon as it starts demonstrating results, there will be more and more demand. Then our job will be done – the company has transformed.”

The hastening snowball of change

Ahmed El Sayed states, “Without freeing up time for our developers today, it would take us another few more years to achieve our dream. Through our use of Azure DevOps and GitHub, we can bring new products to market faster. This gives us more money to invest in new products and the cycle becomes much faster. We can reinvest in our future to grow our value.”

“More and more groups at Vodafone are starting to align to this new modus operandi,” says Ben Connolly. “This is how transformation works in large businesses – it’s not a big bang thing: it’s a pollination, a snowball.”

As the snowball continues to roll on, Vodafone sees a big role for Microsoft Azure and Azure DevOps.

“Microsoft will help us advance our capabilities, not just for the sake of it – but in a way that really drives the behaviour and principles of our engineering culture,” concludes Ben Connolly. “Azure DevOps isn’t just a great tool, it has fuelled the change culturally in what was a classic telco to now becoming a really great software business.”