Even as the City of Saint John was working harder than ever to deliver service to its citizens during an unprecedented pandemic, it was targeted by a crippling ransomware attack. Thanks to its steely resolve and its strong relationship with Microsoft Partner Network member and security expert Bulletproof, the City rebuilt and came back stronger than ever before with a host of Microsoft Security solutions.
“We have visibility into inputs and endpoints from our Microsoft Security solutions that we never had before, and it’s key to educating users and maintaining a more resilient network.”
Stephanie Rackley-Roach, Chief Information Officer, City of Saint John
It might have been a wildly over-imaginative script for a B movie.
Deep into the realm of “you can’t make this up,” on Friday the 13th, November 2020, the City of Saint John in New Brunswick, Canada, was hit by a crippling ransomware demand. The resolute municipality refused to submit to the demands to pay up. Instead, the City’s IT team partnered with 2021 Microsoft Global Security Partner of the Year, Bulletproof, to rebuild critical systems and re-engineer the City’s cybersecurity posture in a more resilient way with Microsoft Security solutions, bringing critical services back online in an ambitious six weeks, avoiding the months—possibly years—that building a new network might take. Theirs is both a cautionary tale and a success story.
Doing all the right things, but .…
The City of Saint John is located along the Bay of Fundy in the eastern province of New Brunswick. Canada’s oldest incorporated city and the largest city geographically in the province, the community is home to about 70,000 people. The City’s 15-person IT team supports 900 users over the vast network of services needed by a thriving community: everything from development to parks to water to public safety services.
A long-standing concern for cybersecurity led the City of Saint John to implement a continuous security improvement program prior to the 2020 attack. Evaluating its security posture after some earlier minor attacks, the City collaborated with a third party to complete a security assessment on its environment, leading its IT team to create updated policies.
It took its next step toward an improved security posture by rolling out a security incident and event management (SIEM) system, adding constant monitoring by the Bulletproof Security Operations Center (SOC). Its roadmap included expanding monitoring to additional servers and user endpoints. Like municipalities everywhere, though, the City of Saint John IT department struggled for budget approval of cybersecurity solutions, often a hard sell for cash-strapped councils. The sympathetic City of Saint John Common Council underwrote improvements as funds became available, but progress was slow.
By early 2020, the City of Saint John entered into a licensing agreement for Microsoft 365. According to Stephanie Rackley-Roach, the City’s Chief Information Officer, “We were still very much an on-premises organization but wanted to begin our migration,” she says. “Our goal was to fully realize the convenience and security of the cloud.”
Surviving every CIO’s nightmare
Rackley-Roach was on vacation when the call came at 11:00 PM on the night of Friday the 13th. “My team had joked that we only had security incidents when I was on vacation,” she says. “When I saw that the incoming call was from one of my management team members, I knew it was bad news before I even answered.” She couldn’t have known just how dire it was.
The City of Saint John had been targeted by a cyberattack, its IT operating environment held hostage for multiple millions of dollars to be paid in Bitcoin. Working through the night, the City’s IT team severed the City’s internet connection and began assessing the damage. Fortunately, the City’s existing contract with Bulletproof for incident response services meant that the Bulletproof team could engage immediately. Alerted to the event that night, Bulletproof Chief Executive Officer Chris Johnston and Chief Operating Officer Jeff Shaw assembled a team that met on site before 9:00 AM the next day, Saturday. Rackley-Roach soon joined them to be briefed on the crisis, develop an immediate plan, and bring City leadership onboard in a process that she describes as “drinking from a firehose.” Says Rackley-Roach: “Bulletproof was our primary partner for containment and restoration. The attack impacted nearly every system.”
While Rackley-Roach worked with the City’s executive leadership team to set up emergency procurement procedures to ensure crucial resources and services could be acquired without delay, the Bulletproof team dove into what Johnston calls managed chaos. “Both the City of Saint John IT team and our Bulletproof people did a phenomenal job in managing everything that was thrown at them in that first surreal 24 hours,” he declares. COVID-19 social distancing restrictions further complicated the operation. “We supported Saint John and our onsite team from a remote command center to ensure we were balancing that risk,” he adds. “I don’t know how to adequately describe it for people who haven’t gone through a high stakes situation of such overwhelming intensity. Any IT executive confronted with a ransomware incident must balance urgent containment actions and communications internally with outside demands, like dealing with media, insurance, and City Council. Imagine everything coming at you all at once—and the pressure to make many critical decisions combined with a multitude of external pressures.”
Putting the pieces together
The Bulletproof team began its initial investigation to determine the source of the breach. “That initial response is a very complicated operation with a lot of moving parts,” explains Shaw. “We had to deal with the complex technical issues of forensic analysis and sever partner connections in addition to the internet. For the CIO, it means navigating operational impact while also working with law enforcement, insurance, external counsel, partner organizations, and other stakeholders.” Collectively, the City’s IT and Bulletproof teams developed a detailed playbook, laying out the recovery steps.
Working with the City’s IT team, the Bulletproof team led the rebuild of the City of Saint John operating environment, layering in security with the end-to-end Microsoft Security solutions stack. It deployed the Bulletproof 365 Enterprise system, which seamlessly merges Microsoft Sentinel with Microsoft 365 to deliver comprehensive intelligent security. The process began by replacing the City’s previous traditional log-based SIEM with Microsoft Sentinel. “We deployed Microsoft Sentinel to fix the ‘blind spot’ that happens with traditional log-based SIEMs,” explains Shaw. “The visibility and capability we get with Microsoft Sentinel far exceeds that of the previous SIEM. We brought the signal into our security operations center with Microsoft Sentinel for a real-time overview of the entire estate. It was critical to spot any threat that could delay reinstating the network and operating systems to fully restore IT functions.”
Fortunately, the City of Saint John had been preparing to deploy the Microsoft Security stack, and Shaw’s team advocated for the immediate rollout of Microsoft Defender for Endpoint to secure the City’s servers and other endpoints, Microsoft 365 Defender, and Microsoft Defender for Cloud. “In this situation, where we needed ultimate confidence to ensure that everything brought back online was highly secure, the Microsoft Defender suite was critical,” says Johnston. “Defender for Endpoint is invaluable in alleviating the fear of residual malware in on-premises servers as systems come back online.”
The team later deployed Microsoft Defender for Cloud Apps to further protect user accounts and applications coming back online and connecting to the City’s network. The visibility of data travel and security policy enforcement would augment security training being led by the City of Saint John.
Winning the good fight
The City aimed to reinstate the core IT operating environment in six weeks. “Our stakeholders found it difficult to believe we could be back online that fast,” Rackley-Roach says. “They thought a timeline of six to eight months was more realistic. But thanks to long hours from our dedicated IT and Bulletproof team, we had our core network, including critical services, up and running in six weeks.” Shaw presented the case for turning on the new network. “Before we could reconnect the City’s communication systems for public safety and law enforcement, we had to jump through a lot of hoops for stakeholders to prove that the new infrastructure was well-protected. We had a solid story to tell because we could show how well we had protected the estate with the Microsoft Security solution stack.”
It took 18 months to completely rebuild most of the City’s network and restore applications. Rackley-Roach wouldn’t endorse the way it came about, but she’s delighted with the City’s modernized and resilient network fueled by the Microsoft Cloud and protected by key Microsoft Security capabilities brought into play with the Defender suite of products. “The silver lining of our ransomware attack is that I go home at night feeling good about the state of our network,” she says. “Now, we work on the assumption that we’re being attacked all the time. Thanks to Microsoft Sentinel, we know that this is the case. The difference is that now we know that we can manage whatever happens through enhanced visibility and response capabilities.”
The City’s IT team meets with Bulletproof every month to review Microsoft Sentinel intelligence and strategize proactively. “We have visibility into inputs and endpoints from our Microsoft Security solutions that we never had before, and it’s key to educating users and maintaining a more resilient network,” Rackley-Roach says.
Even with his deep experience in cybersecurity rescue, Johnston looks back on this as a special case. “This is the story of a lot of hard work by a lot of people to make a faster recovery than anyone thought possible,” he says. For him, the City of Saint John is a testament to the interoperable nature of Microsoft Security solutions. “We’ve all lived in a multi-vendor world with limited visibility into security,” he adds. “That was the best we could hope for until the Microsoft Security solution stack was available, giving us a way to protect customers. We protect our own business with it.”
Months after recovery, the team bonds are as strong as ever. “It’s a hard way to meet people,” says Rackley-Roach. “But we have friends for life with Bulletproof.”
Find out more about City of Saint John on Twitter, Facebook, and LinkedIn.
“We deployed Microsoft Sentinel to fix the ‘blind spot’ that happens with traditional log-based SIEMs. The visibility and capability we get with Microsoft Sentinel far exceeds that of the previous SIEM.”
Jeff Shaw, Chief Operating Officer, Bulletproof
Follow Microsoft