Nuance’s Dragon Ambient eXperience (DAX) helps doctors document care faster so they can spend more time with patients delivering high-quality care with AI on Azure

Diagnosing a need

When we picture a doctor, we probably envision someone listening to lungs with a stethoscope or holding an x-ray up to the light. But along with providing hands-on patient exams and care, doctors also need to write up clinical notes, which often puts them behind a computer screen for part of the appointment. Doctors spend a significant portion of their days doing medical administration and entering patient data into electronic health records (EHRs).

Founded in 1992, Nuance has evolved into a global leader in conversational AI solutions that help transform how people interact. The company currently serves 10,000 medical customers, including healthcare organizations, hospitals, and clinics. Nuance understood the note-taking burden on doctors and how the administrative load in addition to patient care contributes to burnout. According to Medscape, 42 percent of all doctors report burnout, which can lead to medical errors, decreased access to care, and increased costs.

In a clinical setting, a patient explains symptoms and answers questions verbally. Their doctor then needs to summarize and enter that history along with findings from the physical exam into the EHR, recap lab and imaging results reviewed with the patient, and articulate the medical decision making in an assessment and treatment plan. Nuance decided to create a conversational AI solution that records the doctor-patient conversation and automatically documents patient encounters at the point of care using the company’s expertise in healthcare AI solutions and the powerful combination of Microsoft Azure and the PyTorch open-source machine learning framework.

A healthy solution

Nuance wanted to develop its Dragon Ambient eXperience (DAX) solution to capture information at the moment the conversation begins and then use AI to automatically generate proper documentation based on the data captured. But, extracting information from a doctor-patient conversation isn’t easy, and Guido Gallopyn, Vice President of Healthcare Research at Nuance, explains why. First, you need to capture the speech as high-quality audio within the constraints of a clinical workflow. Second, that audio from multiple speakers needs to be accurately transcribed. Third, the transcript needs to be summarized and transformed into a properly formatted clinical report. Finally, the clinical report and possible additional structured data need to be uploaded to an EHR system where the doctor can review the AI-generated document.

The company accomplished this complex process by building its models using Azure and PyTorch. Nuance uses PyTorch to train machine learning models in speech recognition and summarization, using separate models for each stage. “What sets PyTorch apart is that with it, we can solve problems we don’t know how to solve,” says Jeremy Jancsary, Senior Principal Research Scientist at Nuance. “We use it to rapidly iterate and go through multiple ideas to find out what works.”

Nuance is now looking to further accelerate its AI model training by using PyTorch together with Azure Machine Learning, which provides fully managed high-performance computing CPU and GPU clusters. “We take things to another level with Azure Machine Learning because we can basically create infrastructure for training our models as we go,” continues Jancsary. “We’ve found PyTorch and Azure Machine Learning to be very powerful together.”

Nuance also uses Azure Cosmos DB, Azure Blob Storage, and Azure Container Registry to support DAX, which has a fully containerized back end. Build artifacts are stored in Container Registry, and the execution of these images is hosted on Azure Kubernetes Service. At the cloud ingress, Azure Cosmos DB is used to synchronize and coordinate audio upload from multiple parallel clients. The intermediate and final output of processing steps persists in Azure Blob Storage. “We architected our whole solution on Azure, which shows how much the platform has to offer,” says Gallopyn. “We have a leg up on security and privacy that would have been more difficult to achieve in our own datacenters.”

Increased speed and scale

By building DAX with Azure and PyTorch, Nuance created a fast and highly scalable solution. Those are essential qualities to making DAX a success—and to making it possible at all. Gallopyn and Jancsary consider what it would have been like to attempt something similar without AI, considering there are more than 1 million doctors in the United States. “We’d have had to use traditional transcription, but you can’t scale with that approach,” says Jancsary. “There are simply not enough trained transcriptionists, and it’s too slow and costly for the healthcare system. Doctors need the clinical report after a visit to either approve or correct it, and only AI can help us deliver that.”

Before adopting PyTorch, Nuance had used a different machine learning framework. “We train our models two-and-a-half times faster now that we’ve switched to PyTorch, so that was an immediate win,” says Jancsary. “Once we actually trained the models and served them in production, we used only one-sixth of the CPU time that we previously needed to serve them.”

Jancsary acknowledges that moving to any cloud platform would likely have increased Nuance’s ability to scale. “But,” he says, “what we think Azure really gets right is the focus on specific vertical markets and all the certifications that come along with the platform, which are crucial for putting healthcare applications in the cloud. We found development and compliance to be much easier with Azure than with competing offerings.”

Enhanced patient care

DAX may be a young product, but doctors have had enough time to see significant results. First, they save time with it. “Doctors who use DAX save multiple hours a week that they can spend seeing more patients, spending more time per patient, or getting some much-needed rest,” says Gallopyn.

In addition, reducing doctors’ day-to-day administrative burden helps them conduct more focused, higher-quality interactions with their patients. “We hope that DAX is making life a little easier for doctors and patients,” says Gallopyn. “Doctors restore the conversation depth with patients because they don’t have to look at their laptops during the visit or worry about doing extensive documentation after the visit.”

Adds Jancsary, “We’re all affected by the quality of healthcare, and I think we all want that direct conversation with the provider. It’s very exciting to be in a unique position to contribute to that.”

Concludes Gallopyn, “Working on DAX has been a once-in-a-lifetime opportunity. A few years ago, people would have thought this was science fiction. It’s not often that you can really transform the delivery of healthcare, yet that is DAX’s potential. With support from Azure and PyTorch, our solution can fundamentally change how doctors and patients engage and how doctors deliver healthcare.“

Find out more about Nuance on Twitter, Facebook, and LinkedIn.