More time for research: ZEISS supports businesses and researchers with ZEISS arivis Cloud based on Microsoft Azure

The challenge: Automating the manual image segmentation

Pick any industry and chances are it relies in some way on microscopy and image analysis—from life sciences to materials sciences to earth sciences. Professionals in these disciplines must be able to get a close look at what are often incredibly complex structures. Materials scientists are interested in looking at the microstructure of different materials, while bioscientists want to examine human cells. While many aspects of image analysis are highly automated, one crucial step lags behind: image segmentation. However, this is undergoing transformation.

“Segmentation involves dividing an image into different regions that are crucial for subsequent analysis and classification,” says Dr. Sandra Lemke, Product Owner for the AI toolkit on ZEISS arivis Cloud at ZEISS Microscopy. “This step is absolutely essential as it provides researchers with the foundation for the precision of the analyses that follow. It’s how a life sciences researcher defines a cell or a nucleus. Without the help of Artificial Intelligence, they’d even have to manually draw the structures for some images pixel by pixel.”

In addition to demanding a well-trained eye, this task is also time-consuming. After all, researchers typically need to segment not just one microscopy image but an entire set, compounding the workload.  Manual efforts, while often considered highly accurate, are susceptible to errors and may yield non-reproducible results. Additionally, there’s the risk that another individual might later perform the segmentation in a slightly different manner.

“Now, with the ZEISS arivis Cloud, we’re providing researchers with a solution they can use to automate image segmentation and produce replicable results,” says Dr. Sreenivas Bhattiprolu, Head of Digital Solutions at ZEISS Microscopy. “The microscopic structures are so complex that only experts can identify them. There’s no turnkey technology that can do this either. Now, however, the ZEISS arivis Cloud makes it possible to train deep learning models so that the researchers themselves no longer have to manually segment every set of microscopy images.”

The solution: Training deep learning models without the need to code

When researchers use the ZEISS arivis Cloud platform to train their custom model, they upload a series of training images that display a certain variance. This is crucial given the diverse nature of samples and the slight variations in images produced by different microscopes. The appearance of images is also influenced by the conditions under which they were captured. The inclusion of such variability in the training data ensures that the deep learning model generates more robust results.

To perform image segmentation, researchers use the web interface to draw regions of interest, a process commonly referred to as annotating. It’s important to note that researchers typically annotate only a few regions in each image, as not every region requires annotation. Subsequently, a pretrained model is employed to segment the required regions across all images. If the model meets expectations, it can be applied to additional image sets. However, if it fails to achieve the necessary accuracy, researchers iteratively annotate additional regions in the training images and initiate the retraining process. This iterative approach enables researchers to establish what is termed "ground truth" in deep learning, even without any coding experience. Based on this ground truth, the models learn to recognize and segment the relevant regions accurately.

“A couple of years ago, it took an entire night to train a deep learning model. But now, thanks to the Azure cloud platform, it takes about as long as a lunch break,” Lemke says. Furthermore, with the software-as-a-service (SaaS) option, researchers eliminate the need to install any software. They can simply log in to their ZEISS arivis Cloud account and start working. Thus, the substantial computing power necessary for training deep learning models is sourced not from the users' own server rooms but from the on-demand cloud-based infrastructure.

Lemke and her team leverage Azure Machine Learning and its associated Machine Learning Operations solutions to develop, provide workflows, implement deep learning algorithms, and monitor and automate the entire process. “During development, we can take the time we would’ve spent on managing these processes and use it to work on new functions. This naturally ends up benefiting our users, too.”

The ZEISS arivis Cloud also facilitates global collaboration, enabling researchers to seamlessly work together on training a deep learning model, picking up where their colleagues left off. Each team member can export the model for use on their own data, ensuring transparency in the model's training process and the ability to replicate results. If the model fails to maintain satisfactory accuracy on new data, even after 100 segmentations, it can be easily retrained with additional data without any hassle.

“It’s hard to tell what deep learning has in store for the future,” Bhattiprolu says. “Entirely new possibilities are opening up every couple of months. But one thing is clear: our solution is already helping researchers carry out more studies and spend less time on image segmentation. And this situation is set to improve even more in the future.”