Advances in digital tech and artificial intelligence are opening new doors in precision health. One exciting area is cancer immunotherapy, where understanding how tumors interact with the immune system can lead to better treatments. Technologies like multiplex immunofluorescence (mIF) provide detailed insights into the cellular makeup of tumors, but they are costly and hard to scale. Now, a new approach uses AI to generate virtual data, making large-scale studies more feasible and affordable.

Transforming Pathology with AI

Traditional mIF imaging allows scientists to see inside tumors at a cellular level, revealing protein activity and spatial relationships. This information is crucial for predicting whether a tumor will respond to immunotherapy and for designing strategies to make tumors more treatable. However, mIF tests can cost thousands of dollars per sample, limiting how many samples labs can analyze. This bottleneck has slowed progress in understanding the tumor microenvironment at a population level.

To overcome this challenge, researchers developed GigaTIME, a multimodal AI model. GigaTIME can convert standard pathology slides, known as H&E slides, into virtual mIF images. This means that instead of expensive and complex tests, labs can use routine slides to generate detailed protein maps virtually. This breakthrough was achieved through collaboration with Providence and the University of Washington, training the AI on a vast dataset of 40 million cells with paired H&E and mIF images across 21 protein channels.

Scaling Up Tumor Microenvironment Studies

Using GigaTIME, scientists analyzed over 14,000 cancer patients from multiple hospitals and clinics. This effort created a virtual population of around 300,000 mIF images covering 24 different cancer types and over 300 subtypes. With this data, researchers identified more than 1,200 significant links between protein activity in tumors and key clinical features like biomarkers, tumor stage, and patient survival. These findings deepen understanding of the tumor immune microenvironment and could influence future treatments.

To validate their results, the team tested GigaTIME on an external dataset from the Cancer Genome Atlas, involving over 10,000 patients. The validation confirmed many of the initial findings, strengthening confidence in the virtual data’s accuracy. This study marks the first large-scale investigation into tumor immune environments using spatial proteomics, made possible only because of the virtual approach. It opens new doors for research, especially since actual mIF data remains scarce and expensive.

By translating common pathology slides into high-resolution virtual mIF images, GigaTIME offers a new way to explore cancer biology across large populations. This approach can accelerate discovery in immuno-oncology and help identify new biomarkers or treatment strategies. The developers have made GigaTIME freely available through Microsoft Foundry Labs and Hugging Face, aiming to support ongoing research and clinical advancements. Overall, this innovative use of AI stands to significantly speed up progress in understanding and treating cancer.

Inspired by

• https://www.microsoft.com/en-us/research/blog/gigatime-scaling-tumor-microenvironment-modeling-using-virtual-population-generated-by-multimodal-ai/