Results | When AI Has a "Molecular Microscope": SciCore-Omics Unites Pathology Images, Spatial Omics, and Language for the First Time
When a pathologist looks at an H&E-stained slide, they see not only tissue morphology but are inferring the underlying molecular state and biological mechanisms. For AI systems, however, spatial biology information has long remained fragmented: pathology foundation models excel at reading images but do not understand gene expression; omics models are proficient in transcripts but lack spatial morphological awareness; language models can explain biological concepts but cannot truly answer how tissue structure relates to molecular activity.
So, can a single model simultaneously understand:
"what the tissue structure looks like," "what genes the local cells express," and "what these phenomena mean"?
To address this, our group, together with Peking Union Medical College, Nanjing University, and the National Center for Protein Sciences (Beijing), jointly proposed and open-sourced SciCore-Omics — the first tri-modal foundation model that unifies histology images, spatial transcriptomics, and natural language text, offering a new intelligent paradigm for spatial biology.
➤ Paper link
🔗 https://www.biorxiv.org/content/10.64898/2026.05.30.728937v2
➤ Model link
🔗 https://huggingface.co/openbmb/SciCore-Omics
➤ Source code and technical report
🔗 https://github.com/OpenBMB/Scicore-Omics
➤ Online demo
🔗 https://huggingface.co/spaces/Alkaidxxy/SciCore-Omics
The core contradiction: the gap between morphology and the molecular world
H&E images record tissue structure and cell morphology; spatial transcriptomics records the gene expression state at the same location; and natural language text in biology carries functional explanations, pathway mechanisms, and disease knowledge.
Over the past few years, most models could only complete "pairwise" mappings: image ↔ text; omics ↔ text; image ↔ omics.
But there has always been a lack of a unified framework that lets all three kinds of information work together within the same reasoning space.
The result:
Models can recognize tissue but cannot explain the molecular mechanisms behind it;
Models can analyze genes but cannot understand tissue spatial structure;
Models can generate explanations but lack real biological evidence.
The goal of SciCore-Omics is precisely to eliminate this barrier.
SciCore-Omics: building a multimodal brain for spatial biology
The research team built a language-driven unified generative framework. Within it, pathology images and spatial transcriptomics are processed by dedicated encoders; natural language is handled by an LLM for reasoning and generation. Different modalities are ultimately mapped into a unified token space and fed into the same autoregressive generative model. At this point, the model can simultaneously perceive tissue structure, molecular state, and biological knowledge within a single semantic space. Tissue morphology is a "visual language"; gene expression is a "molecular language"; and natural language becomes the unifying explanatory layer between them, offering a possible path toward making latent complex mechanisms interpretable.
Building supervision signals: the first image-gene-text triple paired dataset
Obtaining supervision signals was one of the key challenges in training SciCore-Omics. Currently, there is no large-scale dataset that simultaneously contains tissue images, spatial transcriptomics, and biological explanatory text.
To address this, the authors proposed Molecularly Grounded Semantic Pairing. Highly expressed genes are extracted from spatial transcriptomics, enriched pathways are obtained via ssGSEA, and biological context such as tissue origin is fused in to automatically generate natural language descriptions with biological grounding. In the end, 151,182 spatial-locus-level tri-modal samples were constructed, covering multiple tissue systems including brain, breast, lung, liver, colon, and skin.
Training strategy: learn language first, then omics, then fusion
Directly training three modalities jointly is highly prone to alignment failure. SciCore-Omics therefore adopts progressive three-stage training:
Stage 1: pathology image–language pre-training, starting from a general-domain vision-text model and transferring tissue structure and pathology semantics.
Stage 2: omics–language alignment, bringing the dedicated gene expression encoder closer to the semantic representation space and laying the foundation for finer-grained alignment.
Stage 3: joint image–omics–language optimization, completing unified tri-modal cognition.
Experimental results: comprehensive improvement across scales
01 Gene expression prediction
Predicting spatial gene expression from H&E images alone.
Compared with the current representative method OmiCLIP: PCC improves by 23.6%, MSE decreases by 15.8%.
The spatial expression maps of several key cardiac genes were successfully recovered.
02 Spatial region identification
On the human brain DLPFC spatial transcriptomics dataset:
Macro-F1 reaches 0.850, an 80.9% improvement over the strongest external representation baseline.
This shows that the model has learned to use morphology and transcript information together to understand tissue spatial structure.
03 Zero-shot pathology classification
With no task-specific fine-tuning, on four public pathology benchmarks — MHIST, BACH, CRC100K, and PatchCamelyon:
SciCore-Omics achieves an average accuracy of 56.46%, surpassing GPT-5 and multiple mainstream medical vision-language models.
04 H&E case-level reasoning
Given only a whole-slide pathology image as input, the model can complete:
tumor region discovery, pathology feature extraction, proliferation status assessment, and case-level comprehensive analysis.
It received the highest expert score in breast cancer case evaluation.
Significance: a new large model paradigm for spatial biology
The true value of SciCore-Omics is not just performance improvement. It proves something more important:
Pathology images, spatial omics, and biological knowledge are not inherently disconnected. As long as a unified semantic space is established, large models can automatically capture the connections between tissue morphology and molecular mechanisms.
This means future large models will not only be able to "see cells" but also understand what is happening inside them. From tissue structure to gene expression, from pathology phenomena to biological mechanisms, a truly interpretable chain of scientific reasoning is ultimately formed.
What SciCore-Omics demonstrates may well be an important direction for the next stage of AI4Science:
Not simply larger parameters or higher scores, but deeper integration of scientific cognition and the interpretability of latent mechanisms.
Letting a model possess a "microscope," a "sequencer," and a "biology textbook" at the same time — this may be a key step toward digital life science.