Why Does Multimodal Retrieval ’Favor’ Text? We Found Chaos in the Text
Why can an irrelevant passage push the image that actually contains the answer out of the Top-K? We systematically studied a long-overlooked problem in mixed-modality retrieval: Modality Preference, and propose Trident, which lets text, image, and fused image-text evidence compete more fairly in the same ranking space.
Today's multimodal systems can retrieve over text, images, charts, slides, scanned documents, and other forms of information at once.
So a seemingly natural assumption follows:
Since the model can search text and search images, putting them in the same knowledge base and searching together should be fine too.
But our experiments show it is not that simple.
Many models perform well in a pure image corpus and also well in a pure text corpus; yet once text and images enter the same candidate pool, retrieval performance drops significantly.
More counterintuitively:
A passage that does not contain the answer may even rank above the image that does.
We call this phenomenon: Chaos in the Text
Paper: Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers
*The authors are from the University of Chinese Academy of Sciences, Northeastern University, Tsinghua University, and Nanjing University. The paper, code, models, and training/evaluation data are all publicly available.
Figure 1 | Left: as image documents are progressively replaced by text representations, NDCG@10 shows a clear V-shaped change; right: text occupies the Top-K, and the chart that actually contains the answer is pushed to 12th place.
Strong Alone, Why Do They Fail Together?
We first ran a very direct experiment.
Starting from an image document corpus, we constructed three semantically corresponding representations for each document:
Image View: the original image;
Text View: a detailed textual description of the image content;
Fused View: the original image + a short text summary.
We then kept document identity and knowledge content unchanged and varied only the proportion of Text / Image / Fused forms across the corpus.
If a retrieval model truly ranked only by "semantic relevance," whether the same content appears as text or as an image should not, in theory, cause such drastic performance changes.
Yet we observed a very clear V-shaped curve:
When the database consists almost entirely of images, the model performs well.
As more and more images are replaced by semantically corresponding text, performance begins to drop quickly; when text and images coexist in large numbers, performance bottoms out; and as the database moves further toward pure text, performance recovers.
In other words:
The model can search images, and it can search text.
The real problem is that it is not good at making text and images compete fairly on the same leaderboard.
This phenomenon appears across multimodal retrievers of different architectures and sizes.
This matters especially for real applications.
Real-world knowledge bases are almost never neatly single-modal: PDFs contain body text and charts, web pages contain text and screenshots, enterprise knowledge bases may hold Markdown, slides, scanned documents, and images at once, and Multimodal RAG by nature needs evidence from different modalities to compete together.
Therefore:
Strong single-modality retrieval does not mean mixed-modality retrieval is reliable.
Is It Just "More Candidates," or Is Text Itself Stealing Rank?
To pin down where the problem comes from, we ran a more direct experiment.
Starting from the original image retrieval corpus, we separately added large numbers of:
irrelevant images (image distractors) and irrelevant text (text distractors)
None of these added documents contain the correct answer for the query.
If the performance drop were only because the candidate pool got bigger, adding the same number of irrelevant text and irrelevant images should have roughly similar effects.
The results were nothing like that.
Irrelevant images had a relatively limited effect, while irrelevant text made retrieval performance drop quickly.
With 50K distractors added, the gap in NDCG@10 caused by text versus image distractors reached 0.31–0.51 across models.
Figure 2 | Blue is Text-Doc distractors, orange is Image-Doc distractors. Even though both types are equally irrelevant to the query, the drop caused by text is clearly more severe.
This means:
The problem is not simply that "there are more candidate documents."
Text itself seems to enjoy a systematic ranking advantage.
So we began to ask:
Do multimodal retrieval models really only compare "who is more relevant to the query"?
Or, beyond semantic relevance, does the document's modality itself affect the final score?
The Real Problem: Modality Preference
To answer this, we had the same set of queries retrieve from three semantically corresponding but modality-different corpora:
Text, Image, and Fused.
If the model truly cared only about semantics, then for the same correct content:
the correct text, the correct image, and the correct image-text combination should have comparable competitiveness.
But the results show:
Text positives have systematically higher similarity; image positives are often the lowest; Fused usually sits in between.
In other words, even though all three express the same semantic content, the model still gives different scores because of "different modality."
More importantly, this is not a phenomenon confined to embedding geometry.
It actually changes the final ranking.
Figure 3 | Left: when the same semantic content appears as Text / Image / Fused, the model's similarity scores differ clearly. Right: the horizontal axis is the score of the correct document, the vertical axis is the score of the strongest irrelevant text; points above y=x mean "wrong text > correct document."
On Jina-CLIP-v2, in 94.6% of queries, irrelevant text scores higher than the relevant image.
In other words:
Faced with an image that actually contains the answer, the model may still be more willing to trust a wrong passage that "looks relevant."
This is what we call:
Modality Preference
It is not exactly the same as the Modality Gap often discussed in the past.
The traditional Modality Gap focuses more on whether text embeddings and image embeddings occupy different geometric regions.
But if a candidate pool is entirely of one modality, an overall score offset often does not affect ranking within that modality.
In mixed-modality retrieval, however, Text, Image, and Fused documents must enter the same ranked list.
At that point, any systematic score bias between modalities can directly change the Top-K.
So the real danger is not simply that "image and text representations do not fully overlap," but that:
The model is not only judging "how relevant this document is to the question" — it is also implicitly influenced by "what modality this document is."
The Solution: Our Answer Is Trident 🔱
Once the problem is identified, a natural idea follows:
Since the model tends to treat Text, Image, and Fused as three different kinds of things, why not tell it explicitly during training:
They are three equivalent positive views of the same piece of evidence.
So we propose Trident:
It contains two core designs.
Figure 4 | Trident: on the left, Modality-Balanced Data Construction; on the right, Multi-Positive View InfoNCE. The three positive views and negatives are jointly optimized in the same shared softmax.
① Modality-Balanced Data Construction
For the same image document, we simultaneously construct three positive views:
Text + Image + Fused
where:
the original image serves as the Image View;
Qwen3-VL-235B generates a detailed description as the Text View;
a short summary combined with the original image serves as the Fused View.
This way, a traditional single-positive training sample is expanded into three cross-modal co-equal positives.
But this step alone is not enough.
Because if Text, Image, and Fused each use their own independently normalized InfoNCE, the model may still only learn:
"how to rank within text"
and
"how to rank within images"
without really solving:
whether text scores and image scores can be compared directly.
② Multi-Positive View InfoNCE
The key to Trident is putting the three positive views and all negatives into the same shared softmax denominator.
The training objective is:
It can be further decomposed precisely into:
where the first part, Relevance Discrimination, serves to:
give all correct evidence higher probability mass overall than negatives.
Simply put:
Correct things should score higher than wrong things.
The second part, Positive-View Balance, serves to:
prevent the positive probability mass from concentrating entirely on one modality.
That is:
Do not rely only on Text just because it is easier to match, while ignoring Image and Fused.
So Trident is not just "putting all three modalities into the training data."
What it really does is:
let different modalities compete head-on in the same probability space for the first time.
How Well Does It Work? The Most Visible Change: the V-shaped Curve Is Filled In
Figure 6 | Before training (left), the retrieval performance of the two models varies clearly with the modality ratio of the corpus, with a pronounced performance trough; after Trident training (right), performance improves overall and the surface flattens, showing that the model's robustness to different modality compositions is significantly enhanced.
The pronounced performance trough in the original mixed-modality corpus is significantly reduced.
That is, no matter how the ratio of Text / Image / Fused in the knowledge base changes, the model's performance is more stable.
More importantly, this is not achieved at the expense of original single-modality capability.
On six Visual Document Retrieval benchmarks:
Qwen3-VL-Embedding-2B: Mix Avg NDCG@10 40.12 → 73.90 (+33.78)
Jina-CLIP-v2: Mix Avg NDCG@10 42.68 → 55.51 (+12.83)
And on the original single-modality data:
Qwen3-VL-Embedding-2B: Origin Avg 77.59 → 80.51
This shows that Trident does not simply "trade single-modality performance for mixed-modality capability."
On the contrary, it improves the robustness of mixed-modality retrieval while maintaining or even improving the original single-modality average.
What Changed Inside the Model?
Beyond final retrieval results, we further analyzed the representations before and after training.
Before training, the mean cosine similarity differed clearly across modality pairs.
For example:
Text–Text, Image–Image, Fused–Fused, as well as Text–Image, Text–Fused, and Image–Fused all showed clearly different center positions.
This shows the model still cares a lot internally about:
"Are you text, or are you an image?"
After Trident training, these distributions begin to converge noticeably.
Figure 5 | Before training (top), the distribution centers of different modality pairs differ clearly; after Trident training (bottom), the six distributions align significantly.
Standard deviation measuring the difference among the six similarity centers:
Jina-CLIP-v2: 0.092 → 0.011
Qwen3-VL-Embedding-2B: 0.055 → 0.004
But there is another important conclusion here:
Balance ≠ forcing all modalities to have exactly the same similarity.
Text, Image, and Fused do not carry exactly the same information.
What really matters is not making their cosine similarity values mechanically identical, but:
not letting any single modality enjoy a systematic ranking privilege.
Why We Think This Problem Deserves Attention
Over the past few years, multimodal retrieval models have become increasingly strong.
From CLIP and SigLIP to VLM-based embedding models, and on to Visual RAG and Multimodal RAG, we usually ask separately:
Is text retrieval strong?
Is image retrieval strong?
But the question real systems actually face is:
When text, images, charts, slides, scanned pages, and image-text pages are all mixed together, can they still be placed fairly on the same leaderboard?
Our experiments show the answer is not a given "yes."
A model can have strong image retrieval capability and strong text retrieval capability and still not necessarily be a reliable mixed-modality retriever.
This is also the problem Chaos in the Text aims to point out:
Multimodal capability should not just mean "I can handle every modality."
It should also mean: when different modalities truly compete on the same field, the model still ranks by relevance, not by modality.
🔱 Trident: Making Relevance the Protagonist of Ranking Again
One-sentence summary of this work:
We find that existing multimodal retrievers systematically prefer text, letting irrelevant text outrank genuinely relevant visual evidence in a mixed-modality corpus; Trident uses tri-view positive construction and a shared Multi-Positive View InfoNCE to let different modalities compete more fairly in the same probability space.
What we end up with:
more stable mixed-modality retrieval
weaker modality preference
stronger text-distractor robustness
overall improvement without sacrificing original single-modality average performance
➤ Paper
🔗 https://arxiv.org/abs/2610.11816
➤ Project Page
🔗 https://chaos-in-texts.github.io/
➤ Code
🔗 https://github.com/OpenBMB/Trident
If you are also following Multimodal RAG / VLM, we welcome your attention to our work, and we hope you will try Trident.
Maybe the biggest problem in multimodal retrieval is not that models cannot understand different modalities, but that they do not treat them equally.