- Updated: March 11, 2026
- 7 min read
MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains
Direct Answer
MC-Search introduces the first benchmark that evaluates multimodal, agentic retrieval‑augmented generation (MM‑RAG) systems on long, step‑wise reasoning chains, and it also provides a process‑supervised fine‑tuning method called Search‑Align that improves planning and retrieval fidelity in open‑source multimodal models.
This matters because it surfaces systematic weaknesses in current multimodal agents—such as over‑retrieval, under‑retrieval, and modality‑misaligned planning—and offers a concrete data‑driven path to more reliable, real‑world AI assistants that must reason across text, images, and external knowledge sources.
Background: Why This Problem Is Hard
Modern multimodal large language models (MLLMs) excel at answering single‑turn questions that can be solved with a brief retrieval step followed by generation. However, many practical AI applications—digital assistants, autonomous research agents, and visual‑question‑answering bots—require:
- Multi‑hop reasoning that spans several modalities (e.g., text → image → table).
- Dynamic planning where the agent decides which modality to query next based on intermediate results.
- Verification of each retrieved piece of evidence before it is incorporated into the final answer.
Existing benchmarks (e.g., VQA, ScienceQA, or standard RAG datasets) focus on short, static retrieval chains and provide only answer‑level accuracy. They do not capture the process‑level decisions that an autonomous agent must make, nor do they enforce hop‑wise attribution of evidence. Consequently, developers lack a reliable way to measure whether a model truly understands when to look up an image versus a document, or whether it can correctly stitch together a multi‑modal reasoning path.
From an engineering perspective, this gap translates into brittle pipelines: a model may hallucinate facts, retrieve irrelevant media, or stop the reasoning process prematurely. Without a benchmark that stresses these capabilities, progress in building robust multimodal agents stalls.
What the Researchers Propose
The authors present two tightly coupled contributions:
- MC-Search Benchmark: A curated collection of 3,333 examples, each annotated with a detailed, five‑type reasoning structure. Every example includes:
- Sub‑questions that break down the main query.
- Explicit retrieval modality (text, image, audio, etc.) for each hop.
- Supporting facts and intermediate answers.
- Hop‑wise Attribution and Verification of Evidence (HAVE) to guarantee chain fidelity.
- Search‑Align Framework: A process‑supervised fine‑tuning approach that ingests the verified reasoning chains from MC‑Search. By aligning the model’s internal planning signals with the ground‑truth hops, Search‑Align teaches the agent to:
- Predict the correct modality for the next retrieval step.
- Select the most relevant evidence from a candidate pool.
- Maintain a coherent intermediate answer state.
In essence, MC‑Search supplies the “gold standard” for what a well‑behaved multimodal agent should do, while Search‑Align provides the training recipe to get there.
How It Works in Practice
Conceptual Workflow
The unified MM‑RAG pipeline built for the benchmark follows a loop of three core stages:
- Planning: Given the user query, a planner module predicts the next sub‑question and the modality (e.g., “search image of the Eiffel Tower”).
- Retrieval: A modality‑specific retriever (text search engine, image index, audio database) returns a ranked list of candidates.
- Generation & Verification: The generator consumes the top‑k retrieved items, produces an intermediate answer, and runs a verification check against the HAVE criteria before committing the hop.
This loop repeats until the final answer is assembled, typically after 3–5 hops for the MC‑Search dataset.
Component Interaction
Figure 1 (illustrative) shows the data flow:
Key interactions:
- Planner ↔ Retriever: The planner’s modality prediction conditions the retriever’s index selection, ensuring that the system does not waste resources querying the wrong source.
- Retriever ↔ Generator: Retrieved documents are encoded and concatenated with the current reasoning state before generation, allowing the model to ground its output in concrete evidence.
- Generator ↔ Verifier (HAVE): After each generation step, a verifier checks that the claimed evidence appears verbatim in the retrieved source, enforcing attribution.
What Sets This Apart
Traditional RAG pipelines treat retrieval as a static pre‑processing step and rely on post‑hoc metrics (e.g., BLEU, ROUGE) to evaluate the final answer. MC‑Search flips that paradigm by:
- Embedding modality choice directly into the planning stage.
- Requiring hop‑level evidence verification, which dramatically reduces hallucination.
- Providing a process‑level supervision signal (via Search‑Align) that aligns model internals with the annotated chain, rather than only optimizing for end‑task loss.
Evaluation & Results
Test Scenarios
The authors evaluated six leading MLLMs (including both closed‑source and open‑source variants) on three dimensions:
- Answer Accuracy: Correctness of the final answer compared to the ground truth.
- Planning Fidelity: How often the model selected the right modality and sub‑question at each hop.
- Retrieval Quality: Precision of the retrieved evidence relative to the annotated supporting facts.
Key Findings
| Model | Final Answer Accuracy | Planning Fidelity | Retrieval Precision |
|---|---|---|---|
| Closed‑source MLLM A | 68 % | 55 % | 60 % |
| Open‑source MLLM B (baseline) | 45 % | 38 % | 42 % |
| Open‑source MLLM B + Search‑Align | 61 % | 52 % | 58 % |
Across the board, models suffered from two systematic errors:
- Over‑retrieval: Pulling more items than needed, which diluted the generator’s focus and increased hallucination risk.
- Modality‑misaligned planning: Selecting the wrong retrieval type (e.g., asking for a text snippet when an image was required), leading to dead‑end hops.
Applying Search‑Align closed a substantial portion of this gap, especially for open‑source models, demonstrating that process‑level supervision can be a practical lever for improving multimodal agents without massive model scaling.
Why This Matters for AI Systems and Agents
For practitioners building production‑grade agents, MC‑Search offers a concrete yardstick to measure not just “does the answer look right?” but “did the system get there the right way?” This distinction is critical for several reasons:
- Reliability in High‑Stakes Domains: In fields like medical imaging or legal document analysis, an agent must justify each claim with verifiable evidence. MC‑Search’s hop‑wise verification mirrors those compliance requirements.
- Cost‑Effective Orchestration: By penalizing over‑retrieval, the benchmark encourages agents to be parsimonious with API calls and storage look‑ups, directly reducing operational expenses.
- Modality‑Aware Planning: The explicit modality prediction step aligns with emerging agent orchestration frameworks that schedule heterogeneous tool calls, making integration smoother.
- Fine‑Tuning Roadmap: Search‑Align shows that a modest amount of process‑supervised data can lift open‑source models to near‑state‑of‑the‑art performance, lowering the barrier for teams without access to massive proprietary datasets.
In short, MC‑Search shifts the evaluation focus from “what the model says” to “how the model gets there,” a paradigm that aligns with the next generation of trustworthy, multimodal AI assistants.
What Comes Next
While MC‑Search and Search‑Align mark a significant step forward, several open challenges remain:
- Scalability of Annotation: Creating hop‑wise, multimodal chains is labor‑intensive. Future work could explore semi‑automated annotation pipelines or crowdsourcing strategies.
- Dynamic Knowledge Updates: Real‑world agents must handle evolving corpora (e.g., news feeds). Extending the benchmark to include temporal reasoning would test continual learning capabilities.
- Beyond Five Hops: Some complex tasks may require deeper reasoning. Investigating how performance degrades with longer chains will inform model architecture choices.
- Cross‑Agent Collaboration: In multi‑agent ecosystems, one agent’s output becomes another’s input. Designing benchmarks that capture inter‑agent handoffs is an exciting frontier.
Addressing these directions will likely involve tighter integration of retrieval engines, modality‑specific encoders, and verification modules. Teams interested in experimenting with next‑generation pipelines can start by exploring our multimodal retrieval stack and the future‑research hub for community‑driven datasets.