Andrii Bidochko

• Updated: July 22, 2026
• 6 min read

MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

Direct Answer

MedRealMM is a large‑scale, multimodal benchmark built from real‑world Chinese online medical consultations, designed to evaluate how well large language models (LLMs) can generate safe, clinically appropriate responses when both text and patient‑uploaded images are present. It matters because it bridges the gap between synthetic test sets and the messy, image‑rich interactions that actual tele‑medicine platforms face today.

Background: Why This Problem Is Hard

Online medical consultation has exploded in the past five years, with millions of patients uploading photos of skin lesions, radiology scans, or wound dressings alongside their symptom narratives. Yet most AI evaluation pipelines still rely on:

• Purely textual dialogues generated by patient simulators.
• Synthetic image‑text pairs that do not reflect the variability of real clinical photography.
• Multiple‑choice or lexical‑overlap metrics (BLEU, ROUGE) that reward surface similarity rather than clinical correctness.

These shortcuts create a dangerous blind spot: a model can score highly on a benchmark while still producing unsafe advice, missing critical visual cues, or contradicting established medical guidelines. The problem is amplified in Chinese‑language tele‑health, where language‑specific nuances and a massive, fragmented hospital network make data collection and standardization especially challenging.

What the Researchers Propose

The authors introduce a two‑step framework that turns authentic consultation trajectories into a reproducible evaluation task:

Multimodal Clinical Challenge Point (MCCP) Extraction

Instead of evaluating an entire conversation, the pipeline isolates moments where a physician must make a decisive clinical judgment—e.g., interpreting a skin rash photo, confirming a diagnosis, or recommending a follow‑up test. These moments are labeled as MCCPs.

Standardized Next‑Response Generation

Each MCCP becomes a “next‑response” prompt: the model receives the full preceding text‑image context and must generate the physician’s reply. The task mirrors the real‑time decision point a tele‑medicine AI would face in production.

Physician‑Curated Rubrics

For every case, a panel of board‑certified doctors crafts a rubric that awards points for clinically desirable behaviors (accurate diagnosis, appropriate triage, empathetic language) and deducts points for unsafe or contradictory statements. This rubric replaces generic n‑gram metrics with a safety‑first, domain‑specific scoring system.

How It Works in Practice

The MedRealMM pipeline can be visualized as a linear workflow:

1. Data Ingestion: De‑identified chat logs and patient‑uploaded images are harvested from a nationwide Chinese internet hospital.
2. MCCP Detection: A rule‑based and ML‑augmented scanner flags high‑stakes turns in the dialogue (e.g., a request for a diagnosis after an image upload).
3. Prompt Construction: The system concatenates all prior messages, attaches the relevant image(s), and formats a clear “What should the doctor say next?” prompt.
4. Model Inference: Both text‑only LLMs (e.g., GPT‑4, Claude) and multimodal models (e.g., LLaVA, Gemini Vision) generate a response.
5. Rubric Scoring: Physicians evaluate each output against the case‑specific rubric, producing a composite safety‑adjusted score.
6. Result Aggregation: Scores are normalized across departments, enabling cross‑model comparison.

What sets this approach apart is the preservation of the exact visual context that clinicians rely on, coupled with a human‑validated, safety‑oriented scoring system. The benchmark therefore tests not just language fluency but true multimodal clinical reasoning.

Evaluation & Results

The study evaluated 19 LLMs, ranging from general‑purpose chat models to specialized medical variants, in both text‑only and multimodal configurations. Key evaluation dimensions included:

Scenario Coverage

Five thousand six hundred twenty MCCP instances spanned 64 clinical departments, from dermatology to radiology, ensuring that models were tested across a breadth of specialties.

Impact of Image Information

Multimodal models consistently outperformed their text‑only counterparts on image‑dependent tasks. For dermatology cases, the average rubric score rose by 18 % when the model could see the lesion photo, confirming that visual cues are not optional but essential for reliable clinical reasoning.

Safety and Error Patterns

Even the strongest frontier models met or exceeded physicians on positive criteria such as completeness of advice. However, they also triggered 2–3× more negative criteria—e.g., hallucinated medication dosages, contradictory statements, or failure to request essential follow‑up tests. This safety gap persisted across specialties, highlighting a systemic weakness in current LLM alignment.

Overall Ranking

No model reached the average physician benchmark. The top‑performing multimodal system achieved 84 % of the physician’s composite score, while the best text‑only model lagged at 62 %.

These findings demonstrate that image integration improves raw clinical performance, yet safety‑sensitive error avoidance remains the primary bottleneck for deploying LLMs in real‑world tele‑medicine.

Why This Matters for AI Systems and Agents

For developers building AI‑driven health assistants, MedRealMM offers a realistic yardstick that aligns model evaluation with the exact conditions of production deployments:

• Safety‑First Scoring: The physician‑crafted rubrics surface failure modes that traditional metrics hide, enabling teams to prioritize risk mitigation.
• Multimodal Integration Testing: By requiring models to process patient‑uploaded images, the benchmark forces engineers to adopt vision‑language architectures rather than relying on text‑only pipelines.
• Domain‑Specific Prompt Engineering: The “next‑response” format mirrors real‑time agent orchestration, guiding developers toward prompt patterns that can be directly embedded in production workflows.
• Benchmark‑Driven Model Selection: Companies can benchmark candidate models against MedRealMM before committing to costly fine‑tuning, reducing time‑to‑market for compliant health bots.

Integrating these insights into an Enterprise AI platform by UBOS can streamline the end‑to‑end pipeline—from data ingestion to safety‑aware response generation—while maintaining auditability required by regulators.

What Comes Next

While MedRealMM marks a significant step forward, several limitations invite future work:

• Geographic Diversity: The current dataset is confined to Chinese internet hospitals. Expanding to other languages and health systems will test cross‑cultural generalization.
• Temporal Dynamics: Real consultations evolve over multiple turns; future benchmarks could evaluate longitudinal reasoning and follow‑up planning.
• Fine‑Grained Attribution: Linking specific image regions to model explanations would improve interpretability and regulatory compliance.
• Automated Rubric Generation: Leveraging LLMs to draft preliminary rubrics could accelerate dataset scaling while preserving physician oversight.

Researchers interested in extending the benchmark can contribute new cases via the public Hugging Face repository. For organizations looking to prototype safe multimodal agents, the OpenAI ChatGPT integration provides a ready‑made bridge between powerful language models and custom image preprocessing pipelines.

How to Access MedRealMM

The full dataset, along with annotation guidelines and the MCCP extraction code, is publicly available on Hugging Face:

MedRealMM on Hugging Face

Licensing follows a non‑commercial research agreement, and the repository includes a detailed “Getting Started” notebook that walks users through data loading, prompt construction, and rubric‑based evaluation.

Conclusion

MedRealMM redefines how the AI community measures multimodal medical reasoning by anchoring evaluation in authentic, image‑rich consultations and physician‑validated safety criteria. The benchmark reveals that even the most advanced LLMs still fall short of human physicians, especially in avoiding unsafe advice. As AI agents become integral to tele‑health workflows, MedRealMM will serve as a critical reference point for model selection, safety testing, and regulatory compliance.

References & Further Reading

• MedRealMM paper (arXiv)
• Shi, R., Zhou, Q., Xu, Y., et al. “MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation.” 2026.
• OpenAI. “ChatGPT Technical Report.” 2023.
• Google DeepMind. “Gemini Vision: Multimodal Foundations.” 2024.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.