Andrii Bidochko

• Updated: June 11, 2026
• 7 min read

MIRA: A Bilingual Benchmark for Medical Information Response Audit

Direct Answer

The paper introduces MIRA (Medical Information Response Audit), a bilingual benchmark designed to test whether large language models (LLMs) preserve the same medical content when faced with variations in user language, register, and health‑literacy signals. It matters because inconsistent or diluted health advice can jeopardize patient safety, especially as LLMs become front‑line sources of public‑facing medical information.

Background: Why This Problem Is Hard

Healthcare‑oriented LLMs are increasingly deployed in chatbots, virtual assistants, and patient‑education portals. While traditional safety evaluations focus on factual correctness, they rarely examine information equivalence across different ways users phrase the same health question. Real‑world patients differ in language proficiency, cultural background, and health‑literacy level, leading to a spectrum of prompts ranging from technical jargon to lay‑person simplifications.

Existing benchmarks—such as MedQA, PubMedQA, or the USMLE‑style tests—assess answer accuracy on a single prompt style, typically English‑only, and ignore how a model might truncate or omit critical details when the input is simplified. This blind spot creates a hidden risk: a model could appear correct on a high‑literacy query but provide a dangerously incomplete response to a low‑literacy user, amplifying health disparities.

Moreover, bilingual evaluation adds another layer of complexity. Non‑English speakers often receive translated health content that suffers from cultural nuance loss or mistranslation, yet most safety studies treat language as a binary “English vs. non‑English” factor without probing model‑specific language behaviors.

What the Researchers Propose

The authors present MIRA, a controlled, bilingual benchmark that systematically varies three user‑side signals:

• Language: English and Mandarin Chinese prompts.
• Register: Formal medical terminology versus conversational lay language.
• Health‑Literacy Signal: High‑literacy (detailed) versus low‑literacy (simplified) phrasing.

Each of the 60 medically reviewed, low‑risk health questions is instantiated in 72 prompt variants (2 languages × 2 registers × 2 literacy levels × 3 paraphrase seeds), yielding a total of 4,320 prompts. The benchmark does not merely score correctness; it audits the *comparability* of information across variants, flagging omissions, reduced actionable steps, and weakened support for autonomous decision‑making.

Key components of the MIRA framework include:

1. Prompt Generator: A rule‑based system that rewrites each base question according to the three signal dimensions while preserving the underlying medical intent.
2. Medical Reference Set: Expert‑curated answer keys that enumerate essential facts, recommended next steps, and decision‑support cues for each question.
3. Audit Engine: An automated pipeline that extracts factual nuggets from model outputs, aligns them with the reference set, and computes a “information dilution” score.

The authors coin the observed phenomenon “Differential Information Dilution” (DID), describing how low‑literacy prompts systematically lose key content compared with high‑literacy counterparts.

How It Works in Practice

The MIRA workflow can be visualized as a four‑stage pipeline:

1. Question Selection: Domain experts choose 60 low‑risk health topics (e.g., “How to treat mild seasonal allergies?”).
2. Prompt Diversification: The Prompt Generator creates 72 variants per question by swapping language, adjusting register, and simplifying or enriching the phrasing.
3. Model Interaction: Each LLM under test receives every variant and returns a free‑form answer.
4. Audit & Scoring: The Audit Engine parses each answer, matches extracted facts to the Medical Reference Set, and calculates three metrics:
   • Coverage Ratio – proportion of essential facts retained.
   • Concrete Next‑Step Count – number of actionable recommendations present.
   • Support‑for‑Judgment Score – presence of balanced risk/benefit language.

What sets MIRA apart from prior evaluations is its *controlled* manipulation of user signals. By holding the underlying medical question constant while varying only the user‑side attributes, the benchmark isolates the effect of language, register, and literacy on model behavior. This design enables a clean attribution of information loss to specific user characteristics rather than to question difficulty.

Evaluation & Results

The study evaluated five mainstream LLMs (Claude, Qwen, GPT‑4, Gemini, and Llama‑2) across the full MIRA suite. All models answered every question, confirming basic competence. However, the audit revealed systematic patterns:

• Differential Information Dilution (DID): Low‑literacy prompts caused a statistically significant drop in coverage ratio (average − 12 %) and concrete next‑step count (average − 8 %). The effect persisted across languages, indicating that simplification—not translation—drives dilution.
• Model‑Specific Language Effects: Contrary to the assumption that non‑English prompts are uniformly weaker, some models (e.g., Claude) performed better in Mandarin than in English for certain questions, while others (e.g., GPT‑4) showed the opposite trend.
• Mitigation via Knowledge‑Guided Prompt: Introducing a short “knowledge‑guidance” pre‑prompt (“Please provide a complete, step‑by‑step answer based on current medical guidelines”) reduced DID for most models. Claude’s dilution dropped by ~8 %, Qwen by ~6 %, while GPT‑4 showed marginal improvement.

To validate external relevance, the authors compared MIRA rankings with a separate set of 300 real‑world health queries collected from public forums. The rank‑order correlation (Spearman ρ ≈ 0.71) suggested that MIRA’s difficulty ordering aligns with actual user experiences, supporting its ecological validity.

Overall, the results demonstrate that current LLMs, while factually knowledgeable, are vulnerable to *information erosion* when faced with low‑literacy or simplified inputs—a risk that could translate into sub‑optimal patient guidance in production systems.

Why This Matters for AI Systems and Agents

For developers building health‑focused conversational agents, MIRA offers a concrete diagnostic tool to surface hidden biases in response completeness. The benchmark’s granular metrics enable teams to:

• Identify which user segments (e.g., low‑literacy patients) are most likely to receive incomplete advice.
• Benchmark mitigation strategies such as system‑level prompts, retrieval‑augmented generation, or post‑processing validation.
• Inform prompt‑engineering pipelines that automatically adapt to user literacy signals, ensuring consistent information delivery.

From an orchestration perspective, integrating MIRA‑style audits into continuous‑integration (CI) pipelines can act as a safety gate before deploying new model versions. This aligns with emerging regulatory expectations for AI in healthcare, where explainability and equity are becoming compliance checkpoints.

Practically, organizations can leverage the UBOS platform overview to construct modular agent workflows that embed the knowledge‑guided pre‑prompt as a reusable component. Coupled with the Workflow automation studio, teams can automate the audit loop: generate prompts, invoke the LLM, run the audit engine, and trigger alerts when DID exceeds a predefined threshold.

Furthermore, the benchmark underscores the importance of multilingual support that goes beyond translation. By exposing model‑specific language strengths, MIRA encourages developers to adopt *language‑aware routing*—sending queries to the model that performs best for a given language, a capability that can be orchestrated through ChatGPT and Telegram integration for multilingual patient outreach.

What Comes Next

While MIRA marks a significant step forward, several limitations remain:

• Scope of Medical Topics: The benchmark focuses on low‑risk, well‑understood conditions. High‑risk or rare diseases may exhibit different dilution dynamics.
• User Demographics: Only English and Mandarin are covered. Extending to additional languages and cultural contexts will be essential for global deployment.
• Dynamic Interaction: Real‑world health conversations are multi‑turn. Future work should evaluate how information dilution evolves across dialogue turns.

Future research directions include:

1. Integrating retrieval‑augmented generation (RAG) pipelines that pull up‑to‑date clinical guidelines, potentially reducing dilution.
2. Developing adaptive prompting strategies that detect user literacy in real time and automatically inject knowledge‑guidance.
3. Expanding the benchmark to cover regulatory compliance checks (e.g., HIPAA, GDPR) alongside informational completeness.

From an application standpoint, the insights from MIRA can inform the design of AI marketing agents that need to convey health‑related product information responsibly, as well as the creation of Enterprise AI platform by UBOS solutions that embed rigorous audit layers for any customer‑facing LLM.

Developers interested in experimenting with the benchmark can download the dataset and audit scripts from the MIRA benchmark paper. By incorporating these tools into their development lifecycle, they can proactively safeguard against differential information dilution and deliver more equitable health advice.

Stay ahead of the curve—explore how UBOS can help you build trustworthy, multilingual health agents that maintain information fidelity across every user interaction.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.