Andrii Bidochko

• Updated: July 3, 2026
• 7 min read

Beyond ‘One Language, One Script’: Quantifying Orthographic Bias in Multilingual VLMs with PuMVR

{{IMAGE_PLACEHOLDER}}

Direct Answer

The paper introduces PuMVR (Punjabi Multimodal Visual Reasoning), the first benchmark that quantifies script‑dependent orthographic bias in multilingual vision‑language models (VLMs). It matters because it reveals a systematic “Script Gap”—models can solve the same visual‑reasoning task in one Punjabi script while failing in another, exposing fairness risks for billions of users who write in multiple scripts.

Background: Why This Problem Is Hard

Multilingual VLMs such as CLIP, Flamingo, and Gemini are praised for handling dozens of languages, yet they implicitly assume a one‑to‑one mapping between language and writing system. In reality, many languages—Punjabi, Serbian, Kurdish, Hindi‑Urdu, among others—are written in two or three active scripts. This multi‑script reality creates three intertwined challenges:

• Orthographic variance: The same lexical item appears with completely different glyphs across scripts, breaking token‑level sharing.
• Data imbalance: Public web‑crawled corpora heavily favor the dominant script, leaving minority scripts under‑represented during pre‑training.
• Evaluation blind spots: Standard multilingual benchmarks test language‑level performance but ignore script‑level consistency, so bias remains hidden.

Existing multilingual benchmarks (e.g., XGLUE, XTREME) evaluate translation or classification across languages but do not probe whether a model’s reasoning ability transfers across scripts of the same language. Consequently, developers lack a concrete metric to detect or remediate orthographic bias, and end‑users experience unpredictable model behavior when switching scripts.

What the Researchers Propose

The authors propose a two‑part framework:

1. PuMVR benchmark: A curated set of 375 image‑reasoning items grounded in Punjabi culture, each rendered in three scripts—Gurmukhi, Shahmukhi, and Roman. Every item is a semantic triple (image, question, answer) duplicated across scripts, ensuring that only the orthography changes.
2. Script Consistency Rate (SCR): A metric that measures the proportion of items for which a model produces the same correct answer across all scripts. SCR directly captures orthographic bias, complementing traditional accuracy scores.

Key components of the framework include:

• Script‑balanced data collection: Native speakers authored questions in each script, guaranteeing cultural relevance and linguistic parity.
• Cross‑script pairing: For each visual puzzle, three parallel instances are generated, enabling a controlled comparison of script impact.
• Evaluation pipeline: State‑of‑the‑art VLMs are queried with the same prompt in each script; answers are normalized and compared against ground truth.

How It Works in Practice

Conceptual Workflow

The PuMVR evaluation proceeds through four stages:

1. Image selection: Curators choose culturally salient photographs (e.g., a traditional Punjabi wedding, a street market).
2. Question authoring: Linguists write a reasoning question in Gurmukhi, then translate it into Shahmukhi and Roman while preserving semantics.
3. Model inference: Each VLM receives the image and a script‑specific textual prompt; the model generates an answer.
4. Metric computation: Accuracy is calculated per script, and SCR is derived by checking whether the model’s answer is correct for all three scripts simultaneously.

Interaction Between Components

During inference, the visual encoder processes the image once, producing a shared visual embedding. The textual encoder then consumes the script‑specific prompt, creating distinct language embeddings. These embeddings are fused (e.g., via cross‑attention) to produce a joint representation that feeds the answer head. Because the visual backbone is shared, any performance discrepancy across scripts can be attributed to the language side—precisely the orthographic bias the benchmark aims to surface.

What Makes This Approach Different

• Script‑controlled design: By holding the visual content constant and only varying the script, PuMVR isolates orthographic effects from visual difficulty.
• Metric focus on consistency: Traditional accuracy rewards models that excel in the dominant script; SCR penalizes uneven performance, encouraging developers to build truly script‑agnostic systems.
• Open‑ended reasoning tasks: Unlike classification benchmarks, PuMVR includes multi‑step reasoning (e.g., “What is the relationship between the two people in the photo?”), exposing deeper language‑vision interactions.

Evaluation & Results

Scenarios and Tasks Tested

The authors evaluated ten leading VLMs, including OpenAI’s CLIP‑V2, Meta’s Flamingo‑3B, and Google’s Gemini‑Pro. Each model was tested on three script subsets:

• Gurmukhi (native Punjabi script used in India)
• Shahmukhi (Perso‑Arabic script used in Pakistan)
• Roman (Latin transliteration common in informal digital communication)

Tasks spanned four reasoning categories: object identification, relational inference, cultural knowledge, and counter‑factual reasoning.

Key Findings

• Script Gap: Accuracy deltas between the best‑performing and worst‑performing scripts reached up to 16 % for several models, indicating a pronounced bias.
• Low Script Consistency: SCR values ranged from 24.8 % to 58.3 % across models, meaning that in many cases a model answered correctly in only one or two scripts.
• Visual Input Helps but Doesn’t Close the Gap: Adding the image boosted overall accuracy by 7‑12 % on average, yet SCR improvements were modest (2‑5 % points), suggesting that visual context does not fully compensate for orthographic disparity.
• Cross‑script Transferability Is Limited: Fine‑tuning on one script rarely improved performance on the others, confirming that knowledge does not automatically transfer across scripts.
• Chain‑of‑Thought Divergence: When prompting models to generate step‑by‑step reasoning, the generated chains differed substantially between scripts, even for identical visual inputs.

Why the Findings Matter

These results demonstrate that current multilingual VLMs are not truly multilingual—they are “script‑biased.” For enterprises deploying VLMs in regions where users switch scripts (e.g., social media platforms, e‑commerce sites), the inconsistency can lead to user frustration, reduced trust, and potential regulatory scrutiny around fairness.

Why This Matters for AI Systems and Agents

For AI practitioners building agents that rely on visual‑language understanding, orthographic bias translates into operational risk:

• Inconsistent user experiences: A chatbot that answers correctly in Gurmukhi but fails in Shahmukhi will alienate half of its Punjabi‑speaking audience.
• Bias amplification in downstream pipelines: Agents that feed VLM outputs into recommendation engines or sentiment analyzers may propagate script‑specific errors, skewing business metrics.
• Compliance and fairness: Regulations in the EU and emerging AI governance frameworks demand demonstrable equity across linguistic groups; SCR provides a quantifiable audit metric.

Developers can integrate these insights into their workflows using the Workflow automation studio to automatically flag script‑inconsistent responses during testing. Moreover, the OpenAI ChatGPT integration can be leveraged to generate synthetic script‑paired data, helping to balance training corpora before fine‑tuning.

By adopting SCR as a standard evaluation checkpoint, product teams can ensure that AI agents behave uniformly regardless of the script a user chooses, thereby improving user retention and meeting fairness standards.

What Comes Next

While PuMVR makes a compelling case for script‑aware evaluation, several open challenges remain:

• Scaling beyond Punjabi: Extending the benchmark to other multi‑script languages (e.g., Serbian, Kurdish) will test the generality of SCR.
• Model architecture innovations: Designing encoders that share sub‑character representations across scripts could reduce orthographic bias.
• Data augmentation strategies: Leveraging transliteration tools or multilingual tokenizers that treat script variants as related units may improve cross‑script transfer.
• Human‑in‑the‑loop evaluation: Incorporating native speaker judgments on answer plausibility across scripts can refine SCR thresholds.

Future research could also explore Enterprise AI platform by UBOS for large‑scale script‑balanced pre‑training, or use the UBOS templates for quick start to prototype script‑agnostic VLM pipelines.

In the meantime, practitioners should audit their existing VLM deployments with PuMVR‑style tests, report SCR alongside accuracy, and prioritize script‑balanced data collection. Doing so will move the field toward truly multilingual, script‑inclusive AI systems.

For a deeper dive into the methodology and full dataset, see the original paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.