Andrii Bidochko

• Updated: August 23, 2026
• 7 min read

Diagram-MMU: A Multi‑Modal Benchmark for Scientific Diagrams

Direct Answer

Diagram‑MMU is a newly released multimodal benchmark that measures how well large language models can read, interpret, and convert scientific diagrams into executable code or answer questions about them. It matters because it exposes a critical blind spot in today’s AI assistants—handling visual scientific artifacts with the same fluency as text.

Background: Why This Problem Is Hard

Scientific diagrams—think of circuit schematics, chemical reaction pathways, or statistical plots—are dense visual encodings of domain knowledge. Researchers and engineers routinely embed such figures in papers, patents, and product documentation, expecting AI tools to extract meaning, edit them, or regenerate them programmatically. Yet most multimodal large language models (MLLMs) excel at captioning or answering simple visual questions, while stumbling when asked to produce precise LaTeX TikZ code or to edit an existing diagram.

Two intertwined bottlenecks explain this gap:

• Semantic granularity: Diagrams convey relationships (arrows, nodes, axes) that are not captured by pixel‑level features alone. Translating those relationships into a formal language requires a deep, structured understanding.
• Code synthesis fidelity: Even if a model “knows” what a diagram represents, generating syntactically correct and compilable TikZ code is a non‑trivial programming task. Small syntax errors break the entire rendering pipeline.

Existing evaluation suites—such as VQA‑Science or ImageNet‑Style—focus on classification or short‑answer QA, leaving diagram‑to‑code generation largely untested. Consequently, developers building AI‑augmented scientific workspaces (e.g., OpenAI Prism) lack a reliable yardstick to gauge progress.

What the Researchers Propose

The authors introduce Diagram‑MMU, a multimodal benchmark specifically crafted for scientific diagram parsing and understanding. The benchmark is built around three core tasks that mirror real‑world workflows in collaborative research platforms:

1. Diagram‑to‑code parsing: Convert a raw image of a diagram into a complete LaTeX TikZ script.
2. Diagram‑to‑code editing: Given an existing TikZ snippet and a modified diagram, produce the updated code that reflects the visual changes.
3. Diagram question answering (QA): Answer natural‑language queries that require reasoning over the diagram’s structure.

To stress‑test models under realistic conditions, the benchmark also defines an agentic setting. In this mode, an AI agent can iteratively query a toolchain (e.g., a code compiler or a visual parser) before delivering its final answer, mimicking how a human would use auxiliary utilities during scientific writing.

How It Works in Practice

Diagram‑MMU follows a straightforward, MECE‑compatible workflow:

• Dataset curation: Curators collected 3,700 high‑quality scientific diagrams spanning six domains (physics, chemistry, biology, engineering, statistics, and computer science). Each diagram is paired with 18,300 human‑validated questions and a ground‑truth TikZ representation.
• Task packaging: For each diagram, the benchmark supplies three JSON‑formatted prompts—one per task—so that a model can be fed the same visual input but asked to perform a distinct operation.
• Agentic orchestration: When evaluating under the agentic regime, the model receives an additional “tool‑use” API that lets it call a compiler, a diagram parser, or a retrieval module. The model can loop through these calls, refine its output, and finally submit the answer.
• Scoring pipeline: Parsing and editing are judged by exact‑match on the generated TikZ code after normalizing whitespace and ordering. QA is scored with a combination of exact match and semantic similarity (BLEU‑4, ROUGE‑L).

What sets Diagram‑MMU apart is the tight coupling between visual perception and program synthesis. Unlike generic multimodal benchmarks, it forces models to produce *executable* artifacts, not just textual descriptions.

Evaluation & Results

The authors benchmarked twelve state‑of‑the‑art MLLMs, ranging from open‑source vision‑language models to proprietary agents such as Claude‑4.6 Opus. Each model was tested in both the standard (non‑agentic) and agentic configurations.

Key observations:

• Parsing vs. QA difficulty: All models achieved moderate success on diagram QA (average accuracy ≈ 62 %), indicating that they can reason over visual layouts. However, parsing accuracy lagged dramatically, with the best non‑agentic model hitting only 28 % exact‑match on TikZ generation.
• Agentic boost: Providing tool‑use capabilities improved parsing and editing scores for most models (average gain of 12 % points). The trade‑off was a slight dip in QA performance, suggesting that the extra reasoning steps sometimes over‑complicate simple answer retrieval.
• Claude‑4.6 Opus outlier: This model improved across all three tasks when operating agentically, reaching 41 % parsing accuracy, 35 % editing accuracy, and 68 % QA accuracy. Its balanced performance hints at a more robust internal planning module.

These results collectively demonstrate that current MLLMs are still “visual question answerers” rather than “visual code generators.” The benchmark therefore highlights a concrete research frontier: bridging perception and precise program synthesis.

Why This Matters for AI Systems and Agents

For developers building AI‑augmented scientific workspaces, Diagram‑MMU offers a diagnostic lens that goes beyond generic language metrics. If an agent can answer “What does this plot show?” but fails to emit correct TikZ, the user experience will still be fragmented—manual copy‑pasting and debugging become inevitable.

Integrating a benchmark‑driven evaluation loop can guide product roadmaps in three ways:

1. Feature prioritization: Teams can allocate engineering effort toward improving diagram parsing pipelines, knowing that a measurable lift on Diagram‑MMU translates directly into smoother LaTeX authoring.
2. Tool orchestration: The agentic setting proves that coupling LLMs with external compilers or visual parsers yields tangible gains. Platforms like the Workflow automation studio can expose such toolchains to end‑users without custom code.
3. Competitive benchmarking: By publishing scores on a public leaderboard, companies can benchmark their proprietary models against open‑source baselines, fostering transparent progress.

In practice, a research team could embed Diagram‑MMU into their CI pipeline, automatically flagging regressions in diagram‑to‑code capability before a new model version ships. This proactive stance reduces the risk of releasing an assistant that looks smart on text but breaks on visual tasks.

What Comes Next

While Diagram‑MMU marks a significant step forward, several limitations remain:

• Domain breadth: The current six domains cover many academic fields but omit specialized engineering schematics (e.g., PCB layouts) and medical imaging diagrams.
• Code diversity: TikZ is a powerful but niche language. Extending the benchmark to other diagramming formats (SVG, Mermaid, or even Python‑based Matplotlib scripts) would broaden applicability.
• Human‑in‑the‑loop evaluation: Exact‑match scoring penalizes harmless formatting differences. Future work could incorporate a compiler‑aware metric that tolerates stylistic variance while still ensuring renderability.

Potential research directions include:

1. Developing hybrid architectures that combine vision transformers with program synthesis modules trained on large code corpora.
2. Exploring reinforcement learning from human feedback (RLHF) where the reward signal is the successful compilation of generated code.
3. Building open toolkits that expose a “diagram compiler” API, enabling agents to iteratively refine code—much like the agentic setting in the benchmark.

For practitioners eager to experiment, the benchmark is openly available on its project page. Researchers can download the dataset, plug it into their training pipelines, and report results back to the community.

Beyond research, enterprises can leverage these insights to power next‑generation AI assistants that truly understand visual scientific content. The Enterprise AI platform by UBOS already supports multimodal model deployment, making it a natural home for a Diagram‑MMU‑trained agent.

Finally, startups looking to differentiate their AI products can explore the UBOS for startups program, which offers accelerated access to compute resources and integration tools for cutting‑edge benchmarks like Diagram‑MMU.

References & Further Reading

• Original arXiv paper: Diagram‑MMU: A Multi‑Modal Benchmark for Scientific Diagrams
• OpenAI Prism – a collaborative scientific writing workspace (public documentation)
• Claude‑4.6 Opus model specifications (Anthropic technical blog)

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.