Andrii Bidochko

• Updated: July 19, 2026
• 7 min read

An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation

Direct Answer

The paper introduces a reference‑free evaluation framework that can automatically assess the quality of flowchart image‑to‑code generation at inference time, using only the input diagram and the produced code. This matters because it enables continuous, production‑grade monitoring of Vision‑Language Models (VLMs) without requiring costly ground‑truth annotations.

Background: Why This Problem Is Hard

Enterprises are increasingly deploying VLMs to transform visual documents—such as flowcharts, UML diagrams, or network topologies—into executable code snippets (e.g., Mermaid, GraphViz). In a controlled research setting, model performance is measured against a curated set of ground‑truth code strings. In real‑world pipelines, however, the input space is unbounded: users upload arbitrary diagrams, and no human‑written reference exists for each image.

Without a reference, traditional metrics like BLEU, ROUGE, or exact match become meaningless. Engineers resort to manual spot‑checks, which are slow, error‑prone, and impossible to scale. Moreover, VLMs can hallucinate elements—adding nodes or edges that never appeared in the source diagram—while simultaneously omitting critical steps. Detecting these failures automatically is essential for maintaining trust in downstream automation, such as CI/CD pipelines that ingest generated Mermaid code.

Existing approaches try to sidestep the problem by generating synthetic references or by fine‑tuning models on limited annotated datasets. These workarounds suffer from domain drift, high annotation cost, and an inability to capture the visual nuance of real‑world flowcharts (hand‑drawn symbols, varied fonts, noisy backgrounds). Consequently, a robust, reference‑free quality monitor has been a missing piece for production‑scale AI document processing.

What the Researchers Propose

The authors present a three‑metric suite that together forms a unified quality score, called F1_OCR‑VE. The metrics are:

• Recall_OCR: Estimates how much of the diagram’s textual content is covered by the generated code. It runs an OCR engine on the input image, extracts all recognizable tokens (e.g., node labels, decision texts), and checks whether each token appears in the output code.
• Precision_VE: Measures hallucination by posing a visual entailment (VE) problem. A lightweight VE model receives the original image and a rendered version of the generated code; it predicts whether every visual element in the rendered code is entailed by the source diagram.
• F1_OCR‑VE: The harmonic mean of Recall_OCR and Precision_VE, providing a single, interpretable score that balances coverage and fidelity.

Crucially, none of these components require a human‑written reference. They rely only on off‑the‑shelf OCR and a visual entailment classifier that can be pre‑trained on generic diagram datasets, making the framework lightweight enough for online deployment.

How It Works in Practice

The operational pipeline can be broken down into four stages:

1. Image Ingestion: The flowchart image is received by the VLM service. The image may be a PNG, JPEG, or PDF page.
2. Code Generation: The VLM (e.g., a multimodal transformer) produces a textual representation such as Mermaid syntax.
3. Reference‑Free Scoring:
   • OCR Extraction: An OCR engine (Tesseract, Google Vision, etc.) scans the image and returns a set of textual tokens.
   • Recall Calculation: The system tokenizes the generated code and computes the proportion of OCR tokens that appear in the code.
   • Visual Entailment Check: The generated code is rendered back into an image (using a Mermaid renderer). A VE model then predicts a binary entailment label for each visual element, yielding a precision value.
   • F1 Aggregation: Recall and precision are combined into the final F1_OCR‑VE score.
4. Monitoring & Alerting: The score is logged to a monitoring dashboard. If the score falls below a configurable threshold, an alert triggers a fallback path (e.g., human review or model rollback).

What sets this approach apart is its “online‑only” nature: the evaluation runs in the same request‑response cycle as code generation, incurring minimal latency. Because the metrics are derived from the input itself, they remain valid even as the VLM evolves or as new diagram styles appear.

Evaluation & Results

To validate the framework, the researchers used the FlowVQA benchmark, which pairs thousands of flowchart images with human‑annotated Mermaid code. They compared the reference‑free scores against traditional ground‑truth metrics (Exact Match, BLEU, and Human Rating).

• Recall_OCR achieved an average Pearson correlation of r = 0.97 with the ground‑truth coverage metric, indicating that OCR‑based token matching reliably reflects how much of the diagram is captured.
• Precision_VE correlated at r = 0.91 with human judgments of hallucination, confirming that visual entailment can spot spurious nodes and edges.
• The combined F1_OCR‑VE score showed a correlation of r = 0.94 with overall human quality ratings, outperforming a naïve string‑matching baseline by a wide margin.

Beyond correlation, the authors demonstrated that the framework can flag low‑quality generations in real time. In a simulated production run, 12 % of requests received an F1_OCR‑VE below 0.70, and manual inspection confirmed that each of these cases contained either missing steps or hallucinated branches.

These results prove that the reference‑free metrics are not only statistically aligned with human perception but also practical for continuous quality assurance.

Why This Matters for AI Systems and Agents

For teams building AI‑driven document pipelines, the ability to monitor output quality without a gold standard unlocks several strategic advantages:

• Continuous Deployment Confidence: Engineers can push model updates to production knowing that any regression will be caught instantly by the F1_OCR‑VE monitor.
• Agent‑Level Feedback Loops: Autonomous agents that orchestrate VLM calls can use the score as a reward signal, enabling reinforcement‑learning‑style self‑improvement without human‑in‑the‑loop labeling.
• Scalable Human‑in‑the‑Loop Workflows: Low‑scoring outputs can be routed to a human verification queue, optimizing the allocation of expert time.
• Cross‑Domain Portability: Because the framework relies on generic OCR and visual entailment, it can be repurposed for other diagram‑to‑code tasks (e.g., UML‑to‑Java, network‑topology‑to‑Terraform).

These capabilities align directly with the needs of modern AI orchestration platforms. For example, the Workflow automation studio can ingest the F1_OCR‑VE score as a conditional branch, automatically triggering alternative flows when quality drops. Similarly, AI marketing agents that generate visual campaign diagrams can rely on the same metrics to ensure that downstream rendering engines receive accurate specifications.

What Comes Next

While the framework marks a significant step forward, several open challenges remain:

• OCR Robustness: Hand‑drawn or low‑resolution diagrams still cause OCR failures. Future work could integrate specialized diagram OCR models trained on synthetic flowchart data.
• Fine‑Grained Entailment: The current VE model provides a binary hallucination flag. Extending it to pinpoint the exact spurious element would enable more precise corrective actions.
• Multi‑Modal Grounding: Incorporating textual annotations (e.g., captions) alongside visual cues could improve both recall and precision, especially for complex decision nodes.
• Benchmark Expansion: Evaluating the framework on non‑flowchart domains (e.g., circuit diagrams, architectural blueprints) would test its generality.

From a product perspective, integrating the evaluation suite into the Enterprise AI platform by UBOS would give customers a turnkey solution for quality‑aware code generation. Early adopters could also benefit from the UBOS partner program, which offers co‑development resources for custom OCR or VE models.

Finally, the framework opens a research avenue for “self‑checking” VLMs—models that not only generate code but also produce a confidence score derived from the same visual context. Such self‑assessment could become a standard component of future multimodal agents, reducing reliance on external monitoring infrastructure.

For readers interested in the technical details, the full study is available on arXiv paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.