Andrii Bidochko

• Updated: August 13, 2026
• 6 min read

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Direct Answer

ExtractBench introduces the first large‑scale benchmark that evaluates how well AI agents can perform schema‑guided extraction on real enterprise documents, measuring not only value accuracy but also grounding traceability and operational cost. It matters because it gives product teams a concrete yardstick for building agents that reliably pull structured data from messy business paperwork while keeping expenses predictable.

Background: Why This Problem Is Hard

Enterprises rely on a constant stream of invoices, contracts, forms, and reports. Turning these PDFs or scanned images into structured records is a prerequisite for downstream analytics, compliance, and automation. The difficulty stems from three intertwined factors:

• Heterogeneous layouts: Even documents of the same type (e.g., purchase orders) can vary wildly in column order, font, and language.
• Schema variability: Business users define custom extraction schemas—field names, hierarchies, and validation rules—that change as regulations or processes evolve.
• Traceability requirements: Auditors demand evidence linking each extracted value back to its source location (page number, word span), a demand most existing models ignore.

Current solutions fall into two camps. “One‑shot” large language models (LLMs) can answer extraction questions but often hallucinate or truncate long record lists, sacrificing completeness. On the other hand, code‑generated agents (e.g., Python scripts using OCR libraries) achieve higher fidelity but incur heavy compute costs and require extensive engineering effort. Without a unified benchmark, teams cannot objectively compare these trade‑offs or know which approach scales to thousands of pages per day.

What the Researchers Propose

The authors present ExtractBench, a comprehensive evaluation suite that captures the full lifecycle of schema‑guided extraction:

• Scalable dataset: 4,869 pages drawn from 370 authentic enterprise documents spanning eight business domains and 67 distinct document types.
• Multi‑dimensional metrics:
  • Order‑insensitive value F1 for correctness of extracted fields.
  • Word‑level and page‑level grounding F1 to assess traceability.
  • Measured cost (GPU minutes, API tokens) to reflect operational expense.
• Ground‑truth pipeline: Combines independent‑system agreement on real documents, synthetic list generation for known values, and human verification for form‑based entries, ensuring high‑quality labels.

In essence, ExtractBench treats schema‑guided extraction as a three‑pronged service contract: deliver the right data, show where it came from, and do it affordably.

How It Works in Practice

The benchmark’s workflow can be visualized as a pipeline of four interacting components:

1. Document Ingestion Layer: Accepts PDFs, scanned images, or digital text and runs OCR (if needed) to produce a tokenized representation.
2. Schema Interpreter: Parses the user‑defined JSON‑like schema, mapping each field to a query template that the downstream agent will fulfill.
3. Extraction Agent: This is the “brain”—either a prompting‑based VLM (e.g., GPT‑4V) or a coding‑centric agent (e.g., LlamaExtract). It receives the token stream and schema prompts, then generates a structured JSON output.
4. Grounding & Cost Logger: Post‑processes the agent’s output to attach word‑span and page identifiers, while simultaneously recording compute usage, API calls, and latency.

What sets ExtractBench apart is the enforced “schema fidelity” rule: the agent must not add, drop, or rename fields. Any deviation is penalized in the value‑F1 score. Moreover, the grounding step is not optional; missing source references directly lower the grounding F1, encouraging developers to build explainable pipelines.

Evaluation & Results

The authors evaluated three families of agents across the benchmark:

• Commercial Vision‑Language Models (VLMs): Strong on short forms but frequently truncate long tables, leading to lower completeness.
• Coding Agents (e.g., LlamaExtract Agentic Plus): Maintain high accuracy on both short and long documents but consume significantly more compute.
• Baseline OCR‑plus‑Rule pipelines: Provide deterministic grounding but suffer from brittle rule maintenance.

Key takeaways:

• All agents achieved >85% value‑F1 on simple invoices, confirming that baseline extraction is now “solved” for low‑complexity cases.
• On multi‑page contracts with extensive clause tables, VLMs’ value‑F1 dropped to ~62% due to list truncation, while LlamaExtract retained ~88%.
• Grounding scores mirrored value performance: VLMs often omitted page references for long lists, whereas coding agents consistently hit >90% page‑level F1.
• Cost analysis revealed that LlamaExtract’s superior accuracy came at roughly 3× the GPU‑minute expense of VLMs, highlighting a clear accuracy‑cost frontier.

Overall, the benchmark demonstrates that no single approach dominates across all dimensions; the choice hinges on the organization’s tolerance for cost versus completeness.

Why This Matters for AI Systems and Agents

For AI practitioners building enterprise‑grade agents, ExtractBench offers a concrete, reproducible yardstick that aligns with real‑world SLAs:

• Design‑by‑metric: Teams can now target specific grounding F1 thresholds to satisfy audit requirements, rather than guessing after deployment.
• Cost‑aware optimization: By exposing token and GPU consumption per page, developers can perform cost‑benefit analyses early in the product roadmap.
• Benchmark‑driven iteration: The modular evaluation suite encourages rapid A/B testing of prompting strategies, toolchain upgrades, or hybrid VLM‑coding ensembles.

These capabilities map directly onto the Enterprise AI platform by UBOS, which already provides a plug‑and‑play environment for orchestrating OCR, schema parsing, and LLM inference. By integrating ExtractBench‑style validation into the platform’s Workflow automation studio, product teams can automatically flag records that fail grounding checks, trigger human‑in‑the‑loop review, and log cost metrics for budgeting.

Moreover, the benchmark’s emphasis on schema fidelity resonates with UBOS’s templates for quick start, which let non‑technical users define extraction schemas without writing code. When combined with the benchmark’s grounding feedback loop, these templates become self‑auditing assets that evolve as document layouts change.

What Comes Next

While ExtractBench marks a significant step forward, several open challenges remain:

• Dynamic schema evolution: Real‑world workflows often add or retire fields on the fly. Future benchmarks should simulate schema drift and evaluate agents’ ability to adapt without retraining.
• Multilingual and multimodal extensions: Enterprises operating globally encounter documents in dozens of languages and mixed media (e.g., embedded charts). Adding these dimensions will stress‑test agents’ generalization.
• Human‑in‑the‑loop feedback loops: Incorporating corrective annotations back into the model could improve grounding over time, a research direction that aligns with UBOS’s Openclaw tools for continuous learning.
• Cost‑predictive modeling: Building a meta‑model that forecasts GPU or token usage based on document characteristics would let enterprises provision resources proactively.

Addressing these gaps will unlock truly autonomous document processing pipelines that scale across continents and regulatory regimes. Companies interested in staying ahead can explore UBOS’s pricing plans to access the compute‑optimized inference clusters needed for high‑throughput coding agents, or leverage the UBOS solutions for SMBs to prototype cost‑effective VLM‑based extraction workflows.

For a deeper dive into the methodology and raw numbers, consult the original ExtractBench paper. The dataset and evaluation scripts are publicly available on Hugging Face, enabling anyone to benchmark their own agents against the same standards.

Ready to future‑proof your document pipelines? Visit the UBOS homepage to explore how our platform can integrate ExtractBench insights into your AI strategy today.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.