Andrii Bidochko

• Updated: July 18, 2026
• 2 min read

PolyWorkBench: Benchmarking Multilingual Long‑Horizon LLM Agents

PolyWorkBench: Benchmarking Multilingual Long‑Horizon LLM Agents

Large language model (LLM) agents have demonstrated impressive capabilities in long‑horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks assume a monolingual workflow, ignoring the multilingual reality of many real‑world applications. PolyWorkBench addresses this gap by providing a comprehensive benchmark for evaluating LLM agents on multilingual, long‑horizon workplace workflows.

Key Contributions

• Introduces a benchmark covering 67 tasks across five domains: commerce, knowledge work, legal analysis, localization, and manufacturing.
• Requires agents to process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and generate structured, multilingual outputs.
• Proposes a hybrid evaluation framework that combines structural grading, executable verification, and LLM‑based semantic assessment to capture both functional correctness and linguistic consistency.
• Provides empirical evidence that state‑of‑the‑art LLM agents experience significant performance degradation in multilingual settings compared to monolingual baselines.

Methodology

PolyWorkBench tasks are designed to mimic real‑world workplace workflows where multilingual communication is intrinsic. Each task includes:

1. Multilingual Input: Text, tables, or documents in multiple languages.
2. Iterative Reasoning: Agents must plan, decompose, and reason across language boundaries.
3. Tool Invocation: Calls to external APIs, calculators, or search engines.
4. Structured Output: JSON or tabular results that preserve language fidelity.

The evaluation framework assesses:

• Structural Accuracy: Does the output follow the required schema?
• Executable Verification: Can the output be executed or consumed by downstream systems?
• Semantic Consistency: An LLM‑based grader checks whether the multilingual content preserves meaning.

Results & Insights

Experiments with leading LLM agents (e.g., GPT‑4‑Turbo, Claude‑3) reveal a consistent drop of 15‑30% in task success rates when multilingual inputs are introduced. Error analysis shows that language switching amplifies reasoning errors and tool‑use mismatches, highlighting the need for models that jointly handle language variation and procedural decision‑making.

Implications

PolyWorkBench underscores the importance of multilingual competence for LLM agents operating in global enterprises. It serves as a diagnostic suite for researchers and product teams aiming to build agents that can seamlessly navigate multilingual workflows.

Learn More

Read the full paper on arXiv and explore related resources on ubos.tech.

Stay tuned for future updates and extensions of the benchmark.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.