Andrii Bidochko

• Updated: June 11, 2026
• 6 min read

AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios

Direct Answer

AsyncTool introduces a benchmark that measures how well large‑language‑model (LLM) agents can call external tools asynchronously while juggling multiple tasks. It matters because real‑world AI assistants must keep working during tool latency, and current evaluations ignore this temporal dimension.

Background: Why This Problem Is Hard

LLM‑based agents have become proficient at invoking APIs, databases, or third‑party services to extend their reasoning capabilities. In practice, however, tool responses are rarely instantaneous—network delays, rate limits, or heavy computation can introduce seconds or minutes of waiting time. When an agent is locked in a synchronous call, it stalls, wasting valuable CPU cycles and user patience.

Most existing benchmarks evaluate agents in a single‑task, turn‑based setting where the next prompt arrives only after the tool finishes. This design hides two critical challenges:

• Temporal reasoning: The agent must decide whether to pause, switch to another task, or issue a follow‑up request while a prior tool is still processing.
• Coordination across tasks: In multi‑task environments, dependencies may overlap (e.g., Task A needs data from Tool X, while Task B can proceed with Tool Y). Efficient agents must interleave work to maximize throughput.

Because current evaluations do not simulate latency or concurrent workloads, they overestimate the practical usefulness of tool‑using agents. As enterprises integrate LLM assistants into ticketing systems, data pipelines, or customer‑facing bots, the ability to keep the “conversation” alive while waiting for external services becomes a competitive differentiator.

What the Researchers Propose

The authors present AsyncTool, a benchmark suite that deliberately injects realistic tool‑response delays and presents several heterogeneous tasks at once. The framework consists of three conceptual layers:

1. Task Generator: Creates a batch of independent and interdependent tasks (e.g., summarization, code generation, data retrieval) that must be solved concurrently.
2. Latency Simulator: Wraps each external tool with a configurable delay, mimicking network latency, compute queues, or human‑in‑the‑loop verification.
3. Evaluation Engine: Tracks the agent’s actions at the step, sub‑task, and full‑task levels, and computes efficiency‑oriented metrics such as idle‑time ratio and makespan.

By exposing agents to a “busy” environment, AsyncTool forces them to demonstrate three core capabilities: task switching, dependency tracking, and state maintenance across asynchronous calls.

How It Works in Practice

When an LLM agent is plugged into AsyncTool, the interaction follows a clear workflow:

1. Initialization: The benchmark delivers a JSON payload describing all pending tasks, their required tools, and any known dependencies.
2. Decision Loop: At each turn, the agent chooses one of three actions:
   • Invoke a tool for a specific sub‑task.
   • Switch context to a different pending sub‑task.
   • Perform internal reasoning (e.g., plan, summarize partial results) without external calls.
3. Asynchronous Handling: Once a tool call is issued, the latency simulator returns a “pending” token. The agent can immediately issue another call or continue reasoning, rather than waiting for the first response.
4. Result Integration: When the simulator finally delivers a tool’s output, the agent must merge it into its internal state, resolve any dependent sub‑tasks, and possibly trigger new calls.
5. Completion Check: The loop repeats until all top‑level tasks report success, at which point the benchmark records total wall‑clock time, number of context switches, and correctness of each answer.

This design differs from prior single‑task evaluations by treating tool latency as a first‑class resource that agents must schedule, much like an operating system scheduler handles I/O waits.

Evaluation & Results

To validate AsyncTool, the authors assembled a dataset of 1,200 multi‑task episodes covering domains such as finance, software engineering, and knowledge retrieval. Each episode combined 3–5 sub‑tasks with varying dependency graphs (linear, fork‑join, and cyclic). Tool latency was sampled from a log‑normal distribution ranging from 0.5 s to 30 s, reflecting real‑world API behavior.

The evaluation measured three tiers of performance:

• Step‑level accuracy: Whether each individual tool call produced the correct output.
• Sub‑task completion rate: The proportion of sub‑tasks finished within a predefined deadline.
• Task‑level efficiency: Metrics such as makespan (total elapsed time) and idle‑time ratio (percentage of time the agent spent waiting).

Key findings include:

• All tested models—GPT‑4‑Turbo, Claude‑3.5, and Llama‑2‑70B—experienced a 15‑30 % drop in task‑level success when tool latency was introduced.
• Models that explicitly reasoned about “pending” calls (e.g., by prompting “store this result for later”) outperformed baseline prompting by up to 12 % on makespan.
• Agents that maintained an external “task queue” structure (implemented via a simple Python dict) reduced idle‑time by 40 % compared to those that relied on implicit context.

These results demonstrate that delayed feedback is not a trivial nuisance; it fundamentally reshapes the performance landscape of LLM agents.

Why This Matters for AI Systems and Agents

Enterprises planning to deploy AI assistants in production environments must account for the hidden cost of tool latency. AsyncTool provides a realistic stress test that reveals whether an agent can keep the user experience fluid while external services catch up. The benchmark’s efficiency‑oriented metrics also give product managers concrete levers—such as improving tool parallelism or adding caching layers—to shrink overall response time.

From an engineering standpoint, the findings suggest three actionable design patterns:

1. Explicit state stores: Persist intermediate results outside the LLM’s context window so the model can retrieve them without re‑prompting.
2. Task‑level schedulers: Treat tool calls as jobs in a queue, allowing the agent to prioritize high‑impact tasks while low‑priority calls wait.
3. Latency‑aware prompting: Include meta‑information about expected wait times in the prompt, enabling the model to plan ahead.

These practices align closely with the capabilities of the Workflow automation studio, which lets developers orchestrate asynchronous tool pipelines without writing custom glue code. By integrating AsyncTool‑style evaluation into CI pipelines, teams can catch coordination bugs early, before they surface in production.

What Comes Next

While AsyncTool marks a significant step forward, several limitations remain:

• Scope of tools: The benchmark currently focuses on deterministic APIs; incorporating stochastic services (e.g., generative image models) would broaden its relevance.
• Scalability of latency patterns: Real‑world systems often experience bursty traffic and back‑pressure; future versions could simulate queueing dynamics.
• Agent diversity: Evaluating retrieval‑augmented generation (RAG) pipelines and hybrid symbolic‑neural agents would test the benchmark’s generality.

Future research may explore “self‑optimizing” agents that dynamically adjust their scheduling policy based on observed latency, or meta‑learning approaches that teach a model to predict tool response times and pre‑emptively allocate resources.

Practitioners interested in building next‑generation assistants can start by prototyping on the UBOS platform overview, which offers built‑in support for asynchronous tool orchestration, state persistence, and real‑time monitoring. Combining such infrastructure with insights from AsyncTool will help close the gap between laboratory‑grade LLM reasoning and production‑grade AI agents.

References

• AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
• Kou Shi et al., “AsyncTool” (2026)

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.