Andrii Bidochko

• Updated: July 17, 2026
• 9 min read

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session‑centric Scheduling

Direct Answer

SMetric introduces a session‑centric scheduling framework that balances load across LLM inference clusters while preserving the high key‑value (KV) cache reuse that agents rely on for fast token generation. By routing only the first request of each agent session for load‑balancing and handling subsequent requests with cache‑aware placement, SMetric lifts overall throughput by up to 16 % without sacrificing per‑token latency.

This matters because modern AI agents issue batches of requests that depend on complete responses, shifting the performance goal from per‑token latency to tokens‑per‑second (TPS) and exposing inefficiencies in existing schedulers that over‑concentrate cache‑heavy traffic on a few nodes.

{{IMAGE_PLACEHOLDER}}

Background: Why This Problem Is Hard

Large Language Model (LLM) serving has traditionally been optimized for human‑driven chat or completion workloads. Those workloads exhibit two key characteristics:

• Per‑token latency matters: Users expect each token to appear quickly, so schedulers prioritize minimizing round‑trip time for every token.
• KV cache reuse is modest: Individual conversations share only a fraction of their attention keys and values, typically 50‑60 % of tokens.

Agentic serving flips this paradigm. Autonomous agents—whether orchestrating web searches, generating code, or managing multi‑step business processes—operate on a “complete‑response” model. An agent does not act on partial output; it waits for the full generation before deciding the next step. Consequently:

• The cluster’s tokens‑per‑second (TPS) becomes the primary performance metric, while per‑token latency is a secondary concern.
• Requests from the same agent session share a large portion of their KV cache. In a production trace from the BAILIAN system, more than 80 % of tokens were reusable across requests, compared with 54‑62 % in typical chat workloads.

Existing schedulers—such as round‑robin, least‑loaded, or cache‑aware heuristics—tend to over‑prioritize routing every request to the node that already holds the relevant KV cache. This creates “hot spots” where a few instances become saturated, while the rest of the cluster sits idle. The result is a hard ceiling on TPS, even though the global KV store could supply cache entries to any node.

In short, the challenge is to balance load without destroying the cache locality that agents depend on. Achieving that balance requires a new scheduling metric that captures session‑level information rather than per‑request cache state.

What the Researchers Propose

The authors present SMetric, a lightweight, stateless scheduler that treats each agent session as a first‑class entity. Its core insights are:

1. Global KV store mitigates the need for perfect local reuse. By keeping a fast, disaggregated KV tier, the system can fetch missing cache entries on demand, allowing some degree of load spreading.
2. Intra‑session locality is concentrated in the first request. The first token generation of a session incurs the highest cache miss cost; later requests benefit from the cache built during the initial “prefill”. Balancing only these first requests yields a well‑distributed workload while preserving most of the reuse for subsequent “decode” steps.

SMetric operationalizes these ideas with a two‑phase routing rule:

• Phase 1 – Load‑balanced placement of session‑initial requests: The scheduler selects the least‑loaded instance, ignoring KV cache state, to handle the first request of each session.
• Phase 2 – Cache‑aware placement of follow‑up requests: Subsequent requests are routed to the instance that already holds the relevant KV cache, maximizing reuse and minimizing cross‑node fetches.

Because the “session turn” information (i.e., whether a request is the first in its session) can be derived directly from the user input metadata, SMetric remains stateless—no per‑session tables or complex bookkeeping are required.

How It Works in Practice

Component Overview

SMetric sits between the front‑end request dispatcher and the pool of LLM inference workers. The architecture comprises four logical components:

1. Request Ingress Layer: Receives HTTP/gRPC calls from agents, extracts session identifiers, and tags each request as “first‑in‑session” or “subsequent”.
2. Load‑Balancer Engine: Maintains a lightweight load metric (e.g., active token count) for each worker node. For first‑in‑session requests, it selects the node with the lowest load.
3. Cache‑Awareness Module: Queries a local KV‑cache registry on each worker to determine whether the required cache entries already exist. For subsequent requests, it forwards the request to the node with a cache hit.
4. Global KV Store (optional): A disaggregated, high‑throughput key‑value service that can supply missing cache entries when no local hit is found, ensuring that load‑balancing does not starve any node of data.

Workflow Illustration

The end‑to‑end flow for a typical multi‑turn agent session looks like this:

1. An agent sends its first prompt (e.g., “Generate a market analysis report”). The Ingress Layer tags it as session‑start.
2. The Load‑Balancer Engine picks the least‑busy worker (Node A) and forwards the request.
3. Node A performs a “prefill” pass, populating its local KV cache with attention keys/values for the prompt.
4. The agent receives the full response, decides on the next action, and issues a follow‑up request (e.g., “Summarize the key findings”). This request is tagged as subsequent.
5. The Cache‑Awareness Module detects that Node A already holds the relevant KV entries and routes the request back to Node A, enabling a fast “decode” step.
6. If the follow‑up request involves a new context that exceeds Node A’s cache, the Global KV Store supplies the missing entries, after which the request proceeds on the chosen node.

Because only the first request of each session participates in load‑balancing, the system naturally spreads the heavy prefill workload across the cluster while keeping the high‑reuse decode phase localized.

Key Differentiators

• Statelessness: No per‑session state is stored in the scheduler; decisions are made on‑the‑fly using only the session‑turn flag.
• Minimal coordination overhead: The only shared data structures are lightweight load counters and optional KV‑hit flags, avoiding costly consensus protocols.
• Scalable KV reuse: By leveraging a global KV tier, SMetric can tolerate occasional cache misses without collapsing the load‑balance.

Evaluation & Results

The authors validated SMetric on two production‑grade traces:

• BAILIAN trace: Real‑world agent traffic with >80 % KV reuse.
• Chat‑style trace: Conventional conversational workload with 54‑62 % reuse.

Experiments compared SMetric against three state‑of‑the‑art schedulers:

1. Pure load‑balancing (round‑robin).
2. Cache‑first routing (always pick the node with the most KV hits).
3. Hybrid heuristic (balance load while trying to keep cache locality).

Key Findings

• Throughput boost: Under a mixed prefill‑decode workload with a global KV store, SMetric raised cluster TPS by 10‑16 % compared with the best baseline.
• Prefill efficiency: When the KV store was disaggregated (i.e., no global cache), SMetric still delivered a 2‑34 % increase in prefill TPS, demonstrating robustness to infrastructure variations.
• Latency impact: Per‑token latency improved modestly (≈5 % reduction) because decode steps stayed on cache‑warm nodes.
• Load distribution: Heat‑maps of node utilization showed a far more even spread of active tokens, eliminating the “hot‑spot” phenomenon observed in cache‑first schedulers.

These results confirm the authors’ hypothesis: balancing only the first request of each session is sufficient to achieve near‑optimal load distribution while preserving the majority of KV reuse for the rest of the session.

For readers who want to dive deeper, the full experimental setup and raw numbers are available in the SMetric paper on arXiv.

Why This Matters for AI Systems and Agents

Enterprises that deploy autonomous AI agents—whether for customer support, data analysis, or workflow automation—face two intertwined constraints: cost‑effective scaling and predictable response times. SMetric directly addresses both.

• Cost efficiency: By smoothing the load across all inference nodes, cloud providers can achieve higher utilization, reducing the need to over‑provision hardware for peak spikes.
• Predictable agent behavior: Agents that rely on complete responses can now expect consistent TPS, which simplifies orchestration logic and improves downstream decision‑making.
• Simplified infrastructure: The stateless nature of SMetric means it can be dropped into existing serving stacks without major refactoring, making it attractive for platforms like the UBOS homepage that already manage multi‑model pipelines.
• Future‑proofing: As LLMs grow larger and KV caches become more valuable, a scheduler that can intelligently balance cache reuse versus load will become a core component of any production‑grade AI platform.

In practice, developers building AI marketing agents, code‑generation bots, or real‑time analytics pipelines can leverage SMetric to achieve higher throughput without sacrificing the low‑latency experience that end users expect.

What Comes Next

While SMetric marks a significant step forward, several open challenges remain:

• Dynamic session lengths: Agents with highly variable numbers of turns may benefit from adaptive policies that balance more than just the first request.
• Multi‑model environments: In clusters serving heterogeneous models (e.g., LLMs, diffusion models), extending the session‑centric metric to account for model‑specific cache characteristics is an open research direction.
• Fault tolerance: If a node holding a session’s cache fails mid‑session, graceful fallback to the global KV store or rapid cache migration strategies need to be explored.
• Integration with orchestration frameworks: Embedding SMetric logic into workflow automation studios or agent orchestration layers could enable end‑to‑end optimization from request generation to execution.

Future work may also investigate hybrid metrics that combine session‑turn information with real‑time cache hit rates, or apply reinforcement learning to continuously adapt scheduling policies based on observed TPS trends.

For organizations eager to experiment, the next logical step is to prototype SMetric within an existing AI serving stack—such as the UBOS platform overview—and measure the impact on their specific agent workloads.

For more details, visit our internal page at ubos.tech.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.