Andrii Bidochko

• Updated: August 15, 2026
• 5 min read

LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

Direct Answer

LinearKV introduces a position‑independent caching (PIC) technique that replaces the multiple cached states required by hybrid large language models (LLMs) with a single, linearly‑updated key‑value store. This simplification cuts inference latency dramatically while preserving generation quality, making real‑time LLM serving more practical for production workloads.

Background: Why This Problem Is Hard

Serving LLMs at scale hinges on reusing previously computed attention results. Traditional full‑attention models cache the key‑value (KV) pairs for every token, enabling the next token to attend to the entire context without recomputation. However, this approach becomes a bottleneck for hybrid architectures that blend dense attention with linear recurrences, because each layer may maintain its own distinct cache. The resulting memory footprint grows linearly with model depth, and synchronizing multiple caches introduces latency spikes that undermine real‑time responsiveness.

Existing PIC solutions, such as HYPIC, attempt to decouple positional information from the cache but still require a separate KV state per layer. This multiplicity limits their applicability to newer hybrid LLMs that interleave dense and linear components, where the cache must accommodate both types of recurrence. Consequently, developers face a trade‑off between speed and model fidelity, often resorting to costly hardware or sacrificing model size.

What the Researchers Propose

LinearKV proposes a unified caching framework that treats the entire stack of KV states as a single linear recurrence. By initializing the cache with a decoupled representation of positional embeddings, the method allows each subsequent token to update the cache with a simple linear transformation. In effect, one cached state now carries the information that previously required a separate KV tensor per layer.

The key components of the framework are:

• Decoupled Initialization: Positional encodings are factored out of the KV tensors before caching, enabling a position‑agnostic start.
• Linear Recurrence Update: Each new token applies a linear operator to the cached state, preserving the cumulative attention context.
• Unified Cache Interface: Downstream layers read from and write to the same cache, eliminating redundancy.

How It Works in Practice

The LinearKV workflow can be broken down into three stages:

1. Cache Warm‑up: When generation begins, the model computes an initial KV pair using decoupled positional embeddings and stores it as the sole cache entry.
2. Token Generation Loop: For each new token, the model:
   • Retrieves the current cache.
   • Applies the linear recurrence to incorporate the new token’s representation.
   • Feeds the updated cache back into the attention mechanism of all layers.
3. Cache Propagation: The updated cache is passed unchanged to the next iteration, ensuring that every layer sees a consistent context without allocating additional memory.

This design diverges from prior PIC methods by eliminating the need for per‑layer KV duplication. The linear update is computationally cheap (a matrix‑vector multiply) and can be fused with existing transformer kernels, making it compatible with off‑the‑shelf inference engines.

Evaluation & Results

The authors benchmarked LinearKV against HYPIC across three model families (dense‑only, hybrid, and retrieval‑augmented) and two selector strategies (greedy and nucleus sampling). The evaluation focused on two axes:

• Quality Recovery: Measured by BLEU and ROUGE scores relative to a full‑attention baseline.
• Latency Reduction: Reported as average per‑token inference time on a single A100 GPU.

Key findings include:

• LinearKV recovered >98% of the baseline quality across all models, matching or exceeding HYPIC’s scores.
• Latency dropped by 30‑45% on hybrid models, with the most pronounced gains (up to 52%) on the deepest architectures.
• Memory consumption fell by roughly 40%, enabling larger batch sizes on the same hardware.

These results demonstrate that a single linear cache can deliver near‑identical generation fidelity while delivering tangible speed and memory benefits.

Why This Matters for AI Systems and Agents

For practitioners building AI agents, chatbots, or real‑time recommendation pipelines, inference latency directly translates to user experience and cost. LinearKV’s reduction in per‑token time means:

• Faster response loops for conversational agents, improving engagement metrics.
• Lower GPU utilization, which reduces operational expenses for cloud‑based deployments.
• Simplified integration pipelines, because developers no longer need to manage a hierarchy of caches.

Teams can adopt LinearKV without rewriting model code; the framework plugs into existing transformer libraries via a thin wrapper. This aligns well with the UBOS platform overview, where modular components are orchestrated through a unified runtime. Moreover, the reduced memory footprint opens the door for deploying hybrid LLMs on edge devices, expanding the reach of AI‑driven products.

What Comes Next

While LinearKV delivers impressive speedups, several open challenges remain:

• Generalization to Multi‑Modal Models: Extending the linear cache to vision‑language or audio‑text hybrids may require additional decoupling strategies.
• Dynamic Length Sequences: Very long contexts (>8k tokens) could still strain the linear recurrence’s numerical stability.
• Hardware‑Specific Optimizations: Tailoring the linear update to emerging accelerators (e.g., TPUs, custom ASICs) could unlock further latency reductions.

Future research may explore hybrid caching schemes that combine LinearKV with selective full‑attention windows, offering a tunable trade‑off between precision and speed. Practitioners interested in experimenting can start by integrating LinearKV into the Workflow automation studio, which provides a low‑code environment for testing new inference pipelines.

References

For a complete technical description, see the original LinearKV paper.

Image Placeholder

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.