Andrii Bidochko

• Updated: June 17, 2026
• 7 min read

When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference

Direct Answer

The paper When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference introduces a systematic, stage‑aware benchmarking framework that isolates the true performance contribution of neural processing units (NPUs) versus CPUs during on‑device large language model (LLM) inference. It shows that, contrary to common expectations, CPUs can outpace NPUs in the compute‑heavy prefill stage, while NPUs deliver only modest gains in the memory‑bound decode stage, and that aggressive NPU offloading may even increase overall energy consumption.

Background: Why This Problem Is Hard

Deploying LLMs on smartphones and other edge devices has become a strategic priority for companies that want to offer privacy‑preserving AI services without relying on cloud connectivity. The hardware reality, however, is a heterogeneous system‑on‑chip (SoC) where a general‑purpose CPU coexists with a specialized NPU designed for tensor operations. While NPUs promise higher throughput and lower latency for deep‑learning workloads, the actual inference pipeline of an LLM is not a monolithic matrix multiply. It consists of distinct stages—most notably the prefill (prompt encoding) and decode (token generation) phases—each with different computational and memory characteristics.

Existing performance studies typically report end‑to‑end latency or FLOPs, treating the SoC as a black box. This approach masks two critical bottlenecks:

• Operator‑level mismatch: Some operators (e.g., attention softmax) map well to NPU kernels, while others (e.g., control‑flow heavy token sampling) remain CPU‑bound.
• Cross‑backend overhead: Switching between CPU and NPU incurs data movement, quantization/de‑quantization, and scheduling latency that can dominate the theoretical speedup.

Because mobile LLMs must run under strict power envelopes, overlooking these nuances can lead to sub‑optimal hardware‑software co‑design, wasted battery life, and a false sense of performance security.

What the Researchers Propose

The authors present a stage‑level, multi‑dimensional benchmarking methodology that decomposes the LLM inference pipeline into isolated components. Their framework, built around an OPMASK‑based control mask, enables researchers to:

• Force specific operators to run exclusively on the CPU, exclusively on the NPU, or on both with explicit synchronization points.
• Measure communication latency, quantization overhead, and raw compute time for each stage independently.
• Compare “what‑if” scenarios such as full‑offload, partial offload, or fallback to CPU when the NPU cannot support a particular kernel.

In essence, the framework acts as a diagnostic microscope that reveals where the NPU adds value and where it becomes a liability. The key components are:

1. Stage Scheduler: Divides the LLM execution trace into prefill and decode phases.
2. OPMASK Controller: Applies bit‑mask directives to route each operator to the desired backend.
3. Telemetry Collector: Captures timestamps, power draw, and memory bandwidth for every micro‑stage.

How It Works in Practice

Imagine a mobile app that needs to generate a response to a user query using a 7‑billion‑parameter transformer. The workflow under the proposed methodology looks like this:

1. Input Tokenization (CPU): The raw text is tokenized and embedded; this step remains on the CPU because it is lightweight and does not benefit from NPU acceleration.
2. Prefill Stage (Hybrid): The model processes the entire prompt. The OPMASK forces compute‑intensive matrix multiplications (e.g., QKV projections) onto the NPU, while keeping attention‑softmax and layer‑norm on the CPU to avoid costly data shuffles.
3. Synchronization Barrier: After the prefill, the framework measures the time spent moving intermediate activations between CPU and NPU memory.
4. Decode Stage (Iterative): For each generated token, the decoder runs a small set of operators. The OPMASK can either keep the whole loop on the CPU (fallback) or offload only the GEMM kernels to the NPU, depending on the measured memory bandwidth.
5. Power & Latency Logging: Throughout the process, the telemetry collector records per‑stage energy consumption using the SoC’s built‑in power sensors.

What makes this approach distinct is the deliberate isolation of “what‑if” configurations without rewriting the model code. Researchers can toggle the mask at runtime, enabling rapid exploration of trade‑offs across hardware generations.

Evaluation & Results

The authors evaluated three commercially available mobile SoCs equipped with ARM CPUs and proprietary NPUs. They ran two representative LLMs (a 2.7 B and a 6.7 B parameter model) across three workloads:

• Prompt‑only prefill: A 256‑token prompt with no generation.
• Short‑generation decode: Generating 20 tokens after the prefill.
• Long‑generation decode: Generating 100 tokens to stress memory bandwidth.

Key findings:

• Prefill stage: CPUs outperformed NPUs by up to 1.6×. The dominant factor was the high compute intensity of attention‑softmax and layer‑norm, which suffered from sub‑optimal NPU kernels and extra data movement.
• Decode stage (memory‑bound): NPUs delivered modest speedups of 1.05–1.2×. Because each token generation touches large activation caches, the bottleneck shifted to memory bandwidth rather than raw compute.
• Scheduling overhead: Switching back and forth between backends added 12–18 % latency, eroding most of the theoretical gains.
• Energy impact: Contrary to the expectation that NPUs are more power‑efficient, aggressive offloading increased total energy consumption by up to 51 % due to repeated memory transfers and idle CPU cycles.

These results overturn the simplistic narrative that “NPUs are always faster.” Instead, they highlight a nuanced performance landscape where the optimal backend depends on the specific stage and workload characteristics.

Why This Matters for AI Systems and Agents

For developers building on‑device AI agents—whether personal assistants, real‑time translation tools, or autonomous UI bots—the paper’s insights translate into concrete design decisions:

• Backend selection logic: Instead of a blanket “run everything on the NPU,” agents should implement a dynamic scheduler that routes prefill to the CPU and selectively offloads only the compute‑heavy kernels during decode.
• Power budgeting: Mobile AI agents often run under strict battery constraints. Understanding that NPU offloading can increase energy draw helps engineers set realistic usage caps and informs UI cues (e.g., “low‑power mode”).
• Model architecture tuning: Designers might favor lightweight attention mechanisms or fused operators that map efficiently to the NPU, thereby reducing the CPU‑bound portion of the prefill.
• Orchestration frameworks: Platforms that automate workflow composition—such as Workflow automation studio—can embed the stage‑level profiling data to auto‑tune pipelines for each device generation.
• Edge‑first product strategy: Companies can market “on‑device AI” with confidence, citing empirical evidence that the right mix of CPU and NPU yields the best latency‑energy trade‑off.

What Comes Next

While the study provides a robust measurement methodology, several open challenges remain:

• Broader hardware coverage: Extending the benchmark to emerging AI accelerators (e.g., Qualcomm Hexagon DSPs, Apple Neural Engine) will validate whether the observed patterns hold across architectures.
• Operator‑level kernel optimization: The performance gap in the prefill stage suggests a need for NPU vendors to deliver optimized kernels for softmax, layer‑norm, and other control‑flow heavy operators.
• Adaptive runtime policies: Future SDKs could expose APIs that let applications query real‑time memory bandwidth and automatically switch backends without developer intervention.
• Energy‑aware scheduling: Integrating power‑sensor feedback into the scheduler could enable “energy‑first” execution modes, a feature that would be valuable for battery‑critical devices.
• Cross‑layer co‑design: Model compression techniques (e.g., quantization‑aware training) combined with hardware‑aware architecture search could produce LLM variants that are intrinsically NPU‑friendly.

Addressing these directions will help hardware manufacturers and AI platform providers converge on a unified stack that truly leverages heterogeneous compute. For teams looking to prototype such solutions today, the UBOS platform overview offers a modular environment where CPU‑NPU scheduling policies can be experimented with alongside pre‑built LLM integrations.

Stay updated with the latest research and practical guides on on‑device AI by visiting our UBOS homepage. Join the conversation, share your own benchmarking results, and help shape the next generation of mobile AI agents.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.