Arm Unveils NPU Architecture for On-Device LLM

Arm Holdings has unveiled a next-generation neural processing unit (NPU) architecture purpose-built for running large language models directly on smartphones, laptops, and IoT devices. The new architecture, internally dubbed Ethos-U Next, represents Arm's most aggressive push yet into on-device AI inference, targeting models with up to 13 billion parameters without requiring cloud connectivity.

The announcement positions Arm squarely against competitors like Qualcomm, Apple, and Intel, all of which have been racing to embed more AI processing capability into edge silicon. Unlike previous Arm NPU designs that focused primarily on computer vision and lightweight ML tasks, Ethos-U Next is engineered from the ground up to handle the memory-intensive, transformer-based workloads that power modern LLMs.

Key Takeaways at a Glance

• Performance target: Up to 100 TOPS (tera operations per second) in a mobile power envelope under 5 watts
• Memory efficiency: New compression engine reduces LLM memory footprint by up to 60% compared to standard INT8 quantization
• Model support: Optimized for transformer architectures up to 13B parameters, including Llama 3, Gemma 2, and Phi-3
• Power efficiency: 4x improvement in performance-per-watt over Arm's current Ethos-U85 NPU
• Availability: IP licensing begins Q1 2026, with first commercial silicon expected late 2026
• Ecosystem: Full integration with Arm's existing ACLE toolchain and KleidiAI software libraries

Ethos-U Next Targets the On-Device AI Gap

The core challenge for on-device LLM inference has always been the same: memory bandwidth. Running a 7B parameter model in FP16 precision requires roughly 14 GB of memory, far exceeding what most mobile SoCs can efficiently handle. Arm's new architecture attacks this bottleneck with a proprietary Streaming Memory Compression (SMC) engine that dynamically compresses and decompresses model weights during inference.

According to Arm, SMC achieves near-lossless compression ratios that reduce effective memory requirements by 50-60%. This means a 7B parameter model could run with as little as 6 GB of effective memory bandwidth, bringing it within reach of mainstream smartphones shipping in 2027.

The NPU also introduces a sparse attention accelerator, a dedicated hardware block optimized for the self-attention mechanisms central to transformer models. Traditional NPUs treat attention computation as general matrix multiplication, but Arm's approach exploits the inherent sparsity patterns in attention heads to skip unnecessary calculations. The result is a claimed 2.5x speedup on attention layers compared to dense computation.

Architecture Deep Dive: What Makes It Different

Ethos-U Next departs from conventional NPU design in several fundamental ways. At its core, the architecture uses a tile-based dataflow engine that breaks transformer layers into smaller computational tiles, each processed independently before being reassembled.

This tiling approach serves 2 critical purposes:

• It maximizes data reuse within the NPU's on-chip SRAM, reducing costly off-chip memory accesses
• It enables flexible scaling — licensees can configure anywhere from 4 to 64 compute tiles depending on their target use case
• It allows dynamic power management, shutting down unused tiles during lighter inference workloads
• It supports heterogeneous precision, mixing INT4, INT8, and FP16 operations within a single inference pass

The heterogeneous precision capability is particularly noteworthy. Research from teams at Meta, Microsoft, and academic institutions has shown that not all layers in a transformer model require the same numerical precision. Ethos-U Next can apply INT4 quantization to less sensitive feed-forward layers while preserving FP16 precision for critical attention computations, all managed automatically by the compiler.

Arm is also introducing a KV-cache management unit, a hardware block specifically designed to handle the key-value caches that grow linearly during autoregressive text generation. Managing KV-cache efficiently is one of the biggest challenges in on-device LLM deployment, and dedicated hardware support could significantly reduce latency during long-context conversations.

How Arm Stacks Up Against Qualcomm and Apple

The competitive landscape for on-device AI silicon has intensified dramatically over the past 18 months. Qualcomm's Hexagon NPU in the Snapdragon 8 Elite already delivers 75 TOPS and can run 7B parameter models on-device. Apple's Neural Engine in the M4 chip family powers Apple Intelligence features across iPhone, iPad, and Mac.

Arm's position is unique because it does not manufacture chips directly. Instead, it licenses IP to the vast majority of mobile and embedded chip designers worldwide, including Qualcomm, Samsung, and MediaTek. This means Ethos-U Next's impact could be far broader than any single chip vendor's solution.

Key competitive comparisons:

• Qualcomm Hexagon (Snapdragon 8 Elite): 75 TOPS, optimized for Qualcomm's AI Hub ecosystem
• Apple Neural Engine (M4): ~38 TOPS, tightly integrated with Core ML framework
• Intel NPU (Lunar Lake): ~48 TOPS, focused on Windows AI PC workloads
• Arm Ethos-U Next: Up to 100 TOPS target, licensable IP for any chip designer
• Google Tensor G5 (rumored): Custom TPU-derived NPU for Pixel devices

The critical advantage Arm brings is ecosystem breadth. While Qualcomm and Apple each serve their own platforms, Arm's IP appears in over 99% of smartphones globally. If Ethos-U Next delivers on its promises, it could democratize on-device LLM capability across price tiers and device categories that proprietary solutions cannot reach.

Software Ecosystem Gets a Major Upgrade

Hardware is only half the equation. Arm recognizes that developer adoption hinges on seamless software support. Alongside Ethos-U Next, the company announced significant expansions to its KleidiAI software library, which provides optimized kernels for popular AI frameworks.

KleidiAI now includes dedicated routines for transformer inference, covering multi-head attention, rotary positional embeddings (RoPE), grouped-query attention (GQA), and FlashAttention-style memory-efficient attention. These kernels integrate directly with ONNX Runtime, TensorFlow Lite, PyTorch ExecuTorch, and the MediaPipe LLM inference pipeline.

Arm is also releasing an updated version of its Arm NN SDK with a new LLM-specific profiling tool. This tool lets developers visualize memory bottlenecks, quantization accuracy trade-offs, and per-layer latency breakdowns before deploying models to target hardware.

Perhaps most significantly, Arm announced a partnership with Hugging Face to create a curated collection of Ethos-U Next-optimized model checkpoints. The initial collection will include quantized variants of Llama 3 (7B and 13B), Microsoft Phi-3 Mini, Google Gemma 2 (2B and 7B), and Mistral 7B — all validated for accuracy and performance on the new NPU.

What This Means for Developers and Businesses

For app developers, Ethos-U Next promises to make on-device AI features viable at a much larger scale. Today, most AI-powered mobile apps rely on cloud APIs, incurring latency, cost, and privacy trade-offs. On-device LLM inference eliminates these concerns, enabling features like real-time translation, intelligent assistants, and document summarization without any data leaving the device.

For device manufacturers, particularly in the Android ecosystem, Arm's licensable approach means they can differentiate on AI performance without designing custom NPU IP from scratch. Companies like Samsung, MediaTek, NVIDIA (in its automotive and robotics platforms), and emerging Chinese chipmakers like Unisoc could all benefit.

The enterprise market also stands to gain. Edge AI deployments in healthcare, manufacturing, and retail increasingly demand LLM-class reasoning capabilities in environments where cloud connectivity is unreliable or prohibited by regulation. A standardized NPU architecture from Arm could accelerate adoption across these verticals.

Privacy-conscious markets in the European Union, where GDPR compliance makes cloud-based AI processing more complex, could see particular benefits from robust on-device inference capabilities.

Looking Ahead: The Road to On-Device AI Everywhere

Arm's timeline suggests first commercial devices powered by Ethos-U Next silicon will arrive in late 2026 or early 2027, roughly aligning with the expected lifecycle of next-generation flagship smartphones and AI PCs. The company hinted that at least 3 major chip partners are already in advanced licensing discussions.

The broader trajectory is clear: the AI industry is moving decisively toward a hybrid inference model where cloud and edge processing share the workload. Lightweight queries, personal data processing, and latency-sensitive tasks will run on-device, while heavy-duty training and massive model inference remain in the cloud.

Arm's bet is that this shift will require purpose-built silicon at the edge — not just GPUs repurposed for inference, but dedicated NPU architectures designed specifically for transformer workloads. If Ethos-U Next delivers on its ambitious performance and efficiency targets, it could establish a new baseline for what 'AI-ready' means in consumer and enterprise devices.

The real test will come when silicon hits the market. Until then, Arm has laid down a compelling architectural vision that addresses the 3 biggest barriers to on-device LLMs: memory bandwidth, power efficiency, and developer accessibility. In a market moving at breakneck speed, that combination could prove decisive.