AI at the Edge: NPUs, Small Models and the Shift to On-Device Inference
Not every AI request needs a data centre. Phones and laptops now carry dedicated AI engines, NPUs, rated at tens of trillions of operations per second, and small models of a few billion parameters can run on them with no network, no per-token bill and no data leaving the device. This guide explains what an NPU is, why memory rather than TOPS decides what runs, which small models matter, how devices and the cloud split the work, and why the AI boom is making AI devices more expensive.
๐ฑ Why run AI on the device?
Every part of this series so far has followed AI into data centres. But the cheapest, fastest token is one that never leaves your phone. Running a model locally, at the edge, changes four things at once:
โ๏ธ Cloud vs device, for the same request
Toggle to compare. Qualitative.
The trade-off is capability: a phone has a few watts and a few gigabytes of memory to spare, while a data-centre GPU has a kilowatt and hundreds of gigabytes of HBM. So devices run small models, and hand harder jobs to the cloud.
๐ง What is an NPU?
A neural processing unit is a block on a phone or laptop chip designed only for the matrix maths of neural networks, using low-precision numbers (8-bit or 4-bit integers) to do it with very little power. It sits beside the CPU (general-purpose work) and GPU (graphics and parallel work), sharing the same memory. Vendors rate NPUs in TOPS, trillions of operations per second. Microsoft requires at least 40 TOPS for a laptop to carry its Copilot+ PC label.
๐ Inside a modern laptop or phone chip
Click a block.
๐ The 2026 chips
๐ NPU ratings (vendor peak TOPS)
Higher is not always faster in practice; see the next section.
Qualcomm (X2 Elite 80 TOPS, up to 85 on top models); AMD Ryzen AI 400 ~60 TOPS (secondary reports); Intel Lunar Lake 48 TOPS (2024); Microsoft's Copilot+ threshold 40 TOPS. Apple does not publish a TOPS figure for M5; third-party estimates put its Neural Engine around 38โ40 TOPS, and M5 adds neural accelerators to every GPU core.
Apple's approach shows why TOPS is a poor yardstick. Rather than chasing a headline NPU number, the M5 adds matrix accelerators inside each GPU core, so much AI work now runs on the GPU rather than the Neural Engine. Benchmarks that run real models tell a more useful story than peak TOPS, and the biggest factor in those benchmarks is often memory.
๐พ Why memory, not TOPS, is the limit
As The Great Flip explained, writing each new token requires reading every model weight from memory, so generation speed is set mostly by memory bandwidth, not compute. And the model must fit: a 3-billion-parameter model compressed to 4 bits needs about 1.5โ2 GB, plus room for the conversation's KV cache, on a phone that is also running everything else. That is why on-device AI features have pushed flagship phones to 12 GB of RAM or more, and why Apple reserves its larger on-device model for devices with more memory.
๐งฎ Will it run on my device?
๐๏ธ Model fit and speed estimator
Pick a device class and a model. Rules of thumb, not benchmarks.
Memory need โ parameters ร bits รท 8, plus ~20% for KV cache and runtime. Usable memory assumes the OS and apps keep roughly half of a phone's RAM and a third of a laptop's. Speed โ 70% of bandwidth รท model bytes. Device figures are typical, illustrative values.
๐ฃ The small models
๐ On-device model families (2026)
| Model | Maker | Size | Where it runs | Notes |
|---|---|---|---|---|
| Apple Foundation Models (on-device) | Apple | ~3B parameters (since iOS 18, 2024) | iPhone, iPad, Mac | Developers can call it through Apple's Foundation Models framework; reports describe a larger sparse variant for newer devices with more memory (unconfirmed) |
| Gemini Nano | Small (multiple versions) | Android (via ML Kit GenAI) and Chrome (Prompt API) | Powers on-device summaries, smart replies and web features | |
| Gemma 4 | Google (open weights) | Several sizes; E2B ~2.9 GB at 4-bit | Phones, laptops, edge devices | Released April 2026 under Apache 2.0 |
| Phi-4-mini | Microsoft (open weights) | 3.8B | Laptops, phones | MIT licence; Windows also ships its own on-device models for Copilot+ features |
| Llama, Qwen and others | Meta, Alibaba, many | 1โ8B variants | Anywhere via llama.cpp, MLX, ONNX | The open ecosystem behind most local-AI apps |
Wikipedia (Apple Intelligence, Gemma), Tech Insider, AIMagicX. Sizes and features change quickly.
Small models get good through distillation (training a small model to imitate a large one), careful data curation, and quantization to 4 bits or fewer. They cannot match frontier models on hard reasoning, but they handle summarising, rewriting, classification, simple Q&A and on-screen actions well, which covers a large share of everyday requests.
๐ Hybrid AI: device first, cloud when needed
No device can run everything, so platforms route each request. Apple's design is the clearest example: a small model on the device, larger models on Private Cloud Compute (Apple-run servers announced in June 2024, designed so that personal data is not stored or accessible to Apple), and, for some requests, third-party models. In January 2026 Apple and Google announced a multi-year deal under which Apple's next-generation foundation models are based on Google's Gemini technology; Apple says they run on device and on Private Cloud Compute, though press accounts of exactly which servers run the largest model conflict.
๐งญ Where should this request run?
Pick a task. Illustrative routing logic, not any vendor's actual rules.
๐ The memory crunch: AI makes AI devices dearer
There is an irony at the edge. On-device AI needs more memory, but the same three companies that make phone and laptop DRAM, Samsung, SK hynix and Micron, have shifted capacity toward HBM for data-centre AI (see The Memory Wall). Gartner forecasts memory prices rising about 130% by the end of 2026, pushing average PC prices up about 17% and smartphone prices up about 13% compared with 2025. Memory is 15โ20% of a mid-range phone's bill of materials, so cheaper devices are hit hardest.
๐ 2026 forecasts
Gartner (prices; PCs โ10.4%); IDC (smartphones โ12.9% to 1.12B units, revised from โ16.7%; PCs โ11.3%). IDC Q3 2026 PC shipments โ20.1% to 62.7M units, via secondary report. Forecasts are revised often.
Meanwhile almost every new PC now counts as an "AI PC", but the share depends on the definition. Gartner projects AI PCs at about 55% of the market in 2026 (143 million units), counting any PC with an NPU. A count limited to Copilot+-class machines with 40+ TOPS gives about 44% (attributed to Canalys), and Morgan Stanley's stricter definition about 28%, both via secondary reports.
๐งญ What edge AI means for the rest of the stack
- Less cloud inference for simple tasks. Everyday summarising and rewriting can move to devices, easing some data-centre demand, though agents and reasoning keep the heaviest work in the cloud.
- Cost shifts to consumers. On-device inference is paid for once, in the device price (and now in pricier memory), rather than per token.
- Privacy and regulation. Keeping data local simplifies compliance, which is why health, finance and enterprise apps push on-device processing.
- Competition for memory. Devices and data centres now bid for the same DRAM supply chain.
- Chip design. Phone and laptop chips are adding bandwidth, bigger NPUs and GPU matrix units; the same memory-bandwidth arithmetic rules from phones to racks.
๐ฐ๏ธ Timeline
๐ญ What to watch
- Memory prices in 2027, and whether device makers cut RAM, raise prices or slow AI features.
- Apple's Gemini-based models and how much Siri work runs on device vs Private Cloud Compute.
- Small-model quality: whether 3โ8B models close more of the gap with frontier models for everyday tasks.
- Developer APIs (Apple's Foundation Models framework, Android ML Kit GenAI, Chrome's Prompt API, Windows AI APIs) and the apps built on them.
- Agents on devices: on-screen understanding and app control, a natural fit for local models.
๐ข Reading the numbers
- TOPS are vendor peak figures at low precision and are not comparable across vendors; Apple publishes none for M5.
- AMD's ~60 TOPS comes from secondary reports.
- Apple model details beyond the ~3B on-device model, and where the largest Gemini-based model runs, come from conflicting press summaries.
- AI PC share ranges from ~28% to ~55% depending on the definition; the Canalys and Morgan Stanley figures are via secondary reports.
- Memory and shipment forecasts (Gartner, IDC) are revised frequently; the Q3 PC figure is from an aggregator.
- The calculator uses rules of thumb and typical device values.
๐ References
- Chips and NPUs
- Tech Insider: Snapdragon X2 Elite review
- LocalAIMaster: Best NPU for AI 2026 โ Intel vs Qualcomm vs AMD vs Apple
- Technerdo: Snapdragon X2 Elite vs Apple M5
- SolidAITech: NPU guide โ TOPS, Copilot+ PCs and memory bandwidth
- Jacar: Next-generation NPUs in 2026
- Models and hybrid AI
- Wikipedia: Apple Intelligence
- Wikipedia: Gemma
- Tech Insider: Gemma 4 vs Phi-4-mini vs Llama 3.2
- AIMagicX: On-device AI in 2026
- Mac Observer: Apple explains how Gemini fits into Siri with Private Cloud Compute
- Introl: The AppleโGoogle Gemini partnership
- Memory crunch and shipments
- IDC: Smartphone shipments set for record drop as the memory crisis hits
- Silicon Republic: Chip crunch to hit smartphone and PC shipments
- Technology.org: Why your next laptop costs more
- TechTimes: AI chip imports eclipse PC and phone shipments
- AI Weekly: IDC Q3 PC shipments plunge 20.1%
- Gartner: AI PCs will represent 31% of the market by end of 2025
- AI PC penetration passes 50% in 2026 (definitions compared)
- Related on this site
- The Great Flip ยท The Memory Wall ยท The Token Price Collapse ยท AI Compute Stack at a Glance