Home โ€บ Blog โ€บ AI at the Edge
AI Compute Stack ยท Part 12 ยท Devices

AI at the Edge: NPUs, Small Models and the Shift to On-Device Inference

Not every AI request needs a data centre. Phones and laptops now carry dedicated AI engines, NPUs, rated at tens of trillions of operations per second, and small models of a few billion parameters can run on them with no network, no per-token bill and no data leaving the device. This guide explains what an NPU is, why memory rather than TOPS decides what runs, which small models matter, how devices and the cloud split the work, and why the AI boom is making AI devices more expensive.

FL
FrontierAGI Team
โš ๏ธ How this was researched This draws on web searches run up to 23 October 2026: Gartner and IDC releases (some via secondary reports), chip reviews and comparisons, Apple, Google and Microsoft materials via trade press, and Wikipedia for model release dates. TOPS figures are vendor-quoted peaks and not comparable across vendors; Apple publishes no TOPS figure for M5. Details of Apple's newest models and its Gemini-based infrastructure come from press summaries that conflict. "AI PC" share depends on the definition used. The "will it run" calculator uses simplified rules of thumb. See Reading the numbers.

๐Ÿ“ฑ Why run AI on the device?

Every part of this series so far has followed AI into data centres. But the cheapest, fastest token is one that never leaves your phone. Running a model locally, at the edge, changes four things at once:

โš–๏ธ Cloud vs device, for the same request

Toggle to compare. Qualitative.

The trade-off is capability: a phone has a few watts and a few gigabytes of memory to spare, while a data-centre GPU has a kilowatt and hundreds of gigabytes of HBM. So devices run small models, and hand harder jobs to the cloud.

๐Ÿง  What is an NPU?

A neural processing unit is a block on a phone or laptop chip designed only for the matrix maths of neural networks, using low-precision numbers (8-bit or 4-bit integers) to do it with very little power. It sits beside the CPU (general-purpose work) and GPU (graphics and parallel work), sharing the same memory. Vendors rate NPUs in TOPS, trillions of operations per second. Microsoft requires at least 40 TOPS for a laptop to carry its Copilot+ PC label.

๐Ÿ” Inside a modern laptop or phone chip

Click a block.

๐Ÿ The 2026 chips

๐Ÿ“Š NPU ratings (vendor peak TOPS)

Higher is not always faster in practice; see the next section.

Qualcomm (X2 Elite 80 TOPS, up to 85 on top models); AMD Ryzen AI 400 ~60 TOPS (secondary reports); Intel Lunar Lake 48 TOPS (2024); Microsoft's Copilot+ threshold 40 TOPS. Apple does not publish a TOPS figure for M5; third-party estimates put its Neural Engine around 38โ€“40 TOPS, and M5 adds neural accelerators to every GPU core.

Apple's approach shows why TOPS is a poor yardstick. Rather than chasing a headline NPU number, the M5 adds matrix accelerators inside each GPU core, so much AI work now runs on the GPU rather than the Neural Engine. Benchmarks that run real models tell a more useful story than peak TOPS, and the biggest factor in those benchmarks is often memory.

๐Ÿ’พ Why memory, not TOPS, is the limit

As The Great Flip explained, writing each new token requires reading every model weight from memory, so generation speed is set mostly by memory bandwidth, not compute. And the model must fit: a 3-billion-parameter model compressed to 4 bits needs about 1.5โ€“2 GB, plus room for the conversation's KV cache, on a phone that is also running everything else. That is why on-device AI features have pushed flagship phones to 12 GB of RAM or more, and why Apple reserves its larger on-device model for devices with more memory.

๐Ÿ’ก Rule of thumb Tokens per second โ‰ˆ memory bandwidth รท model size in bytes. A phone with ~60 GB/s of bandwidth running a 2 GB model tops out around 30 tokens a second; the same model on a laptop with ~250 GB/s can go several times faster. Quantization (fewer bits per weight) helps twice: the model fits, and it reads faster.

๐Ÿงฎ Will it run on my device?

๐ŸŽ›๏ธ Model fit and speed estimator

Pick a device class and a model. Rules of thumb, not benchmarks.

Device
Precision

Memory need โ‰ˆ parameters ร— bits รท 8, plus ~20% for KV cache and runtime. Usable memory assumes the OS and apps keep roughly half of a phone's RAM and a third of a laptop's. Speed โ‰ˆ 70% of bandwidth รท model bytes. Device figures are typical, illustrative values.

๐Ÿฃ The small models

๐Ÿ“‹ On-device model families (2026)

ModelMakerSizeWhere it runsNotes
Apple Foundation Models (on-device)Apple~3B parameters (since iOS 18, 2024)iPhone, iPad, MacDevelopers can call it through Apple's Foundation Models framework; reports describe a larger sparse variant for newer devices with more memory (unconfirmed)
Gemini NanoGoogleSmall (multiple versions)Android (via ML Kit GenAI) and Chrome (Prompt API)Powers on-device summaries, smart replies and web features
Gemma 4Google (open weights)Several sizes; E2B ~2.9 GB at 4-bitPhones, laptops, edge devicesReleased April 2026 under Apache 2.0
Phi-4-miniMicrosoft (open weights)3.8BLaptops, phonesMIT licence; Windows also ships its own on-device models for Copilot+ features
Llama, Qwen and othersMeta, Alibaba, many1โ€“8B variantsAnywhere via llama.cpp, MLX, ONNXThe open ecosystem behind most local-AI apps

Wikipedia (Apple Intelligence, Gemma), Tech Insider, AIMagicX. Sizes and features change quickly.

Small models get good through distillation (training a small model to imitate a large one), careful data curation, and quantization to 4 bits or fewer. They cannot match frontier models on hard reasoning, but they handle summarising, rewriting, classification, simple Q&A and on-screen actions well, which covers a large share of everyday requests.

๐Ÿ”€ Hybrid AI: device first, cloud when needed

No device can run everything, so platforms route each request. Apple's design is the clearest example: a small model on the device, larger models on Private Cloud Compute (Apple-run servers announced in June 2024, designed so that personal data is not stored or accessible to Apple), and, for some requests, third-party models. In January 2026 Apple and Google announced a multi-year deal under which Apple's next-generation foundation models are based on Google's Gemini technology; Apple says they run on device and on Private Cloud Compute, though press accounts of exactly which servers run the largest model conflict.

๐Ÿงญ Where should this request run?

Pick a task. Illustrative routing logic, not any vendor's actual rules.

๐Ÿ“‰ The memory crunch: AI makes AI devices dearer

There is an irony at the edge. On-device AI needs more memory, but the same three companies that make phone and laptop DRAM, Samsung, SK hynix and Micron, have shifted capacity toward HBM for data-centre AI (see The Memory Wall). Gartner forecasts memory prices rising about 130% by the end of 2026, pushing average PC prices up about 17% and smartphone prices up about 13% compared with 2025. Memory is 15โ€“20% of a mid-range phone's bill of materials, so cheaper devices are hit hardest.

๐Ÿ“Š 2026 forecasts

Gartner (prices; PCs โˆ’10.4%); IDC (smartphones โˆ’12.9% to 1.12B units, revised from โˆ’16.7%; PCs โˆ’11.3%). IDC Q3 2026 PC shipments โˆ’20.1% to 62.7M units, via secondary report. Forecasts are revised often.

Meanwhile almost every new PC now counts as an "AI PC", but the share depends on the definition. Gartner projects AI PCs at about 55% of the market in 2026 (143 million units), counting any PC with an NPU. A count limited to Copilot+-class machines with 40+ TOPS gives about 44% (attributed to Canalys), and Morgan Stanley's stricter definition about 28%, both via secondary reports.

๐Ÿงญ What edge AI means for the rest of the stack

๐Ÿ•ฐ๏ธ Timeline

2017 ยท Apple's A11 Bionic introduces the Neural Engine; Huawei's Kirin 970 ships an NPU.
2023 ยท Google ships Gemini Nano on Pixel phones; llama.cpp popularises running open models locally.
May 2024 ยท Microsoft launches Copilot+ PCs with a 40 TOPS NPU requirement.
Jun 2024 ยท Apple Intelligence announced, with ~3B on-device models and Private Cloud Compute.
2024 ยท Intel Lunar Lake (48 TOPS) and AMD Ryzen AI chips meet the Copilot+ bar.
Jan 2026 ยท Apple and Google announce Gemini-based Apple foundation models. Snapdragon X2 Elite laptops (80+ TOPS) shown at CES.
Apr 2026 ยท Google releases Gemma 4 open-weight models, including phone-sized variants.
2026 ยท Memory prices surge as DRAM shifts to HBM; forecasts cut phone and PC shipments by 10โ€“13%.
Q3 2026 ยท IDC: PC shipments fall about 20% year on year.

๐Ÿ”ญ What to watch

๐ŸŽฏ The bottom line AI is spreading from data centres to the devices in our pockets and bags. NPUs and GPU matrix units let phones and laptops run small models privately, instantly and without per-token costs, while memory capacity and bandwidth, not headline TOPS, decide what fits and how fast it runs. The future is hybrid: everyday tasks on the device, hard reasoning and agents in the cloud. And in a twist, the data-centre boom is making the memory that edge AI depends on scarcer and dearer.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References