On-Device & Edge Inference

The quantization zoo covered the formats that shrink a model’s weights. On-device inference is where those formats stop being an optimization and become the only way the model fits at all. Here the GPU is a phone NPU or a laptop’s unified memory, the power budget is a battery, and there is no datacenter to fall back on.

Why run locally at all

Cloud inference is the default for good reasons — but four pressures push work to the edge:

• Privacy. The data never leaves the device. For health, keyboard, photos, or on-device assistants, “we don’t send it anywhere” is a feature you cannot fake with a privacy policy.
• Latency. No network round-trip. A local 7B model can start streaming before a cloud request finishes its TLS handshake — and it works on a plane, in a tunnel, on bad Wi-Fi.
• Offline. Works with no connectivity at all.
• Zero per-token cost. This is the throughline turned inside-out. Cloud inference bills every token forever; local inference is paid once (the user’s hardware) and then free to you. At scale, moving inference onto users’ devices can erase your single largest variable cost.

  Cloud:   $ per token, forever, but unlimited compute
  Device:  $0 per token, but YOUR memory/battery/silicon budget

What stops you

The edge takes the constraints from Part 1 and tightens every one:

• Memory. A phone has ~6–12 GB of RAM shared with the whole OS; a laptop maybe 16–64 GB unified. An FP16 7B model is ~14 GB — already too big for most phones before you’ve opened any other app.
• No datacenter GPU. No 80 GB H100, no NVLink. You get a mobile NPU or an integrated GPU with a fraction of the bandwidth.
• Battery and thermals. Sustained inference drains the battery and heats the chip until it throttles. The constraint isn’t just peak speed; it’s speed you can sustain without cooking the phone.

The enablers

What makes local inference feasible is a stack of aggressive compression and purpose-built runtimes:

• Heavy INT4 (and lower) quantization. This is the load-bearing trick. A 7B model:

  Precision	Size	Fits a phone?
  FP16	~14 GB	no
  INT8	~7 GB	borderline
  INT4	~3.5 GB	yes

  Worked: 7×10⁹ params × 0.5 bytes (4-bit) ≈ 3.5 GB. INT4 is what drops a 7B model from “impossible” to “runs on a flagship phone with room to spare.” Going to ~2-bit pushes even bigger models onto laptops, at rising accuracy cost.

• llama.cpp / GGUF. A CPU-and-everything-else inference engine plus a quantized file format that made local LLMs mainstream — runs on Macs, Windows, Raspberry Pis, phones, with k-quant schemes tuned for exactly this regime.

• Apple MLX / Core ML, and unified memory. Apple Silicon shares one memory pool between CPU and GPU, so a quantized model is reachable by the accelerator without copying — a structural advantage for on-device LLMs.

• NPUs. Dedicated neural accelerators (Apple Neural Engine, Qualcomm Hexagon, Google Tensor) run quantized matmuls at far better performance-per-watt than the CPU — the battery-saving piece.

• Smaller models. The other half of the answer: 1–8B models (Llama 3.2 1B/3B, Phi, Gemma, Qwen small) designed to be good enough at a size that fits. You don’t shrink a 70B to a phone; you start from a model built for the phone.

Two knobs, same goal

On-device feasibility = smaller model × heavier quantization. A 3B model at INT4 (~1.5 GB) is a different proposition from a 70B at INT4 (~35 GB). The art is picking the smallest model that still does the job, then quantizing as hard as quality allows.

The trade-offs vs cloud

Local is not strictly better — it’s a different point on the curve.

• Quality ceiling. The frontier 400B+ model will not run on your phone any time soon. On-device means accepting a smaller model’s quality. Many tasks (autocomplete, summarization, classification, simple agents) don’t need the frontier; some do.
• Cold weights, slow decode. Lower memory bandwidth means slower TPOT than a datacenter GPU — fine for a chat-speed assistant, painful for bulk generation.
• Distribution. You now ship multi-GB model files to devices and update them — an app-size and logistics cost that the cloud doesn’t have.

What’s actually feasible today

Comfortable: 1–8B models at INT4 on phones and laptops, for assistant-style and on-device features. Stretch: ~13B on high-RAM laptops, ~30B on workstations with lots of unified memory, at reduced speed. Not happening on a phone: frontier-scale dense models. Hybrid designs route easy turns to the local model and hard ones to the cloud — local-first, cloud-as-fallback.

By the numbers — why local decode is slow

The “slower TPOT” warning above is really a bandwidth ceiling. Decode reads the whole model once per token, so the fastest you can possibly emit tokens is memory bandwidth ÷ model bytes.

  4-bit 7B model ............ ~3.5 GB to read per token
  flagship phone LPDDR5 RAM .. ~tens of GB/s (call it 60 GB/s)
  ceiling = 60 GB/s ÷ 3.5 GB ≈ 17 tokens/sec   (before any compute)

A datacenter GPU’s HBM moves ~1–3 TB/s — 20–50× more — which is the entire gap between a snappy cloud stream and a phone that types at reading speed. It also explains why on-device work leans so hard on smaller models: halving the bytes you must stream per token directly doubles the token ceiling, no better silicon required.

It actually happened — llama.cpp put a real LLM on a laptop (2023)

The local-LLM movement has a precise starting gun. In March 2023, days after Meta’s LLaMA weights leaked, Georgi Gerganov released llama.cpp — a dependency-free C/C++ implementation that ran LLaMA on a MacBook’s CPU using 4-bit quantization, no datacenter GPU anywhere. It made “a useful model on hardware you already own” something people could actually do that week, and a community exploded around it. Its quantized weight format, originally GGML, was superseded in August 2023 by GGUF, designed to be extensible and to carry metadata so one file could load across wildly different machines. Everything in this page’s “enablers” list — k-quant schemes, NPU and Metal backends, phone builds — grew out of that runtime. The lesson: the binding constraint on the edge was never just the math; it was a portable runtime that made heavy quantization practical on commodity silicon.

The cost lens

On-device inference is the most extreme answer to “how do we make it cheaper”: push the per-token cost to zero by spending the user’s memory and battery instead of your GPUs. The bill doesn’t vanish — it moves, and it’s bounded by what a phone can physically hold and sustain. Heavy quantization and small models are simply what it takes to fit a useful model inside that budget.

The architect’s lens

The most extreme answer to “how do we make it cheaper” — move the bill onto the user’s silicon:

• Why does it exist? Four pressures cloud can’t satisfy: privacy (data never leaves the device), latency (no round-trip), offline operation, and zero per-token cost — local inference can erase your single largest variable cost.
• What problem does it solve? Fitting a useful model inside a phone’s ~6–12 GB shared RAM and battery budget: feasibility = smaller model × heavier quantization, e.g. a 7B at INT4 (~3.5 GB) goes from impossible to runs-on-a-flagship.
• What are the trade-offs? A lower quality ceiling (no frontier 400B+ on a phone), slower decode (LPDDR5 ~60 GB/s caps a 4-bit 7B near ~17 tok/s vs a datacenter GPU’s 20–50×), and multi-GB model files to distribute and update.
• When should I avoid it? When the task genuinely needs frontier quality or bulk generation throughput — keep those in the cloud, or use a hybrid that routes easy turns local and hard ones to the cloud.
• What breaks if I remove it? You’re back to paying per token forever and sending user data off-device — fine for many apps, but you lose the privacy, offline, and zero-marginal-cost properties entirely.

Check your understanding

1. Give the four main reasons to run inference on-device rather than in the cloud.
2. Why does INT4 quantization specifically make the difference between a 7B model fitting on a phone or not? Show the arithmetic.
3. Why is “smaller model” listed alongside quantization rather than instead of it?
4. What structural advantage does Apple Silicon’s unified memory give on-device LLMs?
5. Describe a realistic hybrid local/cloud design and why you’d use it.

Show answers

1. Privacy (data never leaves the device), latency (no network round-trip), offline operation, and zero per-token cost to the operator.
2. A 7B model is ~14 GB at FP16 (7×10⁹ × 2 bytes) — too big for a phone — but ~3.5 GB at INT4 (7×10⁹ × 0.5 bytes), which fits in a phone’s RAM with room for the OS.
3. Feasibility is the product of model size and quantization; you can’t quantize a 70B small enough for a phone without destroying it, so you also start from a model built small (1–8B). Both knobs together get you under budget.
4. CPU and GPU share one memory pool, so a quantized model can be used by the accelerator without copying it across separate memory spaces — saving memory and bandwidth.
5. Route easy/short turns to a small local model (free, private, fast) and fall back to a frontier cloud model only for hard requests — minimizing cloud token cost while keeping quality available when needed.