Beyond GPUs: TPUs, LPUs, Custom Silicon

Why inference is memory-bound established the central fact: during decode the GPU’s compute units sit idle waiting on memory bandwidth. That fact is also a business opportunity — if decode is bottlenecked on bandwidth, not FLOPs, then a chip designed around bandwidth (or that sidesteps the bottleneck entirely) can beat a general-purpose GPU. This page surveys the silicon built on that bet.

Why the GPU isn’t sacred

GPUs are general-purpose parallel processors that happened to be excellent at the dense matmuls deep learning needs. But “general-purpose” means transistors and memory hierarchy spent on flexibility you may not need for inference. Remove the flexibility, spend the budget on the one thing decode is starved for — bandwidth, or on-chip memory — and you can win on latency, throughput-per-watt, or cost.

  Decode bottleneck:  read ALL weights → do a sliver of math → emit 1 token
                      ▲ bandwidth-bound, compute idle

  GPU:    fast compute, HBM off-chip   → bandwidth is the wall
  LPU:    keep weights in SRAM on-chip → no HBM wall, deterministic
  TPU:    systolic array               → wins the big-matmul (prefill) side
  Wafer:  enormous on-chip memory      → fit the model in SRAM

TPUs — systolic arrays for big matmuls

Google’s TPU is the longest-running alternative. Its core is a systolic array: a grid of multiply-accumulate units where data flows through and each cell does one MAC per beat, so a value is loaded once and reused across the whole array. That makes it superb at the large, dense matmuls of training and prefill, with strong performance-per-watt. TPUs connect over a custom high-bandwidth interconnect (ICI) into large “pods,” and Google serves much of its own AI on them. They lean compute-and-throughput; the decode-bandwidth story below is more the LPU’s pitch.

LPUs — Groq’s bet on deterministic, in-SRAM decode

Groq’s LPU attacks decode head-on. The move: don’t fight the off-chip memory wall — eliminate it. Keep weights in on-chip SRAM, which has bandwidth measured in tens of TB/s versus an HBM GPU’s ~1–8 TB/s. If the weight read is no longer the bottleneck, decode runs much faster and at very low per-token latency.

The design is also deterministic: no caches, no dynamic scheduling, a statically compiled dataflow where the compiler knows the cycle every operation runs. That kills timing jitter and gives extremely consistent latency — attractive for real-time and agentic loops where p99 matters more than peak.

The catch is the flip side of the win: on-chip SRAM is small (hundreds of MB per chip). A 70B model does not fit on one LPU, so you shard it across many chips wired together — the model lives in the aggregate SRAM of a rack. You trade per-chip simplicity and blazing bandwidth for needing a lot of chips and a fat interconnect.

The core insight

A GPU is bandwidth-bound on decode because the weights live in off-chip HBM. Groq’s answer is to put the weights in on-chip SRAM and spread the model across many chips — turning a bandwidth problem into an interconnect-and-chip-count problem, which for low-latency decode can be the better trade.

Wafer-scale and the cloud ASICs

• Cerebras (wafer-scale). Instead of dicing a wafer into many chips, Cerebras keeps the entire wafer as one processor with a huge pool of fast on-chip SRAM and enormous on-die bandwidth. The pitch is the same as the LPU’s — keep the model in fast on-chip memory — at a different physical extreme. Cost and cooling are the obvious prices.
• AWS Inferentia / Trainium. Amazon’s in-house inference (Inf) and training (Trn) ASICs. The motive is partly technical and largely economic: vertically integrate to cut the GPU-vendor margin and tune price-per-token for cloud customers. They occupy the “cheaper, good-enough, deeply integrated into our cloud” niche rather than the absolute-frontier-flexibility one.

The real cost: ecosystem, not just FLOPs

Specialized silicon almost always wins some benchmark. The question is whether you can actually use it.

	GPU (NVIDIA)	Specialized (TPU/LPU/ASIC)
Peak on its strength	good all-round	often better on its niche
Software maturity	CUDA — huge, battle-tested	smaller, evolving
Flexibility	runs anything	tuned for specific shapes
Availability	broad	often single-vendor/cloud

CUDA is a two-decade moat: every kernel, every framework, every optimization first lands on NVIDIA. A faster chip with a thin compiler, missing ops, or a quirky programming model can be slower in practice once you account for engineering time and the features you can’t run. That ecosystem gap — not raw FLOPs — is why GPUs still dominate despite the alternatives’ benchmark wins.

Don’t chase the benchmark

“3× faster tokens/sec on a demo” rarely survives contact with a real workload: dynamic batching, long context, mixed adapters, and the long tail of ops your stack actually uses. Evaluate alternative silicon on your model and traffic, with software maturity and availability weighted as heavily as peak throughput. The cheapest chip is the one your team can fully exploit.

It actually happened — the first TPU shipped before the boom (2015–2017)

Specialized inference silicon is not new. Google built its first TPU and deployed it in production datacenters around 2015, then described it publicly in the 2017 ISCA paper In-Datacenter Performance Analysis of a Tensor Processing Unit. That chip was inference-only, built around a 256×256 systolic array of 8-bit multiply-accumulate units, and it already powered live services — Search ranking, Translate, and the Go-playing system that beat a world champion. The motivation was exactly this page’s: Google projected that if users spoke to voice search for just a few minutes a day, serving it with neural nets on CPUs would have demanded roughly double their datacenter fleet. The durable lesson is that custom silicon tends to arrive the moment one workload gets big enough that general-purpose hardware’s overhead becomes a line item you can no longer ignore.

The cost lens

The whole “beyond GPUs” story is the throughline as a hardware question. Decode is memory-bound, so the chip that frees the bottleneck — bandwidth via on-chip SRAM, or a systolic array for the compute-bound prefill — can lower latency and dollars-per-token for that phase. Whether it lowers your total cost depends on the unglamorous parts: software maturity, availability, and how much of your real workload it can actually run.

Check your understanding

1. Why does the memory-bound nature of decode open the door to non-GPU silicon?
2. What is a systolic array, and which inference phase does the TPU’s design favor?
3. How does Groq’s LPU sidestep the HBM bandwidth wall, and what does that approach cost?
4. What is the primary motivation behind AWS Inferentia/Trainium, and how does it differ from chasing peak performance?
5. Why do GPUs still dominate even when specialized chips win benchmarks?

Show answers

1. Decode is bottlenecked on memory bandwidth, not compute, so a chip designed around bandwidth (or that keeps weights in fast on-chip memory) can beat a general-purpose GPU whose compute units sit idle waiting on HBM.
2. A grid of multiply-accumulate cells where data flows through and each value is loaded once and reused across the array; it excels at large dense matmuls, favoring the compute-bound prefill/training side.
3. It keeps weights in on-chip SRAM (tens of TB/s, far faster than HBM) and runs a deterministic statically-compiled dataflow; the cost is that SRAM is tiny, so the model must be sharded across many chips wired by a fat interconnect.
4. Mainly economic vertical integration — cut the GPU-vendor margin and tune price-per-token for cloud customers — aiming for cheaper, good-enough, deeply integrated inference rather than absolute peak performance or flexibility.
5. CUDA’s two-decade software ecosystem: frameworks, kernels, and ops land on NVIDIA first, so an alternative with thin software or missing ops can be slower in practice and can’t run parts of a real workload, outweighing raw benchmark wins.