Cost Modeling: Build vs Buy

Evals made sure your optimizations did not cost quality. This page is about the biggest cost decision of all, the one that sits above every technique in this book: do you rent intelligence from an API priced per token, or own a GPU priced per hour and serve the tokens yourself? Both answers can be correct; which one is depends entirely on your numbers. So we model them.

Two pricing models, two cost structures

The build-vs-buy choice is really a choice between two cost shapes.

• Buy (API): you pay $ per 1M tokens. Cost is purely variable — zero traffic costs zero, double the traffic costs double. No GPUs, no ops, no idle time. Someone else absorbed the fixed cost and the utilization risk.
• Build (self-host open weights): you pay $ per GPU-hour, whether or not a single request arrives. Cost is mostly fixed (the GPU is reserved 24/7); your job is to amortize that fixed cost across as many tokens as possible. Your effective $ per token = GPU-hour rate ÷ tokens-per-hour you actually serve.

        cost
         │                          API (pure variable)
         │                       ╱
         │                    ╱
         │                 ╱
         │              ╱
         │           ╱ ───────────────  self-host (fixed floor + cheap/token)
         │        ╱  ╱
         │     ╱  ╱
         │  ╱  ╱
         │╱ ╱
         └──────────────────────────────► volume (tokens/month)
            ▲ break-even
   left of it: API cheaper   right of it: self-host cheaper

The API line goes through the origin (no traffic, no cost) but has a steep slope (every token is marked up). The self-host line starts above the origin (you pay for the GPU even at zero traffic) but has a shallow slope (each token is nearly free once the GPU is bought). They cross at the break-even volume.

Utilization is everything

The self-host slope is not fixed — it depends on how full you keep the GPU. An idle GPU is pure loss: you paid for the hour and served nothing, so those tokens-that-never-happened still divide into the bill, raising every real token’s cost. This is the cost throughline at its most brutal: the same hardware can be 5× cheaper or 5× more expensive per token with no code change, purely on utilization. Everything in Parts 3-5 — continuous batching, PagedAttention, good scheduling — exists to push that utilization up, which is to say, to lower the self-host slope.

This is also why the shape of your traffic decides the answer. Steady, predictable, high-volume traffic keeps a GPU busy → low effective cost → build. Spiky, low, or unpredictable traffic leaves the GPU idle between bursts → high effective cost → the API’s pay-per-use shape wins, because you only pay when you actually serve.

Worked example: break-even volume

Let’s find the crossover for one realistic setup.

Assumptions
  Self-host:  1× H100, rented at ........... $2.00 / GPU-hour
              sustained output throughput .. 4 000 tokens/sec   (from your load test)
  API:        output price .................. $3.00 / 1M tokens
              (ignore input tokens for this back-of-envelope)

Step 1 — self-host cost per token at full utilization:

tokens/hour = 4 000 tok/s × 3 600 s = 14.4M tokens/hour
cost/token  = $2.00 ÷ 14.4M        = $0.139 per 1M tokens

So at 100% utilization, self-hosting is $0.139/1M vs the API’s $3.00/1M — about 21× cheaper per token. But that assumes the GPU never idles.

Step 2 — break-even on a 24/7 rented GPU. The GPU costs $2/hr × 24 × 30 ≈ $1 440/month whether used or not. The API would charge $3 per 1M tokens for the same work. So self-host wins once:

monthly tokens × ($3.00 / 1M) > $1 440
monthly tokens > $1 440 ÷ $3.00 × 1M
monthly tokens > 480M tokens / month

Break-even ≈ 480M output tokens/month. That GPU can produce up to 14.4M/hr × 720 hr ≈ 10.4B tokens/month at full tilt, so 480M is only ~4.6% utilization — meaning above a fairly low steady load, owning beats renting handily. Below it, the idle GPU makes the API cheaper.

Step 3 — the utilization trap. Now suppose your real traffic only keeps the GPU 10% busy (spiky product, mostly daytime). You still pay $1 440/month but serve ~1.04B tokens, so your effective cost is $1 440 ÷ 1.04B = $1.38/1M — still under the API, but you have thrown away ~90% of the value. At 2% utilization you would serve ~208M tokens for $1 440 = $6.92/1M, now more than twice the API price. Same GPU, same code: utilization alone flipped the decision.

The model above is deliberately optimistic

Real self-hosting adds costs the napkin skips: GPU ops/SRE salaries, redundancy for failover (a second GPU mostly idle), model storage and loading, autoscaling headroom you pay for but rarely use, and the engineering time to keep vLLM/TGI tuned. Add a fudge factor — these can easily double the true self-host cost — and the break-even volume moves right. Do not commit to “build” on the bare arithmetic.

When each one wins

Choose the API when…	Choose self-host when…
Volume is low, spiky, or unpredictable	Volume is high and steady (keeps GPUs busy)
You want frontier closed models (GPT, Claude, Gemini)	Open weights meet your quality bar (verify with evals)
You have no GPU/SRE ops capacity	You have the ops muscle to run inference 24/7
Fast time-to-market, no capacity planning	You need data residency / full control
Cost is dominated by variability	You can amortize a fixed cost across real load

The fixed-vs-variable instinct

Buy converts a fixed cost into a variable one (you rent capability per token); build converts a variable cost into a fixed one (you own capacity and must fill it). The right call is whichever matches your demand curve — and the lever you control either way is utilization, which is exactly what the rest of this book taught you to raise.

Under the hood — how a marked-up token still beats your GPU

The puzzle the table hides: if the API marks up every token, how is it ever cheaper than serving the same token yourself? The answer is utilization through pooling. An API provider multiplexes thousands of independent customers onto one shared fleet, so their peaks and troughs cancel — one customer’s 3 a.m. lull is filled by another’s daytime burst. They run their GPUs at a utilization that you, with a single bursty workload, cannot touch, and that high utilization drives their true cost per token far below their list price. Your self-host math only wins once your own traffic is steady enough to keep the GPU as full as theirs is. The markup is real; you pay it to rent someone else’s near-perfect utilization until your volume can earn its own.

This is where Part 8 hands off to the capstone: those two cost shapes — $/token rented vs $/GPU-hour owned — are the entire economic engine of an AI product, and the capstone runs the full economics of one end to end on top of them.

Check your understanding

1. Describe the cost shape of the API option versus the self-host option, and where the two lines cross.
2. Why is an idle GPU “pure loss,” and what does that do to effective cost per token?
3. Using $2/GPU-hour, 4 000 output tok/s, and an API price of $3/1M tokens, reproduce the ~480M tokens/month break-even.
4. At 2% utilization the worked example flipped to favor the API. Explain why the same GPU and code can be cheaper or more expensive than the API.
5. Give two reasons to choose the API even when self-hosting would be cheaper per token at full utilization.

Show answers

1. The API is pure variable cost: a line through the origin (zero traffic, zero cost) with a steep slope (every token marked up). Self-host is a fixed floor (you pay for the GPU even at zero traffic) plus a shallow slope (each token nearly free once bought). They cross at the break-even volume; below it the API is cheaper, above it self-host is cheaper.
2. You pay the GPU-hour regardless of traffic, so unserved capacity is money spent for nothing; those missing tokens still divide into the fixed bill, raising the cost of every token you do serve. Low utilization inflates effective $/token.
3. The GPU costs $2 × 24 × 30 = $1 440/month. Self-host beats the API once monthly tokens × ($3/1M) > $1 440, i.e. monthly tokens > $1 440 ÷ $3 × 1M = 480M tokens/month.
4. Because the GPU’s cost is fixed but its output depends on utilization. Full load spreads $1 440 over billions of tokens (~$0.14/1M); 2% load spreads the same $1 440 over only 208M tokens ($6.92/1M). Utilization alone moves effective cost by orders of magnitude.
5. Any two of: you need frontier closed models; traffic is too spiky/low to keep a GPU busy; you lack GPU/SRE ops capacity; you want faster time-to-market; the optimistic napkin math ignores ops, redundancy, and engineering costs that can double the real self-host price.