Fine-tune vs RAG vs Prompt

LoRA and QLoRA made fine-tuning cheap enough that you can train. This page is about whether you should. Before you spend a single GPU-hour, there are two ways to change a model’s behavior that involve no training at all — and they are often the right answer. The throughline for this whole part lands here: the cheapest training is the training you don’t do, so we hold every approach up against its cost in dollars, effort, and per-request latency.

Three tools, three jobs

Prompt engineering changes behavior by changing the input. No training, no infrastructure — you rewrite the instructions, add a few examples, fix the output format. It is instant and free to iterate, and it is where you should always start. The cost is per-request: a longer prompt is more tokens, and tokens are latency and dollars on every single call, forever.

RAG (Retrieval-Augmented Generation) changes behavior by changing the knowledge available at inference time. You retrieve relevant documents and inject them into the prompt, so the model answers from current, citable facts it was never trained on. RAG is how you get freshness (today’s data, not the training cutoff) and provenance (a link to the source). The cost is a retrieval pipeline to build and run, plus extra context tokens per request.

Fine-tuning changes the weights — it bakes a behavior in. Use it to teach a consistent style or format, to nail a narrow task the base model fumbles, or to shorten prompts by moving instructions-and-examples out of the context window and into the weights. The cost is mostly one-time (the training run) plus hosting the adapter; in return, per-request prompts get shorter, which can lower ongoing inference cost.

The mnemonic

Fine-tune for form, RAG for facts, prompt first. Form = style, tone, output structure, narrow skills (weights). Facts = knowledge that is fresh, large, or must be cited (retrieval). And you almost always try a prompt before either.

They are complementary, not exclusive

The framing “fine-tune or RAG” is a trap. Production systems stack all three: a model fine-tuned to always answer in your support voice and JSON schema, fed retrieved knowledge-base articles for the specific ticket, inside a prompt template that sets the task. Each tool does the job it is best at — the fine-tune fixes form, RAG supplies facts, the prompt orchestrates. They compose.

Cost, effort, and latency at a glance

Dimension	Prompt	RAG	Fine-tune
Upfront cost	~zero	Build retrieval pipeline	One training run (LoRA: cheap)
Per-request cost	Higher (long prompts)	Higher (retrieved tokens)	Lower (shorter prompts)
Iteration speed	Seconds	Hours–days	Hours–days per run
Best for	Behavior, format, quick wins	Fresh / large / citable knowledge	Style, format, narrow tasks
Knowledge freshness	Manual	Live (re-index)	Frozen at train time
Changes the weights?	No	No	Yes

Notice the per-request column: prompt and RAG add tokens to every call, while fine-tuning can remove them by internalizing instructions. At low volume the prompt’s zero upfront cost wins; at high volume, a one-time fine-tune that trims 500 tokens off every request can pay for itself many times over. (The token-economy side of this — and how RAG keeps its injected context lean — is the subject of RAG efficiency.)

A practical decision guide

Need fresher / external / citable knowledge?   → RAG
Need a consistent style, format, or narrow skill? → fine-tune
Just need different behavior or output shape?   → prompt (try this first)
Prompt works but is enormous and high-traffic?  → fine-tune to shorten it
Still wrong after all three?                    → revisit the data / the base model

Work top-down and stop at the first tool that solves it. Reach for fine-tuning when a prompt can express the behavior but is too long, too brittle, or too slow to do so at your traffic — that is when baking it into the weights earns its one-time cost.

Don’t fine-tune to add facts

The most common mistake is fine-tuning to inject knowledge. Models memorize facts poorly and you must retrain to update them — RAG does this better, cheaper, and stays current. Fine-tune to change how the model answers, not what it knows.

It actually happened — fine-tuning new facts can increase hallucination (2024)

The “don’t fine-tune to add facts” rule got empirical backing in 2024. A study from Google Research (Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, Gekhman et al.) split fine-tuning examples into facts the base model already knew versus genuinely new ones. The new-knowledge examples were learned slowly — the model fit them much later in training — and, more tellingly, as the model finally memorized them its tendency to hallucinate on unrelated questions rose. The mechanism is intuitive in hindsight: fine-tuning teaches the model that confidently producing facts from its weights is rewarded, even for things it doesn’t reliably know, nudging it toward fabrication. This is the data behind the mnemonic — fine-tune to change how the model answers, and reach for RAG when the goal is what it knows.

The architect’s lens

This page is a decision, so turn the five questions on the choice itself:

• Why does it exist? Because the three tools change different things — prompt changes the input, RAG changes the retrieved knowledge, fine-tuning changes the weights — and conflating them wastes GPU-hours on problems a prompt could solve.
• What problem does it solve? Spending the right resource: “fine-tune for form, RAG for facts, prompt first,” so you don’t pay a training run for what a prompt edit fixes, nor re-train weights for knowledge that changes weekly.
• What are the trade-offs? Prompt and RAG add tokens to every call (higher per-request cost) while fine-tuning is one-time and can shorten prompts; at low volume the prompt’s zero upfront cost wins, at high volume a fine-tune that trims ~500 tokens/call pays back many times over.
• When should I avoid it? Don’t fine-tune to add facts — models memorize them poorly, you must retrain to update, and a 2024 study found new-knowledge fine-tuning can raise hallucination; use RAG for what the model should know.
• What breaks if I remove it? The composition — treating the three as exclusive is a trap; production stacks layer all of them (fine-tuned voice + retrieved facts + a prompt template), and dropping one forces another to do a job it’s bad at.

Check your understanding

1. What does each of the three tools change — input, retrieval, or weights?
2. State the “form / facts / first” mnemonic and what each word maps to.
3. Why is it a mistake to treat fine-tuning and RAG as mutually exclusive?
4. Per request, how does fine-tuning’s cost profile differ from prompt and RAG, and when does that flip the economics?
5. A team wants their assistant to cite the latest internal docs. Which tool, and why not fine-tuning?

Show answers

1. Prompt changes the input (instructions/examples), RAG changes the retrieved knowledge injected at inference, fine-tuning changes the model’s weights.
2. “Fine-tune for form, RAG for facts, prompt first”: form = style/format/narrow skills (weights), facts = fresh/large/citable knowledge (retrieval), and you try a prompt before either.
3. They solve different problems and compose — a single system can be fine-tuned for form, fed retrieved facts, and orchestrated by a prompt template all at once.
4. Prompt and RAG add tokens to every request (higher per-request cost), while fine-tuning can shorten prompts and lower per-request cost; at high traffic, a one-time fine-tune that trims tokens off every call can outweigh its upfront cost.
5. RAG — it provides fresh, citable knowledge from the live docs; fine-tuning memorizes facts poorly and would need retraining every time the docs change.