RAG or fine-tuning? How to tell which problemyou have

"Should we fine-tune or use RAG?" is the most common first question we hear about AI features, and it is usually the wrong question. They solve different problems. Here is how to tell which problem you actually have.

What each one actually does

Retrieval grounds the model in your documents at query time. The knowledge stays in your corpus, stays current, and every answer can cite its sources. Fine-tuning changes how the model behaves — its tone, its format, its skill at a task — by training on examples.

The confusion comes from demos where both appear to "teach the model" something. Only one of them keeps teaching it after the demo ends.

The decision in one table

• Knowledge changes often → RAG. Documents update; retraining schedules do not keep up. Retrieval reads the current version every time.
• Answers must cite sources → RAG. A fine-tuned model cannot point at the paragraph it used, because there is no paragraph — the knowledge is in the weights.
• Consistent format or tone → fine-tune. House style, structured output schemas, classification labels: behaviour, not knowledge.
• High volume, narrow task → fine-tune a small model. A fine-tuned 8B model answering one question type will beat a frontier model on cost by an order of magnitude — once you have proven the task with retrieval first.

Prototype with retrieval, measure with an evaluation set, and fine-tune only the parts the numbers say retrieval cannot do.

The sequence that works

1. Start with baseline RAG over your real corpus and a labelled set of real questions. This takes days and tells you whether the knowledge is even retrievable — which is the thing most likely to be wrong.
2. Fix the data before the model. Chunking, metadata and deduplication move accuracy more than any model choice. We have never seen an engagement where this was not true.
3. Fine-tune narrowly, if at all. Format adherence, refusal behaviour, domain phrasing — small, testable behaviours with their own spot-checks.
4. Keep the harness running. Whatever you chose, the regression set runs on every change: prompts, models, corpus updates. All of it.

What about training on our data from scratch?

Almost certainly not. Pre-training is a capital expenditure measured in millions, and the result is a model that knows your 2024 documentation the way a graduate remembers a textbook — approximately, and without citations. Retrieval over your live corpus beats it on accuracy, freshness and cost in every engagement we have scoped.

The cost angle

A fine-tuned small model is the cheapest thing to run at volume and among the most expensive things to build well — data curation, training runs, evaluation of the tuned behaviour itself. RAG is the reverse: cheap to start, with costs that scale per query. For anything under a few million queries a month, retrieval wins on total cost. Past that, measure both and let the numbers decide.