Fine-Tuning, RAG, or Prompting: Choosing by Failure Mode

The question arrives at least once per engagement: should we fine-tune. It is almost always the wrong first question, because fine-tuning is a solution to one specific problem and it is usually not the problem the team has.

The useful reframing is to name the failure. If the model does not know something, that is a knowledge gap and training is a poor fix. If the model knows but behaves wrongly — wrong format, wrong tone, wrong decision boundary, too verbose for your interface — that is a behaviour gap, and that is where fine-tuning genuinely wins.

Knowledge gaps belong to retrieval

Fine-tuning facts into weights is expensive, slow to update, and unreliable in exactly the way that matters: the model becomes more confident about the domain without becoming consistently correct, and you cannot cite a source. Your pricing changed last Tuesday. Retraining is not an acceptable update mechanism for that, and it never becomes one.

Retrieval gives you freshness, provenance and per-tenant isolation, all of which matter more in enterprise settings than the marginal fluency a fine-tune buys. The counter-argument I hear is cost — those retrieved tokens are paid for on every call — and prompt caching has largely dissolved it.

Behaviour gaps are where training earns its keep

Where fine-tuning consistently delivered for me: enforcing a rigid output convention that no amount of prompting made reliable, teaching a domain classification boundary that is genuinely idiosyncratic to a customer, matching a house style across thousands of generated documents, and — the most commercially interesting case — distilling a large model's behaviour on a narrow task into a small one you can serve cheaply.

That last one is the strongest economic argument available. Use the expensive model to produce a few thousand high-quality examples on a task you run millions of times, train an adapter on a small model, and serve it at a fraction of the price with lower latency. The quality ceiling is lower, but on a narrow task it is frequently above your requirement.

# LoRA: train a small adapter, not the model. Minutes on one GPU, megabytes to ship.
config = LoraConfig(
    r=16,                      # rank; 8-32 covers most behaviour adaptation
    lora_alpha=32,
    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],
    lora_dropout=0.05,
    task_type='CAUSAL_LM',
)
model = get_peft_model(base_model, config)

# Data quality dominates data quantity. 500 correct, consistent, deduplicated
# examples beat 50,000 scraped ones, and the failure mode of the large set is
# that you teach the model your labelling inconsistencies.
train = load_curated('triage-v4.jsonl')      # held-out split by TENANT, not by row
assert no_leakage(train, holdout)            # near-duplicates inflate every metric

Two details there are where projects actually fail. Splitting the holdout by row rather than by tenant or document leaks near-duplicates into evaluation and produces a number that looks excellent and does not survive deployment. And inconsistent labels in training data are not noise the model averages out — they are a behaviour you are explicitly teaching.

The costs that are not the GPU bill

• A fine-tune pins you to a base model. When a better one ships, you re-run the whole pipeline to take advantage of it.
• You now own an evaluation suite, a data pipeline, a versioned artifact and a rollback story — an ML system, not a prompt.
• Adapters can degrade general capability outside the trained task. Test the things you did not train on.
• Training data is a compliance object. Customer text in a fine-tune is difficult to delete on request.
• Every prompt change afterwards interacts with the trained behaviour; the two are no longer independent.

Fine-tuning teaches behaviour. Retrieval supplies knowledge. Most teams reach for the first when they have a problem shaped like the second.

The sequence I now recommend without much hedging: get the prompt and the schema right, add retrieval if the failures are knowledge-shaped, build the eval suite, and only then consider training — with the eval suite already in place, because without it you cannot tell whether the fine-tune helped. In practice a good schema plus decent retrieval resolves most of what teams bring to me as a fine-tuning request, in about a tenth of the time.