Fine-Tune, RAG, or Prompt? An Honest Decision Framework

There's a specific kind of engineering meeting I've sat through too many times. Someone has an LLM that isn't quite doing what the product needs. It's a little off-format, or it doesn't know the internal jargon, or it won't hold a consistent tone. And someone in the room says the words that quietly commit the team to three weeks of work: "We should probably fine-tune it."

Maybe. Or maybe fine-tuning is the most expensive way to solve a problem a paragraph of prompt would have handled. The trouble is that "fine-tune vs. RAG vs. prompt" has become a tribal argument online instead of an engineering decision. One camp insists fine-tuning is obsolete now that retrieval exists. The other fine-tunes everything and then acts surprised when the model forgets how to count. Both are skipping the only question that matters: which tool fits this problem, at what cost?

Here's the framework I actually use — the one that opens my new book, Fine-Tuning LLMs — stripped down to something you can apply before you spend a single GPU-hour.

Three Tools That Change Three Different Things

Start by being precise about what each option actually changes. Not what it "feels" like — what it mechanically does.

Prompting changes the instructions. You're shaping behavior at inference time with words. It's free, it's instant, and it's reversible. It's also the thing most teams under-invest in before jumping to heavier tools. A well-constructed prompt — clear role, explicit format, a few good examples — solves an astonishing number of "we need to fine-tune" problems. Rule of thumb: if you haven't seriously tried few-shot prompting with representative examples, you have not yet earned the right to complain that the base model can't do the task.

RAG changes what the model knows. Retrieval-augmented generation pulls relevant context into the prompt at query time. This is the right tool when your problem is knowledge — facts that live in your documents, that change over time, that are too numerous or too fresh to bake into weights. Product catalogs, policy docs, a knowledge base that updates weekly. RAG shines here because you can change the data without touching the model. But RAG is weak exactly where knowledge isn't the issue. It won't reliably teach a model your output format, your tone, or a behavior. You can stuff format instructions into the retrieved context, but you're now paying for those tokens on every single call, forever, and still hoping the model complies.

Fine-tuning changes the weights. This is the only option that actually teaches the model a new default behavior. It's the right tool when you need consistency the prompt can't hold: a rigid output structure across thousands of calls, a specific voice, a specialized task the base model does clumsily. The cost is real — you need a dataset, GPU time, and an evaluation process — and it's the one option with a nasty failure mode attached, which we'll get to.

The mistake in both directions is a mismatch. Fine-tuning a problem that was really about changing facts means retraining every time the facts change — miserable. Bolting RAG onto a problem that was really about format means paying a token tax forever for a behavior you could have trained in once. Name what your problem actually is — instructions, knowledge, or behavior — and the tool usually names itself.

A Decision Path You Can Actually Walk

Frameworks are useless if they don't produce a decision. Here's the order I run it in, as rough pseudo-logic:

if problem is "model doesn't follow instructions well enough":
    exhaust prompt engineering first        # cheap, instant, reversible
    if a strong few-shot prompt solves it:  -> STOP. Ship the prompt.

if problem is "model lacks specific/changing knowledge":
    reach for RAG
    if answers just need the right facts at query time: -> RAG. Don't fine-tune.

if problem is "model won't hold a format/tone/behavior consistently":
    AND you can build a few hundred+ good examples of the target behavior:
    AND the behavior is stable enough to be worth baking in:
        -> fine-tune (probably LoRA/QLoRA)

if you need both specialized behavior AND fresh facts:
    -> fine-tune for the behavior, RAG for the knowledge. They compose.

Notice how many branches end before fine-tuning. That's not an accident. The single most valuable thing this framework does is talk you out of fine-tuning when something cheaper wins. "Don't fine-tune" is a legitimate, common, and often correct outcome. An honest framework has to be as willing to say no as yes, or it's just a rationalization for the impressive-sounding option.

The other gate worth calling out explicitly: do you have the data? Fine-tuning isn't a technique you apply to a model, it's a technique you apply to a dataset. If you can't assemble a few hundred clean, representative examples of the behavior you want, the decision is made for you — you're not fine-tuning yet, you're collecting data. In practice, dataset construction is where the real work lives. It's the reason my book spends three full chapters on sourcing, cleaning, formatting, and splitting data, and comparatively few pages on the training command itself.

The Failure Mode That Makes People Regret Fine-Tuning

Say you've run the framework, the answer is genuinely "fine-tune," and you do it. Here's the trap that ambushes first-timers, and the reason the decision deserves respect.

You point a large model at a small, narrow dataset. Training goes great — the loss curve slides down smooth and monotonic, textbook. You try a few prompts, the model nails your target format, and you feel like a wizard. Then someone asks it to do something outside the training set — some basic reasoning or arithmetic it could do perfectly well before — and it fumbles. It got better at the narrow thing and worse at everything else. This is catastrophic forgetting, and the cruelest part is that nothing in the standard tutorial pipeline warned you. The loss curve only measured the narrow thing. It went down the whole time.

This is why the "just fine-tune it" reflex is dangerous without an evaluation practice attached. Techniques like LoRA and QLoRA reduce the risk — by training small adapter layers instead of rewriting the whole model, you disturb the original capabilities less, and you get the bonus of fitting the whole job on a single rented GPU. But reducing risk isn't eliminating it. The only way to actually know whether your fine-tune helped or quietly broke something is to measure before and after, on a held-out set that never touched training, with tests that check the capabilities you didn't fine-tune as well as the one you did.

That's the discipline that separates a fine-tune you can ship from one that embarrasses you in production three days later. It's also, not coincidentally, the chapter most fine-tuning guides skip entirely.

Where This Goes Next

If you run this framework and land on "prompt" or "RAG," that's a win — you just saved yourself weeks. (My own take on when retrieval is the right answer is a whole book of its own; I'd rather you fine-tune the right things than everything.)

But if you land on "fine-tune," you've only made the first decision. Ahead of you: choosing and licensing an open base model, building the dataset, configuring LoRA or QLoRA with understanding instead of copied hyperparameters, getting the chat template right so a silent formatting bug doesn't ruin the run, evaluating honestly, and finally quantizing and serving the thing. That's the whole arc of Fine-Tuning LLMs, and it's built around one project — a customer-support model for a fictional company, taken from this exact decision all the way to a deployed API, entirely on a single rented consumer GPU with real memory and cost numbers.

The goal isn't to make you fine-tune more. It's to make you fine-tune deliberately — and to know, every time, whether closing the laptop and writing a better prompt was the smarter call all along. (I write more about practical AI engineering at shanelarson.com.)

Fine-Tuning LLMs: A Practical Python Guide to Customizing Open Models is available now on Amazon Kindle.