Fine-tuning won't teach the model your price list

What people mean by "trained on our data"

Ask a business owner what they want from AI and you'll often hear some version of: "a model trained on our data." Then ask what they'd use it for. Answer customer questions about our products. Quote our prices. Explain our returns policy. Help new staff find the right procedure.

All of those are about facts. Your facts, which change. And for facts, fine-tuning is the wrong tool.

That isn't just our opinion. OpenAI's own developer guide says it plainly: a common misconception is that fine-tuning can teach models about your data. It isn't the case, the guide says, and if you want your application to know about your data, you should use retrieval.

What fine-tuning actually changes

Fine-tuning takes an existing model and trains it further on examples of inputs and the outputs you want. What it learns is mostly behavior: a tone, a format, a way of handling a particular kind of input.

OpenAI's guide lists when supervised fine-tuning is a good fit: when the output must follow strict guidelines for tone, style or format; when it's easier to show than tell how to handle certain inputs; and when you need inputs processed in a consistent way. It can also let you get the same result with a shorter prompt, and maybe a smaller, cheaper model.

Those are real uses. Sorting incoming emails into your twelve categories. Writing product descriptions in your house style. Pulling the same fields out of messy supplier documents every time. None of them require the model to know what a blue large costs today.

Why facts belong outside the model

Imagine you fine-tune on last month's price list. The model will sound confident about prices. Some will be right. Some will be blended from similar products. And when prices change, it won't know.

Facts change on a Tuesday. A fine-tuned model finds out at the next training run. In between, it will answer in exactly the same confident voice.

Retrieval works the other way round. When a question comes in, the system looks up the relevant facts in your own sources, such as the current price list, the policy page or the product record, and hands them to the model along with the question. The model writes the answer from what's in front of it. Change the price in your system and the next answer changes. You can also show where the answer came from, which is how you catch the wrong ones.

There's a permissions angle too. If you fine-tune on internal documents, whatever was in them is now blended into the model, and anyone who can use the model can coax some of it back out. With retrieval, the system can check who's asking before it looks anything up, so a customer gets the public returns policy and a store manager gets the internal exceptions. That's much harder to arrange once the knowledge is inside the model itself.

The hard part of retrieval isn't the model. It's the sources: which ones are true, who owns each one, and how old versions get retired. We've written about collecting the questions before you index the documents, which is where that work starts.

When fine-tuning is worth it

Fine-tuning earns its place after the basics work. A sensible order:

1. Prompt and retrieval first. Clear instructions, the right facts retrieved at answer time.
2. A test set. Fifty or a hundred real questions with answers someone has checked. Run every change against it.
3. Then ask what's still wrong. If the facts are right but the format wanders, or the tone is off, or a narrow task is inconsistent, fine-tuning might fix that.

Here's what a good case looks like. A distributor receives hundreds of order emails a day, each written differently, and needs the same eight fields pulled out and filed in the same way every time. Prompting gets most of them right, and the rest fail in predictable ways. A few hundred checked examples, used to fine-tune a smaller model, might make that job more consistent and cheaper to run. Notice that the facts still come from the email and the order system. The model only learned the habit.

Fine-tuning also has upkeep. Training examples need to be written and checked. When the underlying model is updated or retired, you may need to train again. OpenAI's own optimization guide pairs fine-tuning with evaluations to measure performance and spot drift, which is a polite way of saying you're signing up to watch it.

Questions to ask whoever is proposing it

If a vendor or developer proposes "training a custom model on your data," ask:

• What exactly will it learn: facts, or behavior?
• When our prices or policies change, how does the system find out, and how fast?
• Can it show which source each answer came from?
• What test set will you use, and can we see the results before and after?
• If you do fine-tune, who writes and checks the training examples, and how many will there be?
• What happens when the base model is retired?

Good answers sound like retrieval for facts, a test set, sources shown, and fine-tuning only for a specific, narrow behavior problem. Vague answers sound like "it learns your business."

Put this into practice

Write down five questions customers or staff actually ask, with the right answers and where those answers live today. If the answers live in one current, owned place, you're ready for a retrieval-based assistant. If they live in three places that disagree, that's the first job, and no amount of training will do it for you.

We build AI applications and agents that answer from your current sources and show where each answer came from, with fine-tuning only where it earns its keep. If someone has quoted you a custom model and you're not sure it's the right tool, talk to us.

Related work: PatientIQ.