Your assistant is still quoting last year's policy

Right in March, wrong in June

A composite, and a very ordinary one. In March your team connected an assistant to the help center and the policy folder. It answered return questions well. In June the return window changed from 60 days to 30. Someone updated the help page. Nobody touched the old PDF in the policy folder, the one the assistant liked to quote because it was more detailed.

Now a customer asks on day 45 whether they can still return a jacket. The assistant says yes, cites the PDF, and links to it. The customer ships the jacket back. Your warehouse refuses it.

This is not a model problem. The model read what you gave it. The question is who owns what it reads.

It's also not a hypothetical risk. In 2024 a British Columbia tribunal found Air Canada responsible for bad bereavement-fare advice its website chatbot gave a customer, and ordered the airline to pay the difference, about $812. The tribunal's reasoning, as CBC reported it, was that the chatbot is part of the website, and it makes no difference whether information comes from a static page or a chatbot. Your customers will see it the same way.

Every source needs an owner and a date

Make a list of everything your assistant reads: help articles, policy documents, product sheets, internal wikis, saved replies. For each one, write two things beside it: the person who owns it, and the date it should next be checked.

The owner is not "support" or "legal." It's a name. The date is not "when it changes." It's a day on a calendar, because things change without anyone telling the assistant.

If a document has no owner, it shouldn't be in the assistant's sources. That rule alone will remove a surprising amount of old material.

Don't stop at documents. Saved replies and macros, product data feeds, the FAQ block on the shipping page, the notes a support lead pasted into a wiki two years ago: if the assistant can read it, it's a source. The saved replies are often the oldest thing in the pile, because they were written once and copied forever.

Retire old versions, don't stack new ones

The most common cause of stale answers isn't that nobody wrote the new policy. It's that the old one is still sitting next to it. When two documents disagree, the assistant may pick either one, and nothing guarantees it picks the newer one. A long, detailed old PDF can easily win.

So when a policy changes, the job is two steps: publish the new version, and remove or clearly retire the old one from the assistant's sources. Put the effective date at the top of every policy document. If you need to keep an archive, keep it somewhere the assistant can't read.

Temporary policies need extra care. A holiday return extension, a free-shipping threshold for one month, a price promise for a launch: each of these should carry an end date in the document itself, and someone should own the day it ends. Otherwise the assistant cheerfully offers the December return window in February, and it will cite your own page when it does.

When a policy changes, the assistant changes too, or it keeps telling customers the old rule in your name. Add "update the assistant's sources" to the same checklist that updates the help page, the email templates and the checkout copy.

Check the answers that carry money

You can't test every possible question. You can test the ones that cost you when they're wrong: refund windows, eligibility, prices, delivery promises, anything involving a deadline or a fee.

Write ten or twenty of those questions down with the correct answer and the source it should come from. Run them against the assistant every week, and every time a policy or a source changes. A person compares the answers. When one is wrong, fix the source first, not the prompt.

Include a few questions the assistant should not answer at all, such as anything that sounds like legal or medical advice, or a request for an exception to a policy. The right answer there is a clean handoff to a person, and it's worth checking that the handoff still works after every change. Keep the list in a plain shared document, with the expected answer and the expected source next to each question, so anyone on the team can run it.

It's a small, recurring job, and the cheapest insurance in the whole setup.

Let the answer show where it came from

Ask the assistant to show which document it used, with the document's effective date. That does two things. Customers and staff can see how current the answer is. And your team can spot a stale source the moment it appears in an answer, instead of when a jacket is refused at the warehouse.

If an answer can't point to a current source, it should say so and hand off to a person. "I'm not sure that's current; here's how to reach the team" is a better answer than a confident wrong one.

The same goes for assistants your staff use internally. A sales rep quoting last quarter's price list, or a new hire reading an old leave policy, causes the same trouble one step removed. Internal sources need owners and dates just as much as the public help center does.

Put this into practice

Today, ask your assistant the three questions where a wrong answer would hurt most. Look at which documents it cites. If any of them is older than your last policy change, you've found your first fix. Then put a recurring hour on someone's calendar for the weekly check, and give them the authority to pull a source that fails it.

We build AI applications and agents that work from the knowledge your business actually trusts, and keep it current. If your assistant is quoting rules you've already changed, talk to us.

Related work: Spin:Market.