Collect the questions before you index the documents.

Everyone's unofficial search engine

Every company has one. Sometimes it's the operations manager who's been there nine years. Sometimes it's someone in finance, or the one support lead who knows how the old system really worked. Their calendar says they have a job, but most of their day goes to answering questions from everyone else. "Where's the latest price list?" "Can we ship this to Canada?" "Why is this order on hold?"

When teams decide to build an internal AI assistant, the usual first step is to gather documents: the shared drive, the wiki, the policy PDFs, the help center. Point a model at all of it, and the assistant can answer anything, right?

In practice, that's backwards. The documents are the easy part. The hard part is knowing which questions matter, where their answers actually come from, and which ones shouldn't be answered by software at all. You find that out from the questions, not from the documents.

Spend a week collecting the questions

Before anyone builds anything, log what people ask your most-interrupted person for one ordinary week. Not a survey, which gets you what people think they ask. The real thing.

Keep it light. A shared sheet, a form, or the expert forwarding messages to a channel works fine. For each question, capture four things:

• The question, in the asker's words.
• Who asked, by role, not name.
• Where the answer came from: a document, a system, the expert's memory, or a phone call to someone else.
• What the asker did next: placed an order, replied to a customer, escalated, waited.

That last column matters more than it looks. It tells you what the answer is for, which is what tells you how bad a wrong answer would be.

A week is usually enough to see the shape. If the work is seasonal, or the expert says "this week was unusual," run it a second week.

Sort the questions by where the answer lives

Now go through the log with the expert and put every question into one of four piles.

The answer is in a document. The returns policy, the price list, the setup guide. This is what most people imagine an assistant doing, and it's a good fit, provided the document is current and you know which version is right.

The answer is in a system. Order status, stock levels, account balances. The assistant can help here too, but it needs a live, permissioned connection to that system, not a copy of last month's export.

The answer is someone's judgment. "Should we make an exception for this customer?" "Is this shipment safe to send as is?" These aren't retrieval questions. The assistant's job here is to gather the context and send it to the right person, not to decide.

The answer doesn't exist yet. The expert gives a different answer each time because nobody ever set a rule. These are the most valuable finds in the whole exercise, and the assistant can't fix them. Somebody has to write the policy.

A single line of the log might read: "Support rep. Can we ship lithium batteries to Hawaii? Answer came from the expert's memory of a carrier call last spring. Rep told the customer yes." That one goes in the judgment pile, and probably the policy pile too, because the answer lives in one person's memory and a customer is relying on it.

Most teams are surprised by how big the last two piles are. That's the point of doing this before you build.

Pick a small set and write down the right answers

From the first two piles, pick a manageable set of questions that come up often and matter. For each one, write down the answer you'd accept, the source it should come from, and what a good next step would be. Have the expert check every one.

That's your test set. Every version of the assistant runs against it before anyone else sees it, and every time you change the prompt, the documents, or the model. If it gets one wrong, you'll know whether the problem is a missing document, a stale one, or the assistant making things up.

It also makes the assistant honest about sources. Each answer should show where it came from, so the person reading it can check in one click. We wrote more about that in Bound the AI answer with a citation pack.

Build the narrow version and watch it

The first version should be deliberately small. It answers questions from the document and system piles, using only the sources you've confirmed. It shows where every answer came from. When it isn't sure, it says so. When a question belongs to the judgment pile, it hands it to the expert with the context already gathered: who's asking, what they're working on, what the assistant found.

One more thing to settle before launch: permissions. The assistant should only show people what they'd be allowed to see if they went looking themselves. If the pricing exceptions folder is restricted to the sales leads, the assistant shouldn't quote it to a new hire in the warehouse. That sounds obvious, but it's easy to lose when every document gets pulled into one search index. Connect sources with the asker's access, not with an all-seeing service account.

Then watch it. Each week, look at the questions it couldn't answer, the ones it answered wrongly, and the ones people asked the expert anyway. Each of those is a missing source, a stale document, a question that needs a policy, or a sign that people don't trust it yet. Add to the test set as you go.

The expert's role changes, but doesn't disappear. Their judgment is still the thing the business depends on. They just stop being the search engine for everything else.

Put it into practice

Start with one person and one week. Log their questions, sort them with them, and see what you've got. Even if you never build an assistant, you'll come out with a list of documents to fix and policies to write, which is useful on its own.

If you do build one, that's the kind of work we do in AI applications and agents: following a real request, finding which sources people trust, and building something that shows where its answers came from and knows when to hand off. If you have someone who's become everyone's search engine, tell us about their week.

Related work: PatientIQ.