twimco.Let’s talk ↗︎
Menu
Field notes / AI & judgment

North Dakota's chatbot answered 81% of questions. The other 19% is where the next decision lives

Dakota BOT moved from one department's website to ND.gov this month, on the strength of answering 81% of nearly 600 questions. That's a respectable start. Whether it's ready for every agency depends on what 'answered' meant, and on what happened to everyone else.

Editorial engraving of a bright state information hall where an officer reads slips from a small pile while a visitor uses a kiosk by prairie windows.

North Dakota expanded Dakota BOT, its AI chatbot, from the Department of Commerce website to ND.gov in early November 2026. The state says that since May the bot fielded nearly 600 questions across more than 400 sessions and successfully answered 81%. Before any public chatbot widens its reach, read the unanswered questions and a sample of the answered ones, define what counts as answered, and check whether the new audience asks different things than the pilot's.

What North Dakota announced

In early November, North Dakota's Department of Commerce and its state IT agency, NDIT, put Dakota BOT on ND.gov, the front page of state government. It's an AI chatbot meant to help residents, businesses and visitors find information and get to the right service.

The bot isn't new. It launched on the Commerce department's own website in May 2026. According to the state, it has since handled nearly 600 questions across more than 400 conversations and "successfully answered 81% of the questions asked." It uses structured workflows and frequently asked questions, was built with input from agencies across government, and drew on North Dakota Tourism's experience running an earlier website chatbot.

The release also says the state will keep evaluating its performance as it expands. That's the right instinct, and it's where this note starts.

What 81% does and doesn't tell you

Eighty-one percent is a respectable figure for a young chatbot built on curated content. It's also a single number covering several different things, and the release doesn't define it. That's normal for a press release. It just means anyone planning a similar rollout should ask what sits underneath.

What counted as answered? It could mean the bot gave a reply on the right topic. It could mean someone checked the reply was correct. It could mean the person didn't ask again. Those produce very different numbers from the same conversations.

What happened to the other 19%? Roughly a hundred questions weren't answered. Some may have been off topic or impossible. Some may be the most important questions the state receives: a deadline, a benefit, a license problem. A bot that answers the easy 81% and fails on the urgent 19% has a different value from one that misses a random fifth.

Was anything answered wrong? A confident wrong answer about a filing date does more harm than "I don't know, here's who to call."

How big is the sample? Nearly 600 questions over about six months is a modest volume, which is fine for a pilot. It means a handful of repeated questions can move the percentage several points either way, and it means the full set is small enough for people to read.

A success rate tells you how often the bot said something. Only the transcripts tell you whether it should have.

Decision guide

Two ways to read the same 81%

ConsiderationWhat the number can meanWhat to check before trusting it
AnsweredThe bot replied with something on topicThe reply was correct, current and pointed to the right agency
Not answeredThe bot said it didn't knowWhether the person found help elsewhere, or gave up
QuestionsNearly 600 from one department's visitorsWhat a statewide audience asks: licenses, benefits, deadlines, complaints
SessionsMore than 400 conversationsHow many ended with the person leaving to call or email anyway
Decision framework for reading a chatbot's success rate; example questions are illustrative, not from Dakota BOT's logs.
A success rate tells you how often the bot said something. Only the transcripts tell you whether it should have.

Why the move to ND.gov changes the test

The pilot ran on the Commerce department's site. Its visitors were mostly asking about the things Commerce does. ND.gov is the front door to everything: taxes, vehicles, health programs, licensing, emergencies, elections. The questions will be broader, more urgent and often owned by an agency whose content wasn't in the original FAQs.

The release says Dakota BOT was developed with input from agencies across state government, which helps. But a statewide audience will find the gaps quickly, and many of them will be questions where the honest answer is "this depends on your situation, talk to someone."

That's not a reason to hold back. It's a reason to treat the first weeks on ND.gov as a new pilot, with its own measures, rather than as a scaled-up version of the old one.

We've seen the same pattern elsewhere. When a county's new AI assistant surfaced a wrong phone number from an old press release, the trouble wasn't the AI. It was content nobody had retired. And a city assistant that speaks 70 languages still needs someone who reads the answers.

What to read before going wider

None of this requires a large evaluation project. It needs an afternoon with the transcripts and someone from each major agency.

  1. Read every unanswered question. At the pilot's scale, that's about a hundred. Group them by topic and by which agency owns the answer.
  2. Read a random sample of the answered ones. Fifty is enough to spot patterns. Mark each as right, wrong, out of date, or "should have sent this person to a human."
  3. Write down the definition. Agree on what "answered" means from now on, so next quarter's figure can be compared with this one.
  4. Decide what stays off the bot. Some topics, such as eligibility decisions, legal deadlines and anything about someone's specific case, may be better handled with a clear handoff to a person or a page than with a generated answer.
  5. Add a way out. Make sure every conversation can end with a phone number, a form or an office, and count how often people use it.
  6. Name an owner per topic. When an answer is wrong, someone at the right agency has to fix the source content, not just the bot's reply.

How the work moves

An afternoon with the transcripts

  1. Pull

    Every unanswered question and a random 50 of the answered ones.

  2. Sort

    Group by topic and by the agency that actually owns the answer.

  3. Mark

    Right, wrong, out of date, or should-have-sent-to-a-person.

  4. Decide

    Fix content, add a handoff, or keep a topic off the bot for now.

Illustrative review sequence for any public chatbot before it widens its reach.

How to report it next time

The most useful thing North Dakota can publish next isn't a bigger number. It's a split one. Show how many questions were answered correctly, how many were handed to a person, and how many went unanswered, with the top topics in each group. Add what changed as a result: new FAQs written, pages fixed, topics routed elsewhere.

That kind of report helps agencies decide where to invest, helps residents trust the tool, and gives other states something real to learn from. It also turns the 19% from a weakness into the bot's best source of improvement.

Agencies can help by treating the chatbot's unanswered questions as a free survey of what residents can't find. If the same question about a renewal date comes up every week, the fix might not be a better bot. It might be a clearer page, a renamed form or a phone tree that finally matches how people describe their problem. Some of the best results from a public chatbot show up on the website around it.

And keep the people who answer phones in the loop. They hear the questions the bot misses, and they'll be the first to notice when a wrong answer starts sending people their way.

If your agency or department is planning a public chatbot, or widening one, and wants help setting up the content, handoffs and review that keep it trustworthy, see our government and public sector work or contact us.

Related work: Sutton|Place

twimco.

We build alongside the people doing the work.
Bring us the problem you’re thinking about.

Related work

Sutton Place ↗︎