Kustomer AI QA review: every conversation gets a score, so the scorecard had better be right

What Kustomer launched, plainly

On October 14, 2026, Kustomer, the customer service platform, launched two features that are available now: Conversation Intelligence and AI QA. Both are built on what Kustomer calls AI Attributes, fields on a conversation that the AI fills in for you.

Conversation Intelligence turns a conversation into structured data. Classifiers sort conversations into labels you define, Scores rate a conversation against a weighted rubric with a reason, and Extractors pull out or summarize information you describe in plain language. That output can feed routing and workflows.

AI QA is the quality-review side. It scores every closed conversation, whether an AI agent handled it, a person did, or the AI handed off to a person partway through, against one weighted scorecard you write in plain language. You can mark some criteria as auto-fail, for things that are never acceptable. Every score comes with the AI's reasoning next to it. If a manager disagrees, they can override the score by hand, and Kustomer says a corrected value won't be silently overwritten the next time the field runs.

Kustomer's pitch is the gap between that and how most QA works today: a sample of conversations, read by a busy lead, plus survey replies from the small share of customers who answer. Kustomer's own release puts survey coverage at 2 to 3 percent of conversations. Whatever your number is, the point holds for most teams: almost nothing gets read.

Our verdict, up front: if you already run support on Kustomer, and especially if an AI agent is talking to your customers, turn this on. Then spend more time on the scorecard than on the setup.

The case for it

It finally checks the bot. This is the strongest argument. Plenty of shops now have an AI agent answering order questions, and nobody reads what it says unless a customer complains. AI QA scores those conversations with the same rubric as your human team, and a workflow can flag a low score to an admin as soon as it happens. In Kustomer's own example, the reasoning shows the agent skipped confirming a resolution timeline, so the admin can fix the instructions. That's a feedback loop most teams don't have at all.

It reads the whole conversation, handoff included. When the AI starts and a person finishes (a manual refund, say), Kustomer says AI QA scores it as one conversation rather than two pieces. Handoffs are where customers get stuck, so a score that sees the join is worth more than two clean halves.

The reasons are visible. A score with no explanation is a number you either believe or don't. A score with a sentence of reasoning is something a manager can check in a minute and argue with. The override that sticks matters too: people will trust the system more if their corrections hold.

It's already where the conversations are. No second login, no export to another tool. Scores behave like any other data on the conversation, so they show up in reports, routing and Kustomer's Data Explorer.

Kustomer is clear about the limit. Its own post says AI QA doesn't make the call on anyone's performance review; it gives the people making that call a fuller picture. That's the right framing, and it's worth holding them to it.

The case against it

The scorecard becomes the policy. Scoring everything doesn't make the score right. It makes the scorecard matter more. A vague criterion like "showed empathy," scored on every conversation, turns into a thousand small judgments nobody wrote down. A criterion that's slightly wrong, such as rewarding fast closes when your policy wants a confirmed fix, now nudges the whole team the wrong way, at scale, with a number attached.

Auto-fail is powerful and blunt. Auto-fail is the right tool for "never share card details" or "never promise a refund outside policy." It's the wrong tool for anything that needs context. Keep the list short.

People will read the number before the reason. Reasoning next to every score only helps if someone reads it. On a busy Monday, a dashboard of low scores gets skimmed, and a rep who handled a genuinely hard customer well can look bad.

It only exists inside Kustomer. This isn't a QA tool you can point at another help desk. If you're not on Kustomer, it's not an option, and it's no reason by itself to switch platforms.

Pricing isn't public. Kustomer's pricing page asks you to fill in an assessment and then talk to someone for a quote, and existing customers are told to ask their Customer Success Manager about rollout. Budget for a conversation, not a line item.

No independent accuracy numbers yet. The launch materials include customer quotes about better categorization, but no published accuracy figures for QA scoring. Your own spot checks are the evidence that counts.

What to do before you switch it on

Write the scorecard as if a new hire will be judged by it, because effectively someone will be. Five or six criteria are plenty. For each, write what a pass looks like in one sentence, using your actual policies: the refund window, the identity check, what counts as resolved. Weight the ones that cost money or trust.

Then calibrate. Take twenty recent conversations, have your best lead score them by hand, and compare with AI QA's scores and reasons. Where they disagree, decide who was right and fix the wording, not the person. Repeat after any policy change.

Keep auto-fail for things that are never acceptable. Decide who reads low AI-agent scores and how fast, and who may override a human rep's score and whether that is logged with a reason.

Who should use it, and the alternatives

Use it if you're on Kustomer, you handle enough conversations that nobody reads most of them, and especially if an AI agent answers customers on your behalf.

Wait if your criteria aren't written down yet. Write them first; the tool will still be there.

Not for you if you're on another help desk. Your options there are a separate QA product that connects to your help desk, or a small scoring job built around your own export: the same idea, with criteria, reasons and overrides you control, and more for you to maintain.

Put this into practice

Pick one criterion your team argues about. Write what a pass looks like in a sentence. Score ten conversations against it by hand this week. If two leads disagree on half of them, that's the work to do before any software scores the rest.

For stores, the money questions tend to be refunds, delivery promises and order changes. We build ecommerce systems where support, orders and policy agree, including the checks on what an AI agent tells your customers. If you want help writing a scorecard that's fair to your team and your bot, talk to us.

Related work: Sutton|Place.