Measure operational AI by the work, not the demo.
Token counts and thumbs-up ratings flatter a demo. For an agent doing real work, ask plainer things. Did the record show the job finished? What happened when it stopped halfway? Have its permissions quietly grown?
Skip vanity model scores. For operational AI, measure task-completion evidence in the system of record, recovery when work stops halfway, exception share handled inside the design, and permission drift over time. Use a measurement card with definitions—not invented industry statistics.
Skip the vanity score
Token counts, latency by itself, thumbs-up on the prose, a leaderboard that has nothing to do with your records. Those flatter a demo. They do not tell you whether the work got done.
For an agent that operates a system, measure the work. Did the record show completion. What happened when the job stopped halfway. How many exceptions stayed inside the design. Whether its permissions drifted.
Four measures, one job
Do this per job, not for "AI" as a department.
Completion evidence. Of the attempts in the window, how many left proof in the system of record that the job finished. A confident message is not proof.
Recovery. When work stopped halfway, how often it was recovered without a duplicate side effect. Count the unsafe retries too. They are the number that matters.
Exception share. Of the awkward cases, how many went through the path you designed, versus a side channel. The side channel is an estimate. Label it as an estimate.
Permission drift. What the agent was allowed to do at the start of the window, what it is allowed to do now, and which of those changes have a recorded approval.
Leave blanks rather than inventing industry averages. Do not average cards across companies and call it a study. The discipline is filling the card from your systems.
How the work moves
Four measures on one job
- Completion evidence
End state in the system of record.
- Recovery
Interrupted work reaches a safe known state.
- Exception share
Designed path vs side channel.
- Permission drift
Runtime agency vs approved matrix.
A fluent answer is not evidence that the job finished.
The card
Job name. The system of record for done. The window. Attempts. Completions with evidence. Interruptions. Safe recoveries. Exceptions inside the design. Exceptions in a side channel, marked as an estimate. Permission changes. Which of those had an approval. Notes.
One card per job. A single score that mixes low-risk drafts with money movement will hide the only failures that matter.
Meaningful versus flattering
Flattering measures move when the demo moves. Meaningful measures stay boring: record-level completion, safe recovery, designed exceptions, permission integrity.
They are slower to celebrate and easier to steer by. If leadership wants one number, offer the four small ones with owners instead.
Decision guide
Meaningful vs vanity
| Consideration | Meaningful | Vanity |
|---|---|---|
| Success | Record shows expected state. | Reply sounded finished. |
| Failure | Recovery path tested. | Error message polished. |
| Volume | Exception share visible. | Only happy-path count. |
| Change | Permission edits reviewed. | Prompt tweaks undocumented. |
A monthly review
Bring the filled card, not a model bake-off. Ask how completion will be inspected in your records. Refuse a dashboard that cannot show recovery and permission changes.
Prefer a narrow job you can measure over a wide assistant you can only admire. When a definition changes, write the date on the card. Otherwise you will argue from memory about whether the month got better.
Put this into practice
Fill one card from your systems this month. Label every estimate. Bring it to the conversation about whether the agent should be allowed to do more.
That work sits under AI applications and agents.
Related work: Spin Market.
We build alongside the people doing the work.
Bring us the problem you’re thinking about.