twimco.Let’s talk ↗︎
Menu
Field notes / AI & judgment

Do you need a model, or a rule nobody wrote down?

A lot of "we need AI for this" turns out to be a rule nobody has written down yet. Here's how to tell which one you're looking at, and what it looks like when a model really earns its place.

Editorial engraving of a bright video studio split in two: a producer pins a single index card beside a simple lever switch, while a young editor studies a monitor of many colored rhythm bands representing different channels' viewing patterns.

If you can write the decision on an index card and the people who make it would agree, it's a rule: automate it with rules. Reach for a model when inputs are messy language or images, when the right answer depends on patterns that differ by context and shift over time, when you have history to learn from, and when a wrong answer is recoverable. Most good systems use rules first and a model for what's left.

How do I tell whether this needs AI at all?

Try the index card test. Ask the people who make the decision today to write down how they make it, on one card. "If the order is over this amount and the customer is new, hold it for review." "If the request mentions a refund, send it to billing." "If the item is out of stock in the main warehouse, check the second one."

If they can write it, and the other people who make the same decision agree with the card, you don't need a model. You need the card. Build it as a rule, test it, and give it an owner. It'll be cheaper to run, easier to explain, and it'll do the same thing every time, which is usually what you want.

We say this on our business process automation page too: AI helps where interpretation is useful, and straightforward rules handle straightforward work. A surprising number of "AI projects" are really this, a decision that lived in someone's head and never got written down. Writing it down is the project.

What does a job that really needs a model look like?

There are a few signs, and the more of them you see, the stronger the case.

The input is messy. Free-text emails, photos, scanned documents, voice notes. Rules struggle when the same thing can be said a hundred ways.

The right answer depends on patterns that vary by context. What's normal for one customer, channel, or region isn't normal for another, and the people deciding are reading signals they can't fully list.

The patterns shift. What worked last quarter doesn't quite work now, and someone keeps adjusting the rules by hand.

You have history to learn from. Past decisions and their outcomes, recorded well enough that a model, or a person checking it, can tell good from bad.

A wrong answer is recoverable. Someone can catch it, or it costs a little rather than a lot.

If the input is clean, the pattern is stable, and a mistake is expensive, lean hard toward rules, even if a model could technically do it.

What about today's language models? They do lower the cost of trying a model, a lot. You can prompt one to sort emails in an afternoon, where training a classifier used to be a project. But cheap to try isn't the same as right for the job. A prompted model is still less predictable than a rule, it can still be confidently wrong, and it still needs a test set and an owner. The bar for "use a model" has moved. The questions above haven't.

The opposite mistake happens too. Sometimes a team spends months maintaining hundreds of rules for something that's really a reading task, like routing free-text requests, and every new phrasing means another rule. That's a sign the job needed interpretation all along.

Decision guide

Write a rule, or bring in a model

ConsiderationWrite a ruleBring in a model
InputClean fields: amounts, codes, dates.Messy text, images, documents, or voice.
PatternStable and easy to state.Varies by context and shifts over time.
HistoryNot needed; the rule is the knowledge.Past decisions and outcomes recorded well enough to learn from.
Cost of a mistakeHigh: money, safety, or a promise.Recoverable: caught, measured, and improved.
Explaining itAnyone can read the card.Needs evidence, review, and limits around it.
Decision framework: assess these conditions against your own decision.
If you can write it on an index card, you don't need a model. You need the card.

What does that look like in a real build?

Howcast is a good example of a job where a model earned its place. Across eleven channels, viewers had different habits. Alongside the systems that processed and delivered video around the world, we built machine-learning tools to schedule publishing around when each channel's audience was most likely to watch.

Here's why that fits the signs above, in our own framing. A single rule, like "publish at 9 a.m.," would have ignored the fact that each channel's audience behaved differently. The patterns depended on context and could change. There was viewing history to learn from. And the decision was about timing, where a less-than-perfect choice could be measured and improved rather than causing lasting harm.

Compare that with deciding whether a refund is allowed. That's a policy. You want it written down, applied the same way every time, and easy to explain to a customer. A model would make it less predictable without making it better.

Can I use both?

Usually, yes, and that's often the best design.

Rules first, model for what's left. Let rules handle the clear cases. Send only the ambiguous remainder to a model, or to a person. You get predictability where it's possible and flexibility where it's needed.

Rules around the model. Even when a model makes the call, set hard limits it can't cross. Never schedule outside these hours. Never approve above this amount. Always send this category to a person. The model works inside the fence.

Model to suggest, rule or person to decide. For anything consequential, the model can draft, sort, or recommend, and a rule or a named person makes the final call.

This combination is also easier to trust. When something goes wrong, you can tell whether it was the rule, the model, or the handoff between them.

How the work moves

Rules first, model for what's left

  1. Rules take the clear cases

    Written, tested, and owned.

  2. Model takes the ambiguous rest

    Only where interpretation is genuinely needed.

  3. Hard limits around the model

    Amounts, hours, and categories it can't cross.

  4. People take the consequential calls

    With the model's suggestion and evidence attached.

Illustrative design pattern, not a client system or measured result.

What should I find out before choosing?

A few things you can learn from your own work, without building anything.

How many cases would a written rule cover cleanly? Take a sample of real cases and try the index card against them. If it handles most of them, start there.

How often do people override the current process? Frequent overrides on a rule usually mean a missing rule, not a need for AI. Frequent overrides that nobody can explain are a better sign that interpretation is involved.

What does a wrong answer cost? Name one bad outcome in plain terms. If it's a mild inconvenience someone can fix, a model's occasional mistakes are tolerable. If it's money, safety, or a customer promise, keep rules and people in charge.

What do you have to learn from? If past decisions and outcomes aren't recorded, a model has nothing to stand on, and the first project is capturing that history.

Where should we start?

With one decision your team makes over and over. Write the index card with the people who make it. Test it against a few weeks of real cases. See what it misses, and look hard at those misses: are they missing rules, or genuine judgment?

That's the conversation we have at the start of most business process automation and AI applications work, because the answer decides which one you're actually buying. If your team is about to start an AI project, bring us the decision, and we'll try the index card first.

twimco.

We build alongside the people doing the work.
Bring us the problem you’re thinking about.

Related work

Howcast ↗︎