Score the job before you buy the model.

The slide and the ticket

The procurement deck is open on one monitor. A ranking chart. Green cells. Someone has circled a model name in red and written ship this. On the other monitor sits a real ticket: a front-end regression with a half-broken layout, a flaky auth redirect, and a comment thread that already ate two hours of a senior engineer’s afternoon. Nobody in the room has run the candidate model against that ticket. They have run the slide.

That gap is the buy decision.

October 2026 coverage of Google’s Gemini 4 / Gemini 4 Argon moment is useful as a hook, not as a verdict. Public reporting and internal voices describe uneven real-world coding—especially front-end work—against strong public benchmarks. Experts reach for the word benchmaxxing: optimizing for leaderboard scores rather than the buyer’s job. Google disputes underperformance. Treat the headlines as weather. The operating question does not depend on which vendor wins the next press cycle.

Before you buy tokens, switch a default router, or point an agent runtime at a frontier model for a production coding, support, or tool-using job, do you have acceptance on your tasks—or only confidence borrowed from someone else’s scoreboard?

When the work is choosing and wiring models into systems people actually operate, start from AI applications & agents.

Leaderboards measure a contest, not your job

A public benchmark answers a narrow question: how did this model do on that suite, under those rules, on that day. It does not answer whether the model will land a clean patch on your design system, keep your support macros inside policy, or finish a multi-step agent job without inventing a tool call.

That distinction is easy to lose in a buying meeting. Ranking tables are shareable. Job-shaped failures are awkward. So the room drifts toward the shareable artifact.

You do not need TWIMCO numbers to see the pattern in the news. Third-party write-ups of Argon’s launch tables have already shown the shape: strong marks on some suites, and publicly reported trailing scores on others—FrontierSWE and Terminal-bench among the ones DataCamp-style press tables have printed—while Google and others dispute what “underperformance” means in the wild. Cite those tables as published third-party context if you need a slide. Do not treat them as your acceptance pack. They were not run on your fixtures, your latency budget, or your fail-closed rules.

Benchmaxxing is not a conspiracy theory. It is what happens when the score becomes the product. Vendors and labs optimize for the suites buyers screenshot. Buyers then confuse the screenshot for a hiring decision.

This note is about the other decision: money and traffic moving to a model. Not how to ban shortcuts on a contest scoreboard. Not how to hold out harness memorization. Not how to freeze agent fixtures. Those are eval-integrity and harness problems. This is the purchase and routing gate.

What the gate forces you to name

Fill one page per production job before you commit spend or default routing. Later you can keep the fields on a Model Job Acceptance Gate if that helps the team; the names matter more than the stationery.

Target job id. Stable name: coding.frontend.ticket-patch.v3, support.triage.macros-12, agent.ops.reconcile-status. If the only name is “Gemini for eng,” you cannot refuse a bad promote later.

Must-pass task pack. Your tickets, fixtures, and traces—not the vendor suite. Minimum: a handful of real recent failures, one happy path, and the messy cases that already cost you people. Front-end layout bugs, auth redirects, partial diffs, and “looks fine in chat / wrong in the repo” belong here if that is the job.

Fail-closed criteria. What must be true to pass: patch applies clean, tests you named stay green, no invented files outside scope, support answer cites allowed sources, agent tool calls stay inside the permission set. Soft “feels better than last week” is not a criterion.

Cost and latency budget. Token, dollar, and wall-clock limits for the pack and for a typical production call. A model that “wins” by burning three times your budget is not a win for the buyer.

Owner who may promote routing. Named role who can move traffic or sign the buy after the pack passes—and who can roll it back. Not “the AI channel.”

Refuse-to-buy-on-leaderboard-alone rule. Explicit: a ranking table, press chart, or vendor demo is insufficient for promote. Write it so procurement and eng share the same refusal.

Planted hard cases. Cases that must fail or pass as designed: a ticket that should refuse an unsafe change, a support case that must escalate, a coding task with a known wrong “clever” fix the model loves. If planted fails still go green, you do not have a gate. You have theater.

Run the pack under the same tool and context limits you will use in production. A chat playground with unlimited paste is not the job.

How buy-on-leaderboard goes wrong

Composites—not defect rates, not TWIMCO benchmark scores.

The screenshot commit. A ranking cell becomes the decision memo. Nobody attaches the must-pass pack. Three weeks later front-end tickets still bounce to humans, and the “win” is defended with a different suite.

The category swap. The model looked strong on a coding board that is mostly backend reasoning. Your pain is CSS, accessibility, and design-token diffs. The buy was for a job you never scored.

The unlimited playground. Acceptance was a Friday afternoon chat where someone pasted a clean repro. Production is a constrained agent with half the context and a cost ceiling. The playground lied politely.

The missing refuse rule. Procurement asked for “best model.” Eng shrugged at a chart. Nobody owned a written refusal to buy on leaderboard alone, so the chart won by default.

The unplanted hard case. Every demo ticket was one the model already liked. The unsafe refactor and the “please escalate” support case were never in the pack. They arrived in production first.

The orphan promote. Traffic flipped because a champion liked the vibe. No named owner, no rollback owner, no re-run of the pack after a model revision.

When you see these, do not add another fairness slide about industry benchmarks. Close the gate: job id, your pack, fail-closed lines, budgets, promote owner, refuse-on-leaderboard rule, planted cases that behave as designed.

If the gap is the durable application layer around that model—routing, permissions, operator recovery—pair the gate with a scoping conversation for web applications as well as AI applications & agents.

Acceptance is not eval integrity

Keep the layers honest so reviews stay short.

Shortcut bans answer: which technically available moves void a score on a tool-using eval. Needed when agents can borrow answers. They do not decide which model you buy.

Harness holdouts answer: whether a self-improving loop memorized the suite. Needed when the harness itself can overfit. They do not decide production routing.

Fixture packs answer: which frozen agent cases you re-run before you promote an agent build. Needed for release discipline. They are not the same as a buyer’s must-pass pack for model selection—though your acceptance tasks can reuse fixtures when the job overlaps.

Job acceptance answers: whether this model may receive spend and traffic for this named job under budgets and fail-closed rules you own.

You can ban shortcuts, hold out harness data, and freeze fixtures, and still buy on a leaderboard. The missing control is the one that says: no promote until our tickets pass under our limits.

A skeptical buyer does not need to settle the Gemini coding debate. They need a pack that would have caught a mismatch before the purchase memo went out.

Put this into practice

1. Pick one production job where a frontier model is about to get money or default traffic—coding assist, support draft, or agent tool use.
2. Write the target job id and the refuse-to-buy-on-leaderboard-alone rule in the same doc procurement and eng will see.
3. Build the must-pass task pack from your tickets and fixtures. Add planted hard cases that must fail or pass as designed.
4. Set fail-closed criteria plus cost and latency budgets. Run under production-like tools and context—not an unlimited playground.
5. Name the owner who may promote routing and who may roll it back. Refuse promote until the pack passes under those rules.
6. Re-run the pack when the vendor ships a new checkpoint or when your job shape changes. A last-quarter chart is not a living gate.
7. If the gap is models and agents you can defend in production—not another ranking screenshot—bring the filled fields into a scoping conversation for AI applications & agents.

A leaderboard is a contest result. A job-shaped pack with fail-closed criteria, budgets, planted cases, and a named promote owner is a buying control. Score the job before you buy the model.

Related work: Sutton|Place.