You cannot promote an agent on a demo transcript.

The demo is not the eval

Someone pastes a chat. The agent sounds careful. It calls the right tools in the screenshot. Leadership asks when it can run without a person watching every turn. The honest answer depends on whether you can re-run the same job tomorrow with the same expectations—and whether "looks careful" is still the bar when the model, the prompt, or the tool schema changes.

A demo transcript is a performance under friendly conditions. An evaluation is a fixed set of cases you can score the same way twice. If you cannot re-run the case, you cannot tell whether a change improved the agent or only improved the story.

This note is a walk-through of the TWIMCO Agent Eval Fixture Pack: one pack for one agent job, filled before you promote permissions, audience, or autonomy. There is no live pack generator to point at. Copy the fields. Fill them with the people who own the tools and the people who own the exceptions.

On work shaped like Sutton|Place, the operating distinction is that a supplied claim is not a checked fact. Treat agent output the same way. A fluent answer is supplied. A fixture result—tools called, state changed or not, human gate recorded—is checked. If the buy or build is really about agents that act inside real workflows, the scoping home is AI applications and agents.

Bring one job the agent is supposed to own. Not "customer support." A job with a trigger, tools it may touch, and a state that must be true afterward. If the demo cannot be turned into that job, the pack stays blank on purpose. Blank is a finding.

Walk the pack field by field

Fill one pack per agent job. An agent with five jobs does not get one vague pack. Start with the job you are about to promote.

Job. Operator language and the tool names the agent is allowed to call. If the agent "helps with orders," split create, update, refund-request, and status-read into separate packs when the authority differs.

Fixture id. A stable name for the case. Not the chat title from the demo. Something you can find in six months.

Input. The fixed trigger: message, record snapshot, or event. Freeze PII. Use realistic shapes. A toy string that never appears in production is a toy fixture.

Allowed tools. The list the agent may call for this job. Everything else is out of scope for the pack even if the model knows the name.

Expected tool trace. The calls that should happen, in order if order matters, including arguments that must match. Soft matching rules belong on the pack—"any id that exists in the fixture store"—not in a reviewer's head.

Forbidden actions. Writes, disclosures, or escalations the agent must not perform even when the user asks nicely. If you cannot write a forbidden line, you are not ready to promote.

Expected end state. What the system of record must show after a pass: fields set, fields unchanged, tickets opened. A green chat is not an end state.

Human gate. The role who may override, approve, or reject before a side effect that cannot be cheaply undone. "A person watches sometimes" is not a gate. Name the role and the moment.

Score rule. Pass, fail, and needs-review—written so two reviewers would mark the same run the same way. Needs-review is not a soft pass. It is a queue with an owner.

That field set is the TWIMCO Agent Eval Fixture Pack. Another studio can tell you to "eval your agents." They do not have this field set, your frozen inputs, or your forbidden list tied to your tools.

Run the pack like a gate, not a vibe check

Walk a promote decision as a short timeline.

Assemble. Ten to thirty fixtures for the job you are promoting—not one happy path. Include at least one refusal, one missing-field case, and one case where the user asks for a forbidden write.

Freeze. Inputs and expected traces live in version control or an equivalent you can diff. A spreadsheet that someone edits after a bad run is not a freeze.

Run. Same model pin, same tool schema, same prompt bundle. Record the actual tool trace and end state. If the platform will not export that, you do not have an eval—you have a chat log.

Score. Apply the pack's pass rule. Two scorers when the job can spend money, change access, or disclose. Keep the lower mark until they re-run together.

Promote rule. Widen permissions only when the pack passes at the bar you wrote before the run—not the bar you negotiated after seeing the transcript. A demo that impressed executives does not rewrite the forbidden list.

Regress. Re-run the pack when the model, prompt, tools, or record schema change. If you skip the re-run, you silently promoted a new agent under the old name.

Prefer fixtures that hurt: the refund ask, the "just this once" override, the partial record. Happy-path-only packs train you to trust fluency. Fluency is cheap. Wrong writes are not.

Demo habits that fake readiness

Composites from delivery work—not client telemetry and not a measured accuracy claim.

The curated transcript. Someone deleted the turns where the agent called the wrong tool. The pack would have kept those turns as fixtures.

The pristine tenant. Every id resolves. Every field is present. Production is messier. Fixtures that never miss a field do not test the escalate path.

The moving prompt. The team "improved" the system prompt between the demo and the review. Without a freeze, you evaluated two agents and reported one.

The missing forbidden list. The agent refused once in the demo. Nobody wrote what it must always refuse. Next week a user asks differently and the write lands.

The human who was in the loop by accident. A reviewer fixed the state by hand and called it a pass. The pack's end state was never checked. The gate was theater.

When you see these, do not add a softer metric. Close the pack: fixed input, expected trace, forbidden actions, end state, human gate. The unreproducible asset is your filled TWIMCO Agent Eval Fixture Pack—your cases, your tool constraints, your score rule—plus one anonymized fail that taught a forbidden line. A blog post that says "we eval continuously" does not name the fixture. Yours does.

Put this into practice

1. Pick one agent job you are about to promote. Put that name on the TWIMCO Agent Eval Fixture Pack.
2. Write ten fixtures minimum: happy path, refusal, missing field, and one forbidden ask.
3. Freeze inputs and expected traces. Run once. Score with the written rule. Keep the lower mark if two scorers disagree.
4. Promote only against the bar you set before the run. Re-run on every model, prompt, or tool change that could change behavior.
5. If the gap is agents that must act inside real workflows with inspectable checks—not demo charm—bring the pack into a scoping conversation for AI applications and agents. Ground the standard in Sutton|Place: supplied fluency stays distinct from checked results.

A demo transcript can sell a meeting. It cannot certify a promote.