Shopify's AI shoppers will now test your storefront for you. The case for listening, and the case against

What Shopify shipped on October 28

Two related changes landed in Shopify's changelog on October 28.

First, Sidekick, the assistant built into the Shopify admin, can now spot a possible improvement to your storefront, build a draft version of it, and compare that draft with your current store using AI-simulated shoppers. It picks changes that can be tried as theme changes and are expected to affect add-to-cart behavior. You review the suggestion before deciding whether to apply it.

Second, SimGym, Shopify's simulation tool, can now run more than theme comparisons. It can send AI shoppers through entire shopping journeys to find friction, test product discovery, which checks whether browsing leads them to what they're looking for, and test add-to-cart confidence, which checks whether your product details give them enough to buy. Recommendations come with numbered markers on the page, a summary of each simulated visit and playback video.

Shopify's own changelog includes a sentence worth reading twice: these recommendations "are not tested on real shoppers and are not live A/B tests."

That's an honest caveat, and it raises the right question for a small store. How much should you trust a shopper who doesn't exist? Here's the best case for listening, then the best case against.

The case for listening

It finds problems you've stopped seeing. Every owner gets used to their own store. You know the size chart is under the second tab and that "Bundle & Save" means the three-pack. A simulated shopper doesn't. If it can't find the returns policy or doesn't understand which variant it picked, that's a useful signal about clarity, and it costs you nothing to hear it.

It works without traffic. A real A/B test needs enough visitors and orders to tell two versions apart. Many small stores don't have that, at least not quickly. A simulation runs whenever you ask, so you get feedback on a product page that sells a few times a week.

It shows its work. Markers on the page, visit summaries and playback mean you're not asked to accept a score. You can watch where the simulated shopper hesitated and decide whether you agree. That's a better way to use AI advice than a single "conversion likely to improve" number.

It's cheap to ignore. Nothing is applied until you approve it. A recommendation you disagree with costs you a minute of reading, which is a fair price for a second pair of eyes on a page you haven't looked at properly in a year.

It's aimed at the right moment. Add-to-cart is where product information either does its job or doesn't. Missing dimensions, unclear shipping costs and vague materials are common, fixable reasons people leave. A tool that checks for them is checking the right thing.

The case against

It isn't your customer. A simulated shopper is a model's idea of a shopper. It doesn't have your customer's budget, taste, loyalty or reasons for coming. A simulated shopper can tell you where the page is confusing. It can't tell you whether anyone wanted the product.

It can prefer what a model likes. AI shoppers read pages the way AI reads pages. They may reward long, explicit product descriptions and penalise a short, confident page that your real customers love because it feels like a boutique. If you follow every recommendation, you may end up with a store that's easier for a machine and blander for a person.

"Expected to affect add-to-cart" isn't "will sell more." Add-to-cart is an early step. A change that gets more people to add an item can also bring more abandoned carts, more returns or lower order values. The simulation stops before any of that happens.

It's easy to mistake for evidence. The playback looks like a real session. When someone on your team reports that a change "tested well," they may mean that a simulation suggested it. That gap matters when you're deciding whether to change a page that's selling well today. And it matters more in the weeks before Black Friday, when there's pressure to do something and no time to undo a change that quietly hurt sales.

Where the two cases agree

Put side by side, the arguments don't really conflict. Both say a simulation is good at spotting friction: missing information, confusing layout, steps that don't make sense. Both say it can't measure demand or prove that a change makes money.

So use it as a reviewer, not a judge. A good human usability review works the same way: five people try your page and you watch where they get stuck. You don't conclude that sales will rise by a certain amount. You conclude that the shipping cost is hidden and you should fix it.

A working rule for small stores

Here's an illustrative rule for a store selling a few hundred orders a month.

Apply without debate when the recommendation fixes missing or confusing facts: dimensions, materials, care instructions, shipping costs, delivery dates, returns. Those are good changes whether a simulation found them or a customer emailed about them.

Think before applying when it changes tone, layout or design choices you made deliberately. Ask whether a real customer has ever complained about this. If not, the simulation may just prefer a different store.

Test on real shoppers when it touches price, offers, or your best-selling page. If you have the traffic, run a real test. If you don't, change one thing at a time and compare orders, not add-to-carts, over the following weeks.

We wrote about putting a holiday offer through a real cart before it shows up in AI search, and about how AI agents found the page you forgot. Both come back to the same point: what real visitors and real orders do decides.

What to do with the next recommendation

When Sidekick or SimGym suggests a change, write down three things: what it found, which of the three rules above it falls under, and what real number you'll check afterwards. Keep a short log. After a couple of months you'll know which kinds of recommendation help your store, and you'll trust the tool for the right things.

If you'd like help deciding which storefront changes are worth making, setting up real tests, or making product pages clearer for people and for AI, that's the work our ecommerce team does. Talk to us and we'll go through your store with you.

Related work: Spin Market.