Blocking AI crawlers doesn't take you out of the answer. It hands the answer to someone else

What the study found

During the final week of Japan's February 8, 2026 election, two researchers asked AI chatbots for voting advice, thousands of times over. Andrew Hall, a professor at Stanford's Graduate School of Business and a senior fellow at the Hoover Institution, and Sho Miyazaki, a visiting researcher at Waseda University, built 36,300 synthetic voter profiles and asked five AI models from three major AI companies which party each should support. Web search was on.

Profiles with left-leaning policy views were steered overwhelmingly toward the Japanese Communist Party, even though other parties held broadly similar positions. According to Stanford's Freeman Spogli Institute, which covered the working paper, the researchers traced the pattern to sources. The party runs a fully open website and a daily newspaper, Akahata, that AI search tools can read freely. Japan's major news outlets, meanwhile, block AI crawlers through robots.txt restrictions, over copyright concerns. The party's site and newspaper ended up among the most-cited sources in the models' recommendations.

The researchers' advice to news organizations is pointed: by restricting access, they may inadvertently cede influence over AI-answered questions to partisan actors, and they may want to consider negotiated access. They also expect political actors to start optimizing their communication for AI.

This is a study about elections. But the mechanism isn't political at all, and it applies to every business with a website.

The mechanism is the same for a product page

When someone asks an AI assistant with web search a question, it reads what it can reach and builds an answer from that. If your pages are blocked, it doesn't say "no comment." It reads your competitors, resellers, review sites, forum threads, and old articles that quote last year's prices.

When you close the door, the assistant doesn't stop answering. It just asks someone else.

That's a decision with real stakes. Customers ask assistants which accounting software suits a small nonprofit, whether a clinic takes their insurance, what a part costs, or which brand of stroller folds smallest. If your own pages, the ones that are accurate and current, are off-limits, the answer gets assembled from whatever is left.

And the study shows something worse. The models couldn't tell editorially independent journalism from content designed to persuade. They treated the party newspaper as a news source. In business terms, a competitor's comparison page or an affiliate's "best of" list can easily become the assistant's main description of your product.

Why businesses block, and why it's often the wrong lever

There are legitimate reasons to block AI crawlers. Publishers worry about their content being used without payment, which is exactly the copyright concern Japanese outlets had. Some businesses don't want their research, pricing models or original writing absorbed for free. Some just see bandwidth bills from aggressive bots.

Those are fair concerns. The trouble is that robots.txt blocking is a blunt tool, and it's often applied to the whole site in one line, copied from a blog post or switched on by a security setting without much thought.

A business usually has two very different kinds of pages. Some pages are the product itself, such as paid research, premium articles or proprietary tools. Others exist to be found: pricing, specs, locations, hours, policies, product descriptions and support answers. Blocking the first kind can make sense. Blocking the second kind mostly means someone else describes you.

We've touched on this before, in Meta's new terms reach past your Page to your website and When Google's AI points to the official page, is it yours? This study adds evidence for the cost of closing everything.

Decide crawler by crawler, page by page

Instead of a single yes or no, make a few separate decisions.

Separate search-style fetching from training. Several AI companies publish different crawler names for different purposes, such as one for training models and another for fetching pages to answer a user's question live. Check each company's current documentation, because names and purposes change. You may decide to refuse training while allowing live answer fetching.

Open the factual pages. Pricing, specs, policies, locations, hours and support content should usually be readable. They're also the pages where a wrong answer costs you most.

Protect what you sell. If the content itself is the product, a block or a negotiated license can make sense. That's the trade the researchers suggest news organizations consider.

Check the result. Ask the assistants your customers use the questions your customers ask. See which sources they cite. If it's never you, find out whether you're blocked, whether your pages are hard to read, or whether the information simply isn't there.

Write it down. Keep a short note of what you allowed, what you blocked and why, next to the robots.txt file itself. The next person who touches that file, or the next security setting that offers to "block all AI bots" with one click, should know a decision was already made on purpose. Revisit it a couple of times a year, because the crawlers, the assistants and your own pages all change.

The caveat, and the argument anyway

This is one study, in one country, about one election, using five models at one point in time. It doesn't prove that any particular business would see the same effect, and it doesn't tell you how many customers ask AI assistants about you. We'd be wary of anyone who claims to know that number for your business without measuring it.

But it shows clearly what assistants do when good sources are out of reach. They don't wait for permission. They build the answer from whatever is open, and they aren't good at telling an independent source from an interested one.

So our argument is simple. Don't block AI crawlers by default. Decide on purpose, page type by page type. Keep the pages that state facts about your business open, accurate and easy to read. Then check what the answers say.

If you want help reviewing your robots.txt, structuring your factual pages so assistants read them correctly and checking what they say about you, our web applications team does that work. Talk to us.

Related work: Sutton Place.