twimco.Let’s talk ↗︎
Menu
Field notes / Product & engineering

Moving your site to a CDN before the holidays? Warm the cache and let Googlebot in before you flip DNS

Plenty of teams put their site behind a CDN in November so it survives the holiday rush. Google has just explained why that first week can go wrong: crawlers speed up, the cache is still empty, and someone panics and blocks every bot. Here's how to do the switch without losing your pages or products from Search.

Editorial engraving of a sunny engineering office where an engineer watches rising charts and signals to wait while a colleague reads from a printed runbook.

Google said on November 11, 2026 that moving a site to a CDN can briefly flood the origin server, because faster responses encourage Googlebot to crawl harder while the CDN's cache is still empty. Before switching, warm the cache with your busiest, upcoming-promotion and most-crawled URLs, and let verified crawlers past bot protection. If the origin still struggles, return 429 or 503 for a few hours, at most a day or two, rather than blocking with 403s or challenge pages.

Why this comes up in November

A lot of sites move behind a content delivery network in the weeks before the holidays. It's a sensible upgrade: the CDN serves copies of pages and images from servers near each visitor, and it's supposed to shield your own server, the origin, from heavy traffic.

On November 11, Google's Search Central blog published a post by Cherry Prommawin and Gary Illyes, "Moving to a CDN without overwhelming your origin (or blocking Googlebot)," with a video to go with it. Its point is that the first days after the switch can produce exactly the problem the CDN was meant to fix, and that the instinctive reaction makes it worse.

Before the runbook, one honest question: does the switch have to happen now? If your site coped last December and the CDN is a nice-to-have, moving it in January is the calmer choice. If you're switching because the origin already struggles under load, or because a promotion is about to send far more traffic than last year, then do it, but do it early enough that the first week isn't Black Friday week.

What goes wrong

Google's crawlers watch how quickly a site responds. When responses slow down, they back off. When responses are fast and steady, Google's systems treat that as healthy infrastructure and, if there's demand, gradually crawl more.

Point your DNS at a fast CDN and response times drop overnight. In Google's words, "it looks like your server just got a massive hardware upgrade," so crawlers start asking for more.

But a new CDN starts with an empty cache. The first request for any page at any edge location has to go back to your origin. Human visitors mostly hit a small set of popular pages, which warm up quickly. Crawlers do the opposite, working steadily through your whole catalog of URLs. So the faster crawl lands on pages the CDN hasn't stored yet, and every one of those requests reaches the origin anyway. A CDN with an empty cache is a faster road to the same old server.

Google adds that this doesn't happen in a millisecond, and that many CDNs pace connections to smooth it out. Still, until the cache is warm, your origin is doing the heavy lifting.

Sequence of work

A CDN switch, from the week before to day three

  1. Week before

    Turn on the CDN's verified-bot allowance; list the URLs to warm; agree who may change security rules.

  2. Day before

    Warm the cache at a steady pace: busiest pages, upcoming promotions, most-crawled URLs.

  3. Switch

    Point DNS at the CDN; watch origin load, cache hit rate and crawler requests together.

  4. Days 1-2

    If the origin struggles, return 429 or 503 to crawlers for hours, not days. No 403s.

  5. Day 3

    Check Search Console crawl stats and Merchant Center for errors; remove temporary limits.

Illustrative runbook based on Google's November 11, 2026 guidance; adapt to your CDN and traffic.
A CDN with an empty cache is a faster road to the same old server.

The reaction that does the damage

Then comes the panic. Someone sees a flood of automated requests in the origin logs, assumes it's a scraper attack and switches on the CDN's blanket bot protection.

Google's crawlers can't pass a challenge page. They see a 403 Forbidden, or a "you've been blocked" page served with a normal 200 status, and to them the site looks broken or deliberately hidden. Google's post says that if this stays in place, crawlers back off and pages "will begin to drop out of Search." According to PPC Land's account of the video, the same goes for products in Shopping and ads, which is the last thing a store wants in late November.

The runbook

None of this is hard to avoid if you plan for it. Google's guidance comes down to three moves, and we'd add a fourth.

Let verified crawlers through before you switch. Most CDNs and web application firewalls have a setting or managed rule that allows known, verified bots past challenges while still filtering everything else. Turn it on and test it. Google publishes the IP ranges its crawlers use, so you can confirm requests claiming to be Googlebot really are.

Warm the cache. Before you route all traffic through the CDN, fetch your most important URLs through it at a controlled pace: the highest-traffic pages, the pages you're about to promote (Black Friday landing pages, gift guides, holiday collections) and the URLs crawlers request most, which you can find in your server logs or Search Console's crawl stats.

Use the right brake. If crawling still puts critical load on the origin while the cache warms, Google suggests rate-limiting rules that return 429 (Too Many Requests) or 503 (Service Unavailable) "for a short window (a few hours, or 1-2 days at most)." Crawlers read those as "slow down." They read 403 as "go away." Apply the brake to crawler traffic specifically if your CDN allows it, so real shoppers aren't the ones being told to come back later. Don't leave 429s or 503s up for days; pages that keep returning them can be dropped.

Decide who may touch security rules during launch. This is our addition. Most of the damage in Google's story comes from a well-meaning person changing a setting under pressure. Name one person who can change bot rules in launch week, and write down what they'll do if traffic spikes. The note can be three lines: check whether the spike is verified crawlers or something else, apply a short 429 rule if the origin is in trouble, and tell the team before touching anything broader.

Decision guide

When crawler traffic spikes

ConsiderationThe panic moveWhat Google suggests instead
Bot protectionBlanket block or challenge for all botsAllow verified crawlers; filter the rest
Status code403 Forbidden, or a challenge page served as 200429 or 503 for a short window
How longUntil someone remembersA few hours, one or two days at most
CacheLeft to fill on its ownWarmed before the switch
Decision framework drawn from Google's November 11, 2026 Search Central guidance as reported by Search Engine Roundtable and PPC Land.

Check it afterwards

For the first few days, watch origin load, the CDN's cache hit rate and crawler requests side by side, so a spike can be read in context. Then look at Search Console's crawl stats for errors and changes in crawl rate, and, if you sell online, Merchant Center for products marked as unavailable or not crawlable.

Remove any temporary limits as soon as the cache is warm. A rate-limit rule written in a hurry on day one has a habit of surviving into January, quietly slowing crawls of your post-holiday sale pages. Write down what you did, too. The next migration, or the next time someone wants to switch on stricter bot rules because AI crawlers are hammering the site, will go better with notes.

If you've dealt with crawler blocking before, our note on what blocked crawlers hand over covers the AI-crawler side, and the Pantheon CDN change covers a migration with a fixed date.

Where we help

A CDN switch touches DNS, caching rules, security settings, your hosting and your search visibility, often owned by different people. If you're planning one before the holidays and want it to be boring, that's part of our web application work. Talk to us.

Related work: Spin:Market

twimco.

We build alongside the people doing the work.
Bring us the problem you’re thinking about.

Related work

Spin:Market ↗︎