A CDN migration checklist for the agency handoff

Consider a plausible handoff failure: the homepage loads quickly after a content delivery network (CDN) migration, so the agency signs off. Later, the hosting team sees rising load on the origin server behind the CDN and enables stricter bot protection. Each decision looks reasonable in isolation; together, they can leave search crawlers unable to reach the site.

Google’s 11 November guidance on moving to a CDN explains the underlying tension. Faster responses can gradually increase crawl capacity when demand exists. But an empty edge cache still sends requests back to the origin, especially as crawlers explore less popular URLs. This is new guidance about a migration risk, not an announcement of a new ranking factor.

For an agency, the useful deliverable is a runbook that joins infrastructure changes to search access. Here is a proposed structure for a substantial content or catalogue site whose public URLs will stay the same. A migration that also changes URLs needs a separate redirect and indexing plan.

Before cutover: record what normal means

Choose a comparable baseline period and record its dates. Include a busy trading period if the migration supports an imminent campaign. Collect origin request volume, response times and errors, alongside the current caching configuration. Assign named owners for DNS, CDN rules, hosting and search monitoring; give one person authority to pause the change.

Export the available Search Console Crawl Stats history. It includes request counts, response times, response categories and host availability. Its example URLs are a sample, and its totals can differ from server logs, so use it as supporting evidence rather than a complete request ledger. Crawl Stats documentation

Build a small reference URL set: homepage, campaign page, older article, deep catalogue page, sitemap, robots.txt and representative CSS, JavaScript and images. Save expected statuses, redirects and content. Agree project-specific limits for origin latency, error rate and capacity with the hosting owner; a universal “safe” number would conceal differences between sites.

Prepare the cache, then inspect both sides at cutover

Before routing all traffic through the CDN, warm important cacheable URLs at a controlled pace, including upcoming campaign pages and frequently crawled content, as Google recommends. Confirm that requests traverse the intended CDN configuration. Repeated success on one page does not demonstrate coverage across the URL inventory or every edge location. Migration guidance

For each reference URL, capture the public response status and body, the CDN’s cache result, and any corresponding origin request. Record timestamps consistently and use a shared request identifier where available. Check redirects and robots.txt against the baseline, then complete a real customer journey through the site using the agency’s website review checklist.

Preserve enough edge logging to investigate requests that never reach the origin. Verify the client IP observed by the CDN against Google’s published crawler IP ranges, or use reverse DNS followed by a matching forward lookup. Use the category appropriate to the crawler being checked. The CDN’s own connection to the origin and a Googlebot user-agent string alone do not establish crawler identity. Verification guidance

An illustrative incident: two different failures

Imagine an agency moving a large product catalogue. After cutover, a popular landing page remains fast, but older product URLs become slow. This is a hypothetical triage exercise, not a measured migration.

First, correlate an affected request across layers. If the edge records a cache miss and the origin records a slow response for the same request, investigate origin work and cache eligibility. Check whether repeated requests become hits, whether query variants fragment the cache, and whether the selected response is intended to be cached at all. Do not make private or personalised content publicly cacheable to improve a graph.

Now consider a different observation: verified Google requests receive a challenge at the edge, with no corresponding request in complete origin logs. Investigate the matching firewall rule. Increasing origin capacity would not remove that interception. Google specifically warns that CAPTCHA challenges block its crawlers; configure an appropriate verified-crawler exemption and verify the resulting response. Bot protection guidance

Stop conditions belong in the handoff

Pause expansion when agreed capacity limits are exceeded, reference URLs return unexpected content, or verified crawlers encounter challenges. Record the failing layer before choosing a scoped rule correction, capacity intervention or prepared rollback. Keep the previous configuration and its restoration steps available.

If genuine overload needs an emergency brake, Google supports temporary 429 or 503 responses to crawl requests, optionally with a Retry-After header. Its documentation describes hours or one to two days at most. Significant numbers of these errors reduce crawling across the whole hostname; persistent errors on a URL can eventually remove it from Search. Assign an owner and removal deadline to any such rule, then verify normal responses after removing it. Emergency crawl-rate guidance

Close the handoff with a short record the next person can act on:

• The baseline period, cutover time and configuration version.
• The reference URLs checked, observed responses and links to edge and origin evidence.
• Each remaining exception, its owner and the agreed trigger for intervention.
• The next review time, who will perform it, and the rollback instructions.

Review the logs again as crawling reaches less frequently visited pages. A successful cutover needs evidence that both customers and intended crawlers can still obtain the right content.

Source and scope

Based on Google’s guidance and primary documentation checked on 12 November 2026. The runbook is editorial advice; the catalogue incident is illustrative. No migration, account test or performance measurement was performed.