Crawl Budget for Large Ecommerce Sites: A Practical Playbook

What crawl budget actually is (and why only big stores need to care)

Crawl budget is the set of URLs Google can and wants to crawl on a host, set by two things: crawl capacity limit (how hard Google can hit your server without slowing it down) and crawl demand (how much Google wants your pages, driven by popularity, freshness and perceived inventory). Source: Google, Crawl Budget Management, updated 2025.

Google itself is clear that most sites do not need to worry about this. The guidance is aimed at sites with over a million pages that change weekly, sites over ten thousand pages that change daily, or sites with a large share of URLs sitting in "Discovered, currently not indexed". Large catalogues hit all three.

This article is the playbook that sits under our wider technical SEO for ecommerce checklist, focused entirely on getting Google to spend its crawl on the pages that earn.

The crawl budget reference table

A one-screen summary of the concept, the drains and the fix levers before we go deeper.

What drains crawl budget on a large catalogue

A store with 5,000 products and twenty filterable attributes can spin up millions of addresses, most near-duplicate. The six biggest drains we see on real catalogues:

• Filter and sort parameters (colour, size, price brackets, sort order).
• Faceted navigation combinations (multiple filters applied together).
• Internal search result pages crawlable and indexable.
• Session IDs and tracking parameters appended to URLs.
• Infinite or unbounded pagination chains.
• A unique URL per variant (every colour and size of one product).

On the facet point, the deep treatment is in our faceted navigation SEO guide. On the near-duplicate URL point, see ecommerce index bloat.

How to detect crawl waste with log file analysis

A log line records the requesting user agent, the URL, the status code, the timestamp and the bytes returned. Verify it really is Googlebot with reverse DNS rather than trusting the user-agent string (Google, verifying Googlebot guidance, 2025).

The practical method: pull a representative window of logs, isolate verified Googlebot, group requests by URL type (product, category, filter/parameter, search, other), and calculate the share of crawl spent on each. The headline metric is the percentage of crawl landing on parameter and search URLs that should never be crawled.

Pair this with the Crawl Stats report in Search Console, which shows total crawl requests, average response time and host availability problems. That is your capacity-limit signal. Site architecture also shapes what gets crawled, which we cover in ecommerce site architecture.

Verify Googlebot by reverse DNS, not the user-agent string. Source: Google, 2025.

The crawl budget calculator

Plug in your catalogue size and Googlebot's daily rate, then see how long Google needs to crawl every URL at your current waste level versus a healthier 10%.

Crawl depth in plain terms is the number of clicks from the homepage to a page. Pages buried six clicks deep get crawled rarely. Our rule: every revenue page within three or four clicks of the homepage.

How to reclaim crawl budget in seven steps

1. Block parameter and sort URLs in robots.txt. Disallow the filter, sort and session parameters that create near-duplicate URLs. Google will not shift freed budget elsewhere unless it is already hitting your serving limit, so do this to stop waste, not as a magic boost. Source: Google, 2025.
2. Canonicalise filtered pages to the parent category. Point every filtered or sorted version at the clean category URL so signals consolidate on one page.
3. Do not rely on noindex for crawl control. Google still has to request a noindexed page before it sees the tag, which spends budget. Use robots.txt to keep it out of the crawl entirely. Source: Google, 2025.
4. Return 404 or 410 for dead URLs, and flatten redirect chains. A 404 is a strong signal to stop crawling a URL. Long redirect chains waste crawl and bleed PageRank at every hop. Source: Google, 2025.
5. Flatten crawl depth and fix orphan pages. Get revenue pages within three or four clicks of the homepage and make sure every important page has internal links pointing at it.
6. Keep XML sitemaps to clean, indexable, canonical URLs only, with accurate lastmod values so Google can prioritise what changed.
7. Speed up server response and re-crawl to confirm. Faster responses raise the crawl capacity limit, so Google can read more. Then re-pull logs and confirm the parameter-crawl share has dropped. Source: Google, 2025.

First-party data: what the numbers say

Faceted navigation handling correlates with ecommerce rank at +0.35 when managed and -0.34 when left unmanaged, and unmanaged faceted navigation wastes 2.7x more crawl budget on parameter URLs. Source: Visionary 2026 Ecommerce SEO Ranking Factor Study, 100,000 pages crawled Q1 2026. Combine with the Botify floor of 40% URLs crawled on unoptimised stores.

Days to crawl a 250,000-URL catalogue, by crawl rate and waste level. Computed from total URLs divided by effective crawl rate. Crawl-rate model per Google, 2025.

How this playbook beats the standard advice

Where this fits, and where to get help

Crawl budget is the first technical fix to make on any large store, ahead of speed and schema work. A fast page Google never crawls earns nothing.

This is exactly the work we do inside our ecommerce SEO services. Log-led crawl audits are the first step on large catalogues.

Methodology and sources

• Google, Crawl Budget Management, Google Search Central, updated 2025.
• Google, verifying Googlebot guidance, Google Search Central, 2025.
• Botify, crawl budget research (carried via our technical SEO for ecommerce pillar).
• Visionary 2026 Ecommerce SEO Ranking Factor Study, 100,000 pages crawled Q1 2026.

Frequently Asked Questions