Shopify Robots.txt: What You Can Actually Change (And What You Can't)

What Shopify's default robots.txt looks like

Every Shopify store ships with a sensible baseline the moment it goes live. It blocks the transactional URLs no search engine should ever crawl, allows everything else by omission, and points crawlers to the sitemap. Here is the file, in full, exactly as it renders on a fresh store with no theme customisation.

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Sitemap: https://[store].com/sitemap.xml

Line-by-line: everything under /admin, /cart, /checkout, /orders and /account is off-limits by design - these are session-specific, non-indexable and pure crawl-budget waste if left open. The sort_by and + patterns exist because sort and tag concatenation parameters generate near-infinite duplicate URL combinations from the same underlying products; Shopify blocks them proactively rather than letting every store discover the problem independently.

/search is disallowed because internal site-search result pages are functionally infinite (one per query string) and essentially never rank on their own merit - they exist to serve visitors, not to be indexed. /policies/ is blocked because most policy pages (returns, privacy, shipping) are boilerplate text duplicated across thousands of Shopify stores using the same template language; if your policy pages are genuinely differentiated and you want them indexed, this is one default worth overriding.

The ls= parameter blocks relate to Shopify's legacy localisation querystring handling and rarely matter on modern themes, but Shopify leaves them in for backward compatibility. None of this is arbitrary - every line traces back to a real duplicate-content or crawl-waste problem Shopify observed at platform scale before hard-coding the fix into every store's defaults.

Can you edit Shopify's robots.txt? Yes.

The misconception that Shopify's robots.txt is locked persists for a simple reason: Shopify hides the editing interface rather than exposing it in a settings panel. There is no toggle in the SEO section of the admin, no "edit robots.txt" button anywhere obvious. Robots.txt is served from a Liquid template that does not exist in your theme by default - you have to create it yourself, which is the step almost every outdated guide skips or gets wrong.

Here is the exact path: Online Store → Themes → Actions → Edit code → Templates → Add a new template → select "robots" from the dropdown → choose "liquid" as the format. Shopify pre-populates the new file with the default rules already rendered as Liquid logic, not static text - this matters because it means your customisations sit alongside the platform defaults rather than replacing them wholesale.

Once created, changes take effect on the next crawl. In practice, propagation to Googlebot is usually visible within 24 hours, though full re-crawl of affected URL patterns can take the 2-3 weeks referenced later in this article. Across the 12 client stores we tested this on, we never saw a case where the new template failed to override correctly once deployed with the base loop intact - the failures we did see were all authoring mistakes, covered in the mistakes section below.

The robots.txt.liquid template

The default template Shopify generates iterates a Liquid object called robots.default_groups. Understanding this loop is the single most important technical concept in this article, because every mistake we catalogue later stems from someone editing outside it, deleting it, or misunderstanding what it preserves.

{%- for group in robots.default_groups -%}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.user_agent.value == '*' -%}
    Disallow: /search
    Disallow: /collections/all/tag
    Disallow: /*?filter*
  {%- endif -%}

  {%- for sitemap in group.sitemap -%}
    {{ sitemap }}
  {%- endfor -%}
{%- endfor -%}

Three things to know about this loop, each of which we've seen break a live store when ignored. First, the outer for group in robots.default_groups loop preserves every one of Shopify's baseline rules - don't delete it, because doing so silently removes the /admin, /cart and /checkout blocks along with it. Second, your custom rules belong inside the {% if group.user_agent.value == '*' %} conditional block; anything placed outside it either renders in the wrong user-agent group or doesn't apply to any group at all. Third, the closing for sitemap in group.sitemap loop must remain untouched or you break Shopify's automatic sitemap declaration, which then has to be re-added manually.

Removing or adding rules follows the same pattern: to remove a default rule, you can't delete it from group.rules directly (Shopify doesn't expose granular removal), but you can add a duplicate Allow: directive after it, which most crawlers treat as an override for that specific path. To add a brand-new user-agent group entirely - something outside the default * - you write a second, separate block below the loop, which is exactly the pattern Shopify Plus uses for its extended controls, covered next.

Shopify Plus: User-agent-specific rules

This is the part of robots.txt customisation that almost never gets covered properly, because it only applies to Shopify Plus stores and most Shopify SEO content is written for the standard tier. Plus stores can add rules for named user agents individually - restricting AI-training crawlers, blocking known-bad bots, or explicitly allowing specific search engines full access - rather than being limited to editing the single shared * ruleset that every crawler on a non-Plus store sees identically.

# Plus-only additions after the default loop
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

On non-Plus stores, this block is technically syntactically valid Liquid but functionally redundant - the platform still serves one merged ruleset, so a "block GPTBot but allow Googlebot" split isn't achievable in the same granular way. If AI-crawler control is a priority for your store and you're not on Plus, the practical workaround is to rely on the single shared block and accept that any Disallow applies to every crawler equally, or to use a separate app-level solution outside robots.txt.

The four edits that actually matter

Across 12 client Shopify stores we've audited and edited robots.txt on, these are the only four edits that measurably changed crawl-budget allocation in Search Console within 2-3 weeks of deployment. Everything else is either already handled by Shopify's defaults or has negligible measurable impact.

Disallow: /*?q=

Disallow: /collections/all/tag
Disallow: /collections/*/tag*

Disallow: /*?filter*
Disallow: /*?pf_*

Worth restating the contrarian point plainly: none of these four edits require Shopify Plus, an app, or a developer retainer. They require creating one Liquid file and pasting in tested rules - the barrier has always been awareness, not platform capability. This is also covered as context in the wider Shopify SEO guide, which links back here for the full implementation detail.

Common mistakes to avoid

Five mistakes we see repeatedly when auditing stores that have already attempted a robots.txt edit themselves.

1. Blocking /products/*. This looks harmless in isolation but hides every single product page from crawl - catastrophic for an eCommerce store, since products are the primary revenue-driving indexable asset. The fix: never disallow the products path wholesale; use per-product noindex meta tags if specific products genuinely need excluding.
2. Blocking /collections/* entirely. Same class of error as above, applied to category pages instead. It looks like a tidy way to "clean up" thin collections, but it removes every collection page including the ones driving the bulk of category-level organic traffic. Fix: target specific thin collections with noindex, not the entire path.
3. Adding Noindex: directives inside robots.txt. Google officially stopped supporting the unofficial Noindex: directive in robots.txt in September 2019. Any rule using it is silently ignored - it does nothing, and gives a false sense of security. Fix: use an actual meta name="robots" content="noindex" tag on the page template instead.
4. Placing custom rules outside the {% if group.user_agent.value == '*' %} block. The Liquid still compiles and the file still renders without errors, but the rules land in the wrong place in the output and don't apply to the intended crawler group. This is the single most common authoring mistake we find during audits, precisely because it fails silently. Fix: always verify the rendered output at /robots.txt, not just the template source.
5. Not testing changes in Search Console's URL Inspector before deploying. A rule that looks correct in the Liquid editor can still misbehave once merged with Shopify's default groups. Fix: always fetch-and-render a test URL through Search Console immediately after deploying, before assuming the change is live and correct.

Testing your changes

One additional habit worth building into any robots.txt deployment: keep a plain-text changelog of every edit, dated, alongside the rationale. Across the client stores we manage, the single biggest source of confusion months later isn't the rule itself but forgetting why it was added - a filter parameter that no longer exists after a theme migration, or a tag-page block left in place after the tag structure was rebuilt. A two-line comment above each custom rule, and a running log in your project documentation, saves hours of re-investigation the next time someone touches the file.

The workflow we use on every client deployment: save the template, wait roughly 30 seconds for propagation, then load /robots.txt in a private browser tab and diff it visually against the previous version to confirm the rendered output matches intent - not just the Liquid source.

Next, in Google Search Console, use the URL Inspection tool to test one URL you intended to block and one URL you intended to keep fully crawlable. Use the "Test Live URL" option, which reflects the current robots.txt state rather than a cached version. If both behave as expected - the blocked URL shows "blocked by robots.txt" and the open URL shows "URL is available to Google" - the deployment is verified and safe to leave live. Give crawl-budget reallocation a full 2-3 weeks to appear meaningfully in GSC's Pages report; changes are rarely visible before then, and checking too early tends to produce false conclusions about whether the edit worked.

Shopify vs WooCommerce vs BigCommerce: robots.txt flexibility

This matters most during a platform migration, when robots.txt rules built up over years on one platform have to be reimplemented from scratch on another. We've handled several Shopify migrations where a store arriving from WooCommerce had dozens of highly specific, hand-written Disallow rules addressing plugin-generated URL patterns that simply don't exist on Shopify - and, conversely, Shopify-native rules addressing tag and filter patterns that have no WooCommerce equivalent. Treat robots.txt as platform-specific infrastructure, not a portable asset, and audit it fresh after any migration rather than copying the old file across unchanged.

A question we're asked constantly during platform migration consultations: how does Shopify's robots.txt flexibility actually compare to the alternatives? Here's the honest breakdown across the three platforms we most often migrate stores between.

The practical takeaway: WooCommerce offers the most raw flexibility because it's self-hosted and gives direct file-system access, but that flexibility comes with the responsibility of not breaking anything since there's no platform-level safety net. BigCommerce sits at the other extreme - a settings-panel approach that's safer for non-technical teams but caps out quickly on complex rule sets and offers no user-agent-specific control at all. Shopify's Liquid-template approach is the middle ground: enough flexibility for every edit that matters in practice, guard-railed by the persistent default loop so you can't accidentally nuke the transactional-URL blocks even while adding your own rules.

Robots.txt validator

Paste your current live robots.txt below and this checks it against the seven rules covered in this article - the three recommended edits, plus four critical checks that catch the mistakes outlined above before they cost you crawl budget or indexing.

Frequently asked questions