# AI Agents and AI Crawlers on Shopify with XSeo {#doc-title}

By [B2B Gold](/about/#how-we-write) Updated 6 November 2026 [XSeo page](/shopify/xseo/) [XSeo on the Shopify App Store](https://apps.shopify.com/xseo?utm_source=b2bgold.app&utm_medium=referral&utm_campaign=docs&utm_content=docs/xseo/ai-agents-and-crawlers&utm_term=docs-header)

## What do AI shopping agents see? {#what-do-ai-shopping-agents-see}

XSeo shows any product the way AI shopping agents get it from your store’s own Shopify catalogue, next to your store’s data, with what’s missing. It also shows which AI crawlers your robots.txt lets in, and can block the ones that only collect pages to train AI models. XSeo never blocks search engines, AI search or shopping assistants.

Open **Listings**, click a product and find **What AI agents see**. XSeo asks for the product through Shopify’s Storefront Catalog MCP, which lets AI agents search one store’s products ([shopify.dev: Storefront Catalog MCP](https://shopify.dev/docs/agents/catalog/storefront-catalog), checked 6 November 2026). It reads live each time, stores nothing, uses no AI items and works behind your store’s password. It shows your Shopify catalogue, not what an assistant chooses to show.

| Row | What it compares |
| --- | --- |
| **Status** | **On the Online Store** or not, and **Hidden from search** |
| **Title**, **Description** | The text agents get, with the description’s **Characters** |
| **Category** | Shopify sends only an ID; XSeo names the category from your admin |
| **Main image** | The image, and its alt text when Shopify’s catalogue sends it; XSeo says when it doesn’t |
| **Options**, **Price**, **Collections** | What agents get, with your options beside them |
| **Barcodes (GTIN)** | **Variants without a barcode**, such as 2 of 4 |
| **Brand and type** | “Not sent to AI agents directly.” |

Below it, **Missing or weak for AI shopping** lists each gap with its fix, such as **Thin description** (under 150 characters) with **Draft a description**. How each fix works: [product data for AI shopping in XSeo](/docs/xseo/ai-shopping-data/).

## Why can’t AI agents find a product? {#why-cant-ai-agents-find-a-product}

AI agents can’t find a Shopify product when your store’s catalogue doesn’t return it, usually because it isn’t active, isn’t published to the Online Store, or is hidden from search engines. XSeo then says “AI agents can’t find this product: Shopify’s catalog didn’t return it.”

If Shopify answers with an error, XSeo says your store may not be eligible for Shopify’s catalogue yet. Shopify Catalog takes products with a title, an image and a price above zero that aren’t **Unlisted**, hidden from search engines or **Uncategorized**, from stores on an eligible Shopify plan that aren’t in private mode ([Shopify: Shopify Catalog requirements](https://help.shopify.com/en/manual/shopify-catalog/requirements), checked 6 November 2026). Next: [why a store isn’t showing up in AI channels](/blog/shopify-agentic-storefronts/#why-isnt-my-store-showing-up).

## Which AI crawlers can read my store? {#which-ai-crawlers-can-read-my-store}

XSeo’s **AI crawlers** page shows which of 13 AI crawlers your Shopify store’s robots.txt lets in, grouped as AI search, assistant visits and model training, and lets you block six model training crawlers, no others. Open **Settings** → **Structured data** → **Manage**, then **Manage** on the **AI crawlers** card.

| Group in XSeo | Crawlers (owner) | Setting |
| --- | --- | --- |
| **AI search** | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity) | **Always allowed** |
| **Assistant visits** | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity) | **Always allowed** |
| **Model training** | GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Applebot-Extended (Apple), CCBot (Common Crawl), Bytespider (ByteDance) | **Allowed** or **Blocked** |
| **Model training** | Google-Extended (Google): “Gemini also uses it to answer with your pages.” | **Always allowed** |

Owners’ pages (checked 6 November 2026): [OpenAI](https://developers.openai.com/api/docs/bots) and [Anthropic](https://support.claude.com/en/articles/8896518) run separate crawlers for training, search and pages a person asks about; [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) says PerplexityBot doesn’t feed AI foundation models; [Apple](https://support.apple.com/en-us/119829) says Applebot-Extended crawls nothing, deciding only whether Applebot’s pages train its models; [Common Crawl](https://commoncrawl.org/ccbot) builds an open web archive; [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) says Google-Extended doesn’t affect Google Search. OpenAI and Perplexity say their assistant fetchers may not follow robots.txt.

Home flags robots.txt rules that keep Googlebot, Bingbot or an AI search crawler off your products: **Search and answer bots kept out of your catalog**.

## How do I block AI training crawlers? {#how-do-i-block-ai-training-crawlers}

To block AI training crawlers, set **Model training** to **Blocked** on XSeo’s **AI crawlers** page, review the change and click **Allow and apply**: XSeo writes the rules into your published Shopify theme’s robots.txt.liquid once you let it edit theme files when Shopify asks.

1. Set **Model training** to **Blocked**, or open **Choose per crawler** and block some.
2. Click **Review changes**: **What changes**, **After the change** (each crawler’s result in your live robots.txt) and **The file**, with **Show the file**.
3. Click **Allow and apply** (later, **Apply**). If you refuse Shopify’s permission, nothing is written.
4. XSeo reads robots.txt back: “Your robots.txt now has these rules.” Shopify can take a few minutes to show them.

Applying uses no AI items, and nothing applies rules automatically.

## What does XSeo change in my theme? {#what-does-xseo-change-in-my-theme}

XSeo writes to your theme only if you apply AI crawler rules, and then only its own marked lines in robots.txt.liquid: a `User-agent` and `Disallow: /` pair per blocked crawler, between `# BEGIN XSeo AI crawler rules` and `# END XSeo AI crawler rules`. The structured-data embed never edits your theme’s files.

- **No robots.txt.liquid yet:** XSeo creates one in your published theme, with Shopify’s default rules first.
- **A file of your own:** XSeo adds its lines at the end and leaves the rest byte for byte, your rules included.
- **Checks:** XSeo reads your live robots.txt first, stops if a change would also block a crawler it never blocks, and refuses a file over 100 KB or its own lines edited by hand.

Shopify calls editing robots.txt.liquid an unsupported customisation that “can result in loss of all traffic” if done wrong; robots.txt rules are advisory. Blocking AI crawlers “affects only open-web discoverability”: Shopify Catalog still sends your product data to the agentic storefronts you’ve activated ([Shopify: editing robots.txt.liquid](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt), checked 6 November 2026).

## How do I remove XSeo’s crawler rules? {#how-do-i-remove-xseos-crawler-rules}

To remove XSeo’s crawler rules, click **Remove XSeo’s rules** on the **AI crawlers** page and apply: XSeo deletes its own lines, and if it created robots.txt.liquid, it deletes the file so Shopify’s own robots.txt comes back. A robots.txt.liquid of your own is never deleted.

**Changes XSeo made** lists each change, with **Undo** on the latest; undo stops if the file changed since or another theme was published. These changes aren’t in **Activity**.

> **Important:** Uninstalling XSeo leaves its rules in your theme, still working. Remove them first, or later delete the lines from `# BEGIN XSeo AI crawler rules` to `# END XSeo AI crawler rules` in your theme’s code.

## Related {#related}

- [Product data for AI shopping in XSeo](/docs/xseo/ai-shopping-data/)
- [Shopify agentic storefronts: keep them on or opt out](/blog/shopify-agentic-storefronts/)
- [AI crawlers when you block countries on Shopify](/blog/geo-blocking-seo-shopify/#what-about-ai-crawlers-like-gptbot-and-claudebot)
- [What XSeo’s site audit checks](/docs/xseo/site-audit/)

[XSeo - SEO & Image Optimizer](/shopify/xseo/)

XSeo is a free Shopify app for SEO and images. It writes SEO titles, meta descriptions and image alt text from templates or AI, compresses images in place with a 60-day backup, adds structured data through a theme app embed, manages redirects and checks the product data AI shopping reads. Runs you start show examples first, and Activity can undo them.

[Install free on the Shopify App Store](https://apps.shopify.com/xseo?utm_source=b2bgold.app&utm_medium=referral&utm_campaign=docs&utm_content=docs/xseo/ai-agents-and-crawlers&utm_term=card) [Plans and pricing](/shopify/xseo/#pricing)

Need help with XSeo? Email [support@b2bgold.app](mailto:support@b2bgold.app).