Can AI companies train on your website?
Unless your robots.txt says otherwise, AI companies' crawlers are free to collect your pages to train their models. You can say no without losing search traffic: search crawlers and training crawlers have different names. Enter your site to see what your robots.txt allows, and get the lines to add.
Free, no signup. Takes a few seconds.What it checks
Google, Bing and Apple's search crawlers, which should stay allowed
AI training crawlers from OpenAI, Google, Anthropic, Apple, Meta, Common Crawl and ByteDance
AI search crawlers (ChatGPT search, Perplexity, Claude), which link back to you
A robots.txt that errors, which makes Google stop crawling your site
Questions
Will blocking AI training hurt my Google ranking?
No. Google-Extended, the token for Gemini training, is separate from Googlebot, and Google says blocking it doesn't affect Search. GPTBot, ClaudeBot and CCBot have nothing to do with search rankings.
Does blocking Google-Extended keep my site out of AI Overviews?
No. AI Overviews are part of Google Search and use the normal Googlebot crawl. Google-Extended covers Gemini's training, not Search.
Does robots.txt actually stop AI crawlers?
It's a request, not a lock. The big AI companies say their named crawlers follow it, but it doesn't remove anything already collected, and unnamed scrapers can ignore it. To enforce it, block the crawlers at your host or CDN too; Cloudflare, for example, has a setting for it.
Should I block AI search crawlers too?
That's your call. ChatGPT search, Perplexity and Claude answer questions with links to the pages they used, so blocking them can cost you visitors. Blocking training crawlers doesn't.
How do I edit robots.txt on my site builder?
On WordPress, most SEO plugins (Yoast, Rank Math) have a robots.txt editor. Wix has one in its SEO settings, and Shopify uses a robots.txt.liquid template in your theme. Squarespace doesn't let you edit the file, but has a setting to block known AI crawlers.
Can it watch for this?
Yes. Add your site to Page Canary (free for one site) and it checks your robots.txt every day, with the rest of your site, and emails you if it ever blocks Google, Bing or Apple's search crawler.