How to Block AI Crawlers From Scraping Your Website (robots.txt Guide)

Want to block AI crawlers from scraping your website, without disappearing from Google? You can, with a few lines in your robots.txt file. This guide lists the main AI bots and what each one does, gives copy-ready robots.txt rules, shows how to add them in WordPress and Cloudflare, and explains the new shared opt-out signal that OpenAI, Google, Microsoft and Anthropic say they will support.

Quick answer: Add a User-agent group with Disallow: / for each AI training bot you want to stop (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent) to the robots.txt file at the root of your site. This doesn’t affect Google or Bing search rankings. Keep search bots such as Googlebot, Bingbot and OAI-SearchBot allowed if you want to keep search traffic. robots.txt is a request, not a lock: reputable companies follow it, but bad scrapers may not, so add a firewall rule (for example Cloudflare’s AI bot settings) for stronger protection.

Training bots vs search bots: block the right ones

AI companies now run several bots with different jobs. Blocking all of them can cost you traffic, because some send visitors from AI search answers. The key is to separate training (your content teaches a model) from search and user requests (your page is fetched to answer a question, often with a link).

User-agent names come from each company’s official crawler pages: OpenAI, Anthropic, Google, Apple, Meta, Common Crawl and Perplexity. Companies add bots from time to time, so check these pages once or twice a year.

robots.txt rules to block AI training crawlers

This set blocks the main AI training bots across your whole site while leaving search engines alone. Add it to your existing robots.txt; don’t delete your other rules or your sitemap line.

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: CCBot
Disallow: /

To block only part of your site, replace Disallow: / with a path such as Disallow: /premium/. To block AI search too (not usually recommended), add groups for OAI-SearchBot, Claude-SearchBot and PerplexityBot.

Don’t put these bots under User-agent: * with Disallow: /, because that would also block Googlebot and Bingbot and remove your site from search. If you are unsure your file is correct, our guide to finding and fixing URLs blocked by robots.txt shows how to test it.

How to block AI crawlers in WordPress

WordPress creates a virtual robots.txt file unless a real file exists in your site’s root folder. You have three easy options:

1. Rank Math: go to Rank Math SEO > General Settings > Edit robots.txt (turn on Advanced Mode first), paste the rules and save. Rank Math says you must delete any physical robots.txt file in the root folder before its editor works (Rank Math help).
2. Yoast SEO or another SEO plugin: most have a robots.txt or file editor under their Tools menu.
3. A real file: create a plain-text file named robots.txt, add your rules and upload it to the root folder (the same folder as wp-config.php) using your host’s file manager or FTP.

Then open yoursite.com/robots.txt in a browser to confirm the new lines appear. Clear any page cache or CDN cache if you still see the old version.

How to block AI bots with Cloudflare

robots.txt relies on bots choosing to obey it. If your site uses Cloudflare, its dashboard can block AI bots at the network edge. In Security Settings, Cloudflare’s Configure AI bot policies option lets you choose separately whether to allow or block AI search, AI agent and AI training bots, on all pages or only on pages with ads (Cloudflare docs). Other hosts and firewalls, such as server rules in nginx or Apache, can also block by user agent, but verify bots by IP where the company publishes IP lists, because some scrapers fake their user agent.

What robots.txt can’t do

• It doesn’t remove past data. Content already collected before you added the rule may still be in training sets.
• It isn’t enforced. Well-known companies say they honour it, but unknown scrapers may ignore it.
• User-requested fetches may bypass it. OpenAI, Meta and Perplexity say some user-triggered fetchers may not follow robots.txt.
• It doesn’t stop copying. For classic content scrapers, see our tips to stop RSS feed scraping and content theft.

Should you block AI crawlers?

It depends on what you publish. Blocking training bots makes sense if your writing, photos or research are your product and you don’t want them reused without a deal. Allowing AI search bots usually makes sense if you want to be cited and linked in ChatGPT, Claude, Perplexity and Gemini answers. Many publishers now do both: block training, allow search. Whatever you choose, review your robots.txt every few months, because bot names and company policies change.

What changed in 2026: a shared AI training opt-out

Updated 1 November 2026.

On 30 October 2026, Anthropic’s Kevin Kelley published an IETF Internet-Draft called Machine-Readable Training Preference Signal, building on the IETF AI Preferences (AIPREF) working group. It defines one shared robots.txt token, __usage_ai-training, that tells every participating company at once whether your content may be used for AI training. Google, OpenAI and Microsoft said the same week that they would support it. The draft’s example looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That means “don’t use my site for AI training, except the press kit”. Important details:

• It isn’t live yet. Google’s crawler documentation says it plans to support the signal in early 2027; Microsoft has said it will honour it from 1 February 2027; OpenAI says “in the coming months”; Anthropic says work has started. Until then, keep the per-bot rules above.
• It is a draft, not a finished standard. It is an individual Internet-Draft and its details may change.
• It covers training, not crawling or search. It doesn’t block crawling and doesn’t change search rankings. Google says that if you opt out with the shared token and don’t mention Google-Extended, it plans to treat that as a Google-Extended block.
• Adding it now is harmless. You can add the group alongside your existing rules so it takes effect as each company switches it on.

FAQ

Does blocking AI crawlers hurt SEO?

Blocking AI training bots such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended doesn’t affect Google or Bing search rankings. Blocking AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) can remove your site from those AI search answers.

What is GPTBot?

GPTBot is OpenAI’s crawler that collects public web content that may be used to train its AI models. OpenAI says disallowing GPTBot tells it your content shouldn’t be used for training.

How do I check which AI bots visit my site?

Look for their user-agent names in your server access logs, your host’s statistics, or Cloudflare’s analytics. Many security plugins also show bot visits.

How long does a robots.txt change take to work?

Crawlers cache robots.txt, so changes can take up to a day or so to be picked up. Meta, for example, says up to 24 hours.