Block AI Crawlers on WordPress Without Blocking Google

Updated 10 November 2026 · Standard formula, shown below

You can discourage AI crawlers from fetching your WordPress pages while leaving Google Search crawling enabled. The usual approach is to add rules for identified crawlers to your site’s robots.txt; this is a request, not access control, and it cannot guarantee that every crawler will comply. Before editing, confirm how your site serves robots.txt and keep a copy of the current file.

What You’ll Need

Administrator access to WordPress or the hosting, server, or CDN settings that manage your site.
The public URL of your robots.txt file, usually https://example.com/robots.txt.
A current list of the crawler user-agent names you want to discourage. Check each operator’s documentation: names and policies can change, and not every AI-related crawler is used for model training.
A backup or saved copy of your existing robots.txt rules.

Step-by-Step

Inspect your current robots.txt. Open https://your-domain.example/robots.txt in a browser, replacing the example with your domain. Save or copy the contents. Look for existing User-agent and Disallow rules, especially rules for Googlebot, *, or your WordPress admin area. Do not replace existing rules without understanding what they do.
Choose the crawlers you want to address. Find the exact user-agent strings published by the crawler operators. You might see names associated with AI search, training, or browsing services, but their functions differ. Add only the agents you intend to exclude; a broad rule can affect unrelated crawlers. Robots rules are matched against user-agent strings, so guessing a name may have no effect.
Find out which system controls the file. WordPress can provide a virtual robots.txt for sites without a physical file, while a file in the site’s web root, a plugin, hosting configuration, or CDN may serve a different one. The version you see at the public URL is the important one. If a robots.txt plugin or SEO plugin has a robots editor, check its documentation and existing settings before using it.
Add separate rules for the selected agents. In the robots.txt editor that actually controls the public file, add a group for each user-agent. For example, using a placeholder only:
User-agent: ExampleBot Disallow: /
Replace ExampleBot with a verified user-agent. The slash means the named agent is asked not to crawl any path. If you want to exclude only a directory, use its path instead, such as Disallow: /private-section/. Keep each group’s directives directly beneath its User-agent line.
Leave Google’s rules intact. Do not add Disallow: / under User-agent: *, because that can request that all crawlers—including Googlebot—stay away. Avoid adding a sitewide disallow under User-agent: Googlebot. If a broad rule already exists, review it before publishing; a specific Google rule can override a wildcard group for Googlebot, but rule matching can be easy to misconfigure. When unsure, remove unintended blocking rules rather than relying on a complicated exception.
Save and check the live file. Visit the public robots.txt URL again, preferably in a private browser window, and confirm that your new rules appear along with the old rules you intended to keep. Check spelling, line breaks, and the user-agent name. If the content does not change, clear the relevant WordPress, hosting, or CDN cache and check again.
Test Google access and monitor results. Use Google Search Console’s available crawl or indexing reports and URL inspection to check important pages. A robots.txt rule is not a guarantee of indexing: Google still needs to be able to fetch a page to see its content and directives. If you already use Search Console, review any robots-related crawl information and correct accidental blocks promptly.

Common Mistakes

Blocking every crawler: User-agent: * followed by Disallow: / applies broadly. It is not an AI-only rule.
Assuming robots.txt is enforcement: Cooperative crawlers may follow it; a crawler can ignore it. It does not prevent access to public pages or hide their URLs from search results.
Using the wrong agent name: Rules only help when the crawler identifies itself with a matching user-agent. Verify names with the operator.
Editing a file that is not served: A plugin may show one configuration while a physical file or CDN serves another. Always verify the public URL after saving.
Confusing crawling with indexing: Blocking a URL from crawling does not necessarily remove it from Google’s results. Google may list a URL without a description if it cannot fetch the page.

Tips

Keep a dated copy of your original rules and note why each crawler was added. This makes rollback and future maintenance easier.
Use the narrowest rule that meets your goal. Blocking a specific directory is less disruptive than blocking an entire site.
Use a password, authentication, or appropriate server access controls for private content. Robots.txt is public and is not a privacy or security feature.
Recheck the file after plugin, hosting, or CDN changes, and periodically verify that crawler documentation has not changed.

Troubleshooting

The new rules do not appear: Check whether a physical robots.txt file, plugin, host, or CDN takes precedence. Purge caches, reload the public URL, and ask your host which system serves the file if the source remains unclear.

An AI crawler still visits: Confirm that the user-agent matches the crawler’s documented identity and that the rule is in the correct group. Some bots disregard robots.txt, change identities, or fetch through services that do not honor the rule. For stronger restrictions, use server or CDN controls, but test carefully so you do not block Googlebot or ordinary visitors.

Google pages stop being crawled: Reopen the live file and look for a wildcard or Googlebot disallow, including rules added by another tool. Restore the previous rules or remove the accidental block, clear caches, and check affected URLs in Search Console. Allow time for Google to recrawl after the correction.

Google still lists a blocked URL: That can happen because blocking a crawl is not the same as requesting removal from search. If removal is needed, use the appropriate Search Console removal process and separately ensure Google can access the page to read its indexing instructions. Do not use robots.txt as a substitute for access control or an indexing directive.

Frequently Asked Questions

Will blocking AI crawlers in robots.txt keep Google Search working?
It can, if the rules target only the intended user-agents and do not block Googlebot or all crawlers. Verify the live file and monitor Search Console.

Does robots.txt stop AI companies from using my content?
No. It asks compliant crawlers not to fetch specified paths. It cannot control copies already collected or stop a crawler that ignores the request.

Can I block AI bots but allow Googlebot?
Yes. Add specific groups for verified AI crawler user-agents and avoid broad disallow rules. Keep any existing Googlebot rules deliberate and check the result.

Will a robots.txt block remove pages already in Google?
Not necessarily. Google may retain a URL it cannot crawl. Removal and indexing controls are separate from crawl permissions.

Can I block only part of my WordPress site?
Yes. In a selected crawler’s group, disallow a specific path rather than /. Confirm the path and how the crawler interprets robots.txt rules.

Where should I edit robots.txt in WordPress?
Use the tool or server configuration that controls the file served at your public robots.txt URL. Check the live URL after saving; the WordPress dashboard is not always the source.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.