ThisCalculator

Everyday

Block AI Crawlers Without Blocking Google Search

Updated · Standard formula, shown below

Block AI Crawlers Without Blocking Google Search

You can discourage many AI crawlers from fetching your website while keeping Google Search crawling enabled by using separate rules in your robots.txt file. This is a request to compliant crawlers, not a technical barrier: bots that ignore the rules may still access public pages.

What You’ll Need

  • Permission to edit your site’s robots.txt file, usually through your hosting control panel, CMS, or deployment workflow.
  • A list of the AI crawler user-agent names you want to disallow. Check each operator’s current documentation because names and policies can change.
  • A way to visit your site’s robots.txt URL and check the live file after publishing, such as https://example.com/robots.txt.

Step-by-Step

  1. Decide what you mean by “block AI crawlers.” Robots.txt can ask named, compliant bots not to fetch specified paths. It cannot reliably prevent scraping, remove content already collected, or control how a search engine uses material it has already crawled. If you need access control, protect the content with authentication or server-side restrictions instead.
  2. Identify the bots and their purposes. Look up the user-agent strings for the particular services you want to address. Some companies operate different crawlers for search, user-requested retrieval, or model training, and their policies may distinguish them. Do not assume that blocking one named bot blocks every crawler from the same operator.
  3. Review the existing robots.txt file before editing. Open https://yourdomain.com/robots.txt and save a copy. Check for existing User-agent, Disallow, and Sitemap lines. A website should have one robots.txt file at the root of each protocol and hostname; for example, the HTTPS and HTTP versions can serve different files.
  4. Add rules for the specific AI user agents. For each bot, add a separate group using the exact user-agent name from its documentation. For example, the structure is:
    User-agent: ExampleBot
    Disallow: /
    Replace ExampleBot with a verified name. Disallow: / requests that this bot avoid all paths on the host covered by the file. If you only want to restrict a section, use its path, such as Disallow: /private-section/, and verify the path-matching behavior for the crawler.
  5. Keep Googlebot rules separate and permissive where needed. Do not put a broad Disallow: / under User-agent: * if you want Google and other general crawlers to access the site. Do not add a site-wide disallow under User-agent: Googlebot. If you already have Google-specific rules, preserve them and check that they do not block pages you want in Google Search. A basic pattern might be:
    User-agent: Googlebot
    Allow: /
    User-agent: ExampleBot
    Disallow: /
    An explicit Googlebot allowance is not always necessary; the key is to avoid a conflicting Googlebot restriction. Google applies the most specific matching user-agent group rather than combining it with the wildcard group.
  6. Preserve useful existing directives. Keep valid sitemap declarations and any intentional path restrictions. Avoid copying a complete robots.txt template over the live file without comparing it first. Directives such as Crawl-delay are not supported by every crawler, so do not rely on them as a universal control.
  7. Publish the file at the correct location. The file must be named exactly robots.txt and served from the site’s root, for example https://example.com/robots.txt. Uploading it into a subfolder will not set rules for the whole host. If your site uses multiple subdomains, create and check a robots.txt file for each one that needs rules.
  8. Check the live response and test important URLs. Visit the public robots.txt URL and confirm it shows your latest text with a successful response, not an old cache, login page, or error. Use Google’s Search Console robots.txt testing or URL inspection tools, where available, to check whether Googlebot can crawl key pages. A robots.txt change affects crawling; it does not guarantee that a URL disappears from search results.
  9. Monitor and update. Review server logs or bot reports to see whether named crawlers continue requesting pages. Confirm their identity using reliable signals, not just a claimed user-agent string, which can be spoofed. Update the file when crawler names or your policy change.

Common Mistakes

  • Blocking everyone with a wildcard: User-agent: * followed by Disallow: / can stop Googlebot and other crawlers that follow robots.txt.
  • Putting unrelated bots in one group: Rules in a group apply to the user-agent or agents named there. Use separate groups to make the intended policy clear and reduce accidental restrictions.
  • Assuming robots.txt is access control: The file is public and advisory. It does not hide a URL, protect confidential information, or force a noncompliant bot to stop.
  • Blocking Google indexing unintentionally: A disallowed URL might still appear in results if Google discovers it elsewhere, even though Google cannot crawl its content. Use appropriate access controls or page-level indexing directives for their intended purposes; do not expect robots.txt alone to remove already indexed URLs.
  • Editing the wrong host or file: Rules for www.example.com do not automatically govern example.com or a subdomain. Check the exact hostname serving the pages.

Tips

  • Start with a short, targeted list of verified user-agent names rather than a copied list of every crawler mentioned online.
  • Keep a dated backup and note why each rule exists. This makes later reviews safer.
  • Use robots.txt to manage compliant crawling, and use server authentication or access controls when pages must not be publicly fetched.
  • Keep in mind that allowing Googlebot to crawl a page does not guarantee Google will index or rank it.

Troubleshooting

Google cannot crawl a page: Check the live robots.txt file for a matching Googlebot group and any path rule that covers the URL. Review redirects, hostname differences, and other access restrictions as well. Test the exact URL in Search Console.

An AI crawler still visits: Confirm the request’s actual source and user-agent, then compare its name with the rule. The crawler may be using another agent name, ignoring robots.txt, or reaching a different host. Robots rules cannot stop a bot that chooses not to comply; use server-side controls if you need to restrict access.

Your changes do not appear: Make sure the file was saved to the web root, check the exact HTTP or HTTPS hostname, and look for CDN or hosting caches. Fetch the public URL again after publishing. Allow time for crawlers to revisit the file.

A page remains in Google results: Robots.txt controls crawling, not guaranteed removal from search. If the page should be private, restrict access. If it should be excluded from results, use an appropriate noindex method that Google can crawl, or follow Google’s removal guidance for eligible cases.

Frequently Asked Questions

Can I block AI crawlers but let Google Search crawl my site?
Yes. Add disallow rules for specific, verified AI crawler user-agents and avoid rules that disallow Googlebot or all crawlers. Test important URLs afterward.

Does robots.txt stop AI companies from using content they already collected?
No. It is a prospective crawling request, not a deletion request or a way to control previously collected material.

Will these rules prevent every AI crawler from accessing my pages?
No. They apply to crawlers that recognize and follow the rules. Use authentication or server-side restrictions for content that must not be publicly accessible.

Should I block User-agent: *?
Not if you want general crawlers, including Googlebot where applicable, to access the site. Use specific groups for the bots you intend to restrict.

Will blocking a crawler remove my pages from Google Search?
No. Robots.txt is not a reliable removal method. A blocked URL can still be listed without a snippet if Google discovers it elsewhere.

Do I need separate rules for subdomains?
Yes. Each hostname, including a subdomain, can have its own robots.txt file and crawling rules.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.