Robots.txt Rules for AI Crawlers: Write and Test Them

Updated 9 November 2026 · Standard formula, shown below

Robots.txt rules tell compliant web crawlers which parts of a site they may request, including crawlers used to collect content for AI services. This guide explains how to identify a crawler, write a rule for it, and check that the rule is published correctly; robots.txt is a request, not access control, and cannot guarantee how a company uses content it has already collected.

What You’ll Need

Permission to edit and publish files at your website’s root, or help from whoever manages hosting or deployment.
Your site’s public origin, such as https://example.com, and a way to open its robots.txt file in a browser.
The crawler’s documented user-agent name and the content paths you want to allow or disallow. Check the crawler operator’s current documentation rather than guessing.
A text editor that saves plain text. Avoid editors that silently add formatting or change the file extension.

Step-by-Step

Choose the exact host to control. A robots.txt file applies to one scheme, host, and port. For example, https://www.example.com/robots.txt does not control https://example.com/robots.txt, a subdomain such as shop.example.com, or the HTTP version. Decide which host serves the pages you want to manage, and check whether redirects lead to a different host.
Identify the crawler and your goal. Find the crawler’s exact user-agent token in its operator’s documentation. Some services use different crawlers for search indexing, user-requested fetches, and model training or data collection. Those purposes may have different tokens and rules. Decide whether you want to block all crawling by that token or only particular URL paths. Do not assume a rule for one crawler covers every AI-related crawler.
Inspect the existing file before changing it. Visit the site’s robots.txt URL and save a copy of its contents. Look for existing User-agent, Allow, and Disallow groups, as well as a sitemap declaration. Preserve rules that serve other crawlers. If the file returns an error, is generated by a content management system, or is managed by a deployment pipeline, find the correct source of truth before editing.
Write a crawler-specific group. Put a User-agent line before the rules that apply to it. For example, the following illustrative group blocks a crawler named ExampleBot from the whole site:
User-agent: ExampleBot Disallow: /
Replace the example with a documented token. A targeted path might instead be Disallow: /private-content/. A slash at the end can make the intended directory boundary clearer. A blank line separates groups; keep each crawler’s rules in the correct group.
Choose path rules carefully. Disallow: / requests that the matched crawler avoid every path on that host. Disallow: /drafts/ targets paths beginning with that directory. Disallow: with nothing after the colon means no path is disallowed for that group. Matching details, including wildcard and end-of-URL support, can differ between crawlers, so use only syntax the crawler documents. Robots rules generally match URL paths, not text that appears on a page.
Keep the file valid and readable. Use plain text with one directive per line. Directives are commonly written as User-agent, Disallow, and Allow; avoid combining unrelated crawler policies into one group. Comments can start with #, but put important rules on their own lines. If a crawler supports Allow, it can be used for a specific exception, but overlapping rules are interpreted according to that crawler’s matching rules. Check its documentation before relying on an exception.
Publish to the correct location. The public file must be available at the root of the relevant host, for example https://www.example.com/robots.txt, not in a subfolder. It should return a successful response and plain-text content, not a login page or an HTML error. If a content delivery network or cache is in front of the site, publishing may not update every response immediately.
Test the live result. Open the exact robots.txt URL in a private browser window or use a command-line request such as curl -i https://www.example.com/robots.txt. Confirm the response is successful, the rules are the version you intended, and there are no unexpected redirects, access challenges, or HTML error pages. Use any crawler-specific testing tool offered by its operator, but treat it as a check of that crawler’s interpretation, not a universal validator.
Test representative URLs and monitor. Check a URL you intend to block and one you intend to leave available, using the crawler’s documented tester or matching rules. Remember that a disallowed URL may still appear in search results if other pages link to it, even when its content is not fetched. After deployment, revisit the live file and review server logs or crawler reports where available. Update the policy if paths or crawler documentation change.

Common Mistakes

Assuming one file covers every subdomain. Each host needs its own robots.txt policy. Check the exact hostname used in published links.
Using robots.txt as a privacy barrier. Disallowed URLs are still publicly discoverable, and compliant crawlers may obey the request without preventing other visitors from accessing the content. Use authentication or server-side access controls for confidential material.
Blocking the wrong token or purpose. A product can operate multiple crawlers. Confirm which one handles the activity you mean to restrict.
Replacing the whole file. Overwriting existing groups can unintentionally block search crawlers or remove useful sitemap information. Merge changes carefully and keep a backup.
Expecting retroactive removal. A new rule does not erase copies already fetched or guarantee that previously indexed URLs disappear. Use the relevant service’s separate removal or content-control process when available.

Tips

Start with the narrowest path rule that meets your goal; use a whole-site block only when that is intentional.
Keep a dated copy of each published version and note why each crawler group exists. This makes later reviews safer.
Check the site’s page templates and internal links for sensitive URLs. Robots.txt is not a substitute for keeping private content off a public site.
Review crawler documentation periodically. User-agent tokens and stated crawler purposes can change.
Make one policy change at a time, then verify the live response and both blocked and allowed examples.

Troubleshooting

The file shows an old version: Check whether your host, content management system, or deployment process generates the file. Purge the relevant cache if you control it, then request the URL again. Confirm you are checking the same host the crawler visits.

The URL returns 404, 403, or an HTML page: Put the file at the host root and check hosting permissions, rewrite rules, and security challenges. A crawler that cannot retrieve the file may not interpret your intended policy as expected; consult that crawler’s documentation for its handling of fetch failures.

A path you meant to block still gets requested: Verify the exact user-agent token and URL path, including capitalization, encoding, query strings, and redirects. Check for a conflicting or misplaced group, and confirm the crawler actually follows robots.txt. Some automated clients do not.

A page is blocked but still appears in search: Robots.txt can prevent crawling but does not reliably remove a URL from an index. Review the search or service operator’s removal process. If the page must not be accessible, restrict it at the server rather than relying on a crawl rule.

An allowed URL is unexpectedly blocked: Inspect all matching groups and the crawler’s rule precedence, then test the exact URL with its supported tool. Simplify overlapping rules where possible and avoid assuming every crawler implements extensions identically.

Frequently Asked Questions

Does robots.txt stop AI systems from using my content?
No. It asks compatible crawlers not to fetch specified URLs. It does not control noncompliant crawlers, content already collected, or every downstream use.

Can I block all AI crawlers with one wildcard?
A wildcard user-agent group can apply to crawlers that follow the standard convention, but it is not a dependable way to identify every AI crawler. Use documented tokens for services you want to address.

Will a robots.txt block hide a URL from search results?
Not necessarily. A URL may be listed without its content if it is discovered through links. Follow the relevant search provider’s removal guidance if you need a result removed.

Should I use Disallow: /?
Only if you intend the matching crawler to avoid the entire host. It can also prevent legitimate crawling for other purposes if the token or group is broader than expected.

How long does a change take to take effect?
There is no universal timing. Crawlers may cache robots.txt and revisit it on different schedules. Confirm the live file is correct, then consult the crawler operator for its cache behavior.

Can I use robots.txt to protect confidential pages?
No. The file is public, and disallowed paths may be visible to anyone. Protect private pages with authentication or other server-side access controls.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.