AI Crawler Checklist: Why Was My Page Never Read?

Updated 9 November 2026 · Standard formula, shown below

An AI crawler checklist helps you find out why a page may not have been fetched, rendered, or made available to an AI service. Check the crawler’s access, your server and robots rules, the page’s response and content, then verify what the specific service says it collects; a page can be crawlable without being used in an answer or training set.

What “never read” can mean

“AI crawler” is not one universal bot. Different organizations operate crawlers for search, retrieval, model training, or other purposes, and their policies and user agents differ. A page might be visited by one crawler but not another. It might also be fetched successfully but omitted from a search index, retrieval system, or generated response.

Separate these stages when diagnosing a problem:

Discovery: Could the crawler find the URL through links or a sitemap?
Access: Did the server allow that crawler to request it?
Rendering: Could it obtain the meaningful content, including any content produced by JavaScript?
Use: Does the service actually index, retrieve, cite, or train on the page? A successful visit does not prove any of these later steps.

There is no reliable way to infer that a particular page was “read” by every AI system. Use server records and the relevant service’s published documentation to answer narrower questions, such as whether a named crawler requested a URL.

Start with this AI crawler checklist

Confirm the exact URL. Check the preferred canonical URL, redirects, capitalization, trailing slash, and whether the page is public. Test the URL without a logged-in session or browser-only access.
Check the HTTP response. The page should return the intended content and an appropriate success response to an ordinary request. Investigate unexpected 403, 404, 429, or 5xx responses, redirect loops, and intermittent failures.
Review robots.txt. Look for rules that disallow the crawler’s documented user agent or block a relevant path. Robots rules guide compliant crawlers; they are not access control and do not guarantee that a crawler will visit.
Inspect page-level directives. Check meta robots tags and HTTP X-Robots-Tag headers for directives such as noindex. Confirm that templates or middleware are not applying restrictions unintentionally.
Check authentication and network controls. A login wall, IP allowlist, bot challenge, firewall, CDN rule, or rate limit can prevent access even when robots.txt permits it.
Verify the content is available in the response. Look at the returned HTML, not only the visual browser page. Important text hidden behind a script, interaction, or delayed request may not be available to every crawler.
Inspect logs and retest. Search access logs for requests to the exact path and time period, including status codes and user-agent strings. After changes, test again and allow for recrawling delays.

How robots rules, indexing, and crawler identity differ

Robots.txt is a site-level text file that can request that compliant crawlers avoid specified paths. It is not a command to crawl, and a rule allowing access does not mean the crawler has discovered the page. A disallow rule also does not remove a URL from every index; the URL might be known through other sources even if its contents cannot be fetched.

Page-level robots directives are different: they are delivered with the page, usually as HTML metadata or an HTTP header. A crawler must be able to fetch the page to see a meta directive in its HTML. If access is blocked before the page is fetched, that directive may not be seen.

Do not identify a bot by user-agent text alone. User-agent strings can be copied, and crawler identity checks vary by operator. If a service publishes verification guidance, follow that method before treating a request as genuine. Avoid loosening firewall protections just because a request claims to be an AI bot.

Common reasons a page is missed

It is hard to discover. An orphan page with few internal links may not be found promptly. A sitemap can help discovery, but it is not a guarantee of crawling or indexing.
The server blocks or limits requests. Security rules, overloaded hosting, rate limits, and transient outages can cause failed or incomplete fetches.
The page is thin or duplicated. A crawler may fetch a page but a downstream system may choose not to index or retrieve it, especially when the content adds little beyond another URL. The reasons are service-specific and may not be disclosed.
The useful text is not in the response. A browser may display content after JavaScript runs, while a crawler that does not execute the same scripts sees a mostly empty shell.
The page changed recently. Crawlers do not necessarily revisit immediately. A corrected robots file or restored page can take time to be observed.
The service does not use that crawler’s data for the outcome you expect. Crawling for search or retrieval is not proof of inclusion in model training or a particular answer.

Check logs, rendering, and content safely

Begin with your hosting, CDN, or web-server logs. Filter for the URL, then note timestamps, status codes, request frequency, and user-agent. A 200 response suggests a request succeeded at the HTTP level, but it does not prove the crawler parsed, stored, or used the page. A missing log entry is not conclusive either: logs may be incomplete, retained briefly, or recorded at another layer.

Compare the raw response with the rendered page. If the essential information appears only after a script call, consider making the core text available in crawlable HTML as well. Use clear headings, descriptive links, and a stable canonical URL. These are sound publishing practices, not guarantees of selection by an AI system.

Keep access controls in place while testing. Do not expose private or personal information to make a page easier to crawl, and do not remove authentication from material that should remain restricted. If you need to block a crawler, use the operator’s documented controls where available and remember that robots.txt is not a security boundary.

What a successful crawl does—and does not—prove

A verified request can establish that a particular crawler reached a URL at a particular time. It cannot establish that the entire page was understood, indexed, retained, or cited. Likewise, a page appearing in an AI-generated answer does not necessarily mean the system fetched it during that query; the service may rely on an index or other sources.

For a useful diagnosis, record the URL, the crawler identity and verification method, the response code, relevant directives, and the date of each test. Then consult the service’s own documentation for its crawling and content-use policies. This makes the question precise: “Did this crawler fetch this page?” is more answerable than “Did AI read my site?”

Frequently Asked Questions

How can I tell whether an AI crawler visited my page?
Check server or CDN access logs for the URL and a request identified as that crawler. Verify the crawler identity using the operator’s published method; a user-agent string alone is not proof.

Does allowing a bot in robots.txt make it crawl my page?
No. It only indicates that the path is not disallowed by that robots file. Discovery, server availability, crawler policy, and later indexing or use are separate matters.

Can JavaScript stop an AI crawler from reading a page?
It can make content unavailable to crawlers that do not execute the relevant scripts or requests. Check the raw HTML response and make essential information available without requiring a browser interaction where practical.

Why does my page return 200 but not appear in an AI answer?
A 200 response confirms an HTTP response, not indexing or inclusion in an answer. The service may not use that crawler’s data for answers, may not have selected the page, or may not have revisited it yet.

Will a sitemap guarantee that an AI bot finds my page?
No. A sitemap can provide URL discovery information, but it does not require a crawler to fetch, index, or use those URLs.

Should I block all AI crawlers?
That depends on your publishing and privacy goals. Review each operator’s documented controls and decide which, if any, to allow. Keep private content protected with real access controls rather than relying on robots.txt.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.