AI crawler access

Overview

AI crawler access tracks how AI company bots interact with your website. It checks your robots.txt and ai.txt files to determine which AI crawlers are allowed or blocked, logs the results over time, and alerts you when access policies change.

If an AI model cannot crawl your website, it cannot learn from your content. AI crawler access ensures you have full visibility into your crawl access posture.

The page is an access check, not a score. It reports three kinds of fact and never blends them: what your robots files say about each named crawler, which crawlers actually fetched your pages (from your access logs), and what your robots files should say. Since 9 September 2026 there is no composite "readiness" number on the page — see Retired: the AI Bot Readiness score.

When your site refuses our reader. Only a "not found" answer means a file isn't there. If your site refuses our reader or doesn't answer, the check says Couldn't read and lists each crawler as Check failed, not as allowed. We only know what our own reader got, so AI crawlers may get the same answer. Blocking automated readers can be your own choice. If you didn't mean to, your firewall or CDN's bot settings are the place to look.

Supported AI Crawlers

Each fetch in your access logs is classified against one registry of crawlers. It includes the crawlers below, among them two known crawlers that are not AI crawlers, recognised so their fetches are never counted as AI crawls. The robots.txt check reads its own, shorter list: every crawler it checked is a row of the access table, and not every crawler below is checked there.

Three figures, never one total (from 21 October 2026; two from 4 September 2026)

The access-logs section of the page shows fetches from uploaded access logs as three separate figures and never adds them together:

• Fetched to answer a question — crawlers that fetch a page because someone asked a question, while the answer is being composed (ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher, Amzn-User, MistralAI-User). This is the figure that tracks whether your site is being used to answer questions.
• Crawled for AI search — an AI search engine's own crawlers, building the index its answers are drawn from (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Amzn-SearchBot, DuckAssistBot). No one asked a question at the moment of the fetch; being in that index is what lets the engine cite you later.
• Crawled for training — crawlers harvesting content for model training, such as GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent and Amazonbot. More of these does not mean more answers cite you.

Where one crawler does more than one job with a single user agent (GoogleOther, YouBot) it is counted under training. That understates live retrieval, which is the safer error: the product never claims a question was answered from your site when it was not. A crawler the registry does not know is counted in neither figure and disclosed as a remainder under the cards.

Known crawlers that are not AI crawlers are counted in neither figure either, and disclosed on their own line — under the cards, or under the empty state when there is no AI-crawler data — never as crawlers the registry could not classify. Two are recognised: OAI-AdsBot, which checks the pages submitted as ads on ChatGPT, and Google-CloudVertexBot, which crawls the pages a site owner asked it to, for that owner's own Vertex AI projects — counting it as AI visibility would credit your own request. Search indexers (Googlebot, Bingbot, Applebot) are counted apart as well, in their own series.

When this reading changed. From 21 October 2026, AI search crawlers are counted as their own class (Crawled for AI search): OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Amzn-SearchBot and DuckAssistBot. Before that date they were counted under Fetched to answer a question. Purpose is read from the stored crawler name each time the page is drawn, so this applies to fetches already stored too; no stored fetch changed figure, as none had been uploaded (measured live 2026-10-21). On 24 September 2026 PerplexityBot moved from training to Fetched to answer a question (Perplexity documents it as a search crawler that is not used for training), Amazonbot moved from "both jobs" to training (still counted under training, as before), Meta-ExternalFetcher was split out of Meta-ExternalAgent as a fetch made for a user, and Amzn-SearchBot, Amzn-User, DuckAssistBot, MistralAI-User, Meta-WebIndexer, OAI-AdsBot and Google-CloudVertexBot were added. Purpose is read from the stored crawler name each time the page is drawn, so the two reclassifications apply to fetches already stored, while the split and the additions apply to fetches stored from this date. No stored fetch was counted under the old reading: there were none (Measured live 2026-09-23). Each fetch is checked against the IP ranges its vendor publishes when it is stored, and keeps that check: a fetch stored before the first IP-range refresh after this change keeps the result it received then, so an OAI-AdsBot fetch may read "outside published ranges".

The same registry groups the access table at the top of the page (Answers questions · Crawls for AI search · Training · Other crawlers · robots.txt permissions). It also keeps a second, robots.txt-guidance reading, the other way round: a mixed crawler is treated as a search crawler there, because advising you to block a crawler that may be answering questions from your site is the costlier mistake in that direction. That reading is stored with each fetch; no recommendation on this page is built from it yet. One list, two readings, each erring on the side that cannot mislead you.

Google-Extended and Applebot-Extended are not crawlers. They are robots.txt directive names that gate what Google and Apple may do with pages their crawlers already fetched. They never appear as a fetching user agent, so they are excluded from every crawler count, listed in their own group at the foot of the access table with a dash where a check time would go, and dropped at log ingest if a line claims them.

Fetch-to-citation lag is the median number of days between an AI bot first fetching a page and that page first appearing as a citation, over pages where the fetch came before the citation. A citation observed before the earliest uploaded fetch says only that your log window started after the fetch that mattered — it is not evidence of lag and is left out. When no page qualifies, the card says so rather than showing a number.

How It Works

1. Policy Check — The system fetches your website's robots.txt, ai.txt, AND llms.txt files and parses the directives for each known AI crawler. Crawler user-agents are matched fuzzily, so versioned user-agent strings like GPTBot/1.0 still match the GPTBot rule. It runs once a night for every live project, and again whenever you click Check access now; each run is a point-in-time check of your files, not live monitoring, and the dashboard shows when it was last checked. It records what your files allow, never whether a crawler fetched anything (that comes from your access logs). From 23 October 2026 the nightly check also runs for a project with no new answer in 31 days; from 7 to 22 October 2026 it was skipped for such a project, so its access table could show an older check date.
2. Status Logging — Each bot's access status (allowed, blocked, or unknown) is recorded in the crawler_logs table with a timestamp.
3. Pattern Detection — By comparing current results with historical logs, the system detects behavioral patterns such as newly blocked bots, newly discovered bots, or status changes.
4. Alerting — Critical patterns (like a previously allowed bot being blocked) are flagged for review.

What the page shows

Top to bottom, three sections — every figure a count or a per-crawler fact, none of them blended:

1. Which AI crawlers your robots.txt allows or blocks — two counts (Crawlers allowed, Crawlers blocked, each "of N crawlers checked") and the access table: one row per named crawler, grouped by what it fetches for, with what robots.txt says, the rule that decided it, when it was last checked, and any change since the last check. Robots.txt permissions (Google-Extended, Applebot-Extended) sit in their own group with a dash for the check time.
2. Which crawlers actually fetched your pages — the live crawler feed (BETA, below), then the two log figures above, the per-bot table (hits, pages, last seen, trend), the bot × day grid, the crawl → citation funnel and the fetch-to-citation lag. This section fills once you connect or upload access logs; until then it explains how to connect them.
3. What your robots files say — whether robots.txt, ai.txt and llms.txt exist, the problems found in them line by line, and what to change. The unblock recommendation covers only crawlers blocked from your whole site (Disallow: /) and counts AI crawlers apart from robots.txt permissions (Google-Extended, Applebot-Extended) and search indexers (Bingbot) — for example "Unblock 2 AI crawlers and allow 1 AI permission". A crawler blocked from part of your site only is never told to unblock: it gets its own, lower-priority line asking you to check that what is blocked is what you mean to keep from AI crawlers. A rule for the Applebot crawler no longer marks the Applebot-Extended permission as blocked. These apply from the first scan of your robots files after the change on 24 September 2026; a scan stored before it keeps its old wording until the next scan.

The first line under the title states the page's scope: site-wide, read from your robots.txt and your access logs, not filtered by question type or capture cycle. There is no lens on this page.

The live crawler feed (BETA)

A project admin connects an Nginx or Apache server by creating a site key, which is shown once, and installing the log reader the card describes: one Python file that sends only the log lines naming a known crawler. The card prints the files' SHA-256 digests and the configuration with the key already in it. Upload a log file points to the upload card; Another host asks which host you use and records the answer.

A site on Cloudflare connects with a Cloudflare Worker instead. The card prints the Worker and its wrangler.toml; the site key goes into a Worker secret at the prompt, never into a file. The Worker counts requests your site served to AI crawlers at the edge, including pages Cloudflare serves from its cache, and sends them to AEO Optima in batches through a Cloudflare Queue; a check-in every 5 minutes keeps quiet days counted as covered. People's requests never leave Cloudflare. If Edge Delivery already runs its Worker on the same route, deploy the Worker from the Edge Delivery page: it is the same Worker and counts too. Cost: on Cloudflare's Free plan, nothing for most small sites; very busy sites may need Cloudflare's $5/month Workers plan (see Cloudflare's Workers pricing page). On the Free plan, set the route's request-limit failure mode to fail open, so the site keeps working if the daily limit is reached.

A site on Vercel, or on Netlify (Enterprise), can connect with a log drain instead (BETA, from 8 October 2026). A drain sends every request record from your site, including visitors' IP addresses, so the card first asks you to confirm that; AEO Optima keeps only the AI-crawler requests and discards everything else in memory, unstored, and stores crawler requests without the IP address. The card then shows a site key once and what to enter at your host: on Vercel, Team Settings → Drains → Add Drain → Logs, with our endpoint and the key as an Authorization custom header; on Netlify, Analytics & metrics → Log Drains, a General HTTP endpoint for traffic logs, with the key in the Authorization header. The privacy policy and the DPA describe what happens to these records.

Once connected, the feed shows each source's health (its last batch, the lines received, kept and refused in the last 24 hours, and the AI-crawler requests it delivered yesterday and today), then two grids on one date axis:

• AI-crawler requests by day, by crawler family, with verified (from the vendor's published addresses), unverified (the vendor publishes no list, or the address was not checked) and outside published ranges (claimed to be the vendor's crawler from outside its list) on separate rows. Search indexers such as Googlebot are counted apart. A request from outside a vendor's published addresses is in no class, on the AI bot traffic page's months, the Overview or the scorecard: the page tells you how many there were. Where a vendor publishes no list, the request is counted and can't be checked. The company, bot and page tables count every request received, including those.
• Answers citing your site, by capture check: on the same days, each check's share of answers that cite your own site, per engine, with the crawlers that feed that engine named beside it.

A day no source covered reads n/a: it was not measured, which is never the same as zero, and the feed never says that no crawler visited. While it waits for the first batch it checks every 10 seconds, and only while the page is open in a visible tab. Each capture check is paired with the verified requests from its engines' crawlers in the days before it; the pairs are listed with their count, read side by side only from 8 paired checks, and as association, never cause.

Changes since the last check

Each check of your robots files is compared with the previous one. A change appears in the access table's Change since last check column against the crawler it concerns, with the date it was seen:

Critical changes trigger alerts. Mark reviewed clears a change from the table once you have addressed it.

Retired: the AI Bot Readiness score

Until 9 September 2026 the page led with a 0–100 "AI Bot Readiness" score and a waterfall of its factors (+20 for a robots.txt, +15 for an AI-policy file, +8 per allowed bot up to +30, −10 per blocked bot down to −25, +15 for no critical issues). The weights were hand-picked and could not be defended, and a crawl check is an access check, not a success measure — so the score was removed and the facts it summed now lead the page as counts and a table. Its definition remains, marked retired, in the Metric definitions appendix for one release.

How to Use

1. Navigate to AI crawler access in the sidebar.
2. Check access once to fill the access table, then review it to confirm the answering and AI search crawlers, such as ChatGPT-User, Perplexity-User, Claude-User, OAI-SearchBot and PerplexityBot, are allowed.
3. Investigate any Newly blocked change — a blocked crawler means an AI model may stop referencing your content.
4. Connect your access logs to see which crawlers actually fetched your pages, and how long fetches take to become citations.
5. Update your robots.txt or ai.txt from the "What to change" list, then re-check.
6. Mark changes reviewed once you have addressed them.

Best Practices

• Allow the answering and AI search crawlers first, such as ChatGPT-User, Perplexity-User, Claude-User, OAI-SearchBot and PerplexityBot. The first fetch pages while answering, and the AI search crawlers keep you in the index those answers are drawn from: together they are what turns into citations in live answers. Allow GPTBot and ClaudeBot if you also want your content in future model training, and allow the Google-Extended permission if you want Google to use your pages to train Gemini models and ground Gemini answers; it does not affect your inclusion in Google Search.
• Review ai.txt and llms.txt in addition to robots.txt. Some organizations maintain separate AI-specific crawler policies; llms.txt is the newer, increasingly adopted standard for per-LLM access and training control.
• Monitor weekly for unexpected changes, especially after website deployments that may overwrite robots.txt.
• Set up alerts so you are notified immediately if a critical crawler gets blocked.

Plan Requirements

AI crawler access is available on all plans that can capture snapshots (no separate feature gate).