Crawler Intelligence

Overview

Crawler Intelligence — Agent access in the sidebar — tracks how AI company bots interact with your website. It checks your robots.txt and ai.txt files to determine which AI crawlers are allowed or blocked, logs the results over time, and alerts you when access policies change.

If an AI model cannot crawl your website, it cannot learn from your content. Agent access ensures you have full visibility into your crawl access posture.

The page is an access check, not a score. It reports three kinds of fact and never blends them: what your robots files say about each named agent, which agents actually fetched your pages (from your access logs), and what your robots files should say. Since 9 September 2026 there is no composite "readiness" number on the page — see Retired: the AI Bot Readiness score.

Supported AI Crawlers

The system monitors the following AI crawlers:

Two figures, never one total (from 4 September 2026)

The access-logs section of the page shows fetches from uploaded access logs as two separate figures and never adds them together:

• Fetched to answer a question — agents that retrieve a page while an answer is being composed for someone (OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-User, Claude-SearchBot). This is the figure that tracks whether your site is being used to answer questions.
• Crawled for training — agents harvesting content for model training or a search index (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent). More of these does not mean more answers cite you.

Where one vendor uses a single agent for both jobs (PerplexityBot, Amazonbot) it is counted under training. That understates live retrieval, which is the safer error: the product never claims a question was answered from your site when it was not. An agent the registry does not know is counted in neither figure and disclosed as a remainder under the cards.

The same registry groups the access table at the top of the page (Answers questions · Training / index · Other agents · robots.txt permissions) and drives the robots.txt guidance, read the other way round: a mixed agent is treated as a search agent there, because advising you to block an agent that may be answering questions from your site is the costlier mistake in that direction. One list, two readings, each erring on the side that cannot mislead you.

Google-Extended and Applebot-Extended are not crawlers. They are robots.txt directive names that gate what Google and Apple may do with pages their other agents already fetched. They never appear as a fetching user agent, so they are excluded from every crawler count, listed in their own group at the foot of the access table with a dash where a check time would go, and dropped at log ingest if a line claims them.

Fetch-to-citation lag is the median number of days between an AI bot first fetching a page and that page first appearing as a citation, over pages where the fetch came before the citation. A citation observed before the earliest uploaded fetch says only that your log window started after the fetch that mattered — it is not evidence of lag and is left out. When no page qualifies, the card says so rather than showing a number.

How It Works

1. Policy Check — The system fetches your website's robots.txt, ai.txt, AND llms.txt files and parses the directives for each known AI crawler. Crawler user-agents are matched fuzzily, so versioned agents like GPTBot/1.0 still match the GPTBot rule. This is an on-demand, point-in-time scan (run when you click "Scan") — not continuous live monitoring; the dashboard shows when it was last checked.
2. Status Logging — Each bot's access status (allowed, blocked, or unknown) is recorded in the crawler_logs table with a timestamp.
3. Pattern Detection — By comparing current results with historical logs, the system detects behavioral patterns such as newly blocked bots, newly discovered bots, or status changes.
4. Alerting — Critical patterns (like a previously allowed bot being blocked) are flagged for review.

What the page shows

Top to bottom, three sections — every figure a count or a per-agent fact, none of them blended:

1. Which AI agents your robots.txt allows or blocks — two counts (Agents allowed, Agents blocked, each "of N agents checked") and the access table: one row per named agent, grouped by what it fetches for, with what robots.txt says, the rule that decided it, when it was last checked, and any change since the last check. Robots.txt permissions (Google-Extended, Applebot-Extended) sit in their own group with a dash for the check time.
2. Which agents actually fetched your pages — the two log figures above, the per-bot table (hits, pages, last seen, trend), the bot × day grid, the crawl → citation funnel and the fetch-to-citation lag. This section fills once you upload access logs; until then it explains how to connect them.
3. What your robots files say — whether robots.txt, ai.txt and llms.txt exist, the problems found in them line by line, and what to change.

The first line under the title states the page's scope: site-wide, read from your robots.txt and your access logs, not filtered by question type or capture cycle. There is no lens on this page.

Changes since the last check

Each check of your robots files is compared with the previous one. A change appears in the access table's Change since last check column against the agent it concerns, with the date it was seen:

Critical changes trigger alerts. Mark reviewed clears a change from the table once you have addressed it.

Retired: the AI Bot Readiness score

Until 9 September 2026 the page led with a 0–100 "AI Bot Readiness" score and a waterfall of its factors (+20 for a robots.txt, +15 for an AI-policy file, +8 per allowed bot up to +30, −10 per blocked bot down to −25, +15 for no critical issues). The weights were hand-picked and could not be defended, and a crawl check is an access check, not a success measure — so the score was removed and the facts it summed now lead the page as counts and a table. Its definition remains, marked retired, in the Metric definitions appendix for one release.

How to Use

1. Navigate to Agent access in the sidebar.
2. Check access once to fill the access table, then review it to confirm the answering agents (OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-User) are allowed.
3. Investigate any Newly blocked change — a blocked agent means an AI model may stop referencing your content.
4. Connect your access logs to see which agents actually fetched your pages, and how long fetches take to become citations.
5. Update your robots.txt or ai.txt from the "What to change" list, then re-check.
6. Mark changes reviewed once you have addressed them.

Best Practices

• Allow the answering agents first — OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-User. These are the fetches that turn into citations in live answers. Allow GPTBot and ClaudeBot if you also want your content in future model training, and allow the Google-Extended directive if you want Gemini and AI Overviews to use pages Googlebot already indexes.
• Review ai.txt and llms.txt in addition to robots.txt. Some organizations maintain separate AI-specific crawler policies; llms.txt is the newer, increasingly adopted standard for per-LLM access and training control.
• Monitor weekly for unexpected changes, especially after website deployments that may overwrite robots.txt.
• Set up alerts so you are notified immediately if a critical crawler gets blocked.

Plan Requirements

Crawler Intelligence is available on all plans that can capture snapshots (no separate feature gate).