Short answer: An AI crawlability audit (an AI technical SEO audit) checks whether the crawlers that feed AI search, mainly Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot, can reach, render and quote your priority pages. It covers five layers: robots.txt rules per crawler, firewall and CDN access verified against each vendor's published IP list, server-rendered text, indexing and snippet controls, and server-log proof that each bot actually receives 200 responses. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are separate decisions that do not affect AI search inclusion.
What does an AI crawlability audit cover?
A classic technical SEO audit asks whether Googlebot can crawl and index your site. An AI technical SEO audit asks the same question for every system that might quote you, and adds one more: is the answer text present in a form the system can lift? We run it on a short list of priority URLs (service pages, pricing pages, top guides, case studies), not on every URL, because AI answers cite specific pages and a 20-page audit done properly beats a 5,000-URL crawl nobody acts on.
| Layer | Question | Typical failure we find |
|---|---|---|
| 1. Policy | Does robots.txt allow the search crawlers you want? | A "block AI bots" plugin added Disallow rules for OAI-SearchBot and PerplexityBot along with GPTBot |
| 2. Network | Do the CDN, WAF and host let verified bots through? | Cloudflare or hosting bot protection returns 403 or a JavaScript challenge |
| 3. Rendering | Is the answer in the HTML a crawler receives? | Prices and FAQs load by JavaScript after page load |
| 4. Indexing and snippets | Is the page indexable, canonical and allowed to be quoted? | Stray noindex from staging, or nosnippet on the main content |
| 5. Evidence | Do server logs show each bot getting 200 responses? | Zero OAI-SearchBot hits in 30 days on an established site |
Content quality and measurement are separate jobs; see content AI search engines cite and our AI visibility tracking framework.
Which robots.txt rules apply to each AI crawler?
Most AI vendors now run separate user agents for search, training and user-triggered fetches. The table summarises each vendor's own documentation as checked on 11 October 2026. The practical rule: allow the search crawlers, decide on training crawlers as a business policy, and understand that user-triggered fetchers may not follow robots.txt at all.
| Vendor | robots.txt token | Job | What blocking it does | Verify with |
|---|---|---|---|---|
| Googlebot | Search crawling, including AI Overviews and AI Mode | Removes crawl access for Search, so no AI Overview citations | Reverse DNS to googlebot.com, google.com or googleusercontent.com (Google) | |
| Google-Extended | Control token for Gemini training and grounding in other Google systems; no separate crawler | Google says it "does not impact a site's inclusion in Google Search" (Google) | Not applicable: it has no HTTP user agent | |
| OpenAI | OAI-SearchBot | ChatGPT search | Sites opted out "will not be shown in ChatGPT search answers" (OpenAI) | openai.com/searchbot.json |
| OpenAI | GPTBot | Training data for foundation models | Opts out of training only; settings are independent | openai.com/gptbot.json |
| OpenAI | ChatGPT-User | Fetches a page when a user asks | OpenAI says robots.txt rules "may not apply" | openai.com/chatgpt-user.json |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results; not used for model training | Removes you from Perplexity's search index (Perplexity) | perplexity.com/perplexitybot.json |
| Perplexity | Perplexity-User | User-requested fetches | "Generally ignores robots.txt rules" | perplexity.com/perplexity-user.json |
| Anthropic | Claude-SearchBot | Indexes content to improve Claude's search results | Content not indexed for Claude search (Anthropic) | claude.com/crawling/bots.json |
| Anthropic | ClaudeBot | Collects content that may be used for training | Future content excluded from training | claude.com/crawling/bots.json |
| Anthropic | Claude-User | Fetches pages for a user's question | May reduce visibility in user-directed web search | claude.com/crawling/bots.json |
| Apple | Applebot | Search features including Siri and Spotlight | Removes Apple search visibility; if Applebot is not named, it follows your Googlebot rules (Apple) | Apple's published guidance |
| Apple | Applebot-Extended | Training opt-out token; "does not crawl webpages" | Pages "can still be included in search results" | Not applicable |
| Common Crawl | CCBot | Open web archive used widely in research and model building | Excludes you from future crawls (Common Crawl) | Reverse DNS to crawl.commoncrawl.org; index.commoncrawl.org/ccbot.json |
| Meta | meta-externalagent | Crawls for uses such as training foundation AI models | Disallow to opt out (Meta) | Meta's crawler page |
Bingbot belongs in the "allow" group as well: Microsoft Copilot and Bing's AI summaries draw on Bing's index, and our ChatGPT search guide explains why Bing coverage also matters for ChatGPT. Two more details from the vendor pages: OpenAI and Perplexity both say robots.txt changes can take about 24 hours to take effect, and Anthropic notes it honours the non-standard Crawl-delay directive and that rules must be repeated for each subdomain's robots.txt.
What should a robots.txt for AI search look like?
A common policy is: visible in every AI search product, opted out of model training. Adapt the training lines to your own decision; there is no search penalty either way according to the vendors above.
| User-agent | Rule | Why |
|---|---|---|
OAI-SearchBot | Allow: / | ChatGPT search |
PerplexityBot | Allow: / | Perplexity results |
Claude-SearchBot | Allow: / | Claude search results |
GPTBot | Disallow: / | Training opt-out (OpenAI) |
ClaudeBot | Disallow: / | Training opt-out (Anthropic) |
Google-Extended | Disallow: / | Gemini training and grounding opt-out |
Applebot-Extended | Disallow: / | Apple training opt-out |
CCBot | Disallow: / | Common Crawl archive opt-out |
* | Disallow: /admin/ | Everyone else, including Googlebot and Bingbot, crawls all public pages |
End the file with a Sitemap: https://www.example.in/sitemap.xml line.
Points to check in your own file:
- A crawler obeys the most specific group that names it and ignores the
*group. If you name OAI-SearchBot, repeat any path exclusions (such as/admin/) inside its group. - Blocking Google-Extended does not remove you from AI Overviews. To stop Google AI features using your content, use the Search Console control or snippet rules covered below.
- If you publish an llms.txt file, it does not replace any of this; see does llms.txt help SEO?
- robots.txt does not keep a page out of Google: a blocked URL can still appear in results without a description. Use
noindexfor that.
How do you check that the firewall or CDN is not blocking AI crawlers?
robots.txt is a request; the firewall is the gatekeeper. A common reason a site with a correct robots.txt is absent from ChatGPT or Perplexity is a security layer: Cloudflare bot fight settings, a hosting provider's WAF, or a WordPress security plugin that challenges unfamiliar user agents.
- List the bot settings in your CDN, WAF and security plugin. Note any "block AI crawlers" toggles and rate limits.
- Allow by user agent and IP together. Perplexity's documentation gives the pattern for Cloudflare: a custom WAF rule where User-Agent contains
PerplexityBotand IP Source Address is in its published ranges, with action Allow. For AWS WAF it advises allow rules with higher priority than blocking rules. The same pattern works for OAI-SearchBot and Claude-SearchBot with their JSON lists. - Never allow by user agent alone. Anyone can send "OAI-SearchBot" as a user agent. The IP list, or reverse DNS for Googlebot and CCBot, is the proof.
- Keep IP lists current. Perplexity recommends treating its JSON endpoints as the source of truth and automating updates; the same applies to OpenAI's and Anthropic's lists.
- Re-test after changes using the log check below, since WAF changes can take time to propagate.
How do you use server logs in an AI crawlability audit?
Logs are the only evidence that a crawler actually received your page. Pull 30 days of access logs from the host or CDN and answer three questions per bot: did it visit, which URLs did it fetch, and what status codes did it get.
A quick command-line count on a standard combined log format:
grep -E "OAI-SearchBot|PerplexityBot|Claude-SearchBot|bingbot|Googlebot" access.log | awk '{print $9}' | sort | uniq -c
Then split by bot and by URL. Our working thresholds for an established site:
| Finding | Interpretation | Action |
|---|---|---|
| No hits from a search crawler in 30 days | Blocked upstream, or the site has no presence in that engine yet | Check WAF events for that user agent; check robots.txt |
| More than 5% of a bot's requests return 403, 429 or 503 | Firewall or rate limit is interfering | Add verified allow rule; raise rate limit for verified IPs |
| Bot fetches only the homepage and robots.txt | Weak internal linking or sitemap not found | Link priority pages from navigation and hubs; list sitemap in robots.txt |
| Hits from a "bot" whose IP is not on the vendor's list | Spoofed user agent | Leave blocked; do not whitelist by name |
| Priority page returns 301 chains or 404 to bots | Broken internal links or migration leftovers | Fix links to point at the final URL |
The 5% threshold is our rule of thumb, not a vendor figure. Shared hosting plans often do not expose raw logs; in that case use the CDN's analytics or ask the host for an export.
Do AI crawlers render JavaScript?
Do not assume they do. Google documents that Googlebot queues pages for rendering in a headless Chromium "once Google's resources allow", and its JavaScript SEO guide still recommends server-side rendering or pre-rendering because "not all bots can run JavaScript". Other AI vendors do not document JavaScript rendering for their search crawlers, so the safe standard is: the answer, the price, the table and the internal links must be in the initial HTML.
Two tests per priority URL:
- View source, not Inspect. Search the raw HTML for the key sentence and price. If it is missing, it is being injected by JavaScript.
- Fetch as the bot.
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.example.in/pricing/ | grep -c "₹". A zero count, a challenge page or a 403 tells you what the crawler sees. Your WAF may treat a spoofed request differently from the real bot, so confirm with logs.
Common Indian-site culprits: React or Next.js builds without server rendering, prices in a tab or accordion populated on click, FAQ widgets loaded from a third-party script, and pricing published only as an image or PDF brochure. If a rebuild is needed, our website development team builds server-rendered pages by default.
How do snippet controls and Google-Extended affect AI features?
Google gives you three levels of control, and they do different things:
| Control | Effect on AI Overviews and AI Mode | Effect on classic Search |
|---|---|---|
nosnippet | Google says it "will also prevent the content from being used as a direct input for AI Overviews and AI Mode" | No text snippet in results |
max-snippet:[n] | Limits how much content may be used as direct input | Limits snippet length |
data-nosnippet on a div, span or section | Excludes that block from snippets; use for disclaimers or gated text | Same |
| Search generative AI control (Search Console Settings) | Excludes the site's links and content from AI Overviews, AI Mode and Discover AI features within 1–2 days | Google says it is not a ranking or inclusion signal for other parts of Search |
| Google-Extended in robots.txt | Not a Search control; covers Gemini training and grounding in other Google systems | None |
Sources: Google's robots meta tag documentation and the Search generative AI control help page. Two audit traps: snippet rules on a page blocked in robots.txt are never read, and for PDFs you must use the X-Robots-Tag HTTP header. Also look for max-snippet:0 or nosnippet set site-wide through an SEO plugin option. For the eligibility side, see our Google AI Overviews guide.
AI technical SEO audit checklist
| Area | Check | Pass condition | Tool |
|---|---|---|---|
| Inventory | Priority URL list | 20–50 revenue and reference pages with owner and target query | Sheet |
| robots.txt | Search crawlers allowed | Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot not disallowed for priority paths | robots.txt tester, manual read |
| robots.txt | Training policy documented | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot set deliberately | Manual read |
| robots.txt | Every subdomain covered | blog., www. and shop. each serve the intended file | Browser |
| Network | No challenge for verified bots | WAF allow rules use UA + IP; no 403/429 in logs | CDN/WAF console, logs |
| Logs | Each search bot seen in 30 days | 200 responses on priority URLs | Access logs |
| HTTP | Status and redirects | 200 on the canonical URL; no redirect chains | Crawler, curl |
| Indexing | Indexable and indexed | No stray noindex; URL Inspection shows "URL is on Google" | Search Console |
| Canonical | Google-selected canonical matches | Same as declared canonical | URL Inspection |
| Snippets | Quoting allowed | No nosnippet or tiny max-snippet on main content; X-Robots-Tag checked | View source, headers |
| Search Console | Generative AI setting | Included (unless you chose to opt out) | Settings |
| Bing | Site verified, sitemap submitted | No crawl errors; IndexNow on if CMS supports it | Bing Webmaster Tools |
| Rendering | Answer in raw HTML | Key sentence, prices and tables present in source | View source, curl |
| Rendering | No content behind clicks | Tabs and accordions present in HTML on load | View source |
| Links | Priority pages discoverable | 2+ contextual internal links; in XML sitemap | Crawler |
| Sitemap | Clean | Only 200, canonical, indexable URLs; accurate lastmod | Crawler |
| Structured data | Valid and matches page | No errors; markup matches visible text | Rich Results Test |
| Performance | Mobile experience | Core Web Vitals pass in field data where available | Search Console, PageSpeed Insights |
Structured data has its own guide: structured data for AI search.
How should the audit report be written?
Each finding needs six fields so a developer can act without a call: the issue, affected URLs, evidence (log lines, header output, screenshot), impact (which engines and pages), the exact fix with owner, and how it will be verified. Prioritise in this order: anything blocking a search crawler sitewide, then template issues affecting many priority pages, then single-page fixes. Re-run the log check two weeks after deployment and record the before-and-after status codes. If you want us to run this audit, it is the first step in our AEO and GEO services.
Sources
- OpenAI: Overview of OpenAI crawlers: OAI-SearchBot, GPTBot, ChatGPT-User roles, IP lists, ~24-hour update.
- Perplexity: Perplexity crawlers: PerplexityBot and Perplexity-User, IP lists, Cloudflare and AWS WAF guidance.
- Anthropic: Does Anthropic crawl data from the web: ClaudeBot, Claude-User, Claude-SearchBot, Crawl-delay, per-subdomain rules, IP list.
- Google: Common crawlers: Google-Extended scope and no separate user agent.
- Apple: About Applebot: Applebot and Applebot-Extended behaviour.
- Common Crawl: CCBot: user agent, opt-out, verification.
- Meta: Web crawlers: meta-externalagent and fetcher behaviour.
- Google: Robots meta tag and X-Robots-Tag: nosnippet and max-snippet effects on AI Overviews and AI Mode.
- Search Console Help: Search generative AI control: scope, timing, no ranking effect.
- Google: JavaScript SEO basics: rendering queue; not all bots run JavaScript.
- Google: Verifying Googlebot: reverse DNS and IP range methods.