AI crawler access is not one switch. Search, training and user-triggered browsing can use different user agents, and the consequences of blocking them are not identical.
OpenAI: OAI-SearchBot matters for search eligibility
OpenAI’s current help documentation says sites should allow OAI-SearchBot if they want to be eligible for inclusion in ChatGPT search results. OpenAI separately operates GPTBot for model training and ChatGPT-User for user-triggered interactions. Treating them as one bot is a common robots.txt mistake.
Check more than robots.txt
A robots file can allow a crawler while a CDN, WAF, security plugin or bot challenge still returns 403, 429 or a JavaScript interstitial. OpenAI explicitly recommends checking web protection and rate limiting when its crawler cannot reach a site.
JavaScript support differs by crawler
Vercel’s crawler analysis found that dedicated crawlers from OpenAI, Anthropic and Perplexity did not render JavaScript in its tests. Google is different: Google says its generative AI search experiences are rooted in Google Search, and Googlebot can render JavaScript. Therefore “AI crawlers do not render JavaScript” is too broad. The practical recommendation is to keep critical business content in the initial HTML.
Do not confuse crawlability with recommendation
Allowing a bot makes the page eligible to be fetched. It does not guarantee indexing, citation or recommendation. OpenAI says placement is not guaranteed; Google says meeting requirements does not guarantee crawling, indexing or serving.
A basic AI crawler access checklist
- robots.txt returns 200 and does not unintentionally disallow search crawlers.
- Key pages return successful HTML to non-browser user agents.
- CDN/WAF rules do not challenge or throttle legitimate crawler traffic.
- Core business facts are present without client-side rendering.
- Sitemaps and canonical URLs resolve cleanly.
The Visible2 Free AI Visibility Scan checks the readiness layer; deeper crawler logs belong in ongoing monitoring.