01
Robots.txt Tester
Tests whether a given URL is allowed or blocked by a robots.txt file, showing which rule…
AI Crawler Access Checker
Reads your robots.txt and shows which AI crawlers are allowed, separating training crawlers from the ones that answer live questions.
SEO
Nothing uploaded
No sign-up
Free, unlimited
Paste something on the left to see the result.
In short
AI crawlers split into two kinds and the difference decides what blocking them costs you. Training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended — collect pages to build models, and blocking them does not remove you from any answer. Retrieval crawlers — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot — fetch your page when someone asks a question right now, and blocking those removes you from the answer entirely, with no citation and no link. Most advice written since 2023 does not separate them, which is how sites end up invisible in ChatGPT and Perplexity while still being trained on from Common Crawl.
How to use the AI Crawler Access Checker
- Paste the whole robots.txt file.
- Set the path you care about.
- Check whether any retrieval crawler is blocked by accident.
About the AI Crawler Access Checker
The AI Crawler Access Checker reads your robots.txt and shows which AI crawlers are allowed, separating training crawlers from the ones that answer live questions. It runs as plain JavaScript inside the page, which means there is no upload step, no queue and no size limit imposed by a server — the only limit is your own device.
What people use it for
- Checking whether a "block AI" rule went further than intended.
- Deciding which crawlers to allow before publishing robots.txt.
- Explaining to a client why they vanished from ChatGPT.
Details
| Training crawlers | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider |
|---|---|
| Retrieval crawlers | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot |
| Google-Extended | Does not affect Google Search indexing — that is Googlebot |
| Most specific group wins | A group naming a bot makes the * group irrelevant to it |
| robots.txt | A request, not access control |
The combination most people actually want is to block the training crawlers and allow the retrieval ones: refuse to be training data while staying quotable and linked. That is a specific list rather than a switch, which is why a blanket "block AI bots" rule so often does the opposite of what was intended.
Where this tool stops
Evaluates the file you paste against the published user-agent tokens. New crawlers appear regularly and undeclared ones ignore the file entirely, so an allowed verdict here is about your rules rather than about what actually reaches your server.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT search results come from <code>OAI-SearchBot</code>, and live page fetches during a conversation come from <code>ChatGPT-User</code>. Blocking GPTBot while leaving those two alone is the usual intent: refuse the training, keep the citations.
Does Google-Extended affect my Google rankings?
Not at all, and this catches people constantly. Google-Extended controls whether your content is used for Gemini training and grounding. Ordinary Search crawling and indexing is Googlebot, which is a separate token — blocking Google-Extended has no effect on how you rank.
Why did my bot-specific rule stop the general rules applying?
Because a crawler obeys the <b>most specific</b> group that names it and ignores every other group, including <code>*</code>. Adding <code>User-agent: GPTBot</code> with one Disallow line means GPTBot now follows only that line — every rule under <code>*</code> stops applying to it.
Does robots.txt actually stop anything?
It stops well-behaved crawlers, which is most of the ones listed here. It is a published request rather than access control, and a page linked or quoted elsewhere can still be summarised without ever being fetched. For content that must not be used, authentication is the only real answer.
Does the AI Crawler Access Checker cost anything?
There is no cost and no cap. The AI Crawler Access Checker is not a demo of a paid product, so nothing is held back or watermarked.
Is the AI Crawler Access Checker safe to use with private information?
No. The AI Crawler Access Checker runs entirely in your browser using JavaScript. Your input never reaches inputtesthub.com or any third party — open your browser's Network tab while you use it and you will see no request go out.
What problem does the AI Crawler Access Checker solve?
Typical uses: checking whether a "block AI" rule went further than intended. Deciding which crawlers to allow before publishing robots.txt. Explaining to a client why they vanished from ChatGPT. It is built for exactly those cases rather than being a general-purpose tool with this bolted on.
What are the limits of the AI Crawler Access Checker?
Evaluates the file you paste against the published user-agent tokens. New crawlers appear regularly and undeclared ones ignore the file entirely, so an allowed verdict here is about your rules rather than about what actually reaches your server. Knowing where a tool stops being trustworthy matters more than the cases where it works.
What are the reference figures here?
Training crawlers is GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider. Retrieval crawlers is OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot. Google-Extended is Does <b>not</b> affect Google Search indexing — that is Googlebot. These are printed on the page itself, so you do not have to take them on trust.
Can I use the AI Crawler Access Checker on mobile?
Yes. The page is responsive and the utility works in Chrome, Safari, Firefox, Edge and Samsung Internet on both Android and iOS. There is no app to install and nothing to sign into.
Related seo tools
All 5602
On-Page SEO Checker
Analyses a page's HTML for title, description, headings, images, links, structured data and…
03
Heading Structure Checker
Extracts the H1 to H6 outline from HTML and flags skipped levels, multiple H1s and empty…
04
Image Alt Text Checker
Lists every image in your HTML and flags missing, empty, duplicated or keyword-stuffed alt…
05
SERP Feature Eligibility Checker
Tells you which search features a page could qualify for — featured snippet, FAQ, how-to…
06
Indexability Checker
Checks page HTML for anything that would stop it being indexed — noindex, canonical conflicts…
07
JSON-LD Validator
Parses JSON-LD structured data, finds every entity, and reports missing required properties…
08
Meta Tag Analyzer
Reads a pasted page head and reports title and description width in pixels, canonical and…
09
Sitemap XML Validator
Checks an XML sitemap against the sitemaps.org rules: the 50,000 URL limit, absolute URLs…
10
Form Label Checker
Checks pasted form markup for inputs with no label, placeholder-only labelling, orphaned label…