AI Crawler Check Which engines can actually reach you.

It depends entirely which crawler. Blocking a training crawler like GPTBot or Google-Extended has no effect on search or on whether you can be cited. Blocking an answer-index crawler like OAI-SearchBot or PerplexityBot removes you from the set of pages that engine can name in a reply — that is not a ranking penalty, it is not being a candidate. Most sites that meant to do the first have done the second by accident, because they reached for User-agent: * .

GPTBot collects text for training OpenAI's models. OAI-SearchBot builds the index ChatGPT Search cites from. Blocking GPTBot keeps your writing out of model weights. Blocking OAI-SearchBot makes you uncitable in ChatGPT today. They are separate tokens precisely so you can make those two decisions independently.

RFC 9309 specifies that a 5xx response to robots.txt should be treated as a full disallow, on the reasoning that a crawler cannot know what it is allowed to do. So a server error on that one file is indistinguishable from a deliberate site-wide block for as long as it lasts. A 404 is the opposite — no file means no restrictions.

The GPTBot group, entirely. A crawler obeys exactly one group — the most specific one whose user-agent matches — and does not combine it with the wildcard group. This is the single most misread rule in robots.txt, and it flips the answer completely. This tool applies it correctly and names the group it used.

The major operators publish their tokens and state that they honour it, and this tool reports what your file tells a compliant crawler. It cannot tell you what a non-compliant one does. If your concern is scrapers that ignore the file entirely, robots.txt is the wrong control and you want request-level blocking instead.