GPTBot, ClaudeBot, PerplexityBot: should you block AI crawlers?

Which bots are we actually talking about?

The user agents in your logs sort into three jobs, and the distinction carries the whole decision:

Each obeys its own robots.txt rule, which means you can block training while staying visible to AI search — or accidentally do the reverse.

What does blocking actually do?

Block the training bots and your content stays out of future model training runs. The visible effect on your business is close to nil: buyer questions get answered from live retrieval, and third-party pages about you get crawled regardless of what your own robots.txt says.

Block the search and fetch bots and the effect is immediate and visible: AI search engines can't read your site, so they can't cite it. Everyone else's description of you becomes the only evidence — failure mode three, self-inflicted.

When does blocking make sense?

When your content is the product. Publishers, stock libraries, data businesses and anyone negotiating licensing deals have a genuine interest in keeping training crawlers out — free training data undercuts the thing they sell. If that's you, block the training bots and think hard about the search bots case by case.

When is blocking a mistake?

When customers finding you is the point. For a business that sells services or products, pages exist to be discovered — and a growing share of discovery now runs through AI answers. Blocking retrieval bots to protect content that was written to attract customers is guarding the shop window by bricking it up.

We say this as a site that got it wrong. This site shipped with a blanket AI opt-out — principled-feeling, and defensible for a compliance product. Then we started selling AI visibility and noticed we were publishing a machine-readable signal telling AI systems to ignore us. We flipped it. The lesson: the setting should follow the business model, not the mood of the debate.

What should a small business do?

Allow the search and fetch bots — for almost any SME the case is one-sided. On training bots, the stakes are small either way: your pages are a rounding error in a training corpus, so opting out costs you little, and opting in mostly signals openness. Decide by what your customers would expect of you, write it down, and stop relitigating it.

Then check what your site actually says, because intent and configuration drift apart: one WordPress plugin update or an over-helpful agency and you're blocking bots you meant to allow. Our free training audit reads your robots.txt, ai.txt and llms.txt and lists every AI crawler's status in about a minute.