Does robots.txt help or hurt AI recommend you?

Robots.txt is one of the most overlooked AI-visibility factors because it can silently do the opposite of what you want: a single stray disallow line can block the very AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others — that you need reading your pages to describe and recommend you. It won't make AI recommend you on its own, but a clean, deliberate robots.txt is table stakes: it ensures the assistants you care about can actually reach your content, your sitemap is discoverable, and you're not accidentally hiding your best pages behind an over-broad rule copied from a template. The nuance is that some site owners deliberately block certain AI crawlers to protect content — a legitimate choice, but one that directly trades away AI visibility. The move is to audit exactly which bots you allow, confirm the assistants you want to appear in can crawl you, and reference your sitemap so discovery is effortless.

FAQ