Review a public website’s robots.txt, check its directives and test whether a URL is allowed or blocked for Googlebot, Bingbot and selected AI crawlers. Edit a local copy of the rules and test again without changing the live file.
A practical starting point from SEO Specialist USA. Use the results to prioritize technical improvements, then verify changes on your website.
robots.txt is a plain text file at the root of a website (https://example.com/robots.txt)
that tells crawlers which parts of the site they may visit. Search engines like Google and Bing and
AI crawlers like GPTBot check it before crawling. One wrong line can stop Google from crawling your
whole site, so it is worth testing every change.
User-agent lines above them. A crawler uses only the most specific group that names it, or the * group if none does.* matches any characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF./Admin/ and /admin/ are different.
Blocking a URL in robots.txt stops crawling, not indexing. If other sites link to a blocked page,
Google can still list it — without a description. To keep a page out of search results, allow
crawling and add a noindex meta tag instead.
User-agent: GPTBot followed by Disallow: /. Common AI tokens are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training) and CCBot (Common Crawl). Blocking Google-Extended does not affect Google Search.Noindex: in robots.txt in 2019. Use a robots meta tag or the X-Robots-Tag header instead.blog.example.com/robots.txt is separate from example.com/robots.txt, and http and https are treated as different hosts too.