Robots.txt Tester

Review a public website’s robots.txt, check its directives and test whether a URL is allowed or blocked for Googlebot, Bingbot and selected AI crawlers. Edit a local copy of the rules and test again without changing the live file.

Google’s matching rules AI crawlers included Edit & re-test

A practical starting point from SEO Specialist USA. Use the results to prioritize technical improvements, then verify changes on your website.

Try: or paste your own robots.txt in the editor below.

robots.txt

Validation

    Test a URL

    All crawlers for this URL

    Rule groups & sitemaps

    What is robots.txt?

    robots.txt is a plain text file at the root of a website (https://example.com/robots.txt) that tells crawlers which parts of the site they may visit. Search engines like Google and Bing and AI crawlers like GPTBot check it before crawling. One wrong line can stop Google from crawling your whole site, so it is worth testing every change.

    How Google reads the rules

    • Groups: rules belong to the User-agent lines above them. A crawler uses only the most specific group that names it, or the * group if none does.
    • Longest match wins: when several rules match a URL, the one with the longest path is used. If an Allow and a Disallow are equally long, Allow wins.
    • Wildcards: * matches any characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF.
    • Case-sensitive paths: /Admin/ and /admin/ are different.
    • No file (404): everything may be crawled. A server error (5xx) makes Google pause crawling.
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    User-agent: GPTBot
    Disallow: /

    Sitemap: https://example.com/sitemap_index.xml

    robots.txt does not remove pages from Google

    Blocking a URL in robots.txt stops crawling, not indexing. If other sites link to a blocked page, Google can still list it — without a description. To keep a page out of search results, allow crawling and add a noindex meta tag instead.

    Frequently Asked Questions

    No. It only reads the live file. Your edits stay in your browser so you can test them; copy or download the result and upload it to your site yourself (in WordPress, most SEO plugins have a robots.txt editor).

    Add a group per crawler, for example User-agent: GPTBot followed by Disallow: /. Common AI tokens are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training) and CCBot (Common Crawl). Blocking Google-Extended does not affect Google Search.

    Googlebot does not support Crawl-delay; it adjusts its speed automatically based on how your server responds. Bing and Yandex do read it.

    No. Google stopped supporting Noindex: in robots.txt in 2019. Use a robots meta tag or the X-Robots-Tag header instead.

    No. Each host needs its own file: blog.example.com/robots.txt is separate from example.com/robots.txt, and http and https are treated as different hosts too.

    More free SEO tools from SEO Specialist USA

    Request Your Free SEO Audit