Crawlmere › Skills › robots.txt tester

robots.txt tester – a skill for Claude

Tests robots.txt the way Google reads it, for search and AI crawlers.

What it checks and does

• RFC 9309 longest-match rules, Allow wins ties, * and $ wildcards
• Most specific user-agent group, merged groups
• Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and any you name
• Explains crawl-block vs noindex and the AI-crawler trade-off

Try asking

“Is /blog/ blocked for GPTBot or Googlebot with this robots.txt?”

Script

robots_test.py – Python 3.8+, standard library only, covered by the pack's tests. Claude runs it and reads only the output, so long files don't fill your context.

When Claude uses it

The skill's description, which Claude matches against your request:

Test a robots.txt file the way Google interprets it (user-agent groups, longest-match Allow/Disallow, * and $ wildcards) against a list of URLs and crawlers (Googlebot, Bingbot, AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended), and write or fix robots.txt rules. Use when the user asks whether a page is blocked, wants to block or allow AI crawlers, has "Blocked by robots.txt" in Search Console, or needs a robots.txt written.

Related skills

• hreflang checker – Audits language and region versions and generates correct hreflang tags.
• SEO client report – Turns exports and audit results into a clear monthly report for a client, a boss or yourself.
• Technical SEO audit – Page-level audit of a built site, saved pages or URLs, sorted into critical, high, medium and low fixes.