surendranb

AI Robots Txt Playbook — AI skill for Claude Code

AI community

AI Search & LLM Crawler robots.txt Playbook.

How to install AI Robots Txt Playbook

Installs to ~/.claude/skills/surendranb-llm-crawler-audit-ai-robots-txt-playbook/SKILL.md

Terminal

mkdir -p ~/.claude/skills/surendranb-llm-crawler-audit-ai-robots-txt-playbook && curl -fsSL https://raw.githubusercontent.com/surendranb/llm-crawler-audit/HEAD/skills/ai-robots-txt-playbook.md -o ~/.claude/skills/surendranb-llm-crawler-audit-ai-robots-txt-playbook/SKILL.md

Restart Claude Code, or start a new session, for it to be picked up.

What AI Robots Txt Playbook does

AI Search & LLM Crawler robots.txt Playbook

A comprehensive guide to configuring RFC 9309 compliant `robots.txt` directives for AI search engines, answer bots, and model training crawlers.

1. The Modern AI Crawler Landscape

AI crawlers fall into two distinct operational categories:

AI Search & Retrieval Bots: Power live web grounding, RAG (Retrieval-Augmented Generation), citations, and real-time user queries in ChatGPT Search, Claude, Perplexity, and Apple Intelligence.

Alternatives in AI

Firecrawl MCP Server — 🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM c 6.1k ★
Geo SEO Claude — GEO-first SEO skill for Claude Code 3.6k ★
Unofficial Claude Code Prompt Playbook — Unofficial playbook for production-grade LLM system prompt architecture, derived from local analysis of Claude 258 ★