lintsinghua

Crawl4Agent Skill — AI skill for Claude Code

AI community

Advanced web crawler skill for AI agents — anti-bot bypass, Chinese platform support, login handling, and template library.

How to install Crawl4Agent Skill

This entry records only its repository, not the path inside it, so there is no exact command to give. Open lintsinghua/Crawl4Agent-skill and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Crawl4Agent Skill does

Advanced web crawler skill for AI agents — anti-bot bypass, Chinese platform support, login handling, and template library.

Alternatives in AI

Geo SEO Claude — GEO-first SEO skill for Claude Code 3.6k ★
Coding Agent Template — Multi-agent AI coding platform powered by Vercel Sandbox and AI Gateway 1.8k ★
Chinese Novelist Skill — 🎭 AI 驱动的中文小说创作助手 1.2k ★

README

Crawl4Agent-skill

Advanced web crawler skill for AI agents — bypasses anti-bot protections, handles login walls, and provides robust browser automation.

Features

Auto-detection: Identifies anti-bot protections (Cloudflare, DataDome, PerimeterX, custom WAF) and selects the right bypass strategy
Multi-fetcher support: Static HTTP, stealth browser, dynamic rendering — auto-chosen based on target site
Chinese platform support: Ready-made integration with MediaCrawler for Chinese social platforms; DrissionPage for custom scripts
Login handling: Interactive login flows (QR code, phone, OAuth) with cookie persistence
API interception: Capture and replay XHR/Fetch API calls
Template library: 9 battle-tested templates for common crawling patterns

Quick Start

1. Auto-detect and crawl

python3 scripts/crawl.py https://example.com

2. Diagnose anti-bot protection

python3 scripts/detect.py https://example.com --verbose

3. Use a template

# Static scraping (fast)
from scrapling import Fetcher
page = Fetcher(auto_match=False).get('https://example.com', impersonate='chrome131')

# Stealth scraping (bypasses anti-bot)
from scrapling import StealthyFetcher
page = StealthyFetcher.fetch('https://protected-site.com', headless=True, solve_cloudflare=True)

# DrissionPage (Chinese sites, no WebDriver detection)
from DrissionPage import ChromiumPage, ChromiumOptions
opts = ChromiumOptions()
opts.headless(True)
opts.set_load_mode("eager")  # CRITICAL: avoids hanging
page = ChromiumPage(opts)
page.get('https://example.com', timeout=15)  # CRITICAL: always set timeout

Project Structure

├── SKILL.md                          # Skill definition and agent instructions
├── scripts/
│   ├── crawl.py                      # Auto-detect + fetch + extract
│   └── detect.py                     # Anti-bot detection
├── templates/
│   ├── scrapling_crawler.py          # General scraping (Scrapling