Web Scraper — Development skill for Claude Code

Web Scraping Skill

Intelligent web scraping with automatic strategy selection and TypeScript-first Apify Actor development.

Overview

This skill provides:

• Adaptive reconnaissance - Phases 0-5 with quality gates that skip unnecessary work (curl first, browser only if needed)
• Framework-aware detection - Identifies site framework before searching, skips irrelevant patterns
• Validated findings - Every claimed selector/path/API is tested before reporting
• Self-critiquing reports - Intelligence reports include gap analysis and staleness warnings
• Iterative implementation - Starts simple, adds complexity only if needed
• Production-ready guidance - TypeScript-first Apify Actor development

Installation

Add this skill to Claude Code by placing this directory in the skills folder.

Quick Start

Scenario 1: Scrape a Website

User: "Scrape https://example.com"

Claude will automatically:
1. Phase 0: curl raw HTML — detect framework, search for data points, check sitemaps
2. QUALITY GATE: All data in HTML? → Skip browser, go to validation
3. Phase 1: Launch stealth browser (only if needed) — capture traffic, rendered DOM
4. Phase 2: Deep scan (only for missing data) — test interactions, sniff APIs
5. Phase 3: Validate every finding — test selectors, replay APIs, confirm paths
6. Phase 4: Protection testing (only if signals detected or user requested)
7. Phase 5: Generate intelligence report with self-critique
8. Implement recommended approach iteratively
9. Test with small batch, then scale

Scenario 2: Create Apify Actor

User: "Make this an Apify Actor"

Claude will:
1. Recommend TypeScript (strongly)
2. Guide through `apify create` command
3. Help choose appropriate template (Cheerio vs Playwright)
4. Port scraping logic to Actor format
5. Configure input schema
6. Test and deploy

Directory Structure

web-scraping/
├── SKILL.md                    # Main entry point (proactive workflow)
├── workflows