bellamarbellaa

Web Scraping Automation With AI Agent — AI skill for Claude Code

AI community

A Firecrawl-based AI automation to source, scrape, and verify research based on a topic before publication.

How to install Web Scraping Automation With AI Agent

This entry records only its repository, not the path inside it, so there is no exact command to give. Open bellamarbellaa/Web-Scraping-Automation-with-AI-Agent and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Web Scraping Automation With AI Agent does

A Firecrawl-based AI automation to source, scrape, and verify research based on a topic before publication. The workflow uses Claude Code + Firecrawl MCP to search Medium website, extract article based on a given topic, independently verify claims, and insert verified references into a Google Sheets file.

Alternatives in AI

Firecrawl MCP Server — 🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM c 6.1k ★
Design Md Chrome — Chrome extension to extract styles from any website and generate DESIGN.md files and design skills for AI base 2.7k ★
Botmux — Bridge Feishu/Lark to AI coding CLIs — Claude Code, Codex, Gemini, OpenCode… every DM, group or topic spawns i 1.2k ★

README

A Firecrawl-based automation system developed for a personal newsletter brand to source and verify research from Medium before it's cited in an issue. The project demonstrates web scraping automation, mixed-quality-source verification, and AI-assisted workflow execution using Claude Code.

How it works

Each issue starts from a single topic or tag and moves through the same repeatable flow:

Search — Medium is searched/mapped for the week's topic or tag, surfacing candidate articles
Filter — candidates are skimmed for tone and quality; video-only and paywalled sources (Medium exclusions, plus sites like Psyche, Aeon, HBR) are dropped
Scrape — the selected article(s) are scraped for full, clean text
Verify — every specific statistic, study name, or named researcher pulled from the text is independently confirmed against the original source or a fresh search — never taken on the article's word
Classify — each point is tagged verified (safe to cite), softened (rewritten without false precision), or dropped (couldn't be confirmed, discarded)

How to use

No install step — this runs on [Firecrawl's](https://firecrawl.dev) MCP tools inside Claude Code, not a local Python environment.

Give Claude a topic and ask it to source research for the newsletter. It reads [workflows/source_and_verify_research.md](workflows/source_and_verify_research.md) and follows the search → filter → scrape → verify → classify flow automatically, rather than needing the process re-explained each time.

Files in this repository include

**Workflow Documentation** — [workflows/source_and_verify_research.md](workflows/source_and_verify_research.md): the step-by-step SOP covering search, filtering, scraping, and the mandatory verification gate.

**Firecrawl Tool Reference** — [firecrawl-cheatsheet.md](firecrawl-cheatsheet.md): which Firecrawl tool to reach for (search vs. map vs. scrape vs. crawl), and which sources to skip entirely.

**Project Brie