Benchmark Claude Pdfs — Development skill for Claude Code

Benchmark: Claude PDF Question Answering

This repository benchmarks Claude's ability to answer questions from corporate sustainability / ESG PDFs using the [pdfQA-Benchmark](https://huggingface.co/datasets/pdfqa/pdfQA-Benchmark) dataset. It supports running evaluations under several configurations:

• Raw — Claude reads PDFs directly with built-in tools, no skill.
• liteparse — first cut of a skill wrapping the local lit CLI for PDF parsing.
• liteparse-targeted — a more directive variant of the same skill.
• effective-liteparse — the current production skill, rewritten after analyzing real traces to enforce parse-once-then-search and other cost-saving patterns.
• microag — a separate harness running the same prompts through MicroAgents (a small Rust-based agent runtime) wired to Anthropic's Sonnet via OpenRouter, with the liteparse library exposed within the builtin Read tool rather than a skill.

Overview

The benchmark downloads publicly available sustainability reports (PDFs) and their corresponding annotated question-answer pairs, then runs Claude (via the `claude-agent-sdk`) over each (PDF, question) pair. Results are saved as structured JSON answers and full interaction traces.

Repository Structure

.
├── data/                          # Downloaded PDFs (gitignored)
├── questions/                     # Annotated Q&A JSON files (gitignored)
├── answers/
│   ├── raw/                       # Answers from raw PDF reading (no skill)
│   ├── liteparse/                 # Answers using the "liteparse" skill
│   ├── liteparse-targeted/        # Answers using the targeted skill variant
│   ├── effective-liteparse/       # Answers using the current production skill
│   └── microag/                   # Answers from the MicroAgents harness
├── traces/
│   ├── raw/                       # Full JSONL interaction traces (Claude Agent SDK)
│   ├── liteparse/
│   ├── liteparse-targeted/