Agent Pr Replay

Agent PR Replay

**Discover how AI coding agents actually navigate your codebase, and where they diverge from human developers.**

The best way to improve an agent's ability to work in a codebase is to observe its default behavior, measure the gap against real human solutions, and steer it based on evidence.

Agent PR Replay takes merged PRs from any repository, reverse-engineers the task prompt, runs Claude Code against it, and compares what the agent did versus what humans actually shipped. The result is targeted, empirical guidance.

**[See example reports →](examples/)**

How It Works

flowchart TB
    subgraph ground["① Ground Truth: Human PRs"]
        H1["PR #1234 merged"] --> H2["Human diff:
auth.py +12 -3"]
    end

    subgraph replay["② Replay"]
        R1["Checkout base commit"] --> R2["Reverse-engineer prompt:
'Fix login validation for empty passwords'"]
    end

    subgraph agent["③ Run Claude Code"]
        A1["Claude explores codebase"] --> A2["Claude diff:
auth.py +28 -3, test_auth.py +45"]
    end

    subgraph compare["④ Compare & Synthesize"]
        C1["Human: minimal fix
Claude: added tests, refactored"] --> C2["Gap: over-engineering"]
        C2 --> C3["CLAUDE.md / AGENTS.md:
'Match scope of request,
don't add tests unless asked'"]
    end

    ground --> replay
    replay --> agent
    ground --> compare
    agent --> compare

1. Ground Truth: Start with real merged PRs as human-validated solutions
2. Replay Setup: Checkout the repo at the PR's base commit, reverse-engineer a task prompt from the diff
3. Run Agent: Execute Claude Code with the prompt (same starting point, same task, different solver)
4. Compare: Diff what Claude changed vs what humans shipped; identify systematic gaps
5. Synthesize: Generate targeted guidance (CLAUDE.md, AGENTS.md, skills) based on observed behavioral patterns

Installation

**Prerequisites:**

• Python 3.11+
• [GitHub CLI](https://cli.github.