Context Files Coding Agents

Do Context Files Help Coding Agents?

**A two-agent ablation study on whether repository context files (`AGENTS.md` / `CLAUDE.md`) change how well LLM coding agents solve real GitHub issues.**

This is the code-and-data release for the paper *"Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories"* (REALM @ EMNLP 2026, under review). It contains the full experimental harness, the safety infrastructure, the benchmark task specifications, the statistical analysis scripts, and an aggregated per-cell results table.

---

TL;DR

We run two production coding agents (**Claude Code** and **OpenAI Codex CLI**) on real, merged pull-request tasks from three Python repositories, under three context-injection strategies, and measure whether the presence of a repository context file changes task **correctness** and **efficiency**.

**Headline finding — a correctness null, replicated across both agents.** Adding the repository's `AGENTS.md` (in full, or via a retrieval-style split) does **not** measurably change the probability that the agent's patch passes the gold PR tests.

Agent	none	always_on	selective
Claude Code (15 tasks)	53.3%	55.6%	55.6%
Codex (17 tasks)	58.8%	56.9%	52.9%

• Within-task omnibus permutation test: p = 1.000 (Claude); no detectable effect on either agent.
• TOST equivalence: effect bounded to ≤10pp (Claude) / ≤15pp (Codex).
• The result is power-limited, and we say so: at n≈15–17 tasks × 3 repeats the minimum detectable effect is >30pp; detecting a 10pp effect at 80% power would need ~120 tasks. We report this as the binding constraint, not a proof of zero.

**Where context *did* move the needle — process efficiency, not outcome.** On the OpShin/opshin tasks, Claude Code with context ran the slow full test suite far less often (blind full-suite runs **3.67 → 2.44 → 1.67** across `none`→`always_on`→`selective`), cutting wall-clock time **