中文说明

---

🎯 Introduction

Alipay-PIBench is a repository-level benchmark proposed by Alipay for evaluating AI coding agents on realistic payment integration tasks. Built around Alipay Open Platform products and business-oriented repositories, it comprises nine product-specific projects and 18 task instances across Basic (Functional Payment Completion) and Advanced (Risk-Aware Payment Hardening) scenarios. The benchmark evaluates functional correctness, reliability, security, and business-state consistency.

• Payment Benchmark Construction. Each payment product is paired with a representative project, and each task combines the target product, business workflow, initial repository, and scenario-specific integration request.
• Rubric-Grounded Evaluation. Scenario-specific rubrics define deterministic checks and LLM-assisted criteria, producing rubric-aligned evidence for structural, executable, and payment-domain requirements.
• Paired Skill Study. Each agent is evaluated on the same task, project, instruction, and environment with and without `alipay-payment-integration`, enabling comparisons by product, scenario, agent, and skill condition.

[](assets/overview.pdf)

🏆 Main Results

Model Capability

[](assets/main-results.pdf)

† Post-publication resul