Gamecraft Bench

Website • Paper • Demos

**GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?**

GameCraft-Bench evaluates whether coding agents can transform natural-language game specifications into complete, playable Godot projects. Unlike traditional coding tasks, game generation depends on scripts, scenes, assets, rendering, runtime configuration, and player-game interaction working together as one executable system.

The benchmark contains **140 tasks** across **15 game families**. Each agent must submit a complete Godot project together with replayable demonstration traces. The verifier launches the project, replays the traces, records gameplay evidence, and scores observed play with a hidden rubric and multimodal judge.

The benchmark runs on top of [Harbor](https://github.com/harbor-framework/harbor) and ships with a custom local-subprocess environment for Docker-less hosts.

---

Benchmark overview

GameCraft-Bench is organized around three desiderata for end-to-end game generation:

• Engine Grounding: games are generated and evaluated inside a concrete game engine and runtime environment.
• Artifact Completeness: agents must deliver complete launchable game projects rather than isolated scripts, scenes, or assets.
• Interactive Verification: games are judged by observed behavior under player input, using replayed demonstrations as standardized gameplay evidence.

The task suite covers diverse 2D game-generation demands, including continuous control and collision, rule and state management, progression and economy, exploration, narrative interaction, and presentation-heavy gameplay.

| Family | Tasks | Family | Task