Recovery Bench

Recovery-Bench

[Recovery-Bench](https://www.letta.com/blog/recovery-bench) is a benchmark for evaluating how well LLM agents recover from mistakes. A weak agent attempts a [Terminal-Bench 2.0](https://harborframework.com/docs/running-tbench) task and fails. We evaluate how well agents can recover after replaying the failed trajectory to reproduce the corrupted environment.

How It Works

Weak agent runs task → fails → trajectory saved
                                        ↓
                          Replay failed commands in fresh env
                                        ↓
                          Recovery agent starts from corrupted state
                                        ↓
                          Measure: did it recover? (reward > 0)

1. Initial traces — An agent (with a weak model) runs Terminal-Bench tasks.
2. Filter failures — Keep only trajectories where the agent failed (reward = 0).
3. Replay — Re-execute the failed agent's commands in a fresh Docker container to reproduce the corrupted state.
4. Recovery — A recovery agent gets the original task, corrupted environment, and optionally context from the failed attempt.
5. Score — Compare recovery success rates across models and agents.

Setup

pip install -e .

# Pull the bundled initial traces (requires Git LFS)
git lfs install
git lfs pull

Add API keys for the models you're testing.

Shared failure set

The `git lfs pull` fetches pre-generated Terminus-2 Haiku 4.5 initial traces into `runs/`. These traces are the common baseline for all experiments — every model and agent is evaluated against the same set of failed tasks and corrupted environments, making results directly comparable across runs.

Evaluating Models (using Terminus-2)

Pick any [LiteLLM model](https://docs.litellm.ai/docs/providers) and run it against the shared Haiku 4.5 failure set using Terminus-2:

python -m recovery_bench.generate_traces \
    --recovery-model