Pi Terminal Bench

pi-terminal-bench

Harbor agent adapter for [pi](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent) coding agent to run [Terminal-Bench](https://tbench.ai/) evaluations.

Installation

# Install uv if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install harbor
uv tool install harbor

# Install this package (for development)
cd ~/workspaces/pi-terminal-bench
uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"

Required: Apply Harbor Fix

There is a bug in Harbor's `upload_dir` function that causes verifier failures when the agent creates a `/tests` directory during task execution. The fix must be applied before running evaluations.

**The Problem:** When using `docker cp /path/to/source container:/target`:

• If /target does NOT exist: copies contents of source into /target ✓
• If /target ALREADY exists: copies source as a subdirectory /target/source/ ✗

The pi agent (and other agents) may create `/tests` during task execution, causing the verifier's test files to end up at `/tests/tests/test.sh` instead of `/tests/test.sh`.

**Apply the fix:**

# Find Harbor's docker.py location
HARBOR_DOCKER=$(python -c "import harbor.environments.docker.docker as m; print(m.__file__)")

# Apply the patch (backs up original first)
cp "$HARBOR_DOCKER" "${HARBOR_DOCKER}.bak"

# Edit the upload_dir function - change this:
#   async def upload_dir(self, source_dir: Path | str, target_dir: str):
#       await self._run_docker_compose_command(
#           ["cp", str(source_dir), f"main:{target_dir}"],
#           check=True,
#       )
#
# To this:
#   async def upload_dir(self, source_dir: Path | str, target_dir: str):
#       # Append /. to source to copy contents, not the directory itself
#       source = str(source_dir).rstrip('/') + '/.'
#       await self._run_docker_compose_command(
#           ["cp", source, f"main:{target_dir}"],
#           check=True,
#       )

Or use this one-lin