Letta Evals

Letta Evals

Letta Evals is a framework for evaluating [Letta](https://github.com/letta-ai/letta) and Letta Code agents. It lets you define an evaluation suite with a dataset, target, extractors, graders, and a reward contract, then run that suite against one or more model configurations.

If you are building agentic systems, high-quality evals are one of the fastest ways to understand how model versions, prompts, tools, or agent configuration changes affect your product.

Requirements

• Python 3.11+
• A running Letta server, either:
  • Self-hosted: follow the Letta installation guide, or
  • Letta Cloud: create an account at app.letta.com and set:

    export LETTA_API_KEY=your-api-key
    export LETTA_PROJECT_ID=your-project-id

    Then use base_url: https://api.letta.com/ in your suite YAML, or pass --base-url https://api.letta.com/ on the CLI.
• Provider API keys for the models you use, such as OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY.

Installation

For local development or custom eval authoring, clone this repository and install with dev dependencies:

uv sync --extra dev

To run existing evals without editing the repo:

pip install letta-evals

Quick start

1. Create a dataset (dataset.jsonl):

{"input": "What's the capital of France?", "ground_truth": "Paris"}
{"input": "Calculate 2+2", "ground_truth": "4"}

1. Create a suite (suite.yaml):

name: my-eval-suite
dataset: dataset.jsonl

target:
  kind: letta_code
  model_handles:
    - openai/gpt-4.1-mini
  base_url: http://localhost:8283

graders:
  correctness:
    kind: tool
    function: contains
    extractor: la