Evalon

Overview

Evalon stores agent runs as traces in local SQLite, turns reviewed traces into dataset cases, and runs deterministic or LLM-judged evals. Everything lives in a Textual TUI — no hosted service, no dashboard, no account. It wraps OpenAI-compatible and Anthropic clients without changing how you call them.

Highlights

• Nested spans and provider wrappers capture inputs, outputs, tokens, latency, and errors without touching control flow
• Datasets version immutably; suites bind a dataset to a Python target with YAML-declared evaluators
• Static graders (exact match, JSON schema, tool order, latency caps) plus binary LLM judges
• CI-friendly: thresholds, JSON/JUnit exports, and exit codes that mean something

Stack

Python 3.11+

SDK, tracing runtime, @observe / @tool

SQLite

Local store for traces, datasets, and eval runs

Textual

Terminal UI for traces and eval history

OpenAI / Anthropic

Drop-in wrappers that turn calls into spans

LiteLLM

Optional per-call cost estimates