Agent evaluation pipeline
Runs an agent programmatically, captures OpenTelemetry traces, and scores both the answer and the trajectory with LLM-as-judge plus deterministic checks.
- Result
- 10+
quality dimensions scored per agent, from traces as well as answers - Role
- Designed the evaluation framework and built the harness
- Domain
- Internal platform for an enterprise client
- Period
- May 2026 – Jul 2026
Problem
The organization wanted an internal marketplace of agents, skills and MCP servers, but had no way to decide which agents were safe to publish. Demo-quality impressions weren't a vetting process.
Approach
Evaluate agents the way production systems are evaluated: from traces. Classify each agent (advisory, generative, workflow), generate a dataset for its class, run it headlessly, and score what it did as well as what it said.
What I built
- A harness that classifies agents, generates and validates datasets, runs the agent programmatically and exports OpenTelemetry traces.
- Contract-based test scenarios per agent class, scored on 10+ quality dimensions with LLM-as-judge, programmatic checks and trajectory metrics (tool-call correctness, retries, failures), rolled into a scorecard and dashboard.
- Trust tiers (approved, community-vetted, beta) and a marketplace prototype: FastAPI and Postgres backend, React frontend.
- A companion spike with release gates and seeded regressions on a 15-case dataset.
Outcome
- The trace view caught what answer-only grading missed: a correct final answer hiding a failed tool call.
- The evaluation framework became the vetting standard proposed for the marketplace.