Skip to main content
An evaluation scores conversations that already happened against criteria you write down. It is how you find out what the agent is doing on real traffic, none of which you scripted.

​
The shape

An evaluation is a definition: what you are measuring and how. Creating one does not score anything by itself. Running it applies it and produces a result.
That split is deliberate. The definition is stable and version-controlled by you; runs accumulate against it, so a score means something over time rather than being a one-off measurement with its own private criteria.

​
Evaluations and transcripts

Evaluations score conversations that already happened, which makes Observe the source of what they measure. Search for the transcripts you care about, then evaluate that set.

​
How this differs from automated testing

Evaluations judge real conversations after the fact. QA does the opposite: it replays scripted conversations you wrote, and asserts the agent still answers the way you decided it should. Use evaluations to find out what is happening in production, and QA to stop a change breaking what already worked.

​
Where to go next

Once you know what to look for, encode it as a test so a regression fails loudly instead of showing up in next month’s numbers.