Skip to main content
A test is a conversation you wrote down, together with the assertions that must hold when it runs. Run it after a change and anything you broke fails right there, instead of turning up in production a week later.

​
Four resources, one idea

A test is the container. Turns are the script. Checks are what must be true. Runs are what happened each time you executed it. Turns and checks are managed independently of the test, so you can add a check to an existing scenario without rewriting the conversation, or extend the conversation without touching the assertions.

​
Creating a suite

Each resource has a batch endpoint, which is what you want when defining a suite from a file rather than clicking through it:

​
Reading results

Runs are searchable rather than merely listable, so the useful question is not “what ran” but “what failed, and since when”. Search runs, then fetch the ones you care about.

​
How this differs from evaluations

These are different jobs and it is worth keeping them apart. QA replays conversations you wrote and fails when the agent stops behaving as decided. Insights scores conversations real users had, and tells you how it is going. One protects against regressions; the other finds things you did not think to test.

​
Where to go next

Run tests against a cloned environment before merging into main, so a failing check blocks the change rather than reporting on it afterwards.