How scenarios fit in
You run a scenario withRunScenarioAsync, passing a scenario name, your agent, the assertions to apply to every item, and the items. Each item runs your agent once.
Items can be defined in code or managed in the dashboard. If you don’t pass items, Veval fetches them for the scenario by name, so you can change test cases without redeploying code.
Item types
An item has either an
input (synthetic) or a trace_id (trace-backed), not both. A trace-backed item’s trace must have recorded steps. If it has none, Veval throws rather than silently calling the live LLM.
A trace-backed item runs live: steps recorded as llm call the model with your current prompt, so a prompt regression fails the item, and every other step is served from the recording, so tools never reach a real system. Set MockModelSteps on the item, or pass ScenarioOptions with MockModelSteps to RunScenarioAsync for every item including those loaded from the dashboard, for a mocked replay instead: free and deterministic, good for code changes, but blind to prompt changes. VevalTestSdk always replays trace-backed items mocked, so tests built on it never make live model calls.
Each replay restarts the trace’s retention window, so a trace your tests replay at least once per window stays. To keep it regardless, click Use as fixture on the trace in the dashboard, which pins it. Each plan allows a set number of pinned traces.
Assertion scope
Results
A scenario run returns whether every item passed, the pass and fail counts, and a result per item with its failure messages.Deep dives
Run a scenario
Define items, apply assertions, and read results.
Assertions
The checks that pass or fail each item.