Skip to main content

​
Overview

Assertions evaluate a completed VevalExecutionContext and return a failure message or null on pass. They can be attached to scenarios (applied to every item) or to individual ScenarioItems.

​
Built-in assertions

​
NoErrors

Fails if any step completed with error status. 
TraceAssert.NoErrors()

​
MaxSteps

Fails if the total number of steps (including nested) exceeds the limit. 
TraceAssert.MaxSteps(10)

​
MaxCost

Fails if the total cost_usd across all steps exceeds the limit. 
TraceAssert.MaxCost(0.10m)   // $0.10 max

​
MaxDuration

Fails if the total step duration exceeds the limit in milliseconds. 
TraceAssert.MaxDuration(5000)   // 5 seconds max

​
StepExists

Fails if no step with the given name exists anywhere in the trace (including nested steps). 
TraceAssert.StepExists("classify")
TraceAssert.StepExists("answer")

​
OutputContains

Fails if no step’s output contains the expected string. 
TraceAssert.OutputContains("summarized")

​
ToolCalled

Fails if no step with type = "tool" and the given name was recorded. Use handle.SetMeta("type", "tool") when recording tool steps. 
TraceAssert.ToolCalled("web_search")

​
Custom assertions

Implement ITraceAssertion to write your own: 
public class OutputLengthAssertion : ITraceAssertion
{
    private readonly int _maxChars;
    public OutputLengthAssertion(int maxChars) => _maxChars = maxChars;

    public string? Evaluate(VevalExecutionContext ctx)
    {
        foreach (var step in ctx.Steps)
        {
            var len = step.Output?.ToString()?.Length ?? 0;
            if (len > _maxChars)
                return $"OutputLength: step '{step.Name}' output is {len} chars (max {_maxChars})";
        }
        return null;  // pass
    }
}

// Use it
TraceAssert.NoErrors(),
new OutputLengthAssertion(500),

​
Scope: scenario vs. per-item

await veval.RunScenarioAsync(
    scenarioName: "my-scenario",
    agent:        service.ExecuteAsync,
    // applied to every item
    scenarioAssertions: [
        TraceAssert.NoErrors(),
        TraceAssert.MaxSteps(10),
    ],
    items: [
        new ScenarioItem
        {
            Name  = "cheap question",
            Input = "What is 2+2?",
            // applied only to this item, on top of scenario assertions
            Assertions = [TraceAssert.MaxCost(0.001m)],
        },
    ]
);