Write assertions

Assertions evaluate a completed VevalExecutionContext and return a failure message or null on pass. You can attach them to scenarios (applied to every item) or to individual ScenarioItems.

Built-in assertions

NoErrors

Fails if any step completed with error status.

TraceAssert.NoErrors()

TraceAssert.no_errors()

TraceAssert.noErrors()

MaxSteps

Fails if the total number of steps (including nested) exceeds the limit.

TraceAssert.MaxSteps(10)

TraceAssert.max_steps(10)

TraceAssert.maxSteps(10)

MaxCost

Fails if the total cost_usd across all steps exceeds the limit.

TraceAssert.MaxCost(0.10m)   // $0.10 max

TraceAssert.max_cost(0.10)   # $0.10 max

TraceAssert.maxCost(0.10)   // $0.10 max

MaxDuration

Fails if the total step duration exceeds the limit in milliseconds.

TraceAssert.MaxDuration(5000)   // 5 seconds max

TraceAssert.max_duration(5000)   # 5 seconds max

TraceAssert.maxDuration(5000)   // 5 seconds max

StepExists

Fails if no step with the given name exists anywhere in the trace (including nested steps).

TraceAssert.StepExists("classify")
TraceAssert.StepExists("answer")

TraceAssert.step_exists("classify")
TraceAssert.step_exists("answer")

TraceAssert.stepExists("classify")
TraceAssert.stepExists("answer")

OutputContains

Fails if no step’s output contains the expected string.

TraceAssert.OutputContains("summarized")

TraceAssert.output_contains("summarized")

TraceAssert.outputContains("summarized")

ToolCalled

Fails if no step with type = "tool" and the given name was recorded. Use handle.SetMeta("type", "tool") when recording tool steps.

TraceAssert.ToolCalled("web_search")

TraceAssert.tool_called("web_search")

TraceAssert.toolCalled("web_search")

Judge

Scores the trace’s output against a rubric using an LLM judge, evaluated server-side. Fails if the judge’s passed verdict is false. The score and reasoning are recorded on the run whether the judge passes or fails.

TraceAssert.Judge(veval, "The answer directly addresses the user's question and cites a source")

TraceAssert.Judge(veval, "The response is polite and on-topic", new JudgeOptions
{
    Model = "claude-sonnet-5",       // must be in the provider's current model catalog
    Threshold = 0.9,                 // stricter than the 0.7 default
    ReferenceOutput = "A polite, on-topic answer that cites at least one source.",
    Samples = 3,                     // median of 3 independent gradings, reduces judge noise
})

TraceAssert.judge(veval, "The answer directly addresses the user's question and cites a source")

TraceAssert.judge(
    veval,
    "The response is polite and on-topic",
    model="claude-sonnet-5",
    threshold=0.9,
    reference_output="A polite, on-topic answer that cites at least one source.",
    samples=3,
)

TraceAssert.judge(veval, "The answer directly addresses the user's question and cites a source")

TraceAssert.judge(veval, "The response is polite and on-topic", {
  model: "claude-sonnet-5",
  threshold: 0.9,
  referenceOutput: "A polite, on-topic answer that cites at least one source.",
  samples: 3,
})

All options are optional:

Option	Default	Notes
`model`	account default	Must appear in the provider’s live model catalog (fetched and cached server-side, ~1h TTL), or the call fails with a 400 rather than silently falling back — this keeps a caller from driving up spend with an arbitrary model.
`threshold`	`0.7`	Pass/fail cutoff on the 0.0–1.0 score. Must be between 0.0 and 1.0.
`referenceOutput` / `reference_output`	none	An example of an output that fully satisfies the rubric, included in the grading prompt to anchor the judge’s scoring.
`samples`	`1`	Runs this many independent gradings and takes the median score, to reduce single-call judge noise. Clamped server-side to 1–5. Each sample is billed separately, so cost scales with this value.

A judge call that fails outright (network error, provider error, invalid model) is treated as an assertion failure rather than a silent pass.

Custom assertions

Implement ITraceAssertion to write your own:

public class OutputLengthAssertion : ITraceAssertion
{
    private readonly int _maxChars;
    public OutputLengthAssertion(int maxChars) => _maxChars = maxChars;

    public string? Evaluate(VevalExecutionContext ctx)
    {
        foreach (var step in ctx.Steps)
        {
            var len = step.Output?.ToString()?.Length ?? 0;
            if (len > _maxChars)
                return $"OutputLength: step '{step.Name}' output is {len} chars (max {_maxChars})";
        }
        return null;  // pass
    }
}

// Use it
TraceAssert.NoErrors(),
new OutputLengthAssertion(500),

from veval import ITraceAssertion

class OutputLengthAssertion(ITraceAssertion):
    def __init__(self, max_chars: int):
        self._max_chars = max_chars

    def evaluate(self, ctx) -> str | None:
        for step in ctx.steps:
            length = len(str(step.output)) if step.output is not None else 0
            if length > self._max_chars:
                return f"OutputLength: step '{step.name}' output is {length} chars (max {self._max_chars})"
        return None  # pass

# Use it
TraceAssert.no_errors(),
OutputLengthAssertion(500),

class OutputLengthAssertion {
  constructor(maxChars) {
    this.maxChars = maxChars;
  }

  evaluate(ctx) {
    for (const step of ctx.steps) {
      const len = step.output != null ? String(step.output).length : 0;
      if (len > this.maxChars)
        return `OutputLength: step '${step.name}' output is ${len} chars (max ${this.maxChars})`;
    }
    return null;  // pass
  }
}

// Use it
TraceAssert.noErrors(),
new OutputLengthAssertion(500),

Scope: scenario vs. per-item

await veval.RunScenarioAsync(
    scenarioName: "my-scenario",
    agent:        service.ExecuteAsync,
    // applied to every item
    scenarioAssertions: [
        TraceAssert.NoErrors(),
        TraceAssert.MaxSteps(10),
    ],
    items: [
        new ScenarioItem
        {
            Name  = "cheap question",
            Input = "What is 2+2?",
            // applied only to this item, on top of scenario assertions
            Assertions = [TraceAssert.MaxCost(0.001m)],
        },
    ]
);

await veval.run_scenario_async(
    scenario_name="my-scenario",
    agent=service.execute_async,
    # applied to every item
    scenario_assertions=[
        TraceAssert.no_errors(),
        TraceAssert.max_steps(10),
    ],
    items=[
        ScenarioItem(
            name="cheap question",
            input="What is 2+2?",
            # applied only to this item, on top of scenario assertions
            assertions=[TraceAssert.max_cost(0.001)],
        ),
    ],
)

await veval.runScenarioAsync(
  "my-scenario",
  (ctx) => service.execute(ctx),
  // applied to every item
  [
    TraceAssert.noErrors(),
    TraceAssert.maxSteps(10),
  ],
  [
    {
      name: "cheap question",
      input: "What is 2+2?",
      // applied only to this item, on top of scenario assertions
      assertions: [TraceAssert.maxCost(0.001)],
    },
  ]
);

Get started

Concepts

Guides

Development

Built-in assertions

NoErrors

MaxSteps

MaxCost

MaxDuration

StepExists

OutputContains

ToolCalled

Judge

Custom assertions

Scope: scenario vs. per-item

What’s next

Catch regressions with snapshots

Run a scenario

​Built-in assertions

​NoErrors

​MaxSteps

​MaxCost

​MaxDuration

​StepExists

​OutputContains

​ToolCalled

​Judge

​Custom assertions

​Scope: scenario vs. per-item

​What’s next

Catch regressions with snapshots

Run a scenario

Built-in assertions

NoErrors

MaxSteps

MaxCost

MaxDuration

StepExists

OutputContains

ToolCalled

Judge

Custom assertions

Scope: scenario vs. per-item

What’s next