Test with replay

VevalTestSdk is a test double for IVevalSdk. It replays a recorded trace so your agent code runs with mocked LLM responses: deterministic, fast, and free. Unlike calling VevalSdk directly with a trace, VevalTestSdk guarantees no live LLM fallthrough. If a step name doesn’t match the recorded trace, it throws immediately.

Setup

// 1. Load a production trace
var trace = await veval.GetTraceAsync("tr_4ec4e79c5d03...");

// 2. Create the test double
var testSdk = new VevalTestSdk(new VevalOptions { ApiKey = "YOUR_API_KEY" })
    .WithReplay(trace);

// 3. Inject into your service (IVevalSdk interface)
var service = new MyAgentService(testSdk, claude);

// 4. Call normally — no live LLM
var result = await testSdk.RunAsync("my-agent", service.ExecuteAsync);

# 1. Load a production trace
trace = await veval.get_trace_async("tr_4ec4e79c5d03...")

# 2. Create the test double
test_sdk = VevalTestSdk(VevalOptions(api_key="YOUR_API_KEY")).with_replay(trace)

# 3. Inject into your service
service = MyAgentService(test_sdk, anthropic)

# 4. Call normally — no live LLM
result = await test_sdk.run_async("my-agent", service.execute_async)

// 1. Load a production trace
const trace = await veval.getTraceAsync("tr_4ec4e79c5d03...");

// 2. Create the test double
const testSdk = new VevalTestSdk({ apiKey: "YOUR_API_KEY" }).withReplay(trace);

// 3. Inject into your service
const service = new MyAgentService(testSdk, anthropic);

// 4. Call normally — no live LLM
const result = await testSdk.runAsync("my-agent", (ctx) => service.execute(ctx));

VevalTestSdk still reports traces and scenario results to the dashboard, so runs count in your history.

In a test project

[Fact]
public async Task Agent_ReturnsExpectedOutput()
{
    var trace = LoadFixture("trace_prompt_caching.json");  // or fetch from API

    var testSdk = new VevalTestSdk(new VevalOptions { ApiKey = TestApiKey })
        .WithReplay(trace);
    var service = new MyAgentService(testSdk, /* mock claude */);

    var result = await testSdk.RunAsync("my-agent", service.ExecuteAsync, input: "test input");

    Assert.Equal("success", testSdk.LastStatus);
    Assert.Contains("caching", result);
}

import pytest

async def test_agent_returns_expected_output():
    trace = load_fixture("trace_prompt_caching.json")  # or fetch from API

    test_sdk = VevalTestSdk(VevalOptions(api_key=TEST_API_KEY)).with_replay(trace)
    service = MyAgentService(test_sdk)

    result = await test_sdk.run_async("my-agent", service.execute_async, "test input")

    assert test_sdk.last_status == "success"
    assert "caching" in result

import { VevalTestSdk } from "@veval/sdk";

test("agent returns expected output", async () => {
  const trace = loadFixture("trace_prompt_caching.json");  // or fetch from API

  const testSdk = new VevalTestSdk({ apiKey: TEST_API_KEY }).withReplay(trace);
  const service = new MyAgentService(testSdk);

  const result = await testSdk.runAsync("my-agent", (ctx) => service.execute(ctx), "test input");

  expect(testSdk.lastStatus).toBe("success");
  expect(result).toContain("caching");
});

Strict mock mode

If your agent calls TrackStepAsync with a step name that isn’t in the recorded trace, VevalTestSdk throws:

InvalidOperationException: Replay mode: no mock output for step 'new_step'.
Available steps: classify, answer.
This would have made a real LLM call.

This catches regressions where code changes add new LLM calls that weren’t in the original trace. The opposite fails too: if the agent never calls a step the trace recorded, the replay fails with

Replay never called 1 recorded step(s) from trace tr_...: 'visit_portal' (recorded step 2).

When several recorded steps share a name, each call is served the one with an identical input, falling back to the next unused one in recorded order. See how calls are matched. A call whose input differs from every recording of that step is reported, not failed. Read the report from the replay result:

var result = await testSdk.ReplayAsync(trace, service.ExecuteAsync, new ReplayOptions { MockLlmResponses = true });

foreach (var b in result.Fidelity!.Bindings.Where(b => b.Kind == ReplayBindingKind.Diverged))
    Console.WriteLine($"{b.StepName}: recorded {b.RecordedInput}, now {b.ActualInput}");

result = await test_sdk.replay_async(trace, service.execute, ReplayOptions(mock_llm_responses=True))

for b in result.fidelity.bindings:
    if b.kind == "diverged":
        print(f"{b.step_name}: recorded {b.recorded_input}, now {b.actual_input}")

const result = await testSdk.replayAsync(trace, (ctx) => service.execute(ctx), { mock_llm_responses: true });

for (const b of result.fidelity.bindings.filter((b) => b.kind === "diverged"))
  console.log(`${b.step_name}: recorded ${b.recorded_input}, now ${b.actual_input}`);

Live replay

To test a prompt change against a recorded trace, replay live: steps recorded as llm call the model, and every other step returns its recorded output, so tools never reach a real system. See Live replay for the rules.

var result = await testSdk.ReplayAsync(trace, service.ExecuteAsync, new ReplayOptions
{
    LiveModelSteps = true,
    Assertions     = [TraceAssert.OutputContains("overdue")],
});

result = await test_sdk.replay_async(trace, service.execute, ReplayOptions(
    live_model_steps=True,
    assertions=[TraceAssert.output_contains("overdue")],
))

const result = await testSdk.replayAsync(trace, (ctx) => service.execute(ctx), {
  live_model_steps: true,
  assertions: [TraceAssert.outputContains("overdue")],
});

Trace-backed scenario items run with VevalSdk replay live by default; VevalTestSdk keeps them mocked.

Skip side effects

Only code inside a step is served from the recording. Work your agent does outside a step runs again on every replay, so check IsReplay before it. IsReplay is true in every replay, mocked or live, and false in a normal run.

var reply = await ctx.TrackStepAsync("write_reply", slot, handle => WriteReplyAsync(slot, handle));

if (!ctx.IsReplay)
    await email.SendAsync(customer, reply);

reply = await ctx.track_step_async("write_reply", slot, write_reply)

if not ctx.is_replay:
    await email.send(customer, reply)

const reply = await ctx.trackStepAsync("write_reply", slot, (handle) => writeReply(slot, handle));

if (!ctx.isReplay) await email.send(customer, reply);

If the side effect’s result matters to the agent, wrap it in a step with SetMeta("type", "tool") instead: replay then returns the recorded result without running it.

ReplayAsync directly

For lower-level control, use ReplayAsync with explicit assertions:

var replayResult = await testSdk.ReplayAsync(
    trace,
    service.ExecuteAsync,
    new ReplayOptions
    {
        MockLlmResponses = true,
        Assertions       = [
            TraceAssert.NoErrors(),
            TraceAssert.StepExists("classify"),
        ],
    }
);

Console.WriteLine(replayResult.Failures.Count == 0 ? "Pass" : "Fail");
foreach (var f in replayResult.Failures)
    Console.WriteLine(f);

from veval import ReplayOptions, TraceAssert

replay_result = await test_sdk.replay_async(
    trace,
    service.execute_async,
    ReplayOptions(
        mock_llm_responses=True,
        assertions=[
            TraceAssert.no_errors(),
            TraceAssert.step_exists("classify"),
        ],
    ),
)

print("Pass" if not replay_result.failures else "Fail")
for f in replay_result.failures:
    print(f)

import { TraceAssert } from "@veval/sdk";

const replayResult = await testSdk.replayAsync(
  trace,
  (ctx) => service.execute(ctx),
  {
    mock_llm_responses: true,
    assertions: [
      TraceAssert.noErrors(),
      TraceAssert.stepExists("classify"),
    ],
  }
);

console.log(replayResult.failures.length === 0 ? "Pass" : "Fail");
for (const f of replayResult.failures)
  console.log(f);

`ReplayResult` member	Description
`Failures`	List of assertion failure messages
`ReplayedContext`	The `VevalExecutionContext` from the run
`Output`	Return value of your agent
`Status`	`"success"` or `"error"`
`Error`	Exception message if status is `"error"`

LastStatus / LastError

After RunAsync, inspect the outcome without catching exceptions:

await testSdk.RunAsync("my-agent", service.ExecuteAsync);

Assert.Equal("success", testSdk.LastStatus);
Assert.Null(testSdk.LastError);

await test_sdk.run_async("my-agent", service.execute_async)

assert test_sdk.last_status == "success"
assert test_sdk.last_error is None

await testSdk.runAsync("my-agent", (ctx) => service.execute(ctx));

expect(testSdk.lastStatus).toBe("success");
expect(testSdk.lastError).toBeNull();

Get started

Concepts

Guides

Development

Setup

In a test project

Strict mock mode

Live replay

Skip side effects

ReplayAsync directly

LastStatus / LastError

What’s next

Catch regressions with snapshots

Write assertions

​Setup

​In a test project

​Strict mock mode

​Live replay

​Skip side effects

​ReplayAsync directly

​LastStatus / LastError

​What’s next

Catch regressions with snapshots

Write assertions

Setup

In a test project

Strict mock mode

Live replay

Skip side effects

ReplayAsync directly

LastStatus / LastError

What’s next