Skip to main content
Replay runs your real agent code against a recorded trace. In a mocked replay, every step returns its recorded output instead of calling the LLM, so tests are deterministic, fast, and free. In a live replay, model steps call the LLM with your current prompt while tools still return their recorded output, so a prompt change shows up without touching real systems.

How replay fits in

VevalTestSdk is a test double for IVevalSdk. Load a trace with WithReplay, inject the test SDK into your agent in place of the real one, and call it normally. When your agent calls TrackStepAsync, the test SDK returns the output recorded for that call. Have your agent accept the IVevalSdk interface, not the concrete class, so your tests can swap in VevalTestSdk. Replay also powers trace-backed scenario items.

Strict mocking

VevalTestSdk never falls through to the live LLM. If your agent calls TrackStepAsync with a step name that isn’t in the recorded trace, it throws:
A code change that adds a new LLM call fails the test instead of quietly making real calls.

How calls are matched to the recording

Each TrackStepAsync call is served a recorded step with the same name. When several recorded steps share that name, replay picks the one whose input is identical to the call’s input (key order doesn’t matter), so calls that changed order still get their own outputs. If none is identical, replay uses the next unused recording of that step in recorded order and marks the call diverged. A step rule can match a step on its name only, or ignore some input fields when matching. In mocked mode, a replay fails if your agent:
  • calls a step more times than the trace recorded (the strict error above), or
  • never calls a step the trace recorded. A skipped branch would otherwise leave your assertions passing on a path the recording doesn’t cover.
A diverged input doesn’t fail the replay on its own: prompts often contain dates or IDs that change on every run. Each replay result reports how every call was matched in Fidelity (fidelity in Python and Node): which calls matched, which diverged with the recorded and actual input side by side, and which recorded steps went unused.

Guarding against drift

To also fail when inputs drift, turn on CompareWithRecording in ReplayOptions. The replay then fails when its step sequence or any step input differs from the trace it’s replaying. List fields that legitimately change on every run in its IgnoreFields; Veval never ignores a field you didn’t list. Mocked replay checks control flow and code changes. It can’t tell you whether a new prompt is better, because every model call returns what the old prompt produced. To test a prompt change, replay live.

Live replay

With LiveModelSteps in ReplayOptions, steps recorded as llm run for real and every other step is served from the recording. A prompt change shows up in the output, while tool calls (a portal visit, a payment) return what they returned in production and never reach a real system. Trace-backed scenario items run with VevalSdk replay this way by default; VevalTestSdk keeps them mocked.
  • A step recorded as tool or left untyped is served from the recording, never run.
  • A model step called more often than recorded runs live too.
  • A model step the recording doesn’t know as one runs live if its type is given where it’s called (StepOptions, see Steps). That covers a step the recording never contained and one saved untyped (custom), such as a model call that threw before SetMeta ran. Without it, Veval can’t tell a model call from a tool call before the step runs: an unrecorded step stops the replay, and an untyped one is served its recorded output. A recorded llm or tool type wins, and a different type at the call site is reported in Fidelity (DeclaredType), so a mislabelled call can’t turn a recorded tool step live.
  • A recorded step the agent didn’t call is reported in Fidelity but doesn’t fail the replay: a live model can take a different path.
  • If no step ran live, the replay fails, because nothing could have caught a prompt change. Mark model steps with StepOptions { Type = StepType.Llm } at the call site, or SetMeta("type", "llm"), when you record.

When the run leaves the recording

A live model can choose differently than it did when the trace was recorded. From that point the recording can’t say what the real tools would have returned, so serving the model a recording from another path could pass a run that would fail for real. Instead, the replay stops at that step and ends inconclusive: it neither passes nor fails. The replay stops when:
  • a step recorded as tool is called with an input that matches none of its unused recordings, or
  • a step that isn’t a model step is called with no unused recording left to serve: a new tool, or an extra call.
Other diverged steps, such as untyped ones, are served the next unused recording and reported as diverged in Fidelity, as before. When a replay stops, its assertions aren’t evaluated, Passed is false, and Inconclusive (inconclusive in Python and Node) reports the step’s position and name, why it stopped, and the recorded and actual input side by side. The step that stopped throws ReplayStoppedException (ReplayStoppedError in Python and Node) into your agent; if your agent catches it, the replay is still inconclusive and every later step throws it again. The replay’s trace is stored with status inconclusive, so it doesn’t raise the error alert. In a scenario, an inconclusive item counts in InconclusiveCount (inconclusive_count), not in the failures, and by default makes the run not pass; ScenarioOptions.MaxInconclusiveRate (max_inconclusive_rate) allows a share of them (see inconclusive items). The dashboard shows the item as inconclusive, with the step where it left the path. A change to your own code that alters how a tool is called stops a live replay too, because the tool’s input no longer matches the recording. To test that kind of change, re-record, or replay mocked.

Step rules

StepRules in ReplayOptions (step_rules in Python and Node) sets, per step name, how calls match their recordings and what a live replay does when none matches. It also takes effect on scenario items, for the whole scenario or one item.
  • Match on name only (MatchOnNameOnly): any call of the step matches its next unused recording. Use it for steps whose input doesn’t change what they return, such as narration whose wording follows the model.
  • Ignore input fields (IgnoreInputFields): drop these property names, at any depth, before comparing inputs. Use it for fields such as a free-text reasoning argument next to the arguments that matter.
  • On divergence (OnDivergence): Stop or Serve, overriding the default for that step. The default stops steps whose recorded type is in StopOnDivergenceTypes, which is tool unless you change it.
An input that is a JSON object encoded as a string, such as a model’s tool-call arguments, is parsed before fields are ignored. OnDivergence only decides what happens to a call that has recordings left to serve. A call with no unused recording left always stops a live replay, even with Serve, because there is nothing to serve it. Model steps always run live in a live replay, so OnDivergence has no effect on them. Matching rules apply to mocked replays too; mocked replays never stop. See Live replay for examples.

Side effects during replay

Replay serves recorded outputs only to code inside a step. Anything your agent does outside a step, like sending an email or writing to a database, runs again in every replay. Either wrap that work in a step recorded as tool, so replay serves it from the recording, or check ctx.IsReplay (ctx.is_replay in Python, ctx.isReplay in Node) and skip it. IsReplay is true in every replay, mocked or live, including trace-backed scenario items and VevalTestSdk.WithReplay. See Skip side effects.

Reporting

VevalTestSdk still reports traces and scenario results to the dashboard, so replay runs count in your history.

Deep dives

Test with replay

Set up VevalTestSdk in your test project.

Snapshots

Compare replays against a known-good baseline.