Tests that don't flake

TL;DR: Resilience is hard to test because it depends on timers and flakiness. NResilience solves this by replacing real time with FakeTimeProvider and real dependencies with Sequence<T>, turning non-deterministic race conditions into predictable, scripted tests.

The "Sleep" Trap

Most resilience tests suffer from the same pattern: they use Task.Delay or Thread.Sleep to simulate a dependency outage, then await for a few seconds and hope the retry logic fired.

This is the primary source of "flaky tests." On a loaded CI server, a 100ms sleep might actually take 200ms, causing a timeout test to fail randomly. Worse, testing a 30-second circuit breaker break duration means your test suite takes 30 seconds to run.

Resilience should be tested as a state machine, not a stopwatch.

Scripting the Dependency

Instead of mocking an interface and trying to maintain an internal counter of how many times a method was called, NResilience provides Sequence<T>.

A sequence is a pre-defined script of outcomes. You define exactly what the dependency does on the first call, the second call, and so on.

using NResilience.Testing;

// 1. Script the dependency: fail twice, then succeed
var calls = Sequence.For<string>()
    .Throws(new HttpRequestException("Transient error"))
    .Throws(new HttpRequestException("Transient error"))
    .Returns("Success!");

// 2. Run the policy against the script
var result = await policy.RunAsync(
    attempt => calls.NextAsync(attempt),
    cancellationToken);

Assert.Equal("Success!", result);
Assert.Equal(3, calls.CallCount);

Because Sequence<T> is a simple list of outcomes, there is no ambiguity about what the dependency returned or when it threw.

Controlling Time

To make these scripts deterministic, you must control the clock. NResilience policies accept a TimeProvider. In tests, you use FakeTimeProvider.

When a policy decides to sleep for a backoff delay, it doesn't block the thread - it asks the TimeProvider to notify it when the time has passed. By using a fake clock, you can "warp" time forward instantaneously.

var time = new FakeTimeProvider();
var policy = TestPolicy.WithClock(time); // Instant, on the test clock

var calls = Sequence.For<string>(time: time)
    .Delays(TimeSpan.FromSeconds(10)) // Simulate a slow call
    .Returns("Done");

var task = policy.RunAsync(attempt => calls.NextAsync(attempt));

// The task is currently suspended on the 10s delay.
// Warp time forward to trigger the completion.
time.Advance(TimeSpan.FromSeconds(10));

var result = await task;

This turns a 10-second test into a 10-millisecond test.

Asserting on Intent, Not Outcome

Asserting that a method returned "Success" is not enough. You need to know how it got there. Did it succeed on the first try? Did it retry twice? Did the circuit breaker trip?

The EventRecorder is a specialized listener that captures every CallEvent raised by the policy. It allows you to assert on the exact sequence of resilience events.

var events = new EventRecorder();
var policy = TestPolicy.InstantHttp with { OnEvent = events.Record };

var calls = Sequence.For<string>()
    .Throws(new HttpRequestException("Transient"))
    .Returns("OK");

await policy.RunAsync(attempt => calls.NextAsync(attempt));

// Assert on the exact order of events
Assert.Equal(
    [CallEventKind.Attempt, CallEventKind.Retrying, CallEventKind.Attempt, CallEventKind.Succeeded],
    events.Kinds
);

By asserting on events.Kinds, you verify the internal logic of the policy (e.g., "did it actually decide to retry?") rather than just the final result.

Testing the HTTP Layer

For those using the HttpClient integration, ScriptedHttpHandler provides the same deterministic scripting but at the HttpMessageHandler level. This allows you to test how your HttpClient handles different status codes without running a local web server.

var transport = new ScriptedHttpHandler()
    .Responds(HttpStatusCode.ServiceUnavailable, times: 2)
    .Responds(HttpStatusCode.OK);

using var client = HttpResilience.CreateClient(policy, innerHandler: transport);
var response = await client.GetAsync(uri);

Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.Equal(3, transport.CallCount);

Measuring the Whole Configuration

Scripts and fake clocks prove what one call does. They say nothing about what a hundred and fifty thousand of them cost the dependency, and that is the question a reviewer actually asks.

Simulate answers it. It runs your real policy - the real executor, breaker, retry budget, classifier and estimators - against a modeled dependency on a virtual clock, and reports what the run cost:

var report = Simulate.Policy(api)
    .Against(Dependency
        .Healthy(p50: TimeSpan.FromMilliseconds(20), p99: TimeSpan.FromMilliseconds(200))
        .Brownout(after: TimeSpan.FromSeconds(30), slower: 8, lasting: TimeSpan.FromMinutes(1)))
    .Under(Load.Constant(perSecond: 500))
    .For(TimeSpan.FromMinutes(5))
    .Run(seed: 42);

Assert.True(report.LoadMultiplier <= 1.2);

Five simulated minutes take about two seconds of wall clock, because nothing sleeps. LoadMultiplier is attempts that reached the dependency per call you made; Amplification is the same ratio in the worst single second, which is the one a dependency feels. Availability, Latency(quantile), BreakerOpens and TimeToRecover round out the report.

Only three things are fake: the clock, the random source and the dependency. Nothing about the decision logic is re-implemented, because a simulator that modeled the library would be worse than no simulator. And the report reproduces to the last byte from a seed - on any operating system and any processor - so Assert.True(report.LoadMultiplier <= 1.2) is a gate rather than an anecdote.

Conclusion

Flaky tests are the result of trying to test asynchronous time with real-world clocks. By combining Sequence<T> for behavior, FakeTimeProvider for time, and EventRecorder for intent, you can write resilience tests that are deterministic and execute in milliseconds. One rule to remember: pass the same TimeProvider to the policy and the script, or a scripted delay becomes a real sleep again.

Stop sleeping in your tests. Script your failures instead.

The Testing documentation covers the full NResilience.Testing package, and Simulation covers the report in detail. For injecting realistic, messy failures into your resilience pipeline, Break things on purpose introduces the Chaos engine and how to run game days in production.