5 min read

Break things on purpose - Chaos testing with NResilience

Stop hoping your resilience policies work. Use the Chaos tool to inject faults through the pipeline and verify your system's behavior under pressure.

TL;DR: Hope is not a resilience strategy. NResilience includes a Chaos tool that lets you inject latency and faults directly into your call path. Because it sits inside your resilience pipeline, it tests whether your classifiers, retries, and breakers actually behave as expected - not just whether your app survives a crash.

The "Hope" Strategy

Most developers write resilience policies and then "hope" they work. They might write a unit test that mocks a dependency to throw an exception, but that usually tests the exception handler, not the resilience policy.

If you wrap a call in a retry policy, you don't just want to know that the app doesn't crash. You want to know:

  • Does the classifier actually see this specific error as Transient?
  • Does the retry budget bound the amplification, or are you about to DDoS your own database?
  • Does the circuit breaker trip exactly when it should, and does it probe for recovery?

To answer these, you need to break things on purpose, and you need to do it in a way that exercises the entire pipeline.

Through-the-Pipeline

The core philosophy of the NResilience Chaos tool is that faults must traverse the real pipeline.

If you inject a fault outside your resilience call, you are bypassing the very logic you're trying to test. You're testing the "outer" world. To test the policy, the fault must be injected inside the callback that the policy executes.

When a fault is injected inside, it is treated exactly like a real failure:

  1. It is passed to the Classifier.
  2. The resulting Verdict determines if it retries.
  3. It is counted against the Circuit Breaker.
  4. It consumes the Retry Budget.

How to Inject Chaos

Wrapping Callbacks

For general async work, you can wrap your callback using the Inject extension method. This replaces your normal work with a version that occasionally fails or slows down based on a Chaos profile.

using NResilience;
using NResilience.Testing;

// 1. Define the chaos profile
var chaos = new Chaos 
{ 
    Enabled = true, 
    FaultRate = 0.1, // 10% of calls fail
    Latency = TimeSpan.FromMilliseconds(500), 
    LatencyRate = 0.2 // 20% of calls are slowed
};

// 2. Wrap your actual work
Func<CancellationToken, Task<string>> work = async ct => 
{
    return await _service.GetDataAsync(ct);
};

var chaoticWork = chaos.Inject(work);

// 3. Run it through your policy
var result = await policy.RunAsync(chaoticWork, ct);

Chaos in the HTTP Pipeline

If you're using HttpClient, the ChaosHandler lets you inject faults into the DelegatingHandler chain. The key is to add it after AddResilience(). In .NET's handler chain, "after" means "inner to," so the fault is injected before the request leaves the app but after the resilience handler has wrapped it.

services.AddHttpClient<OrdersClient>()
    .AddResilience() // Outer: handles the retries/breakers
    .AddHttpMessageHandler(() => new ChaosHandler(chaos)); // Inner: injects the fault

Testing Outcomes vs. Exceptions

Real failures aren't always exceptions. Sometimes a dependency returns a 503 Service Unavailable or a 429 Too Many Requests. If your classifier is tuned to retry these specific status codes, testing with throw new Exception() is useless.

The Chaos tool allows you to inject outcomes - specific return values that the classifier must judge.

// Inject a 503 response instead of throwing an exception
var chaosHandler = new ChaosHandler(chaos, response: () => 
{
    return new HttpResponseMessage(HttpStatusCode.ServiceUnavailable);
});

Game Days and Production Safety

You might be hesitant to leave testing code in production. However, Chaos is designed to be low-cost. When Enabled = false, the Inject method returns the original callback untouched.

At the call site, this is just one boolean check at composition time. There is zero overhead per call when chaos is off. This means you can safely leave Inject in your production code and toggle a "Game Day" on and off via configuration without redeploying.

Determinism and Seeds

Randomness is the enemy of a stable CI build. To make chaos tests repeatable, Chaos accepts a Seed. If you use the same seed and drive your calls sequentially, you will get the exact same sequence of faults every time.

var chaos = new Chaos { Seed = 42, FaultRate = 0.1 };

What's Next?

Chaos testing is great for verifying that your policies react correctly to single faults. But real-world outages are rarely single events - they are sequences of degradations, partial failures, and recovery cycles.

In our next deep dive, we'll look at Simulation. While Chaos injects random noise, Simulation lets you script precise failure scenarios to see exactly how your system evolves over time.


Go deeper: Read the Fault Injection documentation for a full API reference.