4 min readProtecting the dependency

Open the breaker before the dependency drowns

Learn how the circuit breaker protects your system from cascading failures and why consecutive failures, brownout detection, and recovery ramps are critical for stability.

circuit-breakerfailure-handling

TL;DR: When a dependency fails, retries make it worse. A circuit breaker stops the bleeding by refusing calls once a failure threshold is hit. NResilience's breaker goes beyond simple error counting, detecting "brownouts" (latency spikes) and offering a gradual recovery ramp to prevent the "thundering herd" from crashing a recovering service.

The kindness of a refusal

In a distributed system, the kindest thing you can do for a struggling dependency is to stop calling it.

When a service starts failing or slowing down, the instinctive reaction is to retry. But if 100 instances of your app are all retrying 3 times, you've just quadrupled the load on a server that is already drowning. This is how a minor glitch becomes a cascading failure that takes down the entire stack.

The circuit breaker is a state machine that sits between your app and the dependency. Its job is simple: fail fast.

How the breaker trips

Most breakers only look at error rates. NResilience uses three distinct trip conditions to ensure it catches every kind of failure.

1. Consecutive Failures

The classic "circuit breaker" pattern. If the dependency fails 5 times in a row (the default), the breaker trips. This is the best protection against total outages, and it needs no throughput to work.

2. The Brownout (Slow Calls)

A dependency doesn't have to return an error to be broken. A service that returns 200 OK in 30 seconds is often more dangerous than one that returns 500 Internal Server Error in 10ms. The former ties up your threads and connection pools, leading to a "resource exhaustion" failure in your own app.

NResilience detects these brownouts automatically. It measures a baseline of "normal" latency and trips if a significant fraction of calls suddenly exceed a multiple of that baseline.

3. Relative Error Rates

Fixed error thresholds (e.g., "trip at 5%") are often wrong. 5% is catastrophic for a payments API but normal for a flaky search backend. NResilience measures the dependency's normal error rate and trips when the current rate spikes significantly above that baseline.

Configuring one

A breaker is a live object, so you hold it where its lifetime is obvious - a field, a singleton, a property on a dependencies class - and hand the same instance to every policy that should share its state. Most of the time the shape you want is one line:

public Breaker Payments { get; } = Breaker.Of(consecutiveFailures: 5, name: "payments");

Breaker.Of covers the 90% case. Everything else lives in BreakerSettings, which one object fully describes - the name included, so settings with { Name = "orders" } is a differently-configured breaker rather than a label bolted on afterwards:

public Breaker Search { get; } = new(new BreakerSettings
{
    Name = "search",
    ConsecutiveFailures = 5,
    SlowCalls = SlowCalls.Above(3),      // a brownout is 3x normal, whatever normal is
    Recovery = Recovery.Over(0.25),      // ramp back rather than cliff
});

The name is not decoration. It is what appears in log lines, telemetry tags and the health endpoint, and it is how you tell which of five breakers is the one that just opened.

The Lifecycle of a Break

Once the breaker trips, it enters a cycle designed to protect the dependency while allowing for safe recovery.

Open: The Silence

The breaker is Open. Every call is refused with a CallRejectedException. No traffic reaches the dependency, giving it space to recover. To prevent the "synchronized pulse" (where every pod probes at the exact same millisecond), NResilience applies jitter to the break duration. The break duration itself doubles on each consecutive trip, up to a maximum, so a flapping dependency spends longer and longer cold.

Half-Open: The Probe

After the break duration expires, the breaker becomes Half-Open. It allows a small number of concurrent "probe" calls through (one by default).

  • If a probe fails, the breaker re-opens immediately with a doubled break duration.
  • If two probes succeed, the breaker closes.

Recovering: The Ramp

Closing a breaker on a "cliff" (sending 100% of traffic back instantly) often crashes the dependency again. NResilience offers a Recovery Ramp, which is opt-in because a call refused during a ramp would have been served by a breaker that closed immediately.

The ramp admits a small fraction of traffic (5% to start) and gradually increases it. If the admitted traffic is fast, the ramp climbs. If a call comes back slow, the ramp halves the admitted fraction on the spot. One real failure during the ramp re-opens the breaker with the grown break duration. This ensures the dependency warms up without being overwhelmed.

Summary

A circuit breaker isn't just about stopping errors; it's about managing the relationship between your app and its dependencies. By combining consecutive failure counting, brownout detection, and a cautious recovery ramp, NResilience ensures that your app stays responsive even when its dependencies are not.

The breaker stops calls to a failing dependency. But what happens when your fleet of clients all retry at once? Don't be the storm introduces the retry budget that bounds retries across the entire fleet.