Don't be the storm
Learn how to prevent retry storms using a client-side retry budget that bounds retries as a fraction of successful traffic.
TL;DR: Per-call retry limits can't stop a retry storm because every client independently thinks it is being reasonable. NResilience uses a RetryBudget to bound aggregate retries as a fraction of traffic (e.g., 10%), ensuring that a fleet of clients doesn't accidentally DDoS a struggling dependency.
The Multiplier Effect
In a simple app, 3 retries sounds reasonable. But in a modern distributed system, retries compose multiplicatively.
Imagine a request flow: Frontend -> Backend -> Database. If each layer is configured with a "reasonable" limit of 3 attempts, a single user action that hits a failure at the database layer can generate 4 x 4 x 4 = 64 attempts at the bottom. Each layer is blind to the amplification because every service sees only its own three attempts.
This is a retry storm. When a dependency starts to struggle, the very mechanism meant to provide reliability (retries) becomes a weapon that finishes it off. The dependency isn't just failing; it's being hammered by a fleet of clients who are all "just retrying a few times."
The Solution: Token-Based Budgets
The only way to stop a storm is to stop thinking about per-call limits and start thinking about aggregate limits.
NResilience implements this using a Retry Budget. Instead of asking "How many times can I retry this specific call?", the executor asks the budget: "Do I have a token available to fund this retry?"
How the Budget Works
The budget is a token bucket with two ways to get tokens:
- Success Deposits: Every successful attempt deposits a small fraction of a token (default is 0.1). This means that in steady state, retries can add at most 10% on top of your successful traffic.
- The Floor Rate: To prevent a low-traffic client from being completely blocked from retrying, the bucket refills at a small absolute rate (3 tokens per second by default), with a burst capacity of ten seconds of that floor.
When a call fails and the policy wants to retry, it spends a token. If the bucket is empty, the retry is refused - even if you haven't hit your Attempts limit. The first attempt of every call always runs; a budget throttles only retries. A refused retry arrives as StopReason.BudgetExhausted (or a CallRejectedException in throw-style code) after the library's short guarded rejection pause, with a Retry-After hint saying when the floor rate will fund the next one.
Using the Budget
The retry budget is on by default on every policy, not just the presets. Budget has three values and no null:
| Value | Meaning |
|---|---|
RetryBudget.Automatic (the default) |
A bucket private to this policy instance, or to each key when the policy is scoped. |
RetryBudget.None |
No budget. Every retry the policy's other bounds allow is funded. |
| Any other instance | That budget, shared wherever the instance is shared. |
Automatic Budgets
RetryBudget.Automatic gives the policy instance a private bucket. This is the safe default for most cases, and it is what you get for free - a policy you never configured a budget on still has one.
The privacy matters: it is why a storm against payments cannot throttle retries to search. It also means the bucket belongs to the instance, so a policy rebuilt inside a method never accumulates enough history to bound anything. Hold policies in a field, and the NRES005 analyzer will tell you when you have not.
Shared Budgets
If you have multiple policies hitting the same dependency, you should share a budget so they don't each get their own 10% allowance.
var sharedBudget = RetryBudget.Shared("payments-api", fraction: 0.1, minimumPerSecond: 3);
var fastPolicy = Resilience.Http with
{
Budget = sharedBudget,
Deadline = TimeSpan.FromSeconds(1)
};
var slowPolicy = Resilience.Http with
{
Budget = sharedBudget,
Deadline = TimeSpan.FromSeconds(10)
};
One warning: don't share a single process-wide budget across unrelated services. A storm against one service would then throttle retries for another - which is exactly what a shared budget is meant to prevent.
Turning it off
RetryBudget.None is the one spelling of off, and it is worth reaching for only when the dependency is genuinely unshared - a local process, a per-tenant database nobody else touches. Turning the budget off to make retries "work properly" during an incident is how a brownout becomes an outage.
Holding capacity back for what matters
A nightly backfill and a checkout draw on the same bucket, and by default the executor cannot tell them apart. With UseAmbientCriticality set on the policy, a call running at Criticality.Sheddable has to leave half the bucket behind: while the dependency is healthy nothing is refused, and once the bucket starts draining the backfill stops so the capacity left is there for the work someone is waiting on.
The refusal is the ordinary one - StopReason.BudgetExhausted, the same guarded pause, the same RejectedByBudget event - because criticality changes when a refusal happens rather than what it is. Deadlines that travel covers where the level comes from.
Monitoring the Storm
One of the most useful metrics in a resilient system is Budget Utilization.
Because the budget is a shared resource, its utilization (0 to 1) is a leading indicator of dependency health. A budget sitting near 1.0 means your clients are actively refusing retries because the dependency is failing. This is a symptom you should alert on - it's the library telling you that it is currently preventing a retry storm.
On a dashboard, you should track:
nresilience.attempts / nresilience.calls(The actual retry fraction)RetryBudget.Utilization(How close you are to the ceiling)
Measuring It Before You Ship
A dashboard tells you what the multiplier was. A simulation tells you what it will be, before the dependency finds out.
Simulate runs your real policy against a modeled dependency on a virtual clock and reports exactly the ratio the dashboard tracks. Five simulated minutes of a dependency that goes eight times slower for a minute, at 500 calls a second, with the budget off and then on:
| Retry budget off | Retry budget on | |
|---|---|---|
| Load multiplier | 1.168 | 1.016 |
| Worst second | 2.086 | 1.104 |
| Availability | 0.9625 | 0.9026 |
The budget takes the worst second the dependency feels from 2.1x down to 1.1x, and it costs this client six points of availability to do it. That is the whole collective-action trade, priced.
Load.Peers asks the fleet-scale version of the question. Fifty pods offering 500 calls a second each, against a dependency that serves 400 at once, is not a retry problem at all: availability lands at 0.44 and the run ends without recovering. The peers are a term in the dependency's arithmetic rather than fifty simulated policies - honest about being an approximation, and enough to tell you that your budget is not the thing that needs fixing.
Summary
Retries are a collective-action problem. What is reasonable for one call is catastrophic for a fleet. By bounding retries as a fraction of successful traffic, NResilience ensures that your app remains a good citizen of the network, protecting your dependencies from the very storms you're trying to survive.
The Simulation documentation covers modeling a dependency and reading the report. The retry budget protects the dependency from you; you also need to protect it from your normal traffic surges, and Bound what leaves the process introduces the limiters and bulkheads that do exactly that.