TL;DR: Resilience at the HTTP level requires more than just a retry loop. This post covers how NResilience manages independent breakers per host, resolves the conflict between HttpClient and library timeouts, ensures safe retries via idempotency keys, prevents retry storms across microservices using a marker header, and reads the rate limit the dependency publishes so you never have to guess it.

The Problem: The "One Client" Fallacy

When you build an HttpClient with resilience, it is easy to think of it as a single pipe. But in a real app, that client often calls multiple different hosts - a payment gateway, a user profile service, and a legacy SOAP API.

If you use a single global circuit breaker, a brownout at the payment gateway trips the breaker for everyone. Your user profile service is healthy, but your app stops calling it because the "client" is broken. This is a blast-radius problem.

Then there is the wire problem. Retrying a GET is safe. Retrying a POST that creates an order can be catastrophic if the server processed the first request but the network failed on the way back. You need a way to signal to the server that "this is a retry of request X" so it doesn't charge the customer twice.

Finally, in a microservice chain, if Service A retries Service B, and Service B retries Service C, a single failure at the bottom can trigger an exponential explosion of requests. This is a retry storm, and it can turn a minor glitch into a full-site outage.

The Solution: HTTP-Aware Resilience

NResilience doesn't treat HTTP as just another callback. It implements specific wire conventions and scoping rules to handle these distributed systems problems.

1. Per-Host State

To solve the blast-radius problem, NResilience supports per-host breakers and budgets. Instead of one breaker per client, you can have one breaker per unique host.

When you configure the client, you can set MaximumHosts to bound the memory used by the host registry. If you call a new host, the library creates a new breaker for it. If the registry hits the limit, it evicts the least recently used host.

// In practice you get the handler through HttpResilience.CreateClient() or AddResilience()
// on a client builder - both of which accept HttpResilienceOptions.
var options = new HttpResilienceOptions
{
    MaximumHosts = 64 // Track up to 64 unique hosts independently
};

using var client = HttpResilience.CreateClient(options: options);

// The breaker for api.payments.com can be Open
// while the breaker for api.users.com remains Closed.

This ensures that a failure in one dependency does not starve your other dependencies. You can inspect the state of these breakers via handler.BreakersByHost() to see exactly which host is struggling.

2. Resolving the Timeout Conflict

There is a classic conflict between HttpClient.Timeout and resilience deadlines. The transport timeout defaults to 100 seconds and covers the entire request sequence - including all attempts and backoff delays - so it silently caps any policy with a longer deadline, invisibly to the policy.

NResilience solves this with OwnTransportTimeout. When enabled (it is on by default), the library sets HttpClient.Timeout to infinite so the policy's deadline is the only active bound.

var options = new HttpResilienceOptions
{
    OwnTransportTimeout = true // HttpClient.Timeout becomes infinite (the default)
};

This makes the NResilience deadline the single source of truth. Whether it is a 30s global deadline or a 2s attempt timeout, the library controls the clock, and the transport just executes the bytes.

Which raises the obvious question about the bytes. Taking ownership of the transport timeout is the right trade - two silent timeout systems are worse than one - but it removes the only bound that ever applied to the response body. An attempt ends when the headers arrive; the body is a live stream you read afterwards, so it is covered by neither the deadline nor the attempt timeout nor the retry, and now not by a 100-second backstop either. A server that sends headers and then stops writing used to give you a call that never returned.

BoundProgress closes that, and it is on by default. The bound is the AttemptTimeout you already set, applied to the gap between two reads: a body that goes quiet for longer than that fails your read with an AttemptStalledException rather than hanging on it. Because it bounds the gap and not the total, a large download that keeps arriving is never cut off.

The stall is bounded, not retried - the retry loop was over before the body existed. BufferResponses is the other side of that trade:

var options = new HttpResilienceOptions
{
    BufferResponses = true // read the body inside the attempt, so a stalled body is retried
};

The body is now read before the attempt is judged, so both clocks cover it and a broken body is one more transient failure. The cost is memory: the whole body is held before the call returns. Leave it off for a client that downloads large files. For a JSON API whose caller was going to buffer the body a moment later anyway, it is close to free.

3. Idempotency on the Wire

Retrying a POST or PATCH is usually a bad idea. NResilience defaults to not retrying these methods. However, if your server supports idempotency keys, you can safely mark a request as repeatable.

The MarkRepeatable extension method does two things in one call: it tells this client the request may be sent again, and it stamps an idempotency key header (defaulting to Idempotency-Key) so the server can deduplicate the operation. Both halves matter - the option without a key duplicates the order, and the key without the option is never used.

using var request = new HttpRequestMessage(HttpMethod.Post, "https://api.example.com/orders")
{
    Content = JsonContent.Create(new { Item = "Book", Amount = 1 })
};

// Mark as repeatable with a unique key
request.MarkRepeatable(idempotencyKey: "order-123-abc");

// The client now allows retries for this POST request
using var response = await client.SendAsync(request, cancellationToken);

If the first attempt succeeds on the server but fails on the wire, the second attempt sends the same key. The server sees the key, recognizes the duplicate, and returns the original success response without processing the order again.

4. Preventing Retry Storms

In a chain of services, retries can multiply. If Service A retries 3 times, and it calls Service B which also retries 3 times, a single failure at the bottom results in 9 requests.

NResilience uses a marker header to make this visible. A retrying handler adds X-NResilience-Retrying: 1 to every request it can retry - on the first attempt, not just on retries, because by the time a retry goes out the amplification has already happened.

On the server side, you add the UseResilienceNestedRetry middleware. It reads the marker and publishes the flag for the rest of the request, so any outbound calls that request makes report a NestedRetry event in telemetry: the middle hop of the chain can now see an amplification it is part of. The library reports but does not intervene - what you do with the signal (typically fewer attempts and a shorter deadline on the inner layer) is your call.

// In your ASP.NET Core Program.cs
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

// Detect incoming retries and report them for this request's outbound calls
app.UseResilienceNestedRetry();

app.MapGet("/proxy", async (HttpClient client) =>
{
    // If the caller sent X-NResilience-Retrying, every outbound call this
    // request makes reports a NestedRetry event in telemetry.
    return await client.GetStringAsync("https://api.example.com/data");
});

app.Run();

Used together with a lower attempt count on the inner layer, this turns an exponential growth of requests into a linear one, protecting your downstream services from being crushed by their own resilience policies.

5. How Failures are Classified

The library uses Classifier.Http to decide if a response is a failure that should trigger a retry or trip a breaker. Not every non-200 response is a failure.

Transient (Verdict.Transient): 500 Internal Server Error, 408 Request Timeout, or HttpRequestException. These are retried.
Throttled (Verdict.Throttled): 429 Too Many Requests or 503 Service Unavailable. These are retried, but the library respects the Retry-After header to avoid hammering a throttled server.
Ok (Verdict.Ok): 404 Not Found and the other 4xx statuses. These are answers, not failures; retrying will not change the outcome.

This classification ensures that your breakers only trip on actual infrastructure failure, not on user errors.

6. The Quota the Dependency Publishes

Retry-After is the dependency telling you what to do after it has already refused you. Most large APIs also tell you before: how much quota you have, how much is left, and when the window resets - on every response, not just the 429. GitHub, Stripe, Shopify and Atlassian all send it. Nothing else in .NET reads it.

The handler does, and it is on by default. It parses two shapes - the structured fields RateLimit-Policy and RateLimit, and the older X-RateLimit-* triple those fields exist to replace - keeps the numbers per host beside that host's breaker and budget, and refuses an attempt locally once what is left is inside the reserve:

var options = new HttpResilienceOptions
{
    Quota = Quota.Reserving(reserve: 0.1) // hold a tenth of the published allowance back. The default.
};

Reserve is the only number in the feature, and it is a fraction rather than a rate, because the rate is the dependency's to publish. That is the whole difference between this and a limiter: PermitsPerSecond is a number somebody had to guess and then keep in sync with a quota they do not control.

The refusal is a local one, so it gets what every local refusal gets: the long backoff curve, the reset time honored as the pushback, no evidence against the host's breaker, and no charge against the retry budget - because the call never left the process. A RejectedByQuota event carries the time until the window resets, and the exception the caller sees is RateLimitedException.

A host that publishes nothing has no quota and the feature is invisible. A host that publishes a remaining count with no quota to take a fraction of has no denominator for the reserve, so it is refused only at zero. A malformed field is ignored rather than thrown - a header the library cannot parse is not a reason to fail a call that worked.

There is one failure mode worth stating outright. A server publishing a per-account quota while you are one of fifty pods will make every pod believe it owns the whole allowance. The reserve does not fix that; only the 429 does, and the 429 still works exactly as it always has. The honest use is a single-instance client, or a generous reserve - and Quota = null reads no headers at all.

Trade-offs and Limitations

Per-host scoping adds a small amount of memory overhead to track the host registry. While MaximumHosts prevents unbounded growth, in an environment where you call thousands of unique hosts, you may need to tune this value to avoid frequent evictions.

Nested retry detection requires the marker header to be passed through. If you have a non-NResilience proxy or a legacy service in the middle of your chain that strips unknown headers, the suppression signal is lost.

The published quota is read per process, not per fleet. It can slow you down and cannot break you - it only ever refuses an attempt, never permits one the policy would have refused - but a shared per-account quota read by several instances is a number each of them will over-count. Lower the reserve, or turn it off, when that is your shape.

Conclusion

Resilience in HTTP is about the contract between the client and the wire. By moving from global state to per-host state, resolving timeout conflicts, and using marker headers for idempotency and nested retries, you can build a system that fails gracefully without amplifying the outage.

The HTTP documentation covers the wire formats and middleware in detail. For how the same rules translate to the RPC world, gRPC in depth explains per-service scoping, the grpc-timeout math, and why blocking calls are not supported.

HTTP in depth