7 min readRunning it for real

Deadlines that travel

How to propagate deadlines and criticality across services in ASP.NET Core, avoiding the nested-retry storm and the "zombie request" problem.

deadlinescriticalityaspnetcore

TL;DR: A deadline is only as good as its propagation, and it is not the only thing that has to travel. NResilience for ASP.NET Core provides middleware to read inbound deadlines, criticality levels and retry markers, ensuring your outbound calls inherit the remaining time, know whether anyone is waiting, and avoid multiplying load on a struggling dependency.

The "Zombie Request" Problem

Imagine Service A calls Service B, and Service B calls Service C.

Service A has a 10-second deadline. It calls Service B. Service B takes 9 seconds to respond. Service A's deadline expires, and it hangs up. But Service B is still working - it has already sent a request to Service C, and it's waiting for an answer.

Service B is now processing a "zombie request": the caller has already given up, but Service B is still consuming threads and memory to finish a job that no one is listening to. In a cascading failure, these zombies pile up, consuming all available resources and turning a partial outage into a total collapse.

To fix this, the deadline must travel.

Propagating the Clock

The solution is to pass the deadline as a header. When Service A calls Service B, it tells it how long this side is prepared to wait - each attempt carries min(AttemptTimeout, time left on the deadline) in whole milliseconds. Service B reads it and knows exactly how much of its own budget it can spend before the answer stops mattering.

NResilience makes this automatic in ASP.NET Core with UseResilienceDeadline().

var app = builder.Build();

// Register before anything that makes an outbound call
app.UseResilienceDeadline();

app.MapGet("/", () => "Hello World");

This middleware reads the inbound deadline header and publishes it for the rest of the request. Any policy in your app that has UseAmbientDeadline = true is bounded by the tighter of two values: its own configured deadline, or the time the caller is actually still waiting.

Propagating What Matters

A deadline is not the only fact that stops at the process edge. How much a request matters stops there too, and the consequence is sharper than a zombie: Service C sheds the checkout and serves the nightly backfill, because from where it stands the two are identical.

AmbientCriticality carries a four-level label the same way the deadline carries a number - Sheddable, SheddablePlus, Critical, CriticalPlus. The same UseResilienceDeadline() pass reads it, because both values arrive in the same headers collection and a second middleware would be a second walk over it for one lookup.

// The outbound half, on the HTTP options.
var options = new HttpResilienceOptions { PropagateCriticality = true };

// The inbound half, on the policy.
var api = Resilience.Http with { UseAmbientCriticality = true };

// And the label itself, for work that is not an inbound request.
using var scope = AmbientCriticality.Begin(Criticality.Sheddable);

What the library does with a level is deliberately narrow. It does not shed, it does not reorder, and it does not decide what Sheddable should cost your service - that depends on which endpoints are cheap and which have a fallback, and a library cannot see either. What it does is the half your own code cannot reach, because it happens inside the executor:

  • A Sheddable call is never hedged. Hedging spends capacity to buy latency, and there is no latency worth buying when nobody is waiting for the answer.
  • A Sheddable retry has to leave half the retry budget behind. While the dependency is healthy the bucket is full and nothing is refused; once it starts draining, the backfill stops and what remains is there for the checkout.

Two guardrails are worth stating outright. An unlabeled call is Critical, never Sheddable - a default that sheds is a default that loses requests during the first incident after an upgrade. And CriticalPlus cannot arrive from the wire: a caller that could escalate itself would, and a level everyone claims is a level that means nothing. Inbound values clamp at Critical.

Shedding itself stays with Admit, which reads AmbientCriticality.Current like any other ambient fact. The point of the feature is that Admit has a value to read rather than a guess to make.

Detecting the Retry Storm

While deadlines prevent zombies, nested retries create storms.

If Service A retries a call to Service B three times, and Service B retries its call to Service C three times, a single failure at Service C can trigger 9 requests (3x3). If you have a chain of four services, you're looking at 81 requests for one logical operation. This is how a minor glitch becomes a self-inflicted DDoS attack.

UseResilienceNestedRetry() solves this by marking the request.

var app = builder.Build();

app.UseResilienceDeadline();
app.UseResilienceNestedRetry();

app.MapGet("/", () => "Hello World");

When Service A makes a call, its handler adds a marker (e.g., X-NResilience-Retrying: 1) to the request - on the first attempt, not just on retries, because by the time a retry goes out the amplification has already happened. Service B reads this marker with UseResilienceNestedRetry(), which publishes the flag for the rest of the request. When Service B then makes its own outbound calls, its handler fires a NestedRetry event saying: "this request is already inside a retrying client."

The library reports but does not intervene: silently dropping configured retries would be surprising and hard to debug. What you do with the signal is yours - typically reducing the inner attempt count and shortening the inner deadline, so the layer closest to the failing dependency does the retrying and the layers above it pass through. That collapses the exponential growth of retries back into a linear flow.

Mapping Failures to the Wire

When a deadline expires or a retry budget is exhausted, the library throws a specific exception. You don't want these to result in a generic 500 Internal Server Error, which tells the caller nothing.

AddResilienceExceptionHandler() maps these internal exceptions to the correct HTTP responses:

  • Deadline Exceeded -> 504 Gateway Timeout
  • Breaker/Budget/Limiter Refusal -> 503 Service Unavailable (with a Retry-After header)
// In Program.cs
builder.Services.AddResilienceExceptionHandler();
builder.Services.AddProblemDetails(); // Required for the exception handler

var app = builder.Build();
app.UseExceptionHandler(); // The framework middleware that runs the handler

This tells the upstream caller exactly why the request failed. A 503 tells them to back off; a 504 tells them the system is too slow. This feedback loop is what allows the entire fleet to stabilize.

Conclusion

Resilience isn't something you configure in a vacuum; it's a conversation between services. By propagating deadlines and criticality and detecting nested retries, you stop your services from working on dead requests, keep a backfill from spending the capacity a checkout needs, and prevent your retry logic from becoming a weapon.

With three lines of middleware and one service registration, you move from "hoping for the best" to a system that actively coordinates its failure modes.

One more option on that middleware is worth knowing about. RejectExpired refuses a request that arrives with less time left than the Reserve says answering costs, with a 504 and a problem document, rather than running a handler whose every outbound call will fail. It is off by default and stays off: the request may well be answerable from cache, and refusing it is a decision about your service rather than about resilience.

The HTTP Deadline documentation details header configuration and nested-retry detection. For testing these propagation rules deterministically, Tests that don't flake shows how to script failures using Sequence<T> and FakeTimeProvider.