6 min readThe rules of the call

Two clocks

Understand the difference between a call Deadline and an AttemptTimeout to avoid the classic "30s x 3 retries" bug.

timeoutsdeadlines

TL;DR: A resilient call needs two separate time bounds. The Deadline caps the entire operation (including all retries), while the AttemptTimeout caps a single request. Confusing the two is the primary cause of requests that hang for minutes when they should have failed in seconds.

The 90-second bug

Imagine you have a dependency that occasionally hangs. To protect your app, you set a timeout of 30 seconds and allow 3 retry attempts.

If you only have one timeout setting, you've likely created the "90-second bug."

When the dependency hangs, the first attempt waits 30 seconds and fails. The second attempt waits 30 seconds and fails. The third attempt waits 30 seconds and fails. Your user, who was expecting a fast failure or a success, has now been staring at a loading spinner for a minute and a half.

In a high-traffic system, this is catastrophic. Those hanging requests tie up threads, exhaust connection pools, and eventually take down your entire service.

The two bounds

NResilience solves this by separating the "overall" clock from the "per-attempt" clock.

1. The Deadline (The Outer Clock)

The Deadline is the absolute ceiling on getting an answer. If you set a deadline of 30 seconds, the call will end in 30 seconds, regardless of whether it's on the first attempt or the fifth. It is the promise you make to the caller.

For HTTP, "an answer" means the response headers and the status code - the things the policy classifies. Reading the body happens afterwards and is bounded separately; see the third clock below.

2. The AttemptTimeout (The Inner Clock)

The AttemptTimeout caps a single request. If you set this to 2 seconds, any single attempt that takes longer than 2 seconds is cancelled and treated as a failure. This allows the executor to move on to the next retry quickly.

var policy = Resilience.Default with 
{ 
    Deadline = TimeSpan.FromSeconds(30),        // Entire operation max 30s
    AttemptTimeout = TimeSpan.FromSeconds(2)    // Each request max 2s
};

With this configuration, if a dependency hangs, you'll hit three 2-second timeouts and fail the whole call in about 6 seconds - not 90.

How they compose

The executor doesn't just pick one; it calculates the "actual" timeout for every attempt. The ceiling for a single attempt is always the smallest of these three values:

  1. The configured AttemptTimeout.
  2. The remaining time left on the Deadline.
  3. The measured term (a dynamic bound based on recent latency, which we'll cover in the "measured, not guessed" post).

If your Deadline has 1 second left, but your AttemptTimeout is 2 seconds, the attempt is capped at 1 second. The outer clock always wins.

The inner clock has a second job

There is a third thing worth knowing about, and the good news is that it is not a third number. AttemptTimeout also caps the gap between two reads of a response body, and the gap between two elements of a stream.

That gap needed a bound because it had none. An HTTP attempt ends when the response headers arrive - the body is a live stream you read after the policy has classified the attempt and moved on. Neither clock covered it, and neither did HttpClient.Timeout, which the resilience handler sets to infinite so the deadline can be the single bound. A server that sent headers and then stopped writing gave you a call that never finished.

BoundProgress is on by default and fixes that with the number you already set. A read that has been outstanding for longer than the AttemptTimeout fails with an AttemptStalledException:

var policy = Resilience.Http with
{
    AttemptTimeout = TimeSpan.FromSeconds(2)  // one request, and one read of its body
};

Two things about it are deliberate. It bounds the gap, not the total: a 4 GB download that keeps arriving is never cut off for taking a while, because a read that is outstanding is waiting on the far side and a consumer that is slow between reads is not. And it makes the hang finite rather than retryable - the retry loop was over before the stall existed, so the exception ends your read rather than the call. If you would rather have it retried, BufferResponses on the HTTP options reads the body inside the attempt, where both clocks already apply, in exchange for holding the whole body in memory.

Choosing your values

When setting these, ask yourself two different questions:

For the Deadline: "How long is the user/caller willing to wait before this is a total failure?" If the API is for a UI screen, this might be 5-10 seconds. If it's a background job, it might be minutes.

For the AttemptTimeout: "At what point is a single request almost certainly a zombie?" If your p99 latency is 200ms, a 2-second timeout is generous. Setting this too high defeats the purpose of retries; setting it too low creates "false" failures during minor spikes.

Common Pitfalls

The "Unreachable" Timeout

If you set an AttemptTimeout of 60 seconds but a Deadline of 30 seconds, the 60-second bound is unreachable. The call will always be killed by the deadline first. NResilience ships an analyzer (NRES004) that flags this at compile time.

Doing the arithmetic in your head

Three attempts of ten seconds, two backoffs, and a thirty-second deadline: which bound actually ends the call, and what is the last attempt left with? That is a function of the attempt count, both bounds, the backoff curve and its jitter, and it is not a calculation worth doing by hand.

Ask the policy instead:

Console.WriteLine(api.Explain());
  Bound first by:  the deadline. Attempt 3 is clamped to 9.70s, 97% of the 10s attempt timeout.

  Worst case:      attempt 1  0.00s -> 10.00s   min(10s attempt timeout, 30.00s left)
                   backoff    +0ms -> 100ms     exponential x2 from 100ms, full jitter, capped at 30s
                   attempt 2  10.10s -> 20.10s  min(10s attempt timeout, 19.90s left)
                   backoff    +0ms -> 200ms
                   attempt 3  20.30s -> 30.00s  min(10s attempt timeout, 9.70s left)
                   deadline   30.00s            DeadlineExceededException

Those are the shipped defaults, and the answer is that the third attempt gets 9.7 of its 10 seconds - so the deadline binds, but only just. Change any one of the five numbers and the sentence at the top changes with it.

The "Infinite" Deadline

Setting Deadline = Timeout.InfiniteTimeSpan removes the wall-clock ceiling on the whole call. It is a legitimate configuration - long-running operations need it - but then the only bounds left are the AttemptTimeout (or the measured ceiling) per attempt and however many attempts you allow. Be deliberate about it: without a deadline, a flaky dependency stretches your worst-case call duration to roughly AttemptTimeout x Attempts, and Explain() states this in the first line, so you don't have to derive it from two absent numbers.

Summary

Stop thinking about "the timeout." Start thinking about "the deadline" and "the attempt."

  • Deadline: The total budget for getting an answer.
  • AttemptTimeout: The budget for one request - and for one read of the body that request returned.

By separating the two, you ensure that your app fails fast when it needs to, and that a few hanging requests don't turn into a full-scale outage.

The two tokens in RunAsync govern these clocks, and passing the wrong one to your dependency is a silent bug. Who cancels what explains the distinction between the caller's token and the attempt token.