5 min readPer stack

gRPC in depth

Explore how NResilience handles gRPC status classification, per-service scoping, and the critical math behind wire deadlines.

grpcdeadlines

TL;DR: gRPC requires a different approach to resilience than REST. This post covers how NResilience maps RPC status codes to verdicts, implements per-service circuit breakers, manages grpc-timeout with a deliberate slack, and handles the unique constraints of server streaming.

The Problem: gRPC isn't just HTTP

If you look at a network trace, every gRPC call is an HTTP/2 POST. But at the application level, the semantics are entirely different. You don't have status codes like 404 or 503 in the response header; you have a trailing grpc-status.

This creates several technical challenges:

  1. Blast Radius: A single gRPC client often connects to a server that hosts ten different services. If the "Payments" service is down, you don't want to trip the breaker for the "Catalog" service.
  2. The Timer Race: The gRPC stack has its own internal deadline management. If you set a timeout in the library and a timeout on the wire to the exact same millisecond, you have a race condition. You don't know who won, and your telemetry becomes a lie.
  3. Streaming State: Unlike a unary call, a stream is a long-lived conversation. If a stream fails after the client has already received five messages, you can't just "retry" the call - you'd be asking the server to start the whole stream from the beginning.

The Solution: gRPC-Aware Resilience

NResilience integrates directly into the gRPC channel via a ResilienceInterceptor, allowing it to understand the specific needs of RPCs.

1. Per-Service Scoping

By default, NResilience does not use a global breaker for the entire client. Instead, it scopes guards by the gRPC Service.

If you are calling OrdersService and InventoryService on the same host, they each get their own circuit breaker and retry budget. This prevents a failure in one domain from taking down your entire integration.

// Registered on the builder that AddGrpcClient<T>() returns
services.AddGrpcClient<Orders.OrdersClient>(o => o.Address = new Uri("https://orders.internal:5001"))
    .AddGrpcResilience();

// Default behavior:
// ScopeBy = static m => m.ServiceName;

You can tighten this to per-method scoping if you have a specific RPC that is notoriously flaky and shouldn't affect other methods in the same service.

2. Idempotency and the "Single Shot"

In gRPC, calls are considered repeatable by default. While this is usually true for reads, it is dangerous for mutations (like ChargeCard).

You can define which methods are repeatable in the options, or use GrpcResilience.SingleShot() for call-site overrides. This creates an ambient scope that disables retries for any call made within that block, regardless of the policy.

// 1. Policy-level: mark a specific method as non-repeatable
services.AddGrpcClient<Orders.OrdersClient>(o => o.Address = new Uri("https://orders.internal:5001"))
    .AddGrpcResilience(options => {
        options.RepeatableWhen = static m => m.Name != "ChargeCard";
    });

// 2. Call-site: force a single attempt
using (GrpcResilience.SingleShot())
{
    await client.ChargeCardAsync(request); // Will not be retried
}

3. The Math of Wire Deadlines

One of the most critical parts of the gRPC integration is how NResilience handles grpc-timeout.

The library calculates the wire deadline as: min(AttemptTimeout, remaining overall deadline) + DeadlineSlack

The DeadlineSlack (defaulting to 50ms) is a deliberate design choice. By giving the wire deadline a small buffer, NResilience ensures that its own internal timer expires first. This allows the library to throw an AttemptTimeoutException and correctly track the attempt as a failure, rather than receiving a generic StatusCode.DeadlineExceeded from the gRPC stack. It ensures the library, not the transport, remains the authoritative source of truth.

4. Handling Server Streams

Retrying a stream is tricky. NResilience supports server streaming with a specific rule: retries are only attempted until the first message is received.

Once the consumer has received the first element of the IAsyncEnumerable, the call is considered "started." Any failure after that point is propagated directly to the consumer. This prevents the library from accidentally triggering a full stream restart, which would likely lead to duplicate data or missing messages. For a stream, the wire deadline is the whole call's remaining budget rather than one attempt's ceiling. Client-streaming and duplex calls pass through untouched - wrapping one means re-enumerating something the failed attempt already partially consumed, which is not a resilience feature.

5. Why No Blocking Calls?

You will notice that BlockingUnaryCall is not supported and will throw a NotSupportedException.

NResilience is built as an async-first library to maximize throughput and minimize allocations. A blocking call that silently ran without the policy would be the one call in the client with no retry, no breaker, and no deadline - a worse outcome than a visible failure at the one call site that uses it. The same reasoning is why a synchronous Send throws in the HTTP handler. Use the generated client's Async overloads.

Trade-offs and Limitations

The per-service scoping adds a small amount of overhead to the interceptor to resolve the service name for every call. While negligible for most, in extremely high-throughput environments, this is the cost of preventing a total client brownout.

The "first message" rule for streams means that if your stream is designed to be resumable (using offsets or tokens), NResilience's built-in retry won't help you after the first message. You would need to implement resumption logic at the application level.

Conclusion

gRPC resilience isn't about wrapping a call in a loop; it's about managing the contract between the interceptor and the wire. By scoping breakers to services and carefully timing the grpc-timeout, NResilience provides protection that respects the semantics of RPCs.

The Deadlines documentation details the wire format and DeadlineSlack. For the engine underneath both the HTTP and gRPC integrations, One loop, one allocation explains the flat executor and how it keeps a single async frame regardless of policy complexity.