In my .NET web app I have a GRPC endpoint:

rpc GetProductDetails(ProductDetailsRequest) returns (stream ProductDetailsResponse);

This is how the endpoint signature looks like on the service level:

 public override async Task GetProductDetails(ProductDetailsRequest request, 
IServerStreamWriter<ProductDetailsResponse> responseStream,
ServerCallContext context)

...which internally calls a method of a class returning an IAsyncEnumerable that streams an ICollection of ProductDetail per provided product:

public async IAsyncEnumerable<ICollection<ProductDetail>> GetProductDetailsAsync()
{
    // some code
    foreach(var product in products)
    {
         // some validation checks etc 
         var productDetails = await Task.Run(() => GetProductDetails(product));
         yield return productDetails;
    }
}

GetProductDetails is a synchronous call to a third-party API and there is nothing we can do about it.

There is a misunderstanding on the await Task.Run() part: why do we need await Task.Run altogether? Aren't both the thread servicing the request and the thread (potentially) picking up the work in await Task.Run() thread pool threads? Doesn't that mean that we are letting a thread pool thread go do something else by making another thread pool thread busy?

i.e. why not this?

public IEnumerable<ICollection<ProductDetail>> GetProductDetailsSync()
{
    // some code
    foreach(var product in products)
    {
         // some validation checks etc 
         var productDetails = GetProductDetails(product);
         yield return productDetails;
    }
}

In my .NET web app I have a GRPC endpoint:

rpc GetProductDetails(ProductDetailsRequest) returns (stream ProductDetailsResponse);

This is how the endpoint signature looks like on the service level:

 public override async Task GetProductDetails(ProductDetailsRequest request, 
IServerStreamWriter<ProductDetailsResponse> responseStream,
ServerCallContext context)

...which internally calls a method of a class returning an IAsyncEnumerable that streams an ICollection of ProductDetail per provided product:

public async IAsyncEnumerable<ICollection<ProductDetail>> GetProductDetailsAsync()
{
    // some code
    foreach(var product in products)
    {
         // some validation checks etc 
         var productDetails = await Task.Run(() => GetProductDetails(product));
         yield return productDetails;
    }
}

GetProductDetails is a synchronous call to a third-party API and there is nothing we can do about it.

There is a misunderstanding on the await Task.Run() part: why do we need await Task.Run altogether? Aren't both the thread servicing the request and the thread (potentially) picking up the work in await Task.Run() thread pool threads? Doesn't that mean that we are letting a thread pool thread go do something else by making another thread pool thread busy?

i.e. why not this?

public IEnumerable<ICollection<ProductDetail>> GetProductDetailsSync()
{
    // some code
    foreach(var product in products)
    {
         // some validation checks etc 
         var productDetails = GetProductDetails(product);
         yield return productDetails;
    }
}

• c#
• asp-core
• async-await
• grpc

• 2 Yeah, Task.Run is usually "I'm busy doing work, find another thread that can run this code" and await is "I've got no useful work to do until this task is complete". It's rare that combining the two together like this makes sense. – Damien_The_Unbeliever Commented Mar 24 at 13:48
• 1 Do you see a warning like here (no warning)? Without Task.Run you are running everything synchronously. No idea how bad it is in asp, but blocking UI would be bad idea. – Sinatr Commented Mar 24 at 13:54
• 1 @Damien_The_Unbeliever, await Task.Run is common when mixing asynchronous and synchronous code. – Sinatr Commented Mar 24 at 13:58
• 2 "Aren't both the thread servicing the request and the thread (potentially) picking up the work in ... thread pool threads" -- Maybe. The thing is, I don't know whether gRPC uses the .NET thread pool to service incoming requests, or whether it has a different mechanism. I rather suspect that's an implementation detail which means it could change. Give that, I would be wary about relying on any one implementation. Using await Task.Run is obviously correct and safe, and doesn't rely on knowledge of how gRPC handles threads with incoming requests – canton7 Commented Mar 24 at 14:05
• 1 For example, gRPC provide the following guidance for C++: "Favor callback API over other APIs for most RPCs, given that the application can avoid all blocking operations or blocking operations can be moved to a separate thread.". So for C++ at least, they'd rather you blocked one of your own threads rather than one of gRPC's threads. – canton7 Commented Mar 24 at 14:07

1 Answer 1

I'm struggling to find much up-to-date information on exactly how the C# gRPC server threading works.

According to this issue it was possible to deadlock the server (at least back in 2016):

There is a thread pool that is used for handling event continuations. All the continuations for your awaits are scheduled onto that threadpool. Once you are in a method after you've awaited an RPC (GetNameAsync in your example), your code is running on that threadpool. If you then invoke the sync version of an RPC (GetName in your example), you prevent gRPC from scheduling continuations on that thread. If the server happens to need schedule continuations on the same thread to handle that call, you'll see a deadlock, because that thread is blocked by your GetName() invocation.

You might started seeing this problem after (#6712) was merged. The PR effectively limits number of threads on which given call can schedule continuations (which decreases latency, but also requires the user to use async/await correctly without mixing).

And also this issue:

it is not only the servers that complete their async tasks with a dedicated GRPC thread pool, but also client GRPC calls as well. So basically the same problems with long-running CPU work and blocking calls in server handlers also exist after making a client call. The client call is also completed on one of the GRPC threads, and so when returning from an async client call, blocking or doing long-running CPU work would prevent progress from things that the thread is able to do.

...

Only async methods are supported on the server side. When implementing them, you either need to implement the async code according to the .NET best practices (e.g. don't block on the threads and use async primitives instead - e.g. use Task.Delay instead of Thread.Sleep), or you can offload the sync operations to a different thread (e.g. using Task.Run and awaiting the result).

That was back in 2016 however, and it looks like things have changed a little since then.

There's the static function GrpcEnvironmint.SetHandlerInlining(Boolean) which:

By default, gRPC's internal event handlers get offloaded to .NET default thread pool thread (inlineHandlers=false). Setting nlineHandlers to true will allow scheduling the event handlers directly to GrpcThreadPool internal threads. That can lead to significant performance gains in some situations, but requires user to never block in async code (incorrectly written code can easily lead to deadlocks). Inlining handlers is an advanced setting and you should only use it if you know what you are doing. Most users should rely on the default value provided by gRPC library. Note: this method is part of an experimental API that can change or be removed without any prior notice. Note: inlineHandlers=true was the default in gRPC C# v1.4.x and earlier.

gRPC 1.4 was released after the issues linked above, so it looks like people might have run into problems deadlocking by blocking in handlers, and they had to change the default back to what it was.

All in all, to get back to your question:

Aren't both the thread servicing the request and the thread (potentially) picking up the work in await Task.Run() thread pool threads?

The answer is "Maybe, it depends". It sounds like you're probably OK right now as long as inlineHandlers=false, but that means you're dependent on:

1. gRPC not changing the default again in the future.
2. Someone else not changing this value in your codebase in the future.
3. The understanding I've managed to cobble together above being correct.

Such code is fragile: it relies on things like implementation details, and could break at any point in the future.

There's also the issue that doing a blocking call in an async handler is going to be a huge red flag to anyone reading your code (including future you): they'll think that they've discovered a bug.

Weigh that up against the cost of an await Task.Run, which is cheap: much cheaper than all of the expensive networking stuff happening with the rest of the call, or the blocking GetProductDetails call itself.

So, I'd stick with the await Task.Run: it's obviously correct to anyone reading the code, it's not fragile and should continue to work in the future, and it's barely adding any cost.