performance - JsonSerializer.DeserializeAsync Fails Under High Load in ASP.NET Core Application

I'm encountering an issue in my ASP.NET Core application where System.Text.Json.JsonSerializer.DeserializeAsync fails under high load. When multiple requests are made simultaneously, the application throws an error, and responses slow down significantly.

Error Message

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.Text.Json.JsonException: The JSON value could not be converted to System.Collections.Generic.List`1[ServiceContract.Models.City]. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
         at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)
         at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
         at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
         at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.ContinueDeserialize(ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.DeserializeAsync(Stream utf8Json, CancellationToken cancellationToken)
         at CitiesService.CovidService.GetCities() in C:\Users\ahmed\RiderProjects\CitiesService\CovidService.cs:line 28
         at HttpClientExample.Controllers.HomeController.Home() in C:\Users\ahmed\RiderProjects\HttpClientExample\Controllers\HomeController.cs:line 19
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execut
e(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g_
_Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awai
ted|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)     
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)       
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Solution Structure

ClientService
  - CovidService.cs
  - CovidServiceSettings.cs
HttpClientExample (Main Project)
ServiceContract
  - Models
    - City.cs
  - ICovidService.cs

Code

CovidService.cs

using System.Text.Json;
using ServiceContract.Models;
using ServiceContract;

namespace CitiesService;

public class CovidService : ICovidService
{
    private readonly HttpClient _httpClient;
    private readonly CovidServiceSettings _covidServiceSettings;

    public CovidService(HttpClient httpClient, CovidServiceSettings covidServiceSettings)
    {
        _httpClient = httpClient;
        _covidServiceSettings = covidServiceSettings;
    }
    
    public async Task<List<City>> GetCities()
    {
        HttpRequestMessage requestMessage = new()
        {
            RequestUri = new Uri(_covidServiceSettings.Url),
            Method = HttpMethod.Get
        };

        HttpResponseMessage responseMessage = await _httpClient.SendAsync(requestMessage);
        Stream stream = await responseMessage.Content.ReadAsStreamAsync();
        List<City> cities = await JsonSerializer.DeserializeAsync<List<City>>(stream);
        return cities;
    }
}

Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllersWithViews();
builder.Services.AddHttpClient();
builder.Services.AddScoped<ICovidService, CovidService>(serviceProvider =>
{
    var httpClientFactory = serviceProvider.GetService<IHttpClientFactory>();
    var configuration = serviceProvider.GetService<IConfiguration>();

    return new CovidService(httpClientFactory!.CreateClient("CovidService"),
        configuration.GetSection("CovidSettings").Get<CovidServiceSettings>());
});

var app = builder.Build();
app.UseStaticFiles();
app.UseRouting();
app.MapControllers();
app.Run();

HomeController.cs

using Microsoft.AspNetCore.Mvc;
using ServiceContract;

namespace HttpClientExample.Controllers;

public class HomeController : Controller
{
    private readonly ICovidService _covidService;

    public HomeController(ICovidService covidService)
    {
        _covidService = covidService;
    }

    [Route("/")]
    public async Task<IActionResult> Home()
    {
        var cities = await _covidService.GetCities();
        return View(cities);
    }
}

Increased Thread Pool Limits: Adjusted the thread pool using ThreadPool.SetMinThreads, which slightly improved performance under high load. The application successfully handled the first 20–30 requests but quickly degraded after that. Occasionally, one or two additional requests were handled correctly during the high load, but the bottleneck persisted overall.

I expected JsonSerializer.DeserializeAsync to handle high-concurrency requests without significant performance degradation or failures.

Error Message

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.Text.Json.JsonException: The JSON value could not be converted to System.Collections.Generic.List`1[ServiceContract.Models.City]. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
         at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)
         at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
         at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
         at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.ContinueDeserialize(ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.DeserializeAsync(Stream utf8Json, CancellationToken cancellationToken)
         at CitiesService.CovidService.GetCities() in C:\Users\ahmed\RiderProjects\CitiesService\CovidService.cs:line 28
         at HttpClientExample.Controllers.HomeController.Home() in C:\Users\ahmed\RiderProjects\HttpClientExample\Controllers\HomeController.cs:line 19
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execut
e(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g_
_Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awai
ted|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)     
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)       
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Solution Structure

ClientService
  - CovidService.cs
  - CovidServiceSettings.cs
HttpClientExample (Main Project)
ServiceContract
  - Models
    - City.cs
  - ICovidService.cs

Code

CovidService.cs

using System.Text.Json;
using ServiceContract.Models;
using ServiceContract;

namespace CitiesService;

public class CovidService : ICovidService
{
    private readonly HttpClient _httpClient;
    private readonly CovidServiceSettings _covidServiceSettings;

    public CovidService(HttpClient httpClient, CovidServiceSettings covidServiceSettings)
    {
        _httpClient = httpClient;
        _covidServiceSettings = covidServiceSettings;
    }
    
    public async Task<List<City>> GetCities()
    {
        HttpRequestMessage requestMessage = new()
        {
            RequestUri = new Uri(_covidServiceSettings.Url),
            Method = HttpMethod.Get
        };

        HttpResponseMessage responseMessage = await _httpClient.SendAsync(requestMessage);
        Stream stream = await responseMessage.Content.ReadAsStreamAsync();
        List<City> cities = await JsonSerializer.DeserializeAsync<List<City>>(stream);
        return cities;
    }
}

Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllersWithViews();
builder.Services.AddHttpClient();
builder.Services.AddScoped<ICovidService, CovidService>(serviceProvider =>
{
    var httpClientFactory = serviceProvider.GetService<IHttpClientFactory>();
    var configuration = serviceProvider.GetService<IConfiguration>();

    return new CovidService(httpClientFactory!.CreateClient("CovidService"),
        configuration.GetSection("CovidSettings").Get<CovidServiceSettings>());
});

var app = builder.Build();
app.UseStaticFiles();
app.UseRouting();
app.MapControllers();
app.Run();

HomeController.cs

using Microsoft.AspNetCore.Mvc;
using ServiceContract;

namespace HttpClientExample.Controllers;

public class HomeController : Controller
{
    private readonly ICovidService _covidService;

    public HomeController(ICovidService covidService)
    {
        _covidService = covidService;
    }

    [Route("/")]
    public async Task<IActionResult> Home()
    {
        var cities = await _covidService.GetCities();
        return View(cities);
    }
}

I expected JsonSerializer.DeserializeAsync to handle high-concurrency requests without significant performance degradation or failures.

Share Improve this question asked Nov 18, 2024 at 10:14 Ahmed Fotoh 111 silver badge2 bronze badges

could try setting the threadpool to ThreadPool.SetMinThreads(100, 100); try to set "MaxConcurrentConnections": 100,"MaxConcurrentUpgradedConnections": 100 is it possible for you to shar ethe reproducible sample with us? are you testing the application locally or on the server after hosting? you could adding logging code to see if any particular type of request is causing this issue or not. You could use the tool like procdump or debug diagnostic to collect the logs and analyze it – Jalpa Panchal Commented Nov 18, 2024 at 13:04
It does not look like that the Deserialize call fails by itself, but rather that it gets invalid input (at least not a JSON structure that matches List<City>). Maybe the service you call (via _httpClient.SendAsync) returns garbage when under heavy load? You could (a) log/write the JSON that fails to deserialize and/or (b) replace the actual call (SendAsync) with a hard coded List<City> instance "in memory" to further narrow this down. – Christian.K Commented Nov 18, 2024 at 14:34
Thank you, @Christian.K , @JalpaPanchal for your insights! After logging the JSON responses during high load, I realized the issue was caused by an error message from the API: {"error":"API limit reached. Please try again later. Remaining Limit: 0"} This response couldn’t be deserialized into the List<City> model. I replaced the SendAsync call with hardcoded JSON data, and the deserialization worked perfectly, confirming the issue was with the API response under high load. I appreciate your guidance in resolving this! – Ahmed Fotoh Commented Nov 18, 2024 at 23:21

Add a comment |

1 Answer 1

Sorted by: Reset to default 1

During high load, the API returned an error response:

{"error":"API limit reached. Please try again later. Remaining Limit: 0"}

I hadn't accounted for these kinds of error responses in my deserialization logic, so the error message couldn’t be deserialized into my List<City> model.

When I replaced the API call with hardcoded JSON data, the deserialization worked fine, confirming that the issue was due to missing error handling for unexpected responses.

科技改变生活-雨落星辰 - 所有的伟大,都源于一个勇敢的开始

performance - JsonSerializer.DeserializeAsync Fails Under High Load in ASP.NET Core Application - Stack Overflow

1 Answer 1

与本文相关的文章

评论列表(0)