i added new feature to one of my blazor apps. This allows uploading files to blob storage in blocks. This works really nice. I had a look to process memory in visual studio. Where my app had all the time 80mb, with all its stuff. But when i start an upload, I see that the memory consumption increases by 10-20mb and it will not fall back. As well it increases with each upload of a file. Notice i upload files in blocks of 10mb, before i had 100mb and it increased 100-200mb. So now i am not really sure its memory leak (i guess it is) or the gc takes too long.

here my function from inside the service for reference

public async Task UploadFileToBlobStorage(Stream stream, string fileName, string containerName,string contentType,IProgress<long> progress,CancellationToken cancellationToken)
{
    _logger.LogInformation($"Uploading file {fileName} to container {containerName}");
    var blobServiceClient = new BlobServiceClient(new Uri($"https://{accountName}.blob.core.windows"), new DefaultAzureCredential());
    var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
    await containerClient.CreateIfNotExistsAsync(cancellationToken:cancellationToken);
    var blockblobClient = containerClient.GetBlockBlobClient(fileName);

    var blockIds = new List<string>();
    var buffer = new byte[10 * 1024 * 1024]; // 10 MB
    int bytesRead;
    int blockNumber = 0;
    long totalBytesRead = 0;
               

    var blobHttpHeader = new BlobHttpHeaders
    {
        ContentType = contentType,
        ContentDisposition = $"attachment; filename={fileName}",              

    };

    while ((bytesRead = await stream.ReadAsync(buffer.AsMemory( 0, buffer.Length), cancellationToken)) > 0)            
    {
        var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockNumber.ToString("d6")));
        blockIds.Add(blockId);

        using (var blockStream = new MemoryStream(buffer, 0, bytesRead))
        {
            await blockblobClient.StageBlockAsync(blockId, blockStream,cancellationToken: cancellationToken);
        }
        totalBytesRead += bytesRead;
        progress?.Report(totalBytesRead);
        blockNumber++;
    }

    await blockblobClient.CommitBlockListAsync(blockIds,httpHeaders:blobHttpHeader,cancellationToken:cancellationToken);            
    _logger.LogInformation($"File {fileName} uploaded to container {containerName} successfull");
}

The variable stream that is passed to the function is an ibrowserfile that was created with using

using var stream = _selectedFile.OpenReadStream(maxAllowedSize: long.MaxValue);

I would be happy to receive tips.

• Does anyone see a serious mistake?
• Maybe no mistake and my interpretation is wrong?

Thanks a lot

i added new feature to one of my blazor apps. This allows uploading files to blob storage in blocks. This works really nice. I had a look to process memory in visual studio. Where my app had all the time 80mb, with all its stuff. But when i start an upload, I see that the memory consumption increases by 10-20mb and it will not fall back. As well it increases with each upload of a file. Notice i upload files in blocks of 10mb, before i had 100mb and it increased 100-200mb. So now i am not really sure its memory leak (i guess it is) or the gc takes too long.

here my function from inside the service for reference

public async Task UploadFileToBlobStorage(Stream stream, string fileName, string containerName,string contentType,IProgress<long> progress,CancellationToken cancellationToken)
{
    _logger.LogInformation($"Uploading file {fileName} to container {containerName}");
    var blobServiceClient = new BlobServiceClient(new Uri($"https://{accountName}.blob.core.windows"), new DefaultAzureCredential());
    var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
    await containerClient.CreateIfNotExistsAsync(cancellationToken:cancellationToken);
    var blockblobClient = containerClient.GetBlockBlobClient(fileName);

    var blockIds = new List<string>();
    var buffer = new byte[10 * 1024 * 1024]; // 10 MB
    int bytesRead;
    int blockNumber = 0;
    long totalBytesRead = 0;
               

    var blobHttpHeader = new BlobHttpHeaders
    {
        ContentType = contentType,
        ContentDisposition = $"attachment; filename={fileName}",              

    };

    while ((bytesRead = await stream.ReadAsync(buffer.AsMemory( 0, buffer.Length), cancellationToken)) > 0)            
    {
        var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockNumber.ToString("d6")));
        blockIds.Add(blockId);

        using (var blockStream = new MemoryStream(buffer, 0, bytesRead))
        {
            await blockblobClient.StageBlockAsync(blockId, blockStream,cancellationToken: cancellationToken);
        }
        totalBytesRead += bytesRead;
        progress?.Report(totalBytesRead);
        blockNumber++;
    }

    await blockblobClient.CommitBlockListAsync(blockIds,httpHeaders:blobHttpHeader,cancellationToken:cancellationToken);            
    _logger.LogInformation($"File {fileName} uploaded to container {containerName} successfull");
}

The variable stream that is passed to the function is an ibrowserfile that was created with using

using var stream = _selectedFile.OpenReadStream(maxAllowedSize: long.MaxValue);

I would be happy to receive tips.

• Does anyone see a serious mistake?
• Maybe no mistake and my interpretation is wrong?

Thanks a lot

• c#
• memory-management
• azure-blob-storage

• An increase in memory does not necessarily imply a memory leak. You will only have an accurate value just after a full GC, and there can be caches that makes measurements more difficult. The solution is to use a memory profiler that has the appropriate tools to identify if there is a problem, and tools to identify what the problem is. – JonasH Commented 2 days ago
• @JonasH which profiler do you recommend? I could use dotnet-counters / dotnet-dump tools – RoXTar Commented 2 days ago
• 1 The code allocates a huge 10MB buffer on each upload. That's not good and doesn't offer any benefit. It actually adds delays, as 10MBs have to be received first before they're sent. Large objects, ie those larger than 85KB, are collected less frequently than others too. That's what causes the very fast increase in RAM usage. – Panagiotis Kanavos Commented 2 days ago
• @PanagiotisKanavos can you share an improved version of mine. Thanks a lot – RoXTar Commented 2 days ago
• 1 I have mostly used dotMemory, but this is largely due to inertia. I think visual studio includes the most important tools now days, like heap snapshots and diffing snapshots. Third party tools may provide a better UI, at a cost. – JonasH Commented 2 days ago

2 Answers 2

At the first glance I have not found clear leak, but there is one thing you need to consider - you are creating 10 MB buffer (var buffer = new byte[10 * 1024 * 1024]; // 10 MB) for every upload, which is greater than default Large Object Heap (LOH) size:

If an object is greater than or equal to 85,000 bytes in size, it's considered a large object. This number was determined by performance tuning. When an object allocation request is for 85,000 or more bytes, the runtime allocates it on the large object heap.

So every such array will directly go to the LOH which is collected only with 2nd generation and not compacted by default:

Since the LOH is only collected during generation 2 GCs, the LOH segment can only be freed during such a GC

and

But because compaction is expensive, the GC sweeps the LOH; it makes a free list out of dead objects that can be reused later to satisfy large object allocation requests. Adjacent dead objects are made into one free object. ... Because the LOH is not compacted, sometimes the LOH is thought to be the source of fragmentation

So even if the memory is actually not needed (i.e. no references are maintained to it) then it potentially still will not be collected/freed for quite some time. Highly likely your application is running in server GC mode which can be quite greedy/"lazy" and in some cases can rarely perform the 2nd gen collection.

You can try manually triggering the collection and compaction:

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, blocking: true, compacting: true);

But that is not the recommended approach in general case (though you can use it if you know what you are doing or for testing for the memory leak - to see if the memory is actually claimed by the GC).

The first question you should ask is that if such big buffer is actually needed and try using buffer which will fit the Small Object Heap (SOH) i.e. less then 85,000 bytes (I would suggest much less). If for some reason that is not an approach you can use - you can consider using the pooling approach (i.e. ArrayPool<T>), but it still can allocate depending on the number of concurrent uploads. Finally you can play with the GC settings, especially the Large object heap threshold, but I would suggest to go with the first option still (i.e. use smaller buffer).

Moving from comments:

I added the writeable: false flag. That's the key is great for serving memory.

for follow up readers. We ended up with the working code above, but one enhancement. Code works like a charm for a lot of users uploading bigfiles.

public async ValueTask UploadFileToBlobStorage(Stream stream, string fileName, string containerName,string contentType,IProgress<long> progress,CancellationToken cancellationToken)
{
    _logger.LogInformation($"Uploading file {fileName} to container {containerName}");
    var blobServiceClient = new BlobServiceClient(new Uri($"https://{accountName}.blob.core.windows"), new DefaultAzureCredential());
    var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
    await containerClient.CreateIfNotExistsAsync(cancellationToken:cancellationToken);
    var blockblobClient = containerClient.GetBlockBlobClient(fileName);

    var blockIds = new List<string>();
    var buffer = new byte[10 * 1024 * 1024]; // 10 MB
    
    int bytesRead;
    int blockNumber = 0;
    long totalBytesRead = 0;
               

    var blobHttpHeader = new BlobHttpHeaders
    {
        ContentType = contentType,
        ContentDisposition = $"attachment; filename={fileName}",              

    };

    while ((bytesRead = await stream.ReadAsync(buffer.AsMemory( 0, buffer.Length), cancellationToken)) > 0)            
    {
        var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockNumber.ToString("d6")));
        blockIds.Add(blockId);

        using (var blockStream = new MemoryStream(buffer, 0, bytesRead,writable:false))
        {
            await blockblobClient.StageBlockAsync(blockId, blockStream,cancellationToken: cancellationToken);
        }
        totalBytesRead += bytesRead;
        progress?.Report(totalBytesRead);
        blockNumber++;
    }

    await blockblobClient.CommitBlockListAsync(blockIds,httpHeaders:blobHttpHeader,cancellationToken:cancellationToken);
    _logger.LogInformation($"File {fileName} uploaded to container {containerName} successfull");
}

Note the "writable:false" on memory stream.

Have a great time