I need to upload a ZIP stream from the browser's network to backend. The stream can reach hundreds of megabytes in size. Current approach involves uploading the stream to isolated storage for scanning, and only if it's clean, it proceeds with the regular flow. Since the stream can only be read once, I considered two approaches:

1. Copy the stream in memory: This is not ideal due to the potential for high memory usage and the risk of DoS attacks, especially with large files.

2. Read from isolated storage after uploading: This involves writing the stream to isolated storage and then reading it from there to avoid keeping it in memory. However, I am concerned that even if I write to blob storage and then read from it, the reading will still involve memory usage. Also, it is not performant.

Are there any better approaches to overcome this problem while minimizing memory usage and ensuring security?

I need to upload a ZIP stream from the browser's network to backend. The stream can reach hundreds of megabytes in size. Current approach involves uploading the stream to isolated storage for scanning, and only if it's clean, it proceeds with the regular flow. Since the stream can only be read once, I considered two approaches:

1. Copy the stream in memory: This is not ideal due to the potential for high memory usage and the risk of DoS attacks, especially with large files.

2. Read from isolated storage after uploading: This involves writing the stream to isolated storage and then reading it from there to avoid keeping it in memory. However, I am concerned that even if I write to blob storage and then read from it, the reading will still involve memory usage. Also, it is not performant.

Are there any better approaches to overcome this problem while minimizing memory usage and ensuring security?

• memory
• stream
• zip
• blob
• azure-storage

• 1 Not sure if this works for your application, but zipflow streams out a zip file with only a tiny amount of memory needed proportional to the number of files, not their size. – Mark Adler Commented 21 hours ago
• Generate a SAS token from the backend for the Azure Blob location. Upload the file to there from the browser. Then separately, have another process (e.g. Blob Trigger Azure Function) that opens the Blob as a stream for reading using the approach from @MarkAdler – Andrew B Commented 19 hours ago

1 Answer 1

Are there any better approaches to overcome this problem while minimizing memory usage and ensuring security?

I agree with Mark Adler's and Andrew's comment,Using the ZipFlow approach alongside a SAS token to upload directly to Azure Blob Storage can be an effective solution to minimize memory usage while streaming the file from the browser.

You can create Azure storage account SAS token using below code.

Code:

 public static string CreateAccountSasToken(string accountName, string accountKey)
    {
        var sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey);
        // Create Account SAS
        AccountSasBuilder sasBuilder = new AccountSasBuilder
        {
            ExpiresOn = DateTimeOffset.UtcNow.AddHours(1), // Set expiration time
            Services = AccountSasServices.Queues,         // Enable queue service
            ResourceTypes = AccountSasResourceTypes.Service | AccountSasResourceTypes.Container | AccountSasResourceTypes.Object, 
        };

        sasBuilder.SetPermissions(AccountSasPermissions.Add | AccountSasPermissions.Create | AccountSasPermissions.Read | AccountSasPermissions.Write);
        string sasToken = sasBuilder.ToSasQueryParameters(sharedKeyCredential).ToString();
        return sasToken;
    }

Now you can use JavaScript to upload the file directly to Azure Blob Storage using the SAS token.

• You can use a Blob Trigger Azure Function to scan files automatically after they are uploaded. This allows you to process the file as soon as it is available in Azure Blob Storage.
• The function can read the file as a stream and then apply the ZipFlow method to scan the file without needing to load it entirely into memory.

Reference:

• Grant limited access to data with shared access signatures (SAS) - Azure Storage | Microsoft Learn
• Quickstart: Azure Blob storage library v12 - JS Browser - Azure Storage | Microsoft Learn
• Node.js developer reference for Azure Functions | Microsoft Learn