I'm currently working on the task to do a HTTP GET request and download a tar-file from a bunch of files stored in SQL Server (Filestream) varbinary(max) columns. So construct a tar-file and write the varbinary data to that tar-file.

Can this be achieved with .NET native API:s, like TarWriter/TarEntry? I'm currently on .NET8. Or do I need to use third-party libs such as SharpZipLib?

Pseudo code:

    // Create a MemoryStream to store the .tar file in memory
    using var tarStream = new MemoryStream();
    await using var tarWriter = new TarWriter(tarStream);
        
    // Open a connection to SQL Server
    await using var connection = new SqlConnection("YourConnectionString");
    await connection.OpenAsync();
    var command = new SqlCommand("SELECT FileName, FileData FROM FilesTable", connection);

    await using var reader = command.ExecuteReader();
                
    while (reader.Read())
    {                            
       var fileName = reader.GetString(0);
       var fileDataStream = reader.GetStream(1);
       
       // How to construct TarEntry from a stream?
       tarWriter.WriteEntry(fileName, fileDataStream);
    }

    // Then write the tarStream to response stream.
   

I'm currently working on the task to do a HTTP GET request and download a tar-file from a bunch of files stored in SQL Server (Filestream) varbinary(max) columns. So construct a tar-file and write the varbinary data to that tar-file.

Can this be achieved with .NET native API:s, like TarWriter/TarEntry? I'm currently on .NET8. Or do I need to use third-party libs such as SharpZipLib?

Pseudo code:

    // Create a MemoryStream to store the .tar file in memory
    using var tarStream = new MemoryStream();
    await using var tarWriter = new TarWriter(tarStream);
        
    // Open a connection to SQL Server
    await using var connection = new SqlConnection("YourConnectionString");
    await connection.OpenAsync();
    var command = new SqlCommand("SELECT FileName, FileData FROM FilesTable", connection);

    await using var reader = command.ExecuteReader();
                
    while (reader.Read())
    {                            
       var fileName = reader.GetString(0);
       var fileDataStream = reader.GetStream(1);
       
       // How to construct TarEntry from a stream?
       tarWriter.WriteEntry(fileName, fileDataStream);
    }

    // Then write the tarStream to response stream.
   

• c#
• sql-server
• filestream
• tar

• What would be the contents of that TAR file? A CSV text file? An Excel file? – Panagiotis Kanavos Commented Jan 30 at 13:07
• @PanagiotisKanavos Basically different text documents and images (.doc, .docx, .csv, .pdf, .png, ...) – Filip Commented Jan 30 at 13:12
• "Can this be achieved with .NET native API:s" If the answer to this was "yes" would that actually answer your question? – Thom A Commented Jan 30 at 13:26
• @ThomA I prefer that the solution are using native .NET. – Filip Commented Jan 30 at 13:35
• 1 Tarwriter does not have an add method. You can only can add one object to the tar file. If you want to merge files together, you need to be able to unmerge the files. If you have Ascii files you can combine the files together and add a separator between the files that is not in the files so you can separate the files after you unzip. If you have binary files you need to add a byte count before each file so you can separate the files after you unzip. – jdweng Commented Jan 30 at 13:37

1 Answer 1

Your pseudo-code is nearly right.

• Use await reader.ReadAsync and command.ExecuteReaderAsync.
• Set CommandBehavior.SequentialAccess for better performance and memory usage on large binaries.
• Specify leaveOpen: true in the TarWriter constructor, otherwise the MemoryStream will be disposed.
• To write an entry, you need to first create one, set its DataStream, then use tarWriter.WriteEntryAsync
• Dispose the SQL stream.
• Consider passing CancellationToken in each async function.
• Then reset the position of the stream, and pass it back in a FileStreamResult.

var tarStream = new MemoryStream();    // no need to dispose MemoryStream
using (var tarWriter = new TarWriter(tarStream, leaveOpen: true))
{
    await using var connection = new SqlConnection("YourConnectionString");
    const string query = @"
SELECT
  FileName,
  FileData
FROM FilesTable;
";
    await using var command = new SqlCommand(query, connection);
    await connection.OpenAsync();
    await using var reader = command.ExecuteReaderAsync(CommandBehavior.SequentialAccess);

    while (await reader.ReadAsync())
    {                            
        var fileName = reader.GetString(0);
        await using var fileDataStream = reader.GetStream(1);

        var entry = new PaxTarEntry(TarEntryType.RegularFile, fileName)
        {
            DataStream = fileDataStream,
        };
        await tarWriter.WriteEntryAsync(entry);
    }
}

tarStream.Position = 0;
return new FileStreamResult(tarStream, "application/x-tar");