I have an XHR object that downloads 1GB file.

function getFile(callback)
{
    var xhr = new XMLHttpRequest();
    xhr.onload = function () {
        if (xhr.status == 200) {
            callback.apply(xhr);
        }else{
            console.log("Request error: " + xhr.statusText);
        }
    };

    xhr.open('GET', 'download', true);
    xhr.onprogress = updateProgress;
    xhr.responseType = "arraybuffer";
    xhr.send();
}

But the File API can't load all that into memory even from a worker it throws out of memory...

btn.addEventListener('click', function() {
    getFile(function() {
        var worker = new Worker("js/saving.worker.js");
        worker.onmessage = function(e) {
            saveAs(e.data); // FileSaver.js it creates URL from blob... but its too large
        };

        worker.postMessage(this.response);
    });
});

Web Worker

onmessage = function (e) {
    var view  = new DataView(e.data, 0);
    var file = new File([view], 'file.zip', {type: "application/zip"});
    postMessage('file');
};

I'm not trying to compress the file, this file is already compressed from server.

I thought storing it first on indexedDB but i i'll have to load blob or file anyway, even if i do request by range bytes, soon or late i will have to build this giant blob..

I want to create blob: url and send it to user after been downloaded by browser

I'll use FileSystem API for Google Chrome, but i want make something for firefox, i looked into File Handle Api but nothing...

Do i have to build an extension for firefox, in order to do the same thing as FileSystem does for google chrome?

Ubuntu 32 bits

I have an XHR object that downloads 1GB file.

function getFile(callback)
{
    var xhr = new XMLHttpRequest();
    xhr.onload = function () {
        if (xhr.status == 200) {
            callback.apply(xhr);
        }else{
            console.log("Request error: " + xhr.statusText);
        }
    };

    xhr.open('GET', 'download', true);
    xhr.onprogress = updateProgress;
    xhr.responseType = "arraybuffer";
    xhr.send();
}

But the File API can't load all that into memory even from a worker it throws out of memory...

btn.addEventListener('click', function() {
    getFile(function() {
        var worker = new Worker("js/saving.worker.js");
        worker.onmessage = function(e) {
            saveAs(e.data); // FileSaver.js it creates URL from blob... but its too large
        };

        worker.postMessage(this.response);
    });
});

Web Worker

onmessage = function (e) {
    var view  = new DataView(e.data, 0);
    var file = new File([view], 'file.zip', {type: "application/zip"});
    postMessage('file');
};

I'm not trying to compress the file, this file is already compressed from server.

I thought storing it first on indexedDB but i i'll have to load blob or file anyway, even if i do request by range bytes, soon or late i will have to build this giant blob..

I want to create blob: url and send it to user after been downloaded by browser

I'll use FileSystem API for Google Chrome, but i want make something for firefox, i looked into File Handle Api but nothing...

Do i have to build an extension for firefox, in order to do the same thing as FileSystem does for google chrome?

Ubuntu 32 bits

• javascript
• firefox
• large-files

• Did it work when you downloaded a smaller *.zip file? The error looks familiar for me, "out of memory". – user5066707 Commented Aug 31, 2016 at 15:27
• Yeah it worked, just for large realy – Gabriel dos Anjos Commented Aug 31, 2016 at 15:27
• 1 I want to know how many alerts you get if you execute them in the 'click' event of btn – user5066707 Commented Aug 31, 2016 at 15:39
• 2 The goal here is to download a large 1gb file - but what are you doing with a 1gb file in memory anyway? The only way to process this file is streaming it. What is the intent of having this 1gb file using XHR and in memory? Need more clarity. Specifically what is this file and what do you want to do with it? – AntonB Commented Aug 31, 2016 at 15:58
• 1 "// FileSaver.js it creates URL from blob... but its too large" ... so it works up to that point? – the8472 Commented Aug 31, 2016 at 19:12

3 Answers 3

I have a theory that is if you split the file into chunks and store them in the indexedDB and then later merge them together it will work

A blob isn't made of data... it's more like pointers to where a file can be read from

Meaning if you store them in indexedDB and then do something like this (using FileSaver or alternative)

finalBlob = new Blob([blob_A_fromDB, blob_B_fromDB])
saveAs(finalBlob, 'filename.zip')

But i can't confirm this since i haven't tested it, would be good if someone else could

Loading 1gb+ with ajax isn't convenient just for monitoring download progress and filling up the memory.

Instead I would just send the file with a Content-Disposition header to save the file.

There are however ways to go around it to monitor the progress. Option one is to have a second websocket that signals how much you have downloaded while you are downloading normally with a get request. the other option will be described later in the bottom

I know you talked about using Blinks sandboxed filesystem in the conversation. but it has some drawbacks. It may need permission if using persistent storage. It only allows 20% of the available disk that are left. And if chrome needs to free some space then it will throw away any others domains temporary storage that was last used for the most recent file. Beside it doesn't work in private mode.
Not to mention that it has been dropping support for it and may never end up in other browsers - but they will most likely not remove it since many sites still depend on it

The only way to process this large file is with streams. That is why I have created a StreamSaver. This is only going to work in Blink (chrome & opera) ATM but it will eventually be supported by other browsers with the whatwg spec to back it up as a standard.

fetch(url).then(res => {
    // One idea is to get the filename from Content-Disposition header...
    const size = ~~res.headers.get('Content-Length')
    const fileStream = streamSaver.createWriteStream('filename.zip', size)
    const writeStream = fileStream.getWriter()
    // Later you will be able to just simply do
    // res.body.pipeTo(fileStream)
    // instead of pumping

    const reader = res.body.getReader()
    const pump = () => reader.read()
        .then(({ value, done }) => {
            // here you know how large the value (chunk) is and you can
            // figure out the download speed/progress when comparing it to the size

            return done 
                ? writeStream.close()
                : writeStream.write(value).then(pump)
        )

    // Start the reader
    pump().then(() =>
        console.log('Closed the stream, Done writing')
    )
})

This will not take up any memory

Blob is cool until you want to download a large file, there is a 600MB limit(chrome) for blob since it stores everything in memory.