Im building an upload tool using node.js and socket.io, because they usually upload incredibly huge files and normal upload forms wouldnt work. The problem is that they wanted to press the files into a zip before sending them, to increase efficiency in transmission.

I've been researching around pression methods like JSZip or zip.js, but neither of those work well with extremely big files. What could I do?

Im building an upload tool using node.js and socket.io, because they usually upload incredibly huge files and normal upload forms wouldnt work. The problem is that they wanted to press the files into a zip before sending them, to increase efficiency in transmission.

I've been researching around pression methods like JSZip or zip.js, but neither of those work well with extremely big files. What could I do?

• javascript
• html
• node.js
• upload
• pression

• Why ZIP? Can you use another algorighm? Can you make a Flash uploader? – nanofarad Commented Nov 7, 2012 at 13:46
• 1 ZIP because its the most widely accepted. I rather not use Flash. – Enrique Moreno Tent Commented Nov 7, 2012 at 13:57

2 Answers 2

You can press up to 4GB of data with zip.js but:

• it will work only with Chrome
• it will be quite slow (approx. 30min per GB of pressed data on my old laptop)

You can try it online with this demo. You must select the "HDD" option in the "choose temporary storage" input. Then you can import big files from your filesystem and monitor the memory consumption: it should be stable (approx. 300MB on my laptop).

Selecting "HDD" means that zip.js will use File API: Directories and System to store pressed data. This API is currently only available on Chrome and allows to write data into a sandboxed virtual filesystem. The demo uses the temporary storage which does not require a user permission.

Edit: You could also implement your own Writer constructor function to stream the data to your server while zip.js presses it: it wouldn't rely on the filesystem API and should work on every browser you support. A Writer must just implement these 2 methods:

init(callback[, onerror])
// initializes the stream where to write the zipped data

writeUint8Array(array, callback[, onerror])
// writes asynchronously a Uint8Array into the stream.

// getData method is optional

Here is an example of custom Writer and Reader constructors. You can also look at zip.js Writers implementations for more examples.

If you're looking for a pressor on the client side I'm sorry to tell you that JS is not the way to go.

A side from that, your best bet is to tell your users to just press the files before uploading them. Or even use some other kind of tool (FTP maybe?).

EDIT:

Oh, btw, ZIP is really inefficient with random data, so you'll be wasting CPU time for the pression/depression and you won't shrink almost nothing in size.