I'm building a Node package that needs to download some files given the url. However, I don't want to save them into the filesystem directly. Instead, I need them to be preprocessed before saving into disk. Thus, I want to save the file content in a Buffer. How can this be acomplished with pipe() ?

Offcourse I could save a temp file and then open the file, process it and save the result but this brings unnecessary reads/writes into disk.

The files have generally small size and are .txt or .zip files so saving them in memory doesn't bring any harm.

One of the objectives is to find and extract only the pretended file when the downloaded file is a .zip . Using Adm-Zip I can do this, but have to pass a Buffer.

Thak You beforehand.

I'm building a Node package that needs to download some files given the url. However, I don't want to save them into the filesystem directly. Instead, I need them to be preprocessed before saving into disk. Thus, I want to save the file content in a Buffer. How can this be acomplished with pipe() ?

Offcourse I could save a temp file and then open the file, process it and save the result but this brings unnecessary reads/writes into disk.

The files have generally small size and are .txt or .zip files so saving them in memory doesn't bring any harm.

One of the objectives is to find and extract only the pretended file when the downloaded file is a .zip . Using Adm-Zip I can do this, but have to pass a Buffer.

Thak You beforehand.

• javascript
• node.js
• ajax

• Probably need to install another package, such as request. Initiate the download using the library and then see if you can contain the response body inside a Buffer object – ionizer Commented Jul 31, 2018 at 17:35

2 Answers 2

https.get(url, (res) => {
  const data = [];
  res.on('data', (chunk) => {
    data.push(chunk);
  }).on('end', () => {
    let buffer = Buffer.concat(data);
    // Do something with the buffer
  });
}).on('error', (err) => {
  console.log('download error:', err);
});

You can use node-fetch library. Code sample from repository readme:

// if you prefer to cache binary data in full, use arrayBuffer()

fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
    .then(res => res.arrayBuffer())
    .then(buffer => { /* do whatever you want with buffer */ })