I understand how to load a remote file with Node.js + request, and I can then read it and return the png binary blob. Is there a elegant way to do it with one request (or even a one-liner)

something like:

http.createServer(function(req, res) {
    res.writeHead(200, {
        'Content-Type': 'image/png'
    });
    var picWrite = fs.createWriteStream(local);
    var picFetch = fs.createReadStream(local);
    picStream.on('close', function() {
        console.log("file loaded");
    });
    request(remote).pipe(picWrite).pipe(picFetch).pipe(res);
})

To be clear: my aim is to load a remote file from a CDN, cache it locally to the server and then return the file in the original request. In future requests I use fs.exists() to check it exists first.

This is my best effort so far:

http.createServer(function(req, res) {
    var file = fs.createWriteStream(local);
    request.get(remote).pipe(file).on('close', function() {
        res.end(fs.readFileSync(local), 'binary');
    });
})

I understand how to load a remote file with Node.js + request, and I can then read it and return the png binary blob. Is there a elegant way to do it with one request (or even a one-liner)

something like:

http.createServer(function(req, res) {
    res.writeHead(200, {
        'Content-Type': 'image/png'
    });
    var picWrite = fs.createWriteStream(local);
    var picFetch = fs.createReadStream(local);
    picStream.on('close', function() {
        console.log("file loaded");
    });
    request(remote).pipe(picWrite).pipe(picFetch).pipe(res);
})

To be clear: my aim is to load a remote file from a CDN, cache it locally to the server and then return the file in the original request. In future requests I use fs.exists() to check it exists first.

This is my best effort so far:

http.createServer(function(req, res) {
    var file = fs.createWriteStream(local);
    request.get(remote).pipe(file).on('close', function() {
        res.end(fs.readFileSync(local), 'binary');
    });
})

• javascript
• node.js
• request

• each http request - I need it to build up a local cache from a remote source. – sidonaldson Commented Oct 8, 2013 at 1:31

1 Answer 1

Since the request will return a readable stream, we can listen on its data and end events to write to both the HTTP response and a writable stream.

var http = require('http');
var request = require('request');

http.createServer(function(req, res) {
  res.writeHead(200, { 'Content-Type': 'image/png' });
  var file = fs.createWriteStream(local);

  // request the file from a remote server
  var rem = request(remote);
  rem.on('data', function(chunk) {
    // instead of loading the file into memory
    // after the download, we can just pipe
    // the data as it's being downloaded
    file.write(chunk);
    res.write(chunk);
  });
  rem.on('end', function() {
    res.end();
  });
});

The method that you showed first writes the data to disk, then reads it into memory again. This is rather pointless, since the data is already accessible when it's being written to disk.

If you use an event handler, you can write to both the HTTP response and the file stream without needing to pointlessly load the file to memory again. This also solves the problem with using pipe(), because pipe() will consume the data from the readable stream, and can only be done once.

This also solves problems with running out of memory, because if you were to download a large file, then it would effectively run your Node.js process out of memory. With streams, only chunks of a file are loaded into memory at one time, so you don't have this problem.