I have created a simple proxy using Node (Express), where I am reading a pdf file from a remote server. I am able to read the file in chunks, and sending the chunks in response as res.write(). When I log the data, I can see all the streaming, but I can't receive the chunks on the front end. I am calling the endpoints correctly, but I want to get a response for each chunk is read.

Client Side code

    fetch('http://localhost:8003/pdf-cors')
    .then(response => response.text())
    .then(data => console.log(data));

I have created a simple proxy using Node (Express), where I am reading a pdf file from a remote server. I am able to read the file in chunks, and sending the chunks in response as res.write(). When I log the data, I can see all the streaming, but I can't receive the chunks on the front end. I am calling the endpoints correctly, but I want to get a response for each chunk is read.

Client Side code

    fetch('http://localhost:8003/pdf-cors')
    .then(response => response.text())
    .then(data => console.log(data));

Node Express code

app.get('/pdf-cors', (req, res) => {
    https.get('https://www.mazda./globalassets/en/assets/csr/download/2017/2017_all.pdf')
        .on('response', response => {
        response.on('data', data => {
            console.log("[Reading File]...");
            res.write(data);
        })
        response.on('end', _ => {
            console.log("File reading done !");
        })
    })
});

Note: When I put res.end() right after res.write(data) I am able to see the first chunk passed to the client in the console, but then I get an error says Error [ERR_STREAM_WRITE_AFTER_END]: write after the end.

The only thing that I want is to see each chunk being passed to the front end.

• javascript
• node.js
• https
• fetch

• why don't you call res.end() in response.on('end'...)? – Randy Casburn Commented Oct 29, 2020 at 20:25
• because thats called at the very end – Nadeem Ahmad Commented Oct 29, 2020 at 20:26
• If you never close the write stream (with res.end()) - you won't get the output sent to the client. If reading the file has ended (at the end), the writing has ended at that same point. – Randy Casburn Commented Oct 29, 2020 at 20:27
• You could (should maybe) use pipe like this: req.pipe(request.get(target)).pipe(res); – Randy Casburn Commented Oct 29, 2020 at 20:29
• 1 See developer.mozilla/en-US/docs/Web/API/ReadableStream – Estus Flask Commented Oct 29, 2020 at 23:16

2 Answers 2

Fetch API allows to stream data with ReadableStream.

On client side a stream can be read like:

  let res = await fetch('...');
  let reader = res.body.getReader();
  let result;
  let decoder = new TextDecoder('utf8');
  while (!result?.done) {
    result = await reader.read();
    let chunk = decoder.decode(result.value);
    console.log(chunk);
  }

If you only need server to client events you can use EventStream with server-sent events.

export const apiEventSourceStream = async (onMessage) => {
  const url = `${API_IRL}/query-stream`;
  const eventSource = new EventSource(url);

  eventSource.onmessage = (event) => {
    onMessage(event.data);
  };

  eventSource.onerror = (error) => {
    console.error("EventSource error: ", error);
    eventSource.close();
  };

  return () => eventSource.close();
};