I'm relatively new to full-stack development, and currently trying to figure out an effective way to send and fetch large data between my front-end (React) and back-end (Express) while minimizing memory usage. Specifically, I'm building a mapping app which requires me to play around with large JSON files (10-100mb).

My current setup works for smaller JSON files:

Backend:

const data = require('../data/data.json');

router.get('/', function(req, res, next) {
  res.json(data);
});

Frontend:

ponentDidMount() {
      fetch('/')
      .then(res => res.json())
      .then(data => this.setState({data: data}));
  }

However, if data is bigger than ~40mb, the backend would crash if I test on local due to running out of memory. Also, holding onto the data with require() takes quite a bit of memory as well.

I've done some research and have a general understanding of JSON parsing, stringifying, streaming, and I think the answer lies somewhere with using chunked json stream to send the data bit by bit, but am pretty much at a loss on its implementation, especially using a single fetch() to do so (is this even possible?).

Definitely appreciate any suggestions on how to approach this.

I'm relatively new to full-stack development, and currently trying to figure out an effective way to send and fetch large data between my front-end (React) and back-end (Express) while minimizing memory usage. Specifically, I'm building a mapping app which requires me to play around with large JSON files (10-100mb).

My current setup works for smaller JSON files:

Backend:

const data = require('../data/data.json');

router.get('/', function(req, res, next) {
  res.json(data);
});

Frontend:

ponentDidMount() {
      fetch('/')
      .then(res => res.json())
      .then(data => this.setState({data: data}));
  }

However, if data is bigger than ~40mb, the backend would crash if I test on local due to running out of memory. Also, holding onto the data with require() takes quite a bit of memory as well.

I've done some research and have a general understanding of JSON parsing, stringifying, streaming, and I think the answer lies somewhere with using chunked json stream to send the data bit by bit, but am pretty much at a loss on its implementation, especially using a single fetch() to do so (is this even possible?).

Definitely appreciate any suggestions on how to approach this.

• javascript
• json
• reactjs

• I think your user has to scroll at some point to view next set of data, say after 10 sets. Why not create an end point that limits the data sent to the user at once. Then from your front end see whether the user has reached the end of the scroll. If yes, then make another subsequent request. Now handle this subsequent request from your express endpoint and limit to next 10. A simple request from front end would look like fetch('exampel./get-users?page=2') – Singh Commented Jul 13, 2018 at 1:42
• If there is no sensitive data, you could press your data before sending it to the user. The browser can depress it. – user3624390 Commented Jul 13, 2018 at 3:00

1 Answer 1

First off, 40mb is huge and can be inconsiderate to your users especially if there' s a high probability of mobile use.

If possible, it would be best to collect this data on the backend, probably put it onto disk, and then provide only the necessary data to the frontend as it's needed. As the map needs more data, you would make further calls to the backend.

If this isn't possible, you could load this data with the client-side bundle. If the data doesn't update too frequently, you can even cache it on the frontend. This would at least prevent the user from needing to fetch it repeatedly.

Alternatively, you can read the JSON via a stream on the server and stream the data to the client and use something like JSONStream to parse the data on the client.

Here's an example of how to stream JSON from your server via sockets: how to stream JSON from your server via sockets