I have a large amount of data to sort and query, and I can't rely on an internet connection. Ideally, I'd like to store my entire data-set as a JSON object (currently around 17MB, but could get much larger) and use something like jLinq or SQLite to query it, as opposed to having to output numerous smaller files.
I'm interested in finding what the largest remended filesize is for an external getJSON call using JavaScript (jQuery, specifically). 1MB, 20MB, 100MB? Information on the subject is scarce. Information on querying large data-sets client-side is scarce all around.
I have a large amount of data to sort and query, and I can't rely on an internet connection. Ideally, I'd like to store my entire data-set as a JSON object (currently around 17MB, but could get much larger) and use something like jLinq or SQLite to query it, as opposed to having to output numerous smaller files.
I'm interested in finding what the largest remended filesize is for an external getJSON call using JavaScript (jQuery, specifically). 1MB, 20MB, 100MB? Information on the subject is scarce. Information on querying large data-sets client-side is scarce all around.
Share Improve this question edited Aug 10, 2022 at 22:17 halfer 20.4k19 gold badges108 silver badges201 bronze badges asked Mar 28, 2012 at 17:26 S16S16 2,9959 gold badges43 silver badges64 bronze badges 3- This is all happening locally? Are you in a browser environment, or using node.js or something similar? My guess is that this is going to be highly dependent on the memory available in the client machine... the biggest issue with this approach is probably parsing, since you may need to load the entire JSON string into memory in order to parse it. – nrabinowitz Commented Mar 28, 2012 at 17:33
- All client-side, locally, in a browser environment. Cannot rely on them having jode.js installed and running. – S16 Commented Mar 28, 2012 at 17:38
-
Note that the first problem you're going to have is loading the file at all - in general, browser security settings will not let you load a local file with AJAX, even from a local HTML page. You can get around this by loading it as a script, either with JSONP or in the document
<head>. – nrabinowitz Commented Mar 28, 2012 at 18:50
1 Answer
Reset to default 12Your biggest problem will probably be loading time since it will have to convert it from a string of JSON into an actual JavaScript object. The other big problem will be that the entire data set will be in memory for the page. I'm not familiar with any page using 100MB+ of data.
I made a jsfiddle to test loading performance of large JSON strings. It looks like it takes ~500ms just to parse a ~20MB string of JSON (on a Core i7 machine), and in Chrome it uses 80MB more memory than if the JSON string basically empty. So 100MB could take a few seconds to load and use 400MB+ of memory.
This won't solve either of those issues, but have you considered using SQL.js? It is a JavaScript implementation of SQLite. It should make querying that data much easier.