As the title suggests, I have a CSV file (~250mb and 700k rows) which I am not able to load into d3. I tried to load it the way I usually do for csv files but no luck. Currently, it does not error out and I get an empty data array in the console. Not sure if the file is too big or I'm loading it incorrectly. Will appreciate any help on this. Thanks.

            var dataset;
            d3.csv("Consumer_Complaints.csv", function (error, data) {
                if (error) {  
                    console.log(error);  
                } else {
                    console.log("file load successful?");
                    console.log(data);   

                    dataset = data;
                }
            });

As the title suggests, I have a CSV file (~250mb and 700k rows) which I am not able to load into d3. I tried to load it the way I usually do for csv files but no luck. Currently, it does not error out and I get an empty data array in the console. Not sure if the file is too big or I'm loading it incorrectly. Will appreciate any help on this. Thanks.

            var dataset;
            d3.csv("Consumer_Complaints.csv", function (error, data) {
                if (error) {  
                    console.log(error);  
                } else {
                    console.log("file load successful?");
                    console.log(data);   

                    dataset = data;
                }
            });

• javascript
• csv
• d3.js

• How are you running this file? Javascript (in the browser) doesn't have access to your local file system. – Burhan Khalid Commented Feb 19, 2017 at 3:26
• @BurhanKhalid IIS via Visual Studio – sparta93 Commented Feb 19, 2017 at 3:27
• Think about DOM. Look at PapaParse. Don't crush your machine's memory or hit the browser's hard-coded limit :) – Max von Hippel Commented Feb 19, 2017 at 3:57

1 Answer 1

This has nothing to do with D3, but with JavaScript in general. D3 imposes no limit on the size of the file it can load and parse.

Javascript runs (with some exceptions) at the client side. That means your code has to download (if in a different server) all that huge CSV file and, not only that, it has to parse those hundreds of thousands of rows in a huge array of objects. It's simply too much.

So, mon sense tells us to think about:

• the user's connection speed
• the user's processing power
• the user's patience to stare at a blank screen for minutes, waiting until the data is downloaded/parsed.

This is a demo that loads a huge CSV file (from the data.gov site), you can see in the console the amount of data loaded. I also put a console.time to show the total time needed to download and parse the file (if you have patience to wait until the end, I hadn't):

console.time("totalTime:");
d3.csv("https://data.consumerfinance.gov/api/views/s6ew-h6mp/rows.csv")
    .on("progress", function(evt) {
        console.log("Amount loaded: " + evt.loaded)
    })
    .get(function(data) {
        console.timeEnd("totalTime:");
    });

<script src="https://d3js/d3.v4.min.js"></script>