Dynamic loading data using d3.csv in D3 v5

Question

I'm trying to build a dynamically loading bar chart in d3.js which will load data from back-end in parts. Using the d3.csv() function, is there any way to read only first n number of rows from a data for initial draw and then load subsequent data as per my JS logic?

tl;dr I want to selectively access my data inside the d3.csv() function.

I'm trying to run the below code for this :

var margin = {
            top: 20,
            bottom: 30,
            left: 40,
            right: 30
        },
        width = 600 - margin.left - margin.right,
        height = 500 - margin.top - margin.bottom;

    var loadData = function() {
        d3.csv("test_data.csv", function(data) {
            console.log(data.filter(function(d, i) {
                return i < 2;
            }));

            console.log(data.filter(function(d, i) {
                return i < 3;
            }))
        })
    }

    loadData();

However, I'm getting an error in the console:

Uncaught (in promise) TypeError: data.filter is not a function(…)

Which leads me to believe that this data is not an array. Is this the case or am i facing some other issue here?

Also, how do I access columns (inside csv file) inside this d3.csv function? (if for example, my csv data file contains two columns named a and b).

Solution 1

First, there is no way to load/parse just the first n rows of a CSV with d3.csv, I'm afraid that's not possible. Unfortunately you'll have to load/parse all the file, which may be inconvenient if the file is huge, meaning that the user will have to wait for the whole file to be loaded/parsed before the chart is painted. Also, it's worth mentioning that since d3.csv will load all the file the subsequent filter irrelevant: just use the rows of data you want, don't add even more unnecessary tasks to the browser, just use the rows you want for painting the chart.

Back to your main question:

Your data is an array. The problem here is just that you're using d3.csv as if it was a XHR, which was the case of D3 v4... However, in D3 v5, d3.csv is a promise.

So, it has to be:

d3.csv(url).then(callback);

Have a look at the demo below:

var csv = URL.createObjectURL(new Blob([
  `foo,bar,baz
12,43,21
45,54,21
87,13,17
98,69,17`
]));

d3.csv(csv).then(function(data) {
  console.log(data.filter(function(d, i) {
    return i < 2;
  }));
})

<script src="https://d3js.org/d3.v5.min.js"></script>

Regarding your second question, d3.csv exposes the columns in an array property named columns:

var csv = URL.createObjectURL(new Blob([
  `foo,bar,baz
12,43,21
45,54,21
87,13,17
98,69,17`
]));

d3.csv(csv).then(function(data) {
  console.log("columns are: " + data.columns)
})

<script src="https://d3js.org/d3.v5.min.js"></script>

Solution 2

One thing to add to Gerardo Furtado's answer: your example is structured like this:

d3.csv('some_file.csv', someFunction)

In d3.csv V5, if a function is passed as an argument like this here, it gets called once for each row, passed an object representing that row, its index, and an array of column keys, allowing the rows to be altered. So, the specific error is because the first arg in this callback is an object representing a row, not an array representing the data set.

Then, when they're all done, the promise is complete and a callback provided with .then(someFunction) is fired.

d3.csv('some_file.csv', transformRow).then(processData)

A transformRow function is optional; if one is provided, then whatever it returns for each row replaces that row in the data.

The processData callback then gets an array of rows, with a property columns which is an array of the original column names (so if transformRow returns an object with different property keys, the data.columns won't match the properties of each row).

So for example:

    var csv = URL.createObjectURL(new Blob([
     `name,start,end
      SOMETHING,123,321
      INVALID,321,123
      ANOTHER,111,333`
    ]));

    d3.csv(csv, processRow).then(processData)
    
    function processRow (row, index, columnKeys) {
      row[columnKeys[0]] = row[columnKeys[0]].trim().toLowerCase()
      row.duration = row.end - row.start // this new property doesn't change data.columns
      if (row.end > row.start) return row
    }
    
    function processData (data) {
      console.log(data, data.columns)
    }

<script src="https://d3js.org/d3.v5.min.js"></script>

Comments

[D_S_X]

Thanks a lot for the reply .... the idea here is to 1) download only the first 1000 rows (i'll do this selection from back-end) from a huge data set 2) draw initial 15-20 data out of this (will use data.filter for this) on a particular event in browser (scroll or click) 3) redraw the subsequent data (15-20 rows every time) on a event (scroll/click) ... do you think this be inconvenient/slow from user point of view ?

[Gerardo Furtado]

No, but you'll have to load all the data anyway.

[Gerardo Furtado]

Well, right now it's not exactly clear what is your objective. Since this question was solved, I suggest you posting a new question, explaining in details what's the issue.