Processing a large (12K+ rows) array in JavaScript

Question

The project requirements are odd for this one, but I'm looking to get some insight...

I have a CSV file with about 12,000 rows of data, approximately 12-15 columns. I'm converting that to a JSON array and loading it via JSONP (has to run client-side). It takes many seconds to do any kind of querying on the data set to returned a smaller, filtered data set. I'm currently using JLINQ to do the filtering, but I'm essentially just looping through the array and returning a smaller set based on conditions.

Would webdb or indexeddb allow me to do this filtering significantly faster? Any tutorials/articles out there that you know of that tackles this particular type of issue?

Solution 1

http://square.github.com/crossfilter/ (no longer maintained, see https://github.com/crossfilter/crossfilter for a newer fork.)

Crossfilter is a JavaScript library for exploring large multivariate datasets in the browser. Crossfilter supports extremely fast (<30ms) interaction with coordinated views, even with datasets containing a million or more records...

Solution 2

This reminds me of an article John Resig wrote about dictionary lookups (a real dictionary, not a programming construct).

http://ejohn.org/blog/dictionary-lookups-in-javascript/

He starts with server side implementations, and then works on a client side solution. It should give you some ideas for ways to improve what you are doing right now:

• Caching
• Local Storage
• Memory Considerations

Solution 3

If you require loading an entire data object into memory before you apply some transform on it, I would leave IndexedDB and WebSQL out of the mix as they typically both add to complexity and reduce the performance of apps.

For this type of filtering, a library like Crossfilter will go a long way.

Where IndexedDB and WebSQL can come into play in terms of filtering is when you don't need to load, or don't want to load, an entire dataset into memory. These databases are best utilized for their ability to index rows (WebSQL) and attributes (IndexedDB).

With in browser databases, you can stream data into a database one record at a time and then cursor through it, one record at a time. The benefit here for filtering is that this you means can leave your data on "disk" (a .leveldb in Chrome and .sqlite database for FF) and filter out unnecessary records either as a pre-filter step or filter in itself.

Comments

[Dawson Toth]

Not precisely apples-to-apples, but I'm hoping it'll give you some ideas.

[Jeremy Smith]

This is an amazing library that I just found, and so clearly the right answer for OP's problem.

[dave4jr]

6 years later I'm seeing this and it's still amazing. Thank you.