JavaScript library for search engine style searching?

Question

Is there a JavaScript library that can determine if a string matches a search query? It should be efficient and provide advanced query functionality like that of Google or LexisNexis (things like and/or operators, synonyms, and parentheses). Any kind of advanced search features would be great; it doesn't have to be an exact match to any particular search engine.

Motivation: I have an HTML page with a search box followed by a bunch of paragraphs (which have unique ids and are generated from a JavaScript array). When the user types a search query in the box and presses enter, all paragraphs should be hidden (i.e. their display set to none) if they don't match the query.

My current strategy (using jQuery):

1. Separate the query string into an array of keywords by splitting it over whitespace.
2. Hide all paragraphs with $('p').hide().
3. For each keyword, show a paragraph containing it with $('p:contains("'+keyword+'")').show().

Which is an extremely limited search feature that is case-sensitive, treats all keywords as optional, and doesn't provide operators like and, or, or parentheses. It's also inefficient because it goes through each string once for each keyword even if it has already been matched.

Solution 1

Here are some libraries that I am evaluating for projects (in July 2013). Any of these should be able to provide the core of the search feature.

• http://lunrjs.com/
  • stemming, scoring built in
  • 13.8 kb minified
  • updated recently (https://github.com/olivernn/lunr.js/commits/master)
  • 10 contributors
  • no external dependencies
• http://fusejs.io (formerly at http://kiro.me/projects/fuse.html)
  • fuzzy search
  • 1.58 kb minified
  • updated recently (https://github.com/krisk/Fuse/commits/master)
  • 1 contributor
  • no external dependencies
• http://reyesr.github.io/fullproof/
  • uses html5 storage with graceful degradation
  • 459 kb minified
  • last updated 2013 (https://github.com/reyesr/fullproof/commits/master)
  • 2 contributors
  • no external dependencies
• http://eikes.github.io/facetedsearch/
  • pagination, templating built in
  • 5.70 kb minified
  • last updated 2014 (https://github.com/eikes/facetedsearch/commits/master)
  • 1 contributor
  • depends on jquery and underscore

If you feel like building your own, here are implementations of 2 common stemming algorithms to get you started:

• https://github.com/fortnightlabs/snowball-js
• http://tartarus.org/martin/PorterStemmer/

As for handling boolean logic search operators, maybe this question about js query parsers will be useful.

Solution 2

The best (easy and good) way is to use a Vector Search Algorithm.

First take all words in each paragraph and save them in a vector object (how to build explained later) and compare relation to query Vector of each Paragraph Vector

Then on each word use the Porter stemmer to make it cluster things like kid and kids.

var Vector = function(phar) {

var self = this;
self.InitVector = function () {
    var wordArray = self.spltwords(phar);
    self.VectorSize = wordArray .length;
    var stemdWordArray = self.runPotterStemmer(wordArray);
    self.VectoData = self.GroupAndCountWords(stemdWordArray) ;
}
self.VectoData = {}; 

self.runPotterStemmer = function(arr){
    // run potter as seen in link
}

self.spltwords= function(arr) {
    // run split
}

self.GroupAndCountWords = function(arr) {
    for (var i=0; i<arr.length; i++) {
        if (VectoData[arr[i]] === undefined) {
            VectoData[arr[i]] = 0;     
        } else {
            VectoData[arr[i]] = VectoData[arr[i]] +1;        
        }
    }
}  
self.compare = function(queryVector) {
    // compare queryVector to current vector and return a similarity number
    // number of similar words count in query divided by the length of paragraph                       
}                        
self.InitVector()
return self;

Solution 3

I have worked on a few open source javascript projects in the search and NLP space. You could checkout search-index which seems to be close to what you are looking for.

Solution 4

PEG.js and boolean-search.js are both useful libraries for getting some parsing libraries.

https://pegjs.org/

https://www.npmjs.com/package/boolean-parser

https://www.npmjs.com/browse/keyword/lucene

Hope that helps.

Solution 5

Disclaimer - I am an author.

You can also try ItemsJS. This is a search engine in JavaScript which supports full text, faceting and sorting.

Below you will find an interactive example - ItemsJS + VueJS:

var configuration = {
  searchableFields: ['title', 'tags', 'actors'],
  sortings: {
    name_asc: {
      field: 'name',
      order: 'asc'
    }
  },
  aggregations: {
    tags: {
      title: 'Tags',
      size: 10
    },
    actors: {
      title: 'Actors',
      size: 10
    },
    genres: {
      title: 'Genres',
      size: 10
    }
  }
}

// the rows comes from external resources
// https://github.com/itemsapi/itemsapi-example-data/blob/master/jsfiddle/imdb.js
itemsjs = itemsjs(rows, configuration);

var vm = new Vue({
  el: '#el',
  data: function () {

    // making it more generic
    var filters = {};
    Object.keys(configuration.aggregations).map(function(v) {
      filters[v] = [];
    })

    return {
      query: '',
      // initializing filters with empty arrays
      filters: filters,
    }
  },
  methods: {
    reset: function () {
      var filters = {};
      Object.keys(configuration.aggregations).map(function(v) {
        filters[v] = [];
      })

      this.filters = filters;
      this.query = '';
    }
  },
  computed: {
    searchResult: function () {

      var result = itemsjs.search({
        query: this.query,
        filters: this.filters
      })
      return result
    }
  }
});

<script src="https://cdn.rawgit.com/itemsapi/itemsapi-example-data/master/jsfiddle/imdb.js"></script>
<script src="https://cdn.rawgit.com/itemsapi/itemsjs/master/dist/itemsjs.js"></script>
<script src="https://cdn.jsdelivr.net/vue/latest/vue.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.0.0/css/bootstrap.min.css" rel="stylesheet"/>
<div id="el">
  <nav class="navbar navbar-default navbar-fixed-top">
    <div class="container">
      <div class="navbar-header">
        <a class="navbar-brand" href="#" v-on:click="reset()">ItemsJS movies</a>
      </div>
      <div id="navbar">
        <form class="navbar-form navbar-left">
          <div class="form-group">
            <input type="text" v-model="query" class="form-control" placeholder="Search">
          </div>
        </form>
      </div><!--/.nav-collapse -->
    </div>
  </nav>

  <div class="container" style="margin-top: 50px;">

    <h1>List of items ({{ searchResult.pagination.total }})</h1>

    <p class="text-muted">Search performed in {{ searchResult.timings.search }} ms, facets in {{ searchResult.timings.facets }} ms</p>

    <div class="row">
      <div class="col-md-2 col-xs-2">
        <div v-for="facet in searchResult.data.aggregations">
          <h5 style="margin-bottom: 5px;"><strong style="color: #337ab7;">{{ facet.title }}</strong></h5>

          <ul class="browse-list list-unstyled long-list" style="margin-bottom: 0;">
            <li v-for="bucket in facet.buckets">
            <div class="checkbox block" style="margin-top: 0; margin-bottom: 0;">
              <label>
                <!--<input class="checkbox" type="checkbox" v-on:click="updateFilters(facet.name, bucket.key)" v-model="filters[bucket.key]" value="{{ bucket.key }}" v-bind:value="isChecked2()">-->
                <!--<input class="checkbox" type="checkbox" v-on:click="updateFilters(facet.name, bucket.key)" v-model="filters[bucket.key]" v-bind:value="bucket.key">-->
                <input class="checkbox" type="checkbox" v-model="filters[facet.name]" v-bind:value="bucket.key">
                {{ bucket.key }} ({{ bucket.doc_count }}) 
              </label>
            </div>
            </li>
          </ul>
        </div>
      </div>

      <div class="col-md-10 col-xs-10">
        <div class="breadcrumbs"></div>
        <div class="clearfix"></div>
        <!--<h3>List of items ({{ searchResult.pagination.total }})</h3>-->
        <table class="table table-striped">
          <tbody>
          <tr v-for="item of searchResult.data.items">
            <td><img style="width: 100px;" v-bind:src="item.image"></td>
            <td></td>
            <td>
              <b>{{ item.name }}</b>
              <br />
              {{ item.description }}
            </td>
            <td></td>
            <td>
              <span style="font-size: 12px; display: block; float: left; background-color: #dbebf2; border-radius: 5px; padding: 1px 3px 1px 3px; margin: 2px;" v-for="tag in item.tags">{{ tag }}</span>
            </td>
          </tr>
          </tbody>
        </table>
        <div class="clearfix"></div>
      </div>

      <div class="clearfix" style="margin-bottom: 100px;"></div>
    </div>
  </div>
</div>

View more solutions

Comments

[Shreedhar]

to make case insensitive, you can convert the keyword to lowercase and check for the match i,e. keyword.toLowerCase()

[Noitidart]

Thanks for sharing this, I was wondering with you settled on and if you had any improved solutions to add in 2017?

[turtlemonvh]

@Noitidart I haven't had a need for these systems in a while in what I've been working on, but if I was going to do something today my first choice would still be the same as it was then: lunrjs

[timakro]

If you need preview functionality Lunr.js is still the way to go in August 2020. FlexSearch.js is a new contender but at the time of writing is missing a feature to get the position of the match which makes it unsuitable for implementing preview functionality. Fuse.js always did support that but it seems more targeted towards short strings like book names than paragraphs of text - in it's default configuration it can only match words that are no more than 100 characters from the beginning of the string.

[Arjun Kalidas]

Amazing collection of libraries. Thank you very much! I have been using Lunr in my projects and so far, that has yielded best results and comfortable to use.