using hit highlighter in lucene

Question

I have two questions regarding hit highlighter provided with apache lucene:

1. see this function could you explain the use of token stream parameter.

2. I have several large lucene document containing many fields and each field has some strings in it. Now I have found the most relevant document for a particular query. Now this document was found because several words in the query might have matched with the words in the document. I want to find out what words in the query caused this. So for this I plan to use Lucene Hit Highlighter. Example: if the query is "skin doctor delhi" and the document titled "dermatologist" contains the words "skin" and "doctor" then after hit highlighting i should be able to separate out "skin" and "doctor" from the query. I have been trying to write the code for this for several weeks now. Not able to get what i want. Could you help me please?

Thanks in advance.

Update:

Current Approach: I create a query containing all the words in the document.

Field[] field = doc.getFields("description");
String desc = "";
for (int j = 0; j < field.length; ++j) {
     desc += field[j].stringValue() + " ";
}

Query q = qp.parse(desc);
QueryScorer scorer = new QueryScorer(q, reader, "description");
Highlighter highlighter = new Highlighter(scorer);

String fragment = highlighter.getBestFragment(analyzer, "description", text);

It works for small documents but does not work for large documents. The following stacktrace is obtained.

    org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
    at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:152)
    at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:891)
    at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:866)
    at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1213)
    at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1167)
    at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:182)

It is obvious that the approach is unreasonable for large documents. What should be done to correct this?

BTW I am using FuzzyQuery matching.

Solution

EDIT: added some details about explain().

Some general introduction: The Lucene Highlighter is meant to find text snippets from a hit document, and to highlight tokens matching the query.

1. Therefore, The TokenStream parameter is used to break the hit text into tokens. The highlighter's scorer then scores each token, in order to score fragments and choose snippets and tokens to be highlighted.
2. I believe you are doing it wrong. If all you want to do is understand which query terms were matched in the document, you should use the explain() method. Basically, after you have instantiated a searcher, use:

Explanation expl = searcher.explain(query, docId);

String asText = expl.toString();

String asHtml = expl.toHtml();

docId is the raw document id from the search results.

Only if you do need the snippets and/or highlights, you should use the Highlighter. If you still want to use the highlighter, follow Nicholas Hrychan's advice. Beware, though, as he describes the Lucene 2.4.1 API - If you use a more advanced version, you should use "QueryScorer" where he says "SpanScorer" .

Comments

[Rohit Banga]

I have not understood the explain method. It returns an Explanation object, what function is required hereon to get the matched query terms. I am not satisfied with the documentation of Lucene.

[Yuval F]

The Lucene Explanation has a recursive structure. toString() and toHtml() give you the full explanation tree. getDetails() gives you a subtree of the tree at a time. I would try looking at the full tree first, and only if this is too complicated go to the subtrees.

[Rohit Banga]

this is what i get when i search for skin. the 0th document hit produces this with explain(). 0.0 = (NON-MATCH) sum of: how do i interpret it?

[Rohit Banga]

System.out.println(searcher.explain(query, i).toString());

[Rohit Banga]

i is zero in the previous function call

[Yuval F]

Here's a fuller version: stackoverflow.com/questions/1742124/…

[Rohit Banga]

OK thank you finally got it working. Now i am using fuzzy query matching. so if the document is a hit, explain would tell me the word as it occurs in the document and not in the query. how to achieve this?