Save This Page
Home » lucene-3.0.1-src » org.apache » lucene » search » highlight » [javadoc | source]
org.apache.lucene.search.highlight
public class: TokenSources [javadoc | source]
java.lang.Object
   org.apache.lucene.search.highlight.TokenSources
Hides implementation issues associated with obtaining a TokenStream for use with the higlighter - can obtain from TermFreqVectors with offsets and (optionally) positions or from Analyzer class reparsing the stored content.
Method from org.apache.lucene.search.highlight.TokenSources Summary:
getAnyTokenStream,   getAnyTokenStream,   getTokenStream,   getTokenStream,   getTokenStream,   getTokenStream,   getTokenStream,   getTokenStream
Methods from java.lang.Object:
clone,   equals,   finalize,   getClass,   hashCode,   notify,   notifyAll,   toString,   wait,   wait,   wait
Method from org.apache.lucene.search.highlight.TokenSources Detail:
 public static TokenStream getAnyTokenStream(IndexReader reader,
    int docId,
    String field,
    Analyzer analyzer) throws IOException 
    A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
 public static TokenStream getAnyTokenStream(IndexReader reader,
    int docId,
    String field,
    Document doc,
    Analyzer analyzer) throws IOException 
    A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in org.apache.lucene.document.Document to retrieve the TokenStream. This is useful when you already have the document, but would prefer to use the vector first.
 public static TokenStream getTokenStream(TermPositionVector tpv) 
 public static TokenStream getTokenStream(TermPositionVector tpv,
    boolean tokenPositionsGuaranteedContiguous) 
    Low level api. Returns a token stream or null if no offset info available in index. This can be used to feed the highlighter with a pre-parsed token stream In my tests the speeds to recreate 1000 token streams using this method are: - with TermVector offset only data stored - 420 milliseconds - with TermVector offset AND position data stored - 271 milliseconds (nb timings for TermVector with position data are based on a tokenizer with contiguous positions - no overlaps or gaps) The cost of not using TermPositionVector to store pre-parsed content and using an analyzer to re-parse the original content: - reanalyzing the original content - 980 milliseconds The re-analyze timings will typically vary depending on - 1) The complexity of the analyzer code (timings above were using a stemmer/lowercaser/stopword combo) 2) The number of other fields (Lucene reads ALL fields off the disk when accessing just one document field - can cost dear!) 3) Use of compression on field storage - could be faster due to compression (less disk IO) or slower (more CPU burn) depending on the content.
 public static TokenStream getTokenStream(IndexReader reader,
    int docId,
    String field) throws IOException 
 public static TokenStream getTokenStream(Document doc,
    String field,
    Analyzer analyzer) 
 public static TokenStream getTokenStream(String field,
    String contents,
    Analyzer analyzer) 
 public static TokenStream getTokenStream(IndexReader reader,
    int docId,
    String field,
    Analyzer analyzer) throws IOException