|org.apache.lucene.benchmark.quality.trec||Utilities for Trec related quality benchmarking, feeding from Trec Topics and QRels inputs.|
|org.apache.lucene.benchmark.quality.utils||Miscellaneous utilities for search quality benchmarking: query parsing, submission reports.|
|Judge||Judge if a document is relevant for a quality query.||code | html|
|QualityQueryParser||Parse a QualityQuery into a Lucene query.||code | html|
|QualityBenchmark||Main entry point for running a quality benchmark.||code | html|
|QualityQuery||A QualityQuery has an ID and some name-value pairs.||code | html|
|QualityStats||Results of quality benchmark run for a single query or for a set of queries.||code | html|
|QualityStats.RecallPoint||A certain rank in which a relevant doc was found.||code | html|
This package allows to benchmark search quality of a Lucene application.
In order to use this package you should provide:
For benchmarking TREC collections with TREC QRels, take a look at the trec package.
Here is a sample code used to run the TREC 2006 queries 701-850 on the .Gov2 collection:
File topicsFile = new File("topics-701-850.txt"); File qrelsFile = new File("qrels-701-850.txt"); Searcher searcher = new IndexSearcher("index"); int maxResults = 1000; String docNameField = "docname"; PrintWriter logger = new PrintWriter(System.out,true); // use trec utilities to read trec topics into quality queries TrecTopicsReader qReader = new TrecTopicsReader(); QualityQuery qqs = qReader.readQueries(new BufferedReader(new FileReader(topicsFile))); // prepare judge, with trec utilities that read from a QRels file Judge judge = new TrecJudge(new BufferedReader(new FileReader(qrelsFile))); // validate topics & judgments match each other judge.validateData(qqs, logger); // set the parsing of quality queries into Lucene queries. QualityQueryParser qqParser = new SimpleQQParser("title", "body"); // run the benchmark QualityBenchmark qrun = new QualityBenchmark(qqs, qqParser, searcher, docNameField); SubmissionReport submitLog = null; QualityStats stats = qrun.execute(maxResults, judge, submitLog, logger); // print an avarage sum of the results QualityStats avg = QualityStats.average(stats); avg.log("SUMMARY",2,logger, " ");
Some immediate ways to modify this program to your needs are: