LuceneHack之通过缩小搜索结果集来提升性能(2)

作者&＃xff1a;caocao&＃xff08;网络隐士&＃xff09;&＃xff0c;[url]http://www.caocao.name[/url]&＃xff0c;[url]http://www.caocao.mobi[/url]

转载请注明来源&＃xff1a;[url]http://www.iteye.com/topic/80073[/url]



书接前文([url]http://www.iteye.com/topic/78884[/url])&＃xff0c;上回说了个大致的原理&＃xff0c;这回开始上代码。



五、原则

1、不改动lucene-core的代码

肆意改动lucene-core的代码实在是很不道德的事情&＃xff0c;而且会导致后期维护升级的大量问题。如果真的有这等迫切需求&＃xff0c;还不如加入lucene开发组&＃xff0c;尽一份绵薄之力。看官说了&＃xff0c;隐士你怎么不去啊&＃xff0c;唉&＃xff0c;代码比较丑陋&＃xff0c;没脸去人家那里&＃xff0c;后文详述。

2、不改动lucene索引文件格式

道理同上。

3、替换常规搜索的接口尽量少

这样可以方便来回切换标准搜索和这个搜索&＃xff0c;减小代码修改、维护的成本。

4、命名规范

所有增加的类名均以Inaccurate开头&＃xff0c;其余遵循lucene命名规范。



六、限制

1、隐士只做了BooleanWeight2的替代品&＃xff0c;如果Weight不是BooleanWeight2&＃xff0c;则等同于常规搜索。

2、如果搜索结果集小于等于最大允许的结果集&＃xff0c;则等同于常规搜索。



七、文件

[code]

org.apache.lucene.search

InaccurateBooleanScorer2.java // BooleanScorer2的替代品

InaccurateBooleanWeight2.java // BooleanWeight2的替代品

InaccurateHit.java // Hit的替代品

InaccurateHitIterator.java // HitIterator的替代品

InaccurateHits.java // Hits的替代品

InaccurateIndexSearcher.java // IndexSearcher的替代品

org.apache.lucene.util

InaccurateResultAggregation.java // 放搜索统计信息的value object

[/code]



八、实战

1、InaccurateIndexSearcher

InaccurateIndexSearcher extends IndexSearcher&＃xff0c;结构很简单&＃xff0c;增加了两个成员变量&＃xff1a;maxNumberOfDocs和inaccurateResultAggregation&＃xff0c;以及几个methods。

丑陋的部分来了&＃xff1a;

[code]

public void search(Weight weight, Filter filter, final HitCollector results, boolean ascending) throws IOException {

...

if (weight.getClass().getSimpleName().equals("BooleanWeight2")) { // hook BooleanWeight2

InaccurateBooleanWeight2 inaccurateBooleanWeight2 &＃61; new InaccurateBooleanWeight2(

this, weight.getQuery());

float sum &＃61; inaccurateBooleanWeight2.sumOfSquaredWeights();

float norm &＃61; this.getSimilarity().queryNorm(sum);

inaccurateBooleanWeight2.normalize(norm); // bad smell

InaccurateBooleanScorer2 inaccurateBooleanScorer2 &＃61; inaccurateBooleanWeight2

.getInaccurateBooleanScorer2(reader, maxNumberOfDocs);

if (inaccurateBooleanScorer2 !&＃61; null) {

inaccurateResultAggregation &＃61; inaccurateBooleanScorer2

.getInaccurateTopAggregation(collector, ascending);

}

} else {

Scorer scorer &＃61; weight.scorer(reader);

if (scorer !&＃61; null) {

scorer.score(collector);

}

}

...

}

[/code]

由于BooleanWeight2被lucene-core给藏起来了&＃xff0c;instanceof都不能用&＃xff0c;只好丑陋一把用weight.getClass().getSimpleName().equals("BooleanWeight2")。

把BooleanWeight2替换为InaccurateBooleanWeight2后代码老是搜不到任何结果&＃xff0c;经过千辛万苦地调试才发现BooleanWeight2初始化后并不算完&＃xff0c;需要拿到sum、norm&＃xff0c;然后normalize一把&＃xff0c;有点bad smell。

接着从InaccurateBooleanWeight2里拿到InaccurateBooleanScorer2&＃xff0c;调用getInaccurateTopAggregation搜一把&＃xff0c;这里ascending并没有发挥作用&＃xff0c;原因相当复杂&＃xff0c;隐士引入ascending的本意是调整lucene扫描索引的方式&＃xff0c;docID小->大或docID大->小&＃xff0c;后来调整了建索引的方式就不需要这个了&＃xff0c;所以隐士只是留这个接口以后用&＃xff0c;万一以后lucene-core支持双向扫描索引即可启用。

2、InaccurateHits

InaccurateIndexSearcher里面调用search其实是调用new InaccurateHits(this, query, null, sort, ascending)。getMoreDocs会反向调用新写的search方法。

上代码&＃xff1a;

[code]

...

TopDocs topDocs &＃61; (sort &＃61;&＃61; null) ? searcher.search(weight, filter, n,

ascending) : searcher

.search(weight, filter, n, sort, ascending);

length &＃61; topDocs.totalHits;

InaccurateResultAggregation inaccurateResultAggregation &＃61; searcher

.getInaccurateResultAggregation();

if (inaccurateResultAggregation &＃61;&＃61; null) {

totalLength &＃61; length;

} else {

accurate &＃61; inaccurateResultAggregation.isAccurate();

if (inaccurateResultAggregation.isAccurate()) {

totalLength &＃61; inaccurateResultAggregation

.getNumberOfRecordsFound();

} else {

int maxDocID &＃61; searcher.maxDoc();

totalLength &＃61; 1000 * ((int) Math

.ceil((0.001

* maxDocID

/ (inaccurateResultAggregation.getLastDocID() &＃43; 1) * inaccurateResultAggregation

.getNumberOfRecordsFetched()))); // guessing how many records there are

}

}

...

[/code]

代码没什么特别的&＃xff0c;除了一个猜测记录总数的算法。lucene从docID小向大的扫&＃xff0c;由于上回说了扫到一半会跳出来&＃xff0c;那么由最后扫到的lastDocID和maxDocID的比例可以猜测总共有多少条记录&＃xff0c;虽然不是很准&＃xff0c;但是数量级的精度是可以保证的&＃xff0c;反正一般用户只能看到前1000条记录&＃xff0c;具体有多少对用户来说不过是过眼云烟。

3、InaccurateBooleanWeight2

InaccurateBooleanWeight2没什么好说的&＃xff0c;就是个拿到InaccurateBooleanScorer2的跳板。

4、InaccurateBooleanScorer2

InaccurateBooleanScorer2的代码均来自BooleanScorer2&＃xff0c;由于BooleanScorer2从设计上来说并不准备被继承&＃xff0c;隐士只好另起炉灶&＃xff0c;bad smell啊。隐士没有修改任何从BooleanScorer2过来的代码&＃xff0c;只加了getMaxNumberOfDocs、getInaccurateTopAggregation、getAccurateBottomAggregation。getInaccurateTopAggregation是扫描到maxNumberOfDocs后立即跳出来&＃xff0c;所以结果会有所不准&＃xff0c;getAccurateBottomAggregation总是保留最后maxNumberOfDocs个结果&＃xff0c;结果也会有所不准&＃xff0c;但是统计值是准的&＃xff0c;因为每次都走完了所有索引。由两者差异可知getAccurateBottomAggregation性能会差一点&＃xff0c;准确性和性能不可兼得啊。

[code]

public InaccurateResultAggregation getInaccurateTopAggregation(

HitCollector hc, boolean ascending) throws IOException {

// DeltaTime dt &＃61; new DeltaTime();

if (countingSumScorer &＃61;&＃61; null) {

initCountingSumScorer();

}

int lastDocID &＃61; 0;

boolean reachedTheEnd &＃61; true;

int numberOfRecordsFetched &＃61; 0;

while (countingSumScorer.next()) {

lastDocID &＃61; countingSumScorer.doc();

float score &＃61; score();

hc.collect(lastDocID, score);

numberOfRecordsFetched&＃43;&＃43;;

if (numberOfRecordsFetched >&＃61; maxNumberOfDocs) {

reachedTheEnd &＃61; !countingSumScorer.next();

break;

}

}

// System.out.println(dt.getTimeElasped());

/*

* This method might cast the rest away. So it might be inaccurate.

*/

return new InaccurateResultAggregation(lastDocID, ascending,

reachedTheEnd, numberOfRecordsFetched, numberOfRecordsFetched);

}



public InaccurateResultAggregation getAccurateBottomAggregation(

HitCollector hc, boolean ascending) throws IOException {

// DeltaTime dt &＃61; new DeltaTime();

if (countingSumScorer &＃61;&＃61; null) {

initCountingSumScorer();

}

LinkedList resultNodes &＃61; new LinkedList();

boolean isFull &＃61; false;

int lastDocID &＃61; 0;

int index &＃61; 0;

int numberOfRecordsFound &＃61; 0;

while (countingSumScorer.next()) {

lastDocID &＃61; countingSumScorer.doc();

float score &＃61; score();

resultNodes.add(new ResultNode(lastDocID, score));

if (isFull) {

resultNodes.removeFirst();

}

index&＃43;&＃43;;

numberOfRecordsFound&＃43;&＃43;;

if (index >&＃61; maxNumberOfDocs) {

isFull &＃61; true;

index &＃61; 0;

// break;

}

}

for (ResultNode resultNode : resultNodes) {

hc.collect(resultNode.getDoc(), resultNode.getScore());

}

// System.out.println(dt.getTimeElasped());



/*

* Since this method runs full scan against all matched docs, it&＃39;s

* accurate at all.

*/

return new InaccurateResultAggregation(lastDocID, ascending, true,

resultNodes.size(), numberOfRecordsFound);

}

[/code]



九、总结

代码已经打包上传了&＃xff0c;有隐士写的简略注释&＃xff0c;调用方式写在readme.txt里面&＃xff0c;只需要替换几行代码即可。

总的来说只要

1、将Searcher searcher &＃61; new IndexSearcher(reader);替换为InaccurateIndexSearcher searcher &＃61; new InaccurateIndexSearcher(reader, 5000);

2、将Hits hits &＃61; searcher.search(query);替换为InaccurateHits hits &＃61; searcher.search(query, sort, ascending);

就行了。欢迎大家试用&＃xff0c;如果有什么改进&＃xff0c;请务必把改进后的代码也开源给大家&＃xff0c;互相学习&＃xff0c;互相促进。

由于代码里面有几处有bad smell&＃xff0c;隐士实在没脸去lucene开发组那里喊一嗓子。