Lucene（全文检索）入门

Lucene实现全文检索的&＃xff08;一部分为索引过程&＃xff0c;一部分为搜索过程&＃xff09;&＃xff1a;

创建索引的过程&＃xff1a;

一、获得原始文档

    原始文档是指要索引和搜索的内容。原始内容包括互联网上的网页、数据库中的数据、磁盘上的文件等。

二、创建文档对象

    获取原始内容的目的是为了索引&＃xff0c;在索引前需要将原始内容创建成文档&＃xff08;Document&＃xff09;&＃xff0c;文档中包括一个一个的域&＃xff08;Field&＃xff09;&＃xff0c;域中存储内容。

    这里我们可以将磁盘上的一个文件当成一个document&＃xff0c;Document中包括一些Field&＃xff08;file_name文件名称、file_path文件路径、file_size文件大小、file_content文件内容&＃xff09;。

三、分析文档

    将原始内容创建为包含域&＃xff08;Field&＃xff09;的文档&＃xff08;document&＃xff09;&＃xff0c;需要再对域中的内容进行分析&＃xff0c;分析的过程是经过对原始文档提取单词、将字母转为小写、去除标点符号、去除停用词等过程生成最终的语汇单元&＃xff0c;可以将语汇单元理解为一个一个的单词。

四、创建索引

    对所有文档分析得出的语汇单元进行索引&＃xff0c;索引的目的是为了搜索&＃xff0c;最终要实现只搜索被索引的语汇单元从而找到Document&＃xff08;文档&＃xff09;。

查询索引的过程&＃xff1a;

一、用户查询接口

    用于输入查询内容的载体。

二、创建查询

    用户输入查询关键字执行搜索之前需要先构建一个查询对象&＃xff0c;查询对象中可以指定查询要搜索的Field文档域、查询关键字等&＃xff0c;查询对象会生成具体的查询语法。

三、执行查询

    根据查询语法在倒排索引词典表中分别找出对应搜索词的索引&＃xff0c;从而找到索引所链接的文档链表。

四、渲染结果

    以一个友好的界面将查询结果展示给用户&＃xff0c;用户根据搜索结果找自己想要的信息&＃xff0c;为了帮助用户很快找到自己的结果&＃xff0c;提供了很多展示的效果&＃xff0c;比如搜索结果中将关键字高亮显示&＃xff0c;百度提供的快照等。

Lucene使用

一、导包

二、创建索引库以及查询索引库

创建索引库时&＃xff0c;域对象相关的子类介绍&＃xff1a;

查询索引库时&＃xff0c;索引搜索器的搜索方法介绍&＃xff1a;

代码&＃xff1a;

package com.xushuai.lucene;import org.apache.commons.io.FileUtils;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.*;
import org.apache.lucene.index.*;
import org.apache.lucene.search.*;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.Version;
import org.junit.Test;import java.io.File;
import java.io.IOException;/*** Lucene初次使用* Author: xushuai* Date: 2018/5/6* Time: 15:08* Description:*/
public class LuceneDemo {/** 创建索引库的步骤* 第一步&＃xff1a;创建一个java工程&＃xff0c;并导入jar包。* 第二步&＃xff1a;创建一个indexwriter对象。* 1&＃xff09;指定索引库的存放位置Directory对象* 2&＃xff09;指定一个分析器&＃xff0c;对文档内容进行分析。* 第三步&＃xff1a;创建document对象。* 第四步&＃xff1a;创建field对象&＃xff0c;将field添加到document对象中。* 第五步&＃xff1a;使用indexwriter对象将document对象写入索引库&＃xff0c;此过程进行索引创建。并将索引和document对象写入索引库。* 第六步&＃xff1a;关闭IndexWriter对象。*//*** 创建索引库* &＃64;auther: xushuai* &＃64;date: 2018/5/6 15:12* &＃64;throws: IOException*/&＃64;Testpublic void luceneCreateIndexRepository() throws IOException {//存放索引库的路径Directory directory &＃61; FSDirectory.open(new File("D:\\lucene&solr\\lucene\\index"));//创建分析器(使用其子类&＃xff0c;标准分析器类)Analyzer analyzer &＃61; new StandardAnalyzer();IndexWriterConfig indexWriterConfig &＃61; new IndexWriterConfig(Version.LATEST, analyzer);//使用索引库路径和分析器构造索引库写入流IndexWriter indexWriter &＃61; new IndexWriter(directory,indexWriterConfig);//读取原始文档创建相应文档对象&＃xff0c;并设置相关field域File dir &＃61; new File("D:\\lucene&solr\\lucene\\searchsource");//遍历该dir下所有的文件创建文档对象for(File file:dir.listFiles()){//获取其文件名、文件大小、文件位置、文件内容String file_name &＃61; file.getName();String file_path &＃61; file.getPath();Long file_size &＃61; FileUtils.sizeOf(file);String file_content &＃61; FileUtils.readFileToString(file);//为获取到的文件属性创建相应域(参数分别为&＃xff1a;域名城、域值以及是否保存)Field fileNameField &＃61; new TextField("filename",file_name, Field.Store.YES);Field filePathField &＃61; new StoredField("filepath",file_path);Field fileSizeField &＃61; new LongField("filesize",file_size, Field.Store.YES);Field fileContentField &＃61; new TextField("filecontent",file_content,Field.Store.YES);//创建document对象Document document &＃61; new Document();//保存域对象document.add(fileContentField);document.add(fileNameField);document.add(filePathField);document.add(fileSizeField);//将document绑定给写入流indexWriter.addDocument(document);}//释放资源indexWriter.close();}/** 查询索引* 第一步&＃xff1a;创建一个Directory对象&＃xff0c;也就是索引库存放的位置。* 第二步&＃xff1a;创建一个indexReader对象&＃xff0c;需要指定Directory对象。* 第三步&＃xff1a;创建一个indexsearcher对象&＃xff0c;需要指定IndexReader对象* 第四步&＃xff1a;创建一个TermQuery对象&＃xff0c;指定查询的域和查询的关键词。* 第五步&＃xff1a;执行查询。* 第六步&＃xff1a;返回查询结果。遍历查询结果并输出。* 第七步&＃xff1a;关闭IndexReader对象*//*** 查询索引* &＃64;auther: xushuai* &＃64;date: 2018/5/6 15:48* &＃64;throws: IOException*/&＃64;Testpublic void luceneSearchIndexRepository() throws IOException {//指定索引库位置Directory directory &＃61; FSDirectory.open(new File("D:\\lucene&solr\\lucene\\index"));//创建索引库读取流IndexReader indexReader &＃61; DirectoryReader.open(directory);//创建索引搜索器对象IndexSearcher indexSearcher &＃61; new IndexSearcher(indexReader);//创建查询条件对象,第一个参数为&＃xff1a;域名称 第二个参数为&＃xff1a;域值 这里查询条件为&＃xff1a;文件名中含有apache的文档Query query &＃61; new TermQuery(new Term("filename","apache"));//执行查询,第一个参数为&＃xff1a;查询条件 第二个参数为&＃xff1a;结果返回最大个数TopDocs topDocs &＃61; indexSearcher.search(query, 10);//打印结果集长度System.out.println("查询结果总条数&＃xff1a;" &＃43; topDocs.totalHits);//遍历结果集for (ScoreDoc doc:topDocs.scoreDocs) {//获取其查询到的文档对象,ScoreDoc对象的doc属性可以获取document的id值Document document &＃61; indexSearcher.doc(doc.doc);//打印文件名System.out.println("文件名&＃xff1a; " &＃43; document.get("filename"));//打印文件大小System.out.println("文件大小&＃xff1a;" &＃43; document.get("filesize"));//打印文件路径System.out.println("文件路径&＃xff1a;" &＃43; document.get("filepath"));//打印文件内容System.out.println(document.get("filecontent"));//分割线System.out.print("------------------------------------------------------------------------------");}//释放资源indexReader.close();}
}


创建索引库的结果&＃xff08;可使用工具查看索引库&＃xff09;&＃xff1a;

索引数据库的结果&＃xff1a;

查询结果总条数&＃xff1a;2
文件名&＃xff1a; apache lucene.txt
文件大小&＃xff1a;724
文件路径&＃xff1a;D:\lucene&solr\lucene\searchsource\apache lucene.txt
# Apache Lucene README file## IntroductionLucene is a Java full-text search engine. Lucene is not a complete
application, but rather a code library and API that can easily be used
to add search capabilities to applications.* The Lucene web site is at: http://lucene.apache.org/* Please join the Lucene-User mailing list by sending a message to:java-user-subscribe&＃64;lucene.apache.org## Files in a binary distributionFiles are organized by module, for example in core/:* &＃96;core/lucene-core-XX.jar&＃96;:The compiled core Lucene library.To review the documentation, read the main documentation page, located at:
&＃96;docs/index.html&＃96;To build Lucene or its documentation for a source distribution, see BUILD.txt------------------------------------------------------------------------------
文件名&＃xff1a; Welcome to the Apache Solr project.txt
文件大小&＃xff1a;5464
文件路径&＃xff1a;D:\lucene&solr\lucene\searchsource\Welcome to the Apache Solr project.txt
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.Welcome to the Apache Solr project!
-----------------------------------Solr is the popular, blazing fast open source enterprise search platform
from the Apache Lucene project.For a complete description of the Solr project, team composition, source
code repositories, and other details, please see the Solr web site at
http://lucene.apache.org/solrGetting Started
---------------See the "example" directory for an example Solr setup. A tutorial
using the example setup can be found athttp://lucene.apache.org/solr/tutorial.html
or linked from "docs/index.html" in a binary distribution.
Also, there are Solr clients for many programming languages, see http://wiki.apache.org/solr/IntegratingSolrFiles included in an Apache Solr binary distribution
----------------------------------------------------example/A self-contained example Solr instance, complete with a sampleconfiguration, documents to index, and the Jetty Servlet container.Please see example/README.txt for information about running thisexample.dist/solr-XX.warThe Apache Solr Application. Deploy this WAR file to any servletcontainer to run Apache Solr.dist/solr--XX.jarThe Apache Solr libraries. To compile Apache Solr Plugins,one or more of these will be required. The core library isrequired at a minimum. (see http://wiki.apache.org/solr/SolrPluginsfor more information).docs/index.htmlThe Apache Solr Javadoc API documentation and TutorialInstructions for Building Apache Solr from Source
-------------------------------------------------1. Download the Java SE 7 JDK (Java Development Kit) or later from http://java.sun.com/You will need the JDK installed, and the $JAVA_HOME/bin (Windows: %JAVA_HOME%\bin) folder included on your command path. To test this, issue a "java -version" command from your shell (command prompt) and verify that the Java version is 1.7 or later.2. Download the Apache Ant binary distribution (1.8.2&＃43;) from http://ant.apache.org/ You will need Ant installed and the $ANT_HOME/bin (Windows: %ANT_HOME%\bin) folder included on your command path. To test this, issue a "ant -version" command from your shell (command prompt) and verify that Ant is available. You will also need to install Apache Ivy binary distribution (2.2.0) from http://ant.apache.org/ivy/ and place ivy-2.2.0.jar file in ~/.ant/lib -- if you skip this step, the Solr build system will offer to do it for you.3. Download the Apache Solr distribution, linked from the above web site. Unzip the distribution to a folder of your choice, e.g. C:\solr or ~/solrAlternately, you can obtain a copy of the latest Apache Solr source codedirectly from the Subversion repository:http://lucene.apache.org/solr/versioncontrol.html4. Navigate to the "solr" folder and issue an "ant" command to see the available optionsfor building, testing, and packaging Solr.NOTE: To see Solr in action, you may want to use the "ant example" command to buildand package Solr into the example/webapps directory. See also example/README.txt.Export control
-------------------------------------------------
This distribution includes cryptographic software. The country in
which you currently reside may have restrictions on the import,
possession, use, and/or re-export to another country, of
encryption software. BEFORE using any encryption software, please
check your country&＃39;s laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software, to
see if this is permitted. See for more
information.The U.S. Government Department of Commerce, Bureau of Industry and
Security (BIS), has classified this software as Export Commodity
Control Number (ECCN) 5D002.C.1, which includes information security
software using or performing cryptographic functions with asymmetric
algorithms. The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception
ENC Technology Software Unrestricted (TSU) exception (see the BIS
Export Administration Regulations, Section 740.13) for both object
code and source code.The following provides more details on the included cryptographic
software:Apache Solr uses the Apache Tika which uses the Bouncy Castle generic encryption libraries forextracting text content and metadata from encrypted PDF files.See http://www.bouncycastle.org/ for more details on Bouncy Castle.------------------------------------------------------------------------------