什么是一个很好的Web爬虫工具-What'sagoodWebCrawlertool

12  

HTTrack -- http://www.httrack.com/ -- is a very good Website copier. Works pretty good. Have been using it for a long time.

HTTrack - http://www.httrack.com/ - 是一款非常好的网站复印机。工作得很好。已经使用了很长时间了。

Nutch is a web crawler(crawler is the type of program you're looking for) -- http://lucene.apache.org/nutch/ -- which uses a top notch search utility lucene.

Nutch是一个网络爬虫(爬虫是你正在寻找的程序类型) - http://lucene.apache.org/nutch/ - 它使用顶级搜索实用程序lucene。

4  

Crawler4j is an open source Java crawler which provides a simple interface for crawling the Web. You can setup a multi-threaded web crawler in 5 minutes.

Crawler4j是一个开源Java爬虫,它为爬网提供了一个简单的界面。您可以在5分钟内设置多线程Web爬网程序。

You can set your own filter to visit pages or not (urls) and define some operation for each crawled page according to your logic.

您可以设置自己的过滤器以访问页面(URL),并根据您的逻辑为每个已爬网页面定义一些操作。

Some reasons to select crawler4j;

选择crawler4j的一些原因;

1. Multi-Threaded Structure,
2. You can Set Depth to be crawled,

您可以设置深度以进行爬网,

3. It is Java Based and open source,

它是基于Java和开源的,

4. Control for redundant links (urls),

控制冗余链接(网址),

5. You can set number of pages to be crawled,

您可以设置要爬网的页数,

6. You can set page size to be crawled,

您可以设置要抓取的页面大小,

7. Enough documentation

2  

Searcharoo.NET contains a spider that crawls and indexes content, and a search engine to use it. You should be able to find your way around the Searcharoo.Indexer.EXE code to trap the content as it's downloaded, and add your own custom code from there...

Searcharoo.NET包含一个爬行和索引内容的蜘蛛,以及一个使用它的搜索引擎。您应该能够找到绕过Searcharoo.Indexer.EXE代码的方法,以便在下载内容时捕获内容,并从那里添加您自己的自定义代码...

It's very basic (all the source code is included, and is explained in six CodeProject articles, the most recent of which is here Searcharoo v6): the spider follows links, imagemaps, images, obeys ROBOTS directives, parses some non-HTML file types. It is intended for single websites (not the entire web).

这是非常基本的(所有源代码都包含在内,并在六篇CodeProject文章中进行了解释,其中最新的是Searcharoo v6):蜘蛛遵循链接,图像映射,图像,服从ROBOTS指令,解析一些非HTML文件类型。它适用于单个网站(不是整个网站)。

Nutch/Lucene is almost certainly a more robust/commercial-grade solution - but I have not looked at their code. Not sure what you are wanting to accomplish, but have you also seen Microsoft Search Server Express?

Nutch / Lucene几乎肯定是一个更强大/商业级的解决方案 - 但我还没有看过他们的代码。不确定你想要完成什么,但你还看过Microsoft Search Server Express吗?

Disclaimer: I am the author of Searcharoo; just offering it here as an option.

免责声明:我是Searcharoo的作者;只是在这里提供它作为一种选择。

1  

Sphider is pretty good. It's PHP, but it might be of some help.

斯弗德很不错。这是PHP,但它可能会有所帮助。

1  

I use Mozenda's Web Scraping software. You could easily have it crawl all of the links and grab all of the information you need and it's a great software for the money.

我使用Mozenda的Web Scraping软件。您可以轻松地抓取所有链接并获取所需的所有信息,这是一个很棒的软件。

0  

I haven't used this yet, but this looks interesting. The author wrote it from scratch and posted how he did. The code for it is available for download as well.

我还没用过这个,但这看起来很有趣。作者从头开始编写并发布了他的做法。它的代码也可以下载。