最近需要学习Nutch，要搭建在Windows下平台并且需要对Nutch再次开发，方向是垂直搜索。初次接触这方面，搭建平台就问题不断，在百度和谷歌上辗转反侧找不到全面的答案，不得已回到Nutch主页去啃E文。这里翻译了一片使用Nutch的文档。
//需要学习NUTCH 网上的资料很杂不适合学习 所以翻译了这篇官方的文档。 英语很烂～不足之处请谅解。
Requirements
要求
1 Java 1.4.x, either from Sun or IBM on Linux is preferred. Set NUTCH_JAVA_HOME to the root of your JVM installation. Nutch 0.9 requires Sun JDK 1.5 or higher
2 Apache's Tomcat 5.x. or higher.
3 On Win32, cygwin, for shell support. (If you plan to use Subversion on Win32, be sure to select the subversion package when you install, in the "Devel" category.)
4 Up to a gigabyte of free disk space, a high-speed connection, and an hour or so.
这些就不翻译了大致需要 java 1.4.x ，tomcat 5.x, cygwin(如果你在Windows平台）。
Getting Started
(开始～～)
First, you need to get a copy of the Nutch code. You can download a release from http://www.apache.org/dyn/closer.cgi/nutch/. Unpack the release and connect to its top-level directory. Or, check out the latest source code from subversion and build it with Ant.
这也无需翻译了 主要是获得Nutch的代码。可以在这个网址上获得，解压，编译。
Try the following command:
完成上面的步骤后可以尝试运行这个目录bin下的nutch
bin/nutch
This will display the documentation for the Nutch command script.
会显示出关于nutch参数的帮助
Good! You are almost ready to crawl. You need to give your crawler a name. This is required.
如果你看到了参数的帮助 小小恭喜一下，你的JAVA_HOME 和 cygwin处于正常工作状态，同时你需要给你的小蜘蛛（crawler)一个你喜欢的名字。这是必须的～
5 Open up $NUTCH_HOME/conf/nutch-default.xml file
6 打开 你解压nutch的文件夹进入/conf/找到 nutch-default.xml 文件编辑它（推荐ultra edit）
Search for http.agent.name , and give it value 'YOURNAME Spider'
搜索 http.agent.name ,替换成你想要命名的小蜘蛛的名字。
Optionally you may also set http.agent.url and http.agent.email properties.
同样你需要设置 http.agent.url 和 http.agent.email
Now we're ready to crawl. There are two approaches to crawling:
好了现在你就可以开始爬行了，这里有两种方法可以驱动你的小蜘蛛爬行
Ps:Nutch的爬虫抓取网页有两种方式，一种方式是Intranet Crawling，针对的是企业内部网或少量网站，使用的是crawl命令；另一种方式是Whole-web crawling，针对的是整个互联网，使用inject、generate、fetch和updatedb等更底层的命令。
Using the crawl command to perform all the crawl steps with a single command. This is sometimes referred to as Intranet Crawling. Although a simple way to get started, it has limitations.
通过使用crawl的命令执行全部的爬行步骤。这种方式常常用在内网的爬行中，当然也是一种简单的方式，不过这种方式也有局限。
Using the lower level inject, generate, fetch and updatedb commands. Sometimes referred to as Whole-Web Crawling this allows you more control of each step of the process and is required to be able to update existing data.
使用低等级注入方式（说实话我没懂。）例如：generate fetch updatedb的命令。有时候也被称为Whole-web 爬行，它允许你对每一步拥有更多的控制并且可以更新数据。
The Crawl Command（无需翻译。。。）
The crawl command is more appropriate when you intend to crawl up to around one million pages on a handful of web servers.
Crawl命令在你需要爬行100万的页面在一半的web服务器上的时候是非常适合的。
Crawl Command: Configuration（配置）
To configure things for the crawl command you must:
配置你的命令前你需要做的：
Create a directory with a flat file of root urls. For example, to crawl the nutch site you might start with a file named urls/nutch containing the url of just the Nutch home page. All other Nutch pages should be reachable from this page. The urls/nutch file would thus contain: http://lucene.apache.org/nutch/
在/urls/下建立一个文件。例如：我们要爬行nutch网址，你需要在/urls/目录下建立一个nutch的文件包含有nutch网址的url(地址)。并且其他的nutch页面可以在这个地址上获得。这里nutch包含地址：http://lucene.apache.org/nutch/
Ps:在Windows下可以建立文本文档。注意地址结尾要有/
Edit the file conf/crawl-urlfilter.txt and replace MY.DOMAIN.NAME with the name of the domain you wish to crawl. For example, if you wished to limit the crawl to the apache.org domain, the line should read: +^http://([a-z0-9]*\.)*apache.org/ This will include any url in the domain apache.org.
编辑conf目录下crawl-urlfilter.txt文件，并且把MY.DOMAIN.NAME替换成你需要爬行的域名。例如：如果你希望仅仅在apache.org域名下爬行，那么就要更改为：+^http://([a-z0-9]*\.)*apache.org/ 表示会包含apache.org域名下的全部地址（url）
* Until someone could explain this...When I use the file crawl-urlfilter.txt the filter doesn't work, instead of it use the file conf/regex-urlfilter.txt and change the last line from "+." to "-."
我估计我遇不到这个问题所以不翻译。。。
Crawl Command: Running the Crawl（运行小蜘蛛～）
Once things are configured, running the crawl is easy. Just use the crawl command. Its options include:
一旦所有的设定都完成，开启小蜘蛛是一件相当简单的事情。仅仅使用crawl命令就可以做到。命令包含如下:
-dir dir names the directory to put the crawl in.
告诉小蜘蛛crawl在那个文件夹
-threads threads determines the number of threads that will fetch in parallel.
告诉小蜘蛛开启几个分身～～:-)
-depth depth indicates the link depth from the root page that should be crawled.
告诉小蜘蛛从根目录开始计算，最多爬行到那个深度（文件夹套文件夹的形式）
-topN N determines the maximum number of pages that will be retrieved at each level up to the depth.
告诉小蜘蛛在同一目录下检测文件的数目最大值
For example, a typical call might be:
例如:一种典型的调用
bin/nutch crawl urls -dir crawl -depth 3 -topN 50
Typically one starts testing one's configuration by crawling at shallow depths, sharply limiting the number of pages fetched at each level (-topN), and watching the output to check that desired pages are fetched and undesirable pages are not. Once one is confident of the configuration, then an appropriate depth for a full crawl is around 10. The number of pages per level (-topN) for a full crawl can be from tens of thousands to millions, depending on your resources.
一般来说通过配置可以控制爬行是浅层还是深度，通过设定（-topN）可以限制每一层的搜索文档的数目，查看输出的爬行结果来检查是不是需要的文件已经被爬行到了，不需要的没有被爬行到。一旦确定一个配置，在一个全爬行的配置中文件深度设定为10。-topN设置为全部的时候，可以产生数百万到千万的页面，这要依据你的资源情况。
Once crawling has completed, one can skip to the Searching section below.
一旦爬行完成，你可以跳到搜索参考那里继续阅读。
Step-by-Step or Whole-web Crawling（循序渐进的或Whole-web的爬行）
Whole-web crawling is designed to handle very large crawls which may take weeks to complete, running on multiple machines. This also permits more control over the crawl process, and incremental crawling. It is important to note that whole web crawling does not necessarily mean crawling the entire world wide web. We can limit a whole web crawl to just a list of the URLs we want to crawl. This is done by using a filter just like we the one we used when we did the crawl command (above).
Whole-web被设计用来处理非常大的爬行任务它们通常需要几周的时间才能完成，并且运行在多台机器上。并且允许控制增加更多的爬行进程。值得注意的是whole-web并不意味着爬行整个万维网。我们可以限制一个完整的网站哪些URL是我们需要爬行的。我们通过使用过滤命令正如我们之前所使用的方式达到目的。
Step-by-Step: Concepts （循序渐进：概念）
Nutch data is composed of:
Nutch 的数据由这些组成
The crawl database, or crawldb. This contains information about every url known to Nutch, including whether it was fetched, and, if so, when.
Crawl database 或 crawldb.它们包含这Nutch知道的所有URL，包括是否爬行，什么时候爬行的信息。
1 The link database, or linkdb. This contains the list of known links to each url, including both the source url and anchor text of the link.
2 Link database 或 linkdb.包含着每条URL所拥有的链接，包括源url和锚文本的连接。
3 A set of segments. Each segment is a set of urls that are fetched as a unit. Segments are directories with the following subdirectories:a crawl_generate names a set of urls to be fetched
一种段的集合：每一种段是一种以单元的模式爬行的URLS的集合。段目录与下面的子目录:crawl_generate一类被爬行的url
a crawl_fetch contains the status of fetching each url
段：crawl_fetch 包含着每一个爬行URL的状态
a content contains the raw content retrieved from each url
Content 包含着每个url的原始内容
a parse_text contains the parsed text of each url
Parse_text包含着每个URL解析后的文本
a parse_data contains outlinks and metadata parsed from each url
Parse_data 包含着从每个URL解析出来的外部链接和数据
a crawl_parse contains the outlink urls, used to update the crawldb
Crawl_parse 包含outlink url,用来更新crawldb
4 The indexes are Lucene-format indexes.
索引是 Lucene-format索引。
Step-by-Step: Seeding the Crawl DB with a list of URLS
设定CrawlDB为一个URL的列表
Option 1: Bootstrapping from the DMOZ database.
选项1：通过DMOZ数据库获得爬行目标
The injector adds urls to the crawldb. Let's inject URLs from the DMOZ Open Directory. First we must download and uncompress the file listing all of the DMOZ pages. (This is a 200+Mb file, so this will take a few minutes.)
链接器把URLS增加到crawldb。我们从DMOZ开发数据库注入URL。首先我们必须下载解压文件列出所有的DMOZ页。（差不多200MB的文件 这需要一些时间）
wget http://rdf.dmoz.org/rdf/content.rdf.u8.gz
gunzip content.rdf.u8.gz
Next we select a random subset of these pages. (We use a random subset so that everyone who runs this tutorial doesn't hammer the same sites.) DMOZ contains around three million URLs. We select one out of every 5000, so that we end up with around 1000 URLs:
接下来我们挑选任意部分页。（我们使用随机这样任何人运行这篇教程的时候不会伤害到同样的服务器（防止DDOS））DMOZ包含差不多300万个URLs.每5000个中我们选择一个，所以我们最终选择了1000个URLs
mkdir dmoz
bin/nutch org.apache.nutch.tools.DmozParser content.rdf.u8 -subset 5000 > dmoz/urls
The parser also takes a few minutes, as it must parse the full file. Finally, we initialize the crawl db with the selected urls.
分析器也需要几分钟,它必须如此分析完整的文件。最后,我们初始化了crawl db与选择的网址。
bin/nutch inject crawl/crawldb dmoz
Now we have a web database with around 1000 as-yet unfetched URLs in it.
现在我们有了一个web数据库,大约有1000仍还未爬行的url。
Option 2. Bootstrapping from an initial seed list.
选项2：手工初始化爬行目标
Instead of bootstrapping from DMOZ, we can create a text file called urls, this file should have one url per line. We can initialize the crawl db with the selected urls.
我们也可以用这种方法代替DMOZ获得目标的方法：我们建立一个文件名为urls,文件中每个url用一行表示，我们可以初始化crawl db通过这些我们说选择的urls
bin/nutch inject crawl/crawldb urls
NOTE: version 0.8 and higher requires that we put this file into a subdirectory, e.g. seed/urls - in this case the command looks like this:
注:版本0.8和更高的要求,我们把这文件分成一个子目录,如种子/ url——在这种情况下命令看起来像这样:
bin/nutch inject crawl/crawldb seed
Step-by-Step: Fetching（爬行ing~）
To fetch, we first generate a fetchlist from the database:
为了爬行，我们首先要从database生成一个爬行列表
bin/nutch generate crawl/crawldb crawl/segments
This generates a fetchlist for all of the pages due to be fetched. The fetchlist is placed in a newly created segment directory. The segment directory is named by the time it's created. We save the name of this segment in the shell variable s1:
命令生成了一个存放了所有需要被爬行的页面的列表。这个列表（fetchlist）被放置在一个新生成的段目录里。这个段目录但创建的时候就被命名了。我们保持这个段的名字是shell传递的参数s1。
s1=`ls -d crawl/segments/2* | tail -1`
echo $s1
Now we run the fetcher on this segment with:
接下来我们爬行这一段的fetchlist列表
bin/nutch fetch $s1
When this is complete, we update the database with the results of the fetch:
但这完成的时候，我们更新爬行的结果到database
bin/nutch updatedb crawl/crawldb $s1
Now the database contains both updated entries for all initial pages as well as new entries that correspond to newly discovered pages linked from the initial set.
现在database包含了初始化的页面的更新记录 和 从最初页面爬行后新发现的记录。
Now we generate and fetch a new segment containing the top-scoring 1000 pages:
现在我们生成和爬行了新的段差不多1000个初始页面。
bin/nutch generate crawl/crawldb crawl/segments -topN 1000
s2=`ls -d crawl/segments/2* | tail -1`
echo $s2
bin/nutch fetch $s2
bin/nutch updatedb crawl/crawldb $s2
Let's fetch one more round:
让我们多爬行一轮吧～
bin/nutch generate crawl/crawldb crawl/segments -topN 1000
s3=`ls -d crawl/segments/2* | tail -1`
echo $s3
bin/nutch fetch $s3
bin/nutch updatedb crawl/crawldb $s3
By this point we've fetched a few thousand pages. Let's index them!
通过这种方式我们爬行了几千个页面。让我们给他创建索引吧
Step-by-Step: Indexing（创建索引）
Before indexing we first invert all of the links, so that we may index incoming anchor text with the pages.
在转换前，我们需要反选所有链接，使我们可以用页面的方式索引锚文本。
bin/nutch invertlinks crawl/linkdb -dir crawl/segments
NOTE: the invertlinks command only applies to Nutch 0.8 and higher.
注意:invertlinks命令只适用于Nutch 0.8和更高版本。
To index the segments we use the index command, as follows:
为段建立索引，我们通过以下命令:
bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
Now we're ready to search!
现在我们可以搜索!
Command Line Searching
Simplest way to verify the integrity of your crawl is to launch NutchBean from command line:
最简单的办法来验证的完整性是推进新一轮的爬行NutchBean从命令行:
bin/nutch org.apache.nutch.searcher.NutchBean apache
where apache is the search term (note that NutchBean will only search pages in the crawl directory, so if you named the crawl directory something else,NutchBean will not find any results). After you have verified that the above command returns results you can proceed to setting up the web interface.
Apache是一个搜索词（注意NutchBean只会在搜索在crawl目录下的页面，因此,如果你给爬目录别的东西,NutchBean找不到任何结果）在你证实了上面的操作返回结果你可以按照原定计划开始假设web界面
Installing in Tomcat（在Tomacat下安装）
To search you need to put the nutch war file into your servlet container. (If instead of downloading a Nutch release you checked the sources out of SVN, then you'll first need to build the war file, with the command ant war.)
你需要找到nutch的war文件放到你的服务器目录下（如果你是通过SVN下载nutch发现的代码，那么你需要编译建立war文件，通过命令:ant war）
Assuming you've unpacked Tomcat as ~/local/tomcat, then the Nutch war file may be installed with the commands:
假设你解压Tomcat到~/local/tomcat文件夹内，那么安装Nutch war文件需要如下命令:
mkdir ~/local/tomcat/webapps/nutch
cp nutch*.war ~/local/tomcat/webapps/nutch/
jar xvf ~/local/tomcat/webapps/nutch/nutch-1.1.war
rm nutch-1.1.war;
The webapp finds its indexes in ./crawl, relative to where you start Tomcat, so use a command like:
~/local/tomcat/bin/catalina.sh start
Then visit（访问）: http://localhost:8080/nutch/
后记：如开头所说为了方便学习nutch而着手翻译本文的。翻译过程中为了图方便借助了翻译软件。能力有限，错误之处还请见谅。
E-mail:guhuatian<a>gmail.com