|
Nutch相关框架安装使用最佳指南（原创）
一、nutch1.2
步骤和二大同小异，在步骤 5、配置构建路径 中需要多两个操作：在左部Package Explorer的 nutch1.2文件夹上单击右键 > Build Path > Configure Build Path... > 选中Source选项 > Default output folder:修改nutch1.2/bin为nutch1.2/_bin，在左部Package Explorer的 nutch1.2文件夹下的bin文件夹上单击右键 > Team > 还原 二中黄色背景部分是版本号的差异，红色部分是1.2版本没有的，绿色部分是不一样的地方，如下： 1、Add JARs... > nutch1.2 > lib ，选中所有的.jar文件 > OK 2、crawl-urlfilter.txt 3、将crawl -urlfilter.txt.template改名为crawl -urlfilter.txt 4、修改crawl-urlfilter.txt，将 # accept hosts in MY.DOMAIN.NAME +^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/ # skip everything else
-. 5、cd /home/ysc/workspace/nutch1.2 nutch1.2是一个完整的搜索引擎，nutch1.5.1只是一个爬虫。nutch1.2可以把索引提交给SOLR，也可以直接生成LUCENE索引，nutch1.5.1则只能把索引提交给SOLR： 1、cd /home/ysc 2、wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-7/v7.0.29/bin/apache-tomcat-7.0.29.tar.gz 3、tar -xvf apache-tomcat-7.0.29.tar.gz 4、在左部Package Explorer的 nutch1.2文件夹下的build.xml文件上单击右键 > Run As > Ant Build... > 选中war target > Run 5、cd /home/ysc/workspace/nutch1.2/build 6、unzip nutch-1.2.war -d nutch-1.2 7、cp -r nutch-1.2 /home/ysc/apache-tomcat-7.0.29/webapps 8、vi /home/ysc/apache-tomcat-7.0.29/webapps/nutch-1.2/WEB-INF/classes/nutch-site.xml 加入以下配置： <property> <name>searcher.dir</name> <value>/home/ysc/workspace/nutch1.2/data</value> <description> Path to root of crawl. This directory is searched (in order) for either the file search-servers.txt, containing a list of distributed search servers, or the directory "index" containing merged indexes, or the directory "segments" containing segment indexes. </description> </property> 9、vi /home/ysc/apache-tomcat-7.0.29/conf/server.xml 将 <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"/> 改为 <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="utf-8"/> 关于nutch1.2更多的BUG修复及资料，请参看我在CSDN发布的资源：http://download.csdn.net/user/yangshangchuan
二、nutch1.5.1
1、下载并解压eclipse（集成开发环境） 下载地址：http://www.eclipse.org/downloads/，下载Eclipse IDE for Java EE Developers 2、安装Subclipse插件（SVN客户端） 插件地址：http://subclipse.tigris.org/update_1.8.x， 3、安装IvyDE插件（下载依赖Jar） 插件地址：http://www.apache.org/dist/ant/ivyde/updatesite/ 4、签出代码 File > New > Project > SVN > 从SVN 检出项目 创建新的资源库位置 > URL：https://svn.apache.org/repos/asf/nutch/tags/release-1.5.1/ > 选中URL > Finish 弹出New Project向导，选择Java Project > Next，输入Project name：nutch1.5.1 > Finish 5、配置构建路径 在左部Package Explorer的 nutch1.5.1文件夹上单击右键 > Build Path > Configure Build Path... > 选中Source选项 > 选择src > Remove > Add Folder... > 选择src/bin, src/java, src/test 和 src/testresources（对于插件，需要选中src/plugin目录下的每一个插件目录下的src/java ， src/test文件夹） > OK 切换到Libraries选项 > Add Class Folder... > 选中nutch1.5.1/conf > OK Add JARs... > 需要选中src/plugin目录下的每一个插件目录下的lib目录下的jar文件 > OK Add Library... > IvyDE Managed Dependencies > Next > Main > Ivy File > Browse > ivy/ivy.xml > Finish 切换到Order and Export选项> 选中conf > Top 6、执行ANT 在左部Package Explorer的 nutch1.5.1文件夹下的build.xml文件上单击右键 > Run As > Ant Build 在左部Package Explorer的 nutch1.5.1文件夹上单击右键 > Refresh 在左部Package Explorer的 nutch1.5.1文件夹上单击右键 > Build Path > Configure Build Path... > 选中Libraries选项 > Add Class Folder... > 选中build > OK 7、修改配置文件nutch-site.xml 和regex-urlfilter.txt 将nutch-site.xml.template改名为nutch-site.xml 将regex-urlfilter.txt.template改名为regex-urlfilter.txt 在左部Package Explorer的 nutch1.5.1文件夹上单击右键 > Refresh 将如下配置项加入文件nutch-site.xml： <property> <name>http.agent.name</name> <value>nutch</value> </property> <property> <name>http.content.limit</name> <value>-1</value> </property> 修改regex-urlfilter.txt，将 # accept anything else +. 替换为： +^http://([a-z0-9]*\.)*news.163.com/ -. 8、开发调试 在左部Package Explorer的 nutch1.5.1文件夹上单击右键 > New > Folder > Folder name: urls 在刚新建的urls目录下新建一个文本文件url，文本内容为：http://news.163.com 打开src/java下的org.apache.nutch.crawl.Crawl.java类，单击右键Run As > Run Configurations > Arguments > 在Program arguments输入框中输入: urls -dir data -depth 3 > Run 在需要调试的地方打上断点Debug As > Java Applicaton 9、查看结果 查看segments目录： 打开src/java下的org.apache.nutch.segment.SegmentReader.java类 单击右键Run As > Java Applicaton，控制台会输出该命令的使用方法 单击右键Run As > Run Configurations > Arguments > 在Program arguments输入框中输入: -dump data/segments/* data/segments/dump 用文本编辑器打开文件data/segments/dump/dump查看segments中存储的信息 查看crawldb目录：
打开src/java下的org.apache.nutch.crawl.CrawlDbReader.java类 单击右键Run As > Java Applicaton，控制台会输出该命令的使用方法 单击右键Run As > Run Configurations > Arguments > 在Program arguments输入框中输入: data/crawldb -stats 控制台会输出 crawldb统计信息 查看linkdb目录：
打开src/java下的org.apache.nutch.crawl.LinkDbReader.java类 单击右键Run As > Java Applicaton，控制台会输出该命令的使用方法 单击右键Run As > Run Configurations > Arguments > 在Program arguments输入框中输入: data/linkdb -dump data/linkdb_dump 用文本编辑器打开文件data/linkdb_dump/part-00000查看linkdb中存储的信息 10、全网分步骤抓取 在左部Package Explorer的 nutch1.5.1文件夹下的build.xml文件上单击右键 > Run As > Ant Build cd /home/ysc/workspace/nutch1.5.1/runtime/local #准备URL列表 wget http://rdf.dmoz.org/rdf/content.rdf.u8.gz gunzip content.rdf.u8.gz mkdir dmoz bin/nutch org.apache.nutch.tools.DmozParser content.rdf.u8 -subset 5000 > dmoz/url #注入URL bin/nutch inject crawl/crawldb dmoz #生成抓取列表 bin/nutch generate crawl/crawldb crawl/segments #第一次抓取 s1=`ls -d crawl/segments/2* | tail -1` echo $s1 #抓取网页 bin/nutch fetch $s1 #解析网页 bin/nutch parse $s1 #更新URL状态 bin/nutch updatedb crawl/crawldb $s1 #第二次抓取 bin/nutch generate crawl/crawldb crawl/segments -topN 1000 s2=`ls -d crawl/segments/2* | tail -1` echo $s2 bin/nutch fetch $s2 bin/nutch parse $s2 bin/nutch updatedb crawl/crawldb $s2 #第三次抓取 bin/nutch generate crawl/crawldb crawl/segments -topN 1000 s3=`ls -d crawl/segments/2* | tail -1` echo $s3 bin/nutch fetch $s3 bin/nutch parse $s3 bin/nutch updatedb crawl/crawldb $s3 #生成反向链接库 bin/nutch invertlinks crawl/linkdb -dir crawl/segments 11、索引和搜索
cd /home/ysc/ wget http://mirror.bjtu.edu.cn/apache/lucene/solr/3.6.1/apache-solr-3.6.1.tgz tar -xvf apache-solr-3.6.1.tgz cd apache-solr-3.6.1 /example NUTCH_RUNTIME_HOME=/home/ysc/workspace/nutch1.5.1/runtime/local APACHE_SOLR_HOME=/home/ysc/apache-solr-3.6.1 cp ${NUTCH_RUNTIME_HOME}/conf/schema.xml ${APACHE_SOLR_HOME}/example/solr/conf/
如果需要把网页内容存储到索引中，则修改 schema.xml文件中的 <field name="content" type="text" stored="false" indexed="true"/> 为 <field name="content" type="text" stored="true" indexed="true"/> 修改${APACHE_SOLR_HOME}/example/solr/conf/solrconfig.xml,将里面的<str name="df">text</str>都替换为<str name="df">content</str>
把${APACHE_SOLR_HOME}/example/solr/conf/schema.xml中的 <schema name="nutch" version="1.5.1">修改为<schema name="nutch" version="1.5">
#启动SOLR服务器 java -jar start.jar cd /home/ysc/workspace/nutch1.5.1/runtime/local
#提交索引 bin/nutch solrindex http://127.0.0.1:8983/solr/ crawl/crawldb -linkdb crawl/linkdb crawl/segments/* 使用以下命令分页查看所有索引的文档：
http://127.0.0.1:8983/solr/select/?q=*%3A*&version=2.2&start=0&rows=10&indent=on 标题包含“网易”的文档： http://127.0.0.1:8983/solr/select/?q=title%3A%E7%BD%91%E6%98%93&version=2.2&start=0&rows=10&indent=on 12、查看索引信息
cd /home/ysc/ wget http://luke.googlecode.com/files/lukeall-3.5.0.jar java -jar lukeall-3.5.0.jar Path: /home/ysc/apache-solr-3.6.1/example/solr/data 13、配置SOLR的中文分词
cd /home/ysc/ wget http://mmseg4j.googlecode.com/files/mmseg4j-1.8.5.zip unzip mmseg4j-1.8.5.zip -d mmseg4j-1.8.5 APACHE_SOLR_HOME=/home/ysc/apache-solr-3.6.1 mkdir $APACHE_SOLR_HOME/example/solr/lib mkdir $APACHE_SOLR_HOME/example/solr/dic cp mmseg4j-1.8.5/mmseg4j-all-1.8.5.jar $APACHE_SOLR_HOME/example/solr/lib cp mmseg4j-1.8.5/data/*.dic $APACHE_SOLR_HOME/example/solr/dic 将${APACHE_SOLR_HOME}/example/solr/conf/schema.xml文件中的 <tokenizer class="solr.WhitespaceTokenizerFactory"/> 和 <tokenizer class="solr.StandardTokenizerFactory"/> 替换为 <tokenizer class="com.chenlb.mmseg4j.solr.MMSegTokenizerFactory" mode="complex" dicPath="/home/ysc/apache-solr-3.6.1/example/solr/dic"/> #重新启动SOLR服务器 java -jar start.jar #重建索引，演示在开发环境中如何操作
打开src/java下的org.apache.nutch.indexer.solr.SolrIndexer.java类 单击右键Run As > Java Applicaton，控制台会输出该命令的使用方法 单击右键Run As > Run Configurations > Arguments > 在Program arguments输入框中输入: http://127.0.0.1:8983/solr/ ; data/crawldb -linkdb data/linkdb data/segments/* 使用luke重新打开索引就会发现分词起作用了 三、nutch2.0
nutch2.0和二中的nutch1.5.1的步骤相同，但在8、开发调试之前需要做以下配置： 在左部Package Explorer的 nutch2.0文件夹上单击右键 > New > Folder > Folder name: data并指定数据存储方式，选如下之一： 1、使用mysql作为数据存储 1）、在nutch2.0/conf/nutch-site.xml中加入如下配置： <property> <name>storage.data.store.class</name> <value>org.apache.gora.sql.store.SqlStore</value> </property> 2）、将nutch2.0/conf/gora.properties文件中的 gora.sqlstore.jdbc.driver=org.hsqldb.jdbc.JDBCDriver gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest gora.sqlstore.jdbc.user=sa gora.sqlstore.jdbc.password= 修改为 gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver gora.sqlstore.jdbc.url=jdbc:mysql://127.0.0.1:3306/nutch2 gora.sqlstore.jdbc.user=root gora.sqlstore.jdbc.password=ROOT 3）、打开nutch2.0/ivy/ivy.xml中的mysql-connector-java依赖 4）、sudo apt-get install mysql-server 2、使用hbase作为数据存储 1）、在nutch2.0/conf/nutch-site.xml中加入如下配置： <property> <name>storage.data.store.class</name> <value>org.apache.gora.hbase.store.HBaseStore</value> </property> 2）、打开nutch2.0/ivy/ivy.xml中的gora-hbase依赖 3）、cd /home/ysc 4）、wget http://mirror.bit.edu.cn/apache/hbase/hbase-0.90.5/hbase-0.90.5.tar.gz 5）、tar -xvf hbase-0.90.5.tar.gz 6）、vi hbase-0.90.5/conf/hbase-site.xml 加入以下配置： <property> <name>hbase.rootdir</name> <value>file:///home/ysc/hbase-0.90.5-database</value> </property> 7)、hbase-0.90.5/bin/start-hbase.sh 8)、将/home/ysc/hbase-0.90.5/hbase-0.90.5.jar加入开发环境eclipse的build path 四、配置SSH
三台机器 devcluster01， devcluster02， devcluster03，分别在每一台机器上面执行如下操作： 1、sudo vi /etc/hosts 加入以下配置： 192.168.1.1 devcluster01 192.168.1.2 devcluster02 192.168.1.3 devcluster03 2、安装SSH服务： sudo apt-get install openssh-server 3、(有提示的时候回车键确认） ssh-keygen -t rsa 该命令会在用户主目录下创建 .ssh 目录，并在其中创建两个文件：id_rsa 私钥文件。是基于 RSA 算法创建。该私钥文件要妥善保管，不要泄漏。id_rsa.pub 公钥文件。和 id_rsa 文件是一对儿，该文件作为公钥文件，可以公开。 4、cp .ssh/id_rsa.pub .ssh/authorized_keys 把 三台机器 devcluster01， devcluster02， devcluster03 的文件/home/ysc/.ssh/authorized_keys的内容复制出来合并成一个文件并替换每一台机器上的/home/ysc/.ssh/authorized_keys文件 在devcluster01上面执行时，以下两条命令的主机为02和03 在devcluster02上面执行时，以下两条命令的主机为01和03 在devcluster03上面执行时，以下两条命令的主机为01和02 5、ssh-copy-id -i .ssh/id_rsa.pub ysc@ devcluster02 6、ssh-copy-id -i .ssh/id_rsa.pub ysc@ devcluster03 以上两条命令实际上是将 .ssh/id_rsa.pub 公钥文件追加到远程主机 server 的 user 主目录下的 .ssh/authorized_keys 文件中。 五、安装Hadoop Cluster（伪分布式运行模式）并运行Nutch
步骤和四大同小异，只需要1台机器 devcluster01，所以黄色背景部分全部设置为devcluster01，不需要第11步 六、安装Hadoop Cluster（分布式运行模式）并运行Nutch
三台机器 devcluster01， devcluster02， devcluster03(vi /etc/hostname) 使用用户ysc登陆 devcluster01： 1、cd /home/ysc 2、wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-1.1.1/hadoop-1.1.1-bin.tar.gz 3、tar -xvf hadoop-1.1.1-bin.tar.gz 4、cd hadoop-1.1.1 5、vi conf/masters 替换内容为 ： devcluster01 6、vi conf/slaves 替换内容为 ： devcluster02 devcluster03 7、vi conf/core-site.xml 加入配置： <property> <name>fs.default.name</name> <value>hdfs://devcluster01:9000</value> <description> Where to find the Hadoop Filesystem through the network. Note 9000 is not the default port. (This is slightly changed from previous versions which didnt have "hdfs") </description> </property> <property> <name>hadoop.security.authorization</name> <value>true</value> </property> 编辑conf/hadoop-policy.xml 8、vi conf/hdfs-site.xml 加入配置： <property> <name>dfs.name.dir</name> <value>/home/ysc/dfs/filesystem/name</value> </property> <property>
<name>dfs.data.dir</name> <value>/home/ysc/dfs/filesystem/data</value> </property> <property>
<name>dfs.replication</name> <value>1</value> </property> <property>
<name>dfs.block.size</name> <value>671088640</value> <description>The default block size for new files.</description> </property> 9、vi conf/mapred-site.xml 加入配置： <property> <name>mapred.job.tracker</name> <value>devcluster01:9001</value> <description> The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. Note 9001 is not the default port. </description> </property> <property>
<name>mapred.reduce.tasks.speculative.execution</name> <value>false</value> <description>If true, then multiple instances of some reduce tasks may be executed in parallel.</description> </property> <property>
<name>mapred.map.tasks.speculative.execution</name> <value>false</value> <description>If true, then multiple instances of some map tasks may be executed in parallel.</description> </property> <property>
<name>mapred.child.java.opts</name> <value>-Xmx2000m</value> </property> <property>
<name>mapred.tasktracker.map.tasks.maximum</name> <value>4</value> <description> the core number of host </description> </property> <property>
<name>mapred.map.tasks</name> <value>4</value> </property> <property>
<name>mapred.tasktracker.reduce.tasks.maximum</name> <value>4</value> <description> define mapred.map tasks to be number of slave hosts.the best number is the number of slave hosts plus the core numbers of per host </description> </property> <property>
<name>mapred.reduce.tasks</name> <value>4</value> <description> define mapred.reduce tasks to be number of slave hosts.the best number is the number of slave hosts plus the core numbers of per host </description> </property> <property>
<name>mapred.output.compression.type</name> <value>BLOCK</value> <description>If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. </description> </property> <property>
<name>mapred.output.compress</name> <value>true</value> <description>Should the job outputs be compressed? </description> </property> <property>
<name>mapred.compress.map.output</name> <value>true</value> <description>Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. </description> </property> <property>
<name>mapred.system.dir</name> <value>/home/ysc/mapreduce/system</value> </property> <property>
<name>mapred.local.dir</name> <value>/home/ysc/mapreduce/local</value> </property> 10、vi conf/hadoop-env.sh 追加： export JAVA_HOME=/home/ysc/jdk1.7.0_05 export HADOOP_HEAPSIZE=2000 #替换掉默认的垃圾回收器，因为默认的垃圾回收器在多线程环境下会有更多的wait等待 export HADOOP_OPTS="-server -Xmn256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70" 11、复制HADOOP文件 scp -r /home/ysc/hadoop-1.1.1 ysc@devcluster02:/home/ysc/hadoop-1.1.1 scp -r /home/ysc/hadoop-1.1.1 ysc@devcluster03:/home/ysc/hadoop-1.1.1 12、sudo vi /etc/profile 追加并重启系统： export PATH=/home/ysc/hadoop-1.1.1/bin:$PATH 13、格式化名称节点并启动集群 hadoop namenode -format start-all.sh 14、cd /home/ysc/workspace/nutch1.5.1/runtime/deploy mkdir urls echo http://news.163.com > urls/url hadoop dfs -put urls urls bin/nutch crawl urls -dir data -depth 2 -topN 100 15、访问 http://localhost:50030 可以查看 JobTracker 的运行状态。访问 http://localhost:50060 可以查看 TaskTracker 的运行状态。访问 http://localhost:50070 可以查看 NameNode 以及整个分布式文件系统的状态，浏览分布式文件系统中的文件以及 log 等 16、通过stop-all.sh停止集群 17、如果NameNode和SecondaryNameNode不在同一台机器上，则在SecondaryNameNode的conf/hdfs-site.xml文件中加入配置： <property> <name>dfs.http.address</name> <value>namenode:50070</value> </property> 七、配置Ganglia监控Hadoop集群和HBase集群
1、服务器端（安装到master devcluster01上） 1）、ssh devcluster01 2）、useradd ganglia -g ganglia 3）、sudo apt-get install ganglia-monitor ganglia-webfront gmetad //补充：在Ubuntu10.04上，ganglia-webfront这个package名字叫ganglia-webfrontend //如果install出错，则运行sudo apt-get update，如果update出错，则删除出错路径 4）、vi /etc/ganglia/gmond.conf 先找到setuid = yes,改成setuid =no; 在找到cluster块中的name，改成name =”hadoop-cluster”; 5）、sudo apt-get install rrdtool 6)、vi /etc/ganglia/gmetad.conf 在这个配置文件中增加一些datasource，即其他2个被监控的节点，增加以下内容： data_source “hadoop-cluster” devcluster01:8649 devcluster02:8649 devcluster03:8649 gridname "Hadoop" 2、数据源端（安装到所有slaves上） 1)、ssh devcluster02 useradd ganglia -g ganglia sudo apt-get install ganglia-monitor useradd ganglia -g ganglia 2)、ssh devcluster03 useradd ganglia -g ganglia sudo apt-get install ganglia-monitor useradd ganglia -g ganglia 3）、ssh devcluster01 scp /etc/ganglia/gmond.conf devcluster02:/etc/ganglia/gmond.conf scp /etc/ganglia/gmond.conf devcluster03:/etc/ganglia/gmond.conf 3、配置WEB 1）、ssh devcluster01 2）、sudo ln -s /usr/share/ganglia-webfrontend /var/www/ganglia 3）、vi /etc/apache2/apache2.conf 添加： ServerName devcluster01 4、重启服务 1）、ssh devcluster02 sudo /etc/init.d/ganglia-monitor restart ssh devcluster03 sudo /etc/init.d/ganglia-monitor restart 2）、ssh devcluster01 sudo /etc/init.d/ganglia-monitor restart sudo /etc/init.d/gmetad restart sudo /etc/init.d/apache2 restart 5、访问页面 http:// devcluster01/ganglia 6、集成hadoop 1）、ssh devcluster01 2）、cd /home/ysc/hadoop-1.1.1 3）、vi conf/hadoop-metrics2.properties # 大于0.20以后的版本用ganglia31 *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31 *.sink.ganglia.period=10 # default for supportsparse is false *.sink.ganglia.supportsparse=true *.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both *.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40 #广播IP地址，这是缺省的，统一设该值(只能用组播地址239.2.11.71) namenode.sink.ganglia.servers=239.2.11.71:8649 datanode.sink.ganglia.servers=239.2.11.71:8649 jobtracker.sink.ganglia.servers=239.2.11.71:8649 tasktracker.sink.ganglia.servers=239.2.11.71:8649 maptask.sink.ganglia.servers=239.2.11.71:8649 reducetask.sink.ganglia.servers=239.2.11.71:8649 dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 dfs.period=10 dfs.servers=239.2.11.71:8649 mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 mapred.period=10 mapred.servers=239.2.11.71:8649 jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 jvm.period=10 jvm.servers=239.2.11.71:8649 4）、scp conf/hadoop-metrics2.properties root@devcluster02:/home/ysc/hadoop-1.1.1/conf/hadoop-metrics2.properties 5）、scp conf/hadoop-metrics2.properties root@devcluster03:/home/ysc/hadoop-1.1.1/conf/hadoop-metrics2.properties 6）、stop-all.sh 7）、start-all.sh 7、集成hbase 1）、ssh devcluster01 2）、cd /home/ysc/hbase-0.92.2 3）、vi conf/hadoop-metrics.properties(只能用组播地址239.2.11.71) hbase.extendedperiod = 3600 hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 hbase.period=10 hbase.servers=239.2.11.71:8649 jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 jvm.period=10 jvm.servers=239.2.11.71:8649 rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 rpc.period=10 rpc.servers=239.2.11.71:8649 4）、scp conf/hadoop-metrics.properties root@devcluster02:/home/ysc/ hbase-0.92.2/conf/hadoop-metrics.properties 5）、scp conf/hadoop-metrics.properties root@devcluster03:/home/ysc/ hbase-0.92.2/conf/hadoop-metrics.properties 6）、stop-hbase.sh 7）、start-hbase.sh 八、Hadoop配置Snappy压缩
1、wget http://snappy.googlecode.com/files/snappy-1.0.5.tar.gz 2、tar -xzvf snappy-1.0.5.tar.gz 3、cd snappy-1.0.5 4、./configure 5、make 6、make install 7、scp /usr/local/lib/libsnappy* devcluster01:/home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64/ scp /usr/local/lib/libsnappy* devcluster02:/home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64/ scp /usr/local/lib/libsnappy* devcluster03:/home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64/ 8、vi /etc/profile 追加： export LD_LIBRARY_PATH=/home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64 9、修改mapred-site.xml <property> <name>mapred.output.compression.type</name> <value>BLOCK</value> <description>If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. </description> </property> <property>
<name>mapred.output.compress</name> <value>true</value> <description>Should the job outputs be compressed? </description> </property> <property>
<name>mapred.compress.map.output</name> <value>true</value> <description>Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. </description> </property> <property>
<name>mapred.map.output.compression.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> <description>If the map outputs are compressed, how should they be compressed? </description> </property> <property>
<name>mapred.output.compression.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> <description>If the job outputs are compressed, how should they be compressed? </description> </property> 九、Hadoop配置Lzo压缩
1、wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz 2、tar -zxvf lzo-2.06.tar.gz 3、cd lzo-2.06 4、./configure --enable-shared 5、make 6、make install 7、scp /usr/local/lib/liblzo2.* devcluster01:/lib/x86_64-linux-gnu scp /usr/local/lib/liblzo2.* devcluster02:/lib/x86_64-linux-gnu scp /usr/local/lib/liblzo2.* devcluster03:/lib/x86_64-linux-gnu 8、wget http://hadoop-gpl-compression.apache-extras.org.codespot.com/files/hadoop-gpl-compression-0.1.0-rc0.tar.gz 9、tar -xzvf hadoop-gpl-compression-0.1.0-rc0.tar.gz 10、cd hadoop-gpl-compression-0.1.0 11、cp lib/native/Linux-amd64-64/* /home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64/ 12、cp hadoop-gpl-compression-0.1.0.jar /home/ysc/hadoop-1.1.1/lib/(这里hadoop集群的版本要和compression使用的版本一致) 13、scp -r /home/ysc/hadoop-1.1.1/lib devcluster02:/home/ysc/hadoop-1.1.1/ scp -r /home/ysc/hadoop-1.1.1/lib devcluster03:/home/ysc/hadoop-1.1.1/ 14、vi /etc/profile 追加： export LD_LIBRARY_PATH=/home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64 15、修改core-site.xml <property> <name>io.compression.codecs</name> <value>com.hadoop.compression.lzo.LzoCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value> <description>A list of the compression codec classes that can be used for compression/decompression.</description> </property> <property>
<name>io.compression.codec.lzo.class</name> <value>com.hadoop.compression.lzo.LzoCodec</value> </property> <property>
<name>fs.trash.interval</name> <value>1440</value> <description>Number of minutes between trash checkpoints. If zero, the trash feature is disabled. </description> </property> 16、修改mapred-site.xml <property> <name>mapred.output.compression.type</name> <value>BLOCK</value> <description>If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. </description> </property> <property>
<name>mapred.output.compress</name> <value>true</value> <description>Should the job outputs be compressed? </description> </property> <property>
<name>mapred.compress.map.output</name> <value>true</value> <description>Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. </description> </property> <property>
<name>mapred.map.output.compression.codec</name> <value>com.hadoop.compression.lzo.LzoCodec</value> <description>If the map outputs are compressed, how should they be compressed? </description> </property> <property>
<name>mapred.output.compression.codec</name> <value>com.hadoop.compression.lzo.LzoCodec</value> <description>If the job outputs are compressed, how should they be compressed? </description> </property> 十、配置zookeeper集群以运行hbase
1、ssh devcluster01 2、cd /home/ysc 3、wget http://mirror.bjtu.edu.cn/apache/zookeeper/stable/zookeeper-3.4.5.tar.gz 4、tar -zxvf zookeeper-3.4.5.tar.gz 5、cd zookeeper-3.4.5 6、cp conf/zoo_sample.cfg conf/zoo.cfg 7、vi conf/zoo.cfg 修改：dataDir=/home/ysc/zookeeper 添加： server.1=devcluster01:2888:3888 server.2=devcluster02:2888:3888 server.3=devcluster03:2888:3888 maxClientCnxns=100 8、scp -r zookeeper-3.4.5 devcluster01:/home/ysc scp -r zookeeper-3.4.5 devcluster02:/home/ysc scp -r zookeeper-3.4.5 devcluster03:/home/ysc 9、分别在三台机器上面执行： ssh devcluster01 mkdir /home/ysc/zookeeper（注：dataDir是zookeeper的数据目录，需要手动创建） echo 1 > /home/ysc/zookeeper/myid ssh devcluster02 mkdir /home/ysc/zookeeper echo 2 > /home/ysc/zookeeper/myid ssh devcluster03 mkdir /home/ysc/zookeeper echo 3 > /home/ysc/zookeeper/myid 10、分别在三台机器上面执行： cd /home/ysc/zookeeper-3.4.5 bin/zkServer.sh start bin/zkCli.sh -server devcluster01:2181 bin/zkServer.sh status 十一、配置Hbase集群以运行nutch-2.1(Region Servers会因为内存的问题宕机)
1、nutch-2.1使用gora-0.2.1， gora-0.2.1使用hbase-0.90.4，hbase-0.90.4和hadoop-1.1.1不兼容，hbase-0.94.4和gora-0.2.1不兼容，hbase-0.92.2没问题。hbase存在系统时间同步的问题，并且误差要再30s以内。 sudo apt-get install ntp sudo ntpdate -u 210.72.145.44 2、HBase是数据库，会在同一时间使用很多的文件句柄。大多数linux系统使用的默认值1024是不能满足的。还需要修改 hbase 用户的 nproc，在压力下，如果过低会造成 OutOfMemoryError异常。 vi /etc/security/limits.conf 添加： ysc soft nproc 32000 ysc hard nproc 32000 ysc soft nofile 32768 ysc hard nofile 32768 vi /etc/pam.d/common-session 添加： session required pam_limits.so 3、登陆master，下载并解压hbase ssh devcluster01 cd /home/ysc wget http://apache.etoak.com/hbase/hbase-0.92.2/hbase-0.92.2.tar.gz tar -zxvf hbase-0.92.2.tar.gz cd hbase-0.92.2 4、修改配置文件hbase-env.sh vi conf/hbase-env.sh 追加： export JAVA_HOME=/home/ysc/jdk1.7.0_05 export HBASE_MANAGES_ZK=false export HBASE_HEAPSIZE=10000 #替换掉默认的垃圾回收器，因为默认的垃圾回收器在多线程环境下会有更多的wait等待 export HBASE_OPTS="-server -Xmn256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70" 5、修改配置文件hbase-site.xml vi conf/hbase-site.xml <property> <name>hbase.rootdir</name> <value>hdfs://devcluster01:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>devcluster01,devcluster02,devcluster03</value> </property> <property> <name>hbase.client.scanner.caching</name> <value>100</value> <description>Number of rows that will be fetched when calling next on a scanner if it is not served from (local, client) memory. Higher caching values will enable faster scanners but will eat up more memory and some calls of next may take longer and longer times when the cache is empty. Do not set this value such that the time between invocations is greater than the scanner timeout; i.e. hbase.regionserver.lease.period </description> </property> <property> <name>hfile.block.cache.size</name> <value>0.25</value> <description> Percentage of maximum heap (-Xmx setting) to allocate to block cache used by HFile/StoreFile. Default of 0.25 means allocate 25%. Set to 0 to disable but it's not recommended. </description> </property> <property> <name>hbase.regionserver.global.memstore.upperLimit</name> <value>0.4</value> <description>Maximum size of all memstores in a region server before new updates are blocked and flushes are forced. Defaults to 40% of heap </description> </property> 6、修改配置文件regionservers vi conf/regionservers devcluster01 devcluster02 devcluster03 7、因为HBase建立在Hadoop之上，Hadoop使用的hadoop*.jar和HBase使用的 必须 一致。所以要将 HBase lib 目录下的hadoop*.jar替换成Hadoop里面的那个，防止版本冲突。 cp /home/ysc/hadoop-1.1.1/hadoop-core-1.1.1.jar /home/ysc/hbase-0.92.2/lib rm /home/ysc/hbase-0.92.2/lib/hadoop-core-1.0.3.jar 8、复制文件到regionservers scp -r /home/ysc/hbase-0.92.2 devcluster01:/home/ysc scp -r /home/ysc/hbase-0.92.2 devcluster02:/home/ysc scp -r /home/ysc/hbase-0.92.2 devcluster03:/home/ysc 9、启动hadoop并创建目录 hadoop fs -mkdir /hbase 10、管理HBase集群: 启动初始 HBase 集群： bin/start-hbase.sh 停止HBase 集群： bin/stop-hbase.sh 启动额外备份主服务器，可以启动到 9 个备份服务器 (总数10 个)： bin/local-master-backup.sh start 1 bin/local-master-backup.sh start 2 3 启动更多 regionservers, 支持到 99 个额外regionservers (总100个)： bin/local-regionservers.sh start 1 bin/local-regionservers.sh start 2 3 4 5 停止备份主服务器: cat /tmp/hbase-ysc-1-master.pid |xargs kill -9 停止单独 regionserver： bin/local-regionservers.sh stop 1 使用HBase命令行模式: bin/hbase shell 11、web界面 http://devcluster01:60010 http://devcluster01:60030 12、如运行nutch2.1则方法一： cp conf/hbase-site.xml /home/ysc/nutch-2.1/conf cd /home/ysc/nutch-2.1 ant cd runtime/deploy unzip -d apache-nutch-2.1 apache-nutch-2.1.job rm apache-nutch-2.1.job cd apache-nutch-2.1 rm lib/hbase-0.90.4.jar cp /home/ysc/hbase-0.92.2/hbase-0.92.2.jar lib zip -r ../apache-nutch-2.1.job ./* cd .. rm -r apache-nutch-2.1 13、如运行nutch2.1则方法二： cp conf/hbase-site.xml /home/ysc/nutch-2.1/conf cd /home/ysc/nutch-2.1 cp /home/ysc/hbase-0.92.2/hbase-0.92.2.jar lib ant cd runtime/deploy zip -d apache-nutch-2.1.job lib/hbase-0.90.4.jar 启用snappy压缩：
1、vi conf/gora-hbase-mapping.xml 在family上面添加属性：compression="SNAPPY" 2、mkdir /home/ysc/hbase-0.92.2/lib/native/Linux-amd64-64 3、cp /home/ysc/hadoop-1.1.1/lib/native/Linux-amd64-64/* /home/ysc/hbase-0.92.2/lib/native/Linux-amd64-64 4、vi /home/ysc/hbase-0.92.2/conf/hbase-site.xml 增加： <property> <name>hbase.regionserver.codecs</name> <value>snappy</value> </property> 十二、配置Accumulo集群以运行nutch-2.1(gora存在BUG)
1、wget http://apache.etoak.com/accumulo/1.4.2/accumulo-1.4.2-dist.tar.gz 2、tar -xzvf accumulo-1.4.2-dist.tar.gz 3、cd accumulo-1.4.2 4、cp conf/examples/3GB/standalone/* conf 5、vi conf/accumulo-env.sh export HADOOP_HOME=/home/ysc/cluster3 export ZOOKEEPER_HOME=/home/ysc/zookeeper-3.4.5 export JAVA_HOME=/home/jdk1.7.0_01 export ACCUMULO_HOME=/home/ysc/accumulo-1.4.2 6、vi conf/slaves devcluster01 devcluster02 devcluster03 7、vi conf/masters devcluster01 8、vi conf/accumulo-site.xml <property> <name>instance.zookeeper.host</name> <value>host6:2181,host8:2181</value> <description>comma separated list of zookeeper servers</description> </property> <property>
<name>logger.dir.walog</name> <value>walogs</value> <description>The directory used to store write-ahead logs on the local filesystem. It is possible to specify a comma-separated list of directories.</description> </property> <property>
<name>instance.secret</name> <value>ysc</value> <description>A secret unique to a given instance that all servers must know in order to communicate with one another. Change it before initialization. To change it later use ./bin/accumulo org.apache.accumulo.server.util.ChangeSecret [oldpasswd] [newpasswd], and then update this file. </description> </property> <property>
<name>tserver.memory.maps.max</name> <value>3G</value> </property> <property>
<name>tserver.cache.data.size</name> <value>50M</value> </property> <property>
<name>tserver.cache.index.size</name> <value>512M</value> </property> <property>
<name>trace.password</name> <!-- change this to the root user's password, and/or change the user below --> <value>ysc</value> </property> <property>
<name>trace.user</name> <value>root</value> </property> 9、bin/accumulo init 10、bin/start-all.sh 11、bin/stop-all.sh 12、web访问：http://devcluster01:50095/ 修改nutch2.1：
1、cd /home/ysc/nutch-2.1 2、vi conf/gora.properties 增加： gora.datastore.default=org.apache.gora.accumulo.store.AccumuloStore gora.datastore.accumulo.mock=false gora.datastore.accumulo.instance=accumulo gora.datastore.accumulo.zookeepers=host6,host8 gora.datastore.accumulo.user=root gora.datastore.accumulo.password=ysc 3、vi conf/nutch-site.xml 增加： <property> <name>storage.data.store.class</name> <value>org.apache.gora.accumulo.store.AccumuloStore</value> </property> 4、vi ivy/ivy.xml 增加： <dependency org="org.apache.gora" name="gora-accumulo" rev="0.2.1" conf="*->default" /> 5、升级accumulo cp /home/ysc/accumulo-1.4.2/lib/accumulo-core-1.4.2.jar /home/ysc/nutch-2.1/lib cp /home/ysc/accumulo-1.4.2/lib/accumulo-start-1.4.2.jar /home/ysc/nutch-2.1/lib cp /home/ysc/accumulo-1.4.2/lib/cloudtrace-1.4.2.jar /home/ysc/nutch-2.1/lib 6、ant 7、cd runtime/deploy 8、删除旧jar zip -d apache-nutch-2.1.job lib/accumulo-core-1.4.0.jar zip -d apache-nutch-2.1.job lib/accumulo-start-1.4.0.jar zip -d apache-nutch-2.1.job lib/cloudtrace-1.4.2.jar 十三、配置Cassandra 集群以运行nutch-2.1（Cassandra 采用去中心化结构）
1、vi /etc/hosts（注意：需要登录到每一台机器上面，将localhost解析到实际地址） 192.168.1.1 localhost 2、wget http://labs.mop.com/apache-mirror/cassandra/1.2.0/apache-cassandra-1.2.0-bin.tar.gz 3、tar -xzvf apache-cassandra-1.2.0-bin.tar.gz 4、cd apache-cassandra-1.2.0 5、vi conf/cassandra-env.sh 增加： MAX_HEAP_SIZE="4G" HEAP_NEWSIZE="800M" 6、vi conf/log4j-server.properties 修改： log4j.appender.R.File=/home/ysc/cassandra/system.log 7、vi conf/cassandra.yaml 修改： cluster_name: 'Cassandra Cluster' data_file_directories: - /home/ysc/cassandra/data commitlog_directory: /home/ysc/cassandra/commitlog saved_caches_directory: /home/ysc/cassandra/saved_caches - seeds: "192.168.1.1"
listen_address: 192.168.1.1 rpc_address: 192.168.1.1 thrift_framed_transport_size_in_mb: 1023
thrift_max_message_length_in_mb: 1024 8、vi bin/stop-server 增加： user=`whoami` pgrep -u $user -f cassandra | xargs kill -9 9、复制cassandra到其他节点： cd .. scp -r apache-cassandra-1.2.0 devcluster02:/home/ysc scp -r apache-cassandra-1.2.0 devcluster03:/home/ysc 分别在devcluster02和devcluster03上面修改： vi conf/cassandra.yaml listen_address: 192.168.1.2 rpc_address: 192.168.1.2 vi conf/cassandra.yaml listen_address: 192.168.1.3 rpc_address: 192.168.1.3 10、分别在3个节点上面运行 bin/cassandra bin/cassandra -f 参数 -f 的作用是让 Cassandra 以前端程序方式运行，这样有利于调试和观察日志信息，而在实际生产环境中这个参数是不需要的（即 Cassandra 会以 daemon 方式运行） 11、bin/nodetool -host devcluster01 ring bin/nodetool -host devcluster01 info 12、bin/stop-server 13、bin/cassandra-cli 修改nutch2.1：
1、cd /home/ysc/nutch-2.1 2、vi conf/gora.properties 增加： gora.cassandrastore.servers=host2:9160,host6:9160,host8:9160 3、vi conf/nutch-site.xml 增加： <property> <name>storage.data.store.class</name> <value>org.apache.gora.cassandra.store.CassandraStore</value> </property> 4、vi ivy/ivy.xml 增加： <dependency org="org.apache.gora" name="gora-cassandra" rev="0.2.1" conf="*->default" /> 5、升级cassandra cp /home/ysc/apache-cassandra-1.2.0/lib/apache-cassandra-1.2.0.jar /home/ysc/nutch-2.1/lib cp /home/ysc/apache-cassandra-1.2.0/lib/apache-cassandra-thrift-1.2.0.jar /home/ysc/nutch-2.1/lib cp /home/ysc/apache-cassandra-1.2.0/lib/jline-1.0.jar /home/ysc/nutch-2.1/lib 6、ant 7、cd runtime/deploy 8、删除旧jar zip -d apache-nutch-2.1.job lib/cassandra-thrift-1.1.2.jar zip -d apache-nutch-2.1.job lib/jline-0.9.1.jar 十四、配置MySQL 单机服务器以运行nutch-2.1
1、apt-get install mysql-server mysql-client 2、vi /etc/mysql/my.cnf 修改： bind-address = 221.194.43.2 在[client]下增加： default-character-set=utf8 在[mysqld]下增加： default-character-set=utf8 3、mysql –uroot –pysc SHOW VARIABLES LIKE '%character%'; 4、service mysql restart 5、mysql –uroot –pysc GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "ysc"; 6、vi conf/gora-sql-mapping.xml 修改字段的长度 <primarykey column="id" length="333"/> <field name="content" column="content" /> <field name="text" column="text" length="19892"/> 7、启动nutch之后登陆mysql ALTER TABLE webpage MODIFY COLUMN content MEDIUMBLOB; ALTER TABLE webpage MODIFY COLUMN text MEDIUMTEXT; ALTER TABLE webpage MODIFY COLUMN title MEDIUMTEXT; ALTER TABLE webpage MODIFY COLUMN reprUrl MEDIUMTEXT; ALTER TABLE webpage MODIFY COLUMN baseUrl MEDIUMTEXT; ALTER TABLE webpage MODIFY COLUMN typ MEDIUMTEXT; ALTER TABLE webpage MODIFY COLUMN inlinks MEDIUMBLOB; ALTER TABLE webpage MODIFY COLUMN outlinks MEDIUMBLOB; 修改nutch2.1：
1、cd /home/ysc/nutch-2.1 2、vi conf/gora.properties 增加： gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver gora.sqlstore.jdbc.url=jdbc:mysql://host2:3306/nutch?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf8 gora.sqlstore.jdbc.user=root gora.sqlstore.jdbc.password=ysc 3、vi conf/nutch-site.xml 增加： <property> <name>storage.data.store.class</name> <value>org.apache.gora.sql.store.SqlStore </value> </property> <property>
<name>encodingdetector.charset.min.confidence</name> <value>1</value> <description>A integer between 0-100 indicating minimum confidence value for charset auto-detection. Any negative value disables auto-detection. </description> </property> 4、vi ivy/ivy.xml 增加： <dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/> 十五、nutch2.1 使用DataFileAvroStore作为数据源
1、cd /home/ysc/nutch-2.1 2、vi conf/gora.properties 增加： gora.datafileavrostore.output.path=datafileavrostore gora.datafileavrostore.input.path=datafileavrostore 3、vi conf/nutch-site.xml 增加： <property> <name>storage.data.store.class</name> <value>org.apache.gora.avro.store.DataFileAvroStore</value> </property> <property>
<name>encodingdetector.charset.min.confidence</name> <value>1</value> <description>A integer between 0-100 indicating minimum confidence value for charset auto-detection. Any negative value disables auto-detection. </description> </property> 十六、nutch2.1 使用AvroStore作为数据源
1、cd /home/ysc/nutch-2.1 2、vi conf/gora.properties 增加： gora.avrostore.codec.type=BINARY gora.avrostore.input.path=avrostore gora.avrostore.output.path=avrostore 3、vi conf/nutch-site.xml 增加： <property> <name>storage.data.store.class</name> <value>org.apache.gora.avro.store.AvroStore</value> </property> <property>
<name>encodingdetector.charset.min.confidence</name> <value>1</value> <description>A integer between 0-100 indicating minimum confidence value for charset auto-detection. Any negative value disables auto-detection. </description> </property> 十七、配置SOLR
配置tomcat： 1、wget http://www.fayea.com/apache-mirror/tomcat/tomcat-7/v7.0.35/bin/apache-tomcat-7.0.35.tar.gz 2、tar -xzvf apache-tomcat-7.0.35.tar.gz 3、cd apache-tomcat-7.0.35 4、vi conf/server.xml 增加URIEncoding="UTF-8"： <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8"/> 5、mkdir conf/Catalina 6、mkdir conf/Catalina/localhost 7、vi conf/Catalina/localhost/solr.xml 增加： <Context path="/solr"> <Environment name="solr/home" type="java.lang.String" value="/home/ysc/solr/configuration/" override="false"/> </Context> 8、cd .. 下载SOLR:
1、wget http://mirrors.tuna.tsinghua.edu.cn/apache/lucene/solr/4.1.0/solr-4.1.0.tgz 2、tar -xzvf solr-4.1.0.tgz 复制资源：
1、mkdir /home/ysc/solr 2、cp -r solr-4.1.0/example/solr /home/ysc/solr/configuration 3、unzip solr-4.1.0/example/webapps/solr.war -d /home/ysc/apache-tomcat-7.0.35/webapps/solr 配置nutch：
1、复制schema： cp /home/ysc/nutch-1.6/conf/schema-solr4.xml /home/ysc/solr/configuration/collection1/conf/schema.xml 2、vi /home/ysc/solr/configuration/collection1/conf/schema.xml 在<fields>下增加： <field name="_version_" type="long" indexed="true" stored="true"/> 配置中文分词：
1、wget http://mmseg4j.googlecode.com/files/mmseg4j-1.9.1.v20130120-SNAPSHOT.zip 2、unzip mmseg4j-1.9.1.v20130120-SNAPSHOT.zip 3、cp mmseg4j-1.9.1-SNAPSHOT/dist/* /home/ysc/apache-tomcat-7.0.35/webapps/solr/WEB-INF/lib 4、unzip mmseg4j-1.9.1-SNAPSHOT/dist/mmseg4j-core-1.9.1-SNAPSHOT.jar -d mmseg4j-1.9.1-SNAPSHOT/dist/mmseg4j-core-1.9.1-SNAPSHOT 5、mkdir /home/ysc/dic 6、cp mmseg4j-1.9.1-SNAPSHOT/dist/mmseg4j-core-1.9.1-SNAPSHOT/data/* /home/ysc/dic 7、vi /home/ysc/solr/configuration/collection1/conf/schema.xml 将文件中的 <tokenizer class="solr.WhitespaceTokenizerFactory"/> 和 <tokenizer class="solr.StandardTokenizerFactory"/> 替换为 <tokenizer class="com.chenlb.mmseg4j.solr.MMSegTokenizerFactory" mode="complex" dicPath="/home/ysc/dic"/> 配置tomcat本地库：
1、wget http://apache.spd.co.il/apr/apr-1.4.6.tar.gz 2、tar -xzvf apr-1.4.6.tar.gz 3、cd apr-1.4.6 4、./configure 5、make 6、make install 1、wget http://mirror.bjtu.edu.cn/apache/apr/apr-util-1.5.1.tar.gz
2、tar -xzvf apr-util-1.5.1.tar.gz 3、cd apr-util-1.5.1 4、./configure --with-apr=/usr/local/apr 5、make 6、make install 1、wget http://mirror.bjtu.edu.cn/apache//tomcat/tomcat-connectors/native/1.1.24/source/tomcat-native-1.1.24-src.tar.gz
2、tar -zxvf tomcat-native-1.1.24-src.tar.gz 3、cd tomcat-native-1.1.24-src/jni/native 4、./configure --with-apr=/usr/local/apr \ --with-java-home=/home/ysc/jdk1.7.0_01 \ --with-ssl=no \ --prefix=/home/ysc/apache-tomcat-7.0.35 5、make 6、make install 7、vi /etc/profile 增加： export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ysc/apache-tomcat-7.0.35/lib:/usr/local/apr/lib 8、source /etc/profile 十八、Nagios监控
服务端： 1、apt-get install apache2 nagios3 nagios-nrpe-plugin 输入密码：nagiosadmin 2、apt-get install nagios3-doc 3、vi /etc/nagios3/conf.d/hostgroups_nagios2.cfg define hostgroup { hostgroup_name nagios-servers alias nagios servers members devcluster01,devcluster02,devcluster03 } 4、cp /etc/nagios3/conf.d/localhost_nagios2.cfg /etc/nagios3/conf.d/devcluster01_nagios2.cfg vi /etc/nagios3/conf.d/devcluster01_nagios2.cfg 替换： g/localhost/s//devcluster01/g g/127.0.0.1/s//192.168.1.1/g 5、cp /etc/nagios3/conf.d/localhost_nagios2.cfg /etc/nagios3/conf.d/devcluster02_nagios2.cfg vi /etc/nagios3/conf.d/devcluster02_nagios2.cfg 替换： g/localhost/s//devcluster02/g g/127.0.0.1/s//192.168.1.2/g 6、cp /etc/nagios3/conf.d/localhost_nagios2.cfg /etc/nagios3/conf.d/devcluster03_nagios2.cfg vi /etc/nagios3/conf.d/devcluster03_nagios2.cfg 替换： g/localhost/s//devcluster03/g g/127.0.0.1/s//192.168.1.3/g 7、vi /etc/nagios3/conf.d/services_nagios2.cfg
将hostgroup_name改为nagios-servers 增加： # check that web services are running define service { hostgroup_name nagios-servers service_description HTTP check_command check_http use generic-service notification_interval 0 ; set > 0 if you want to be renotified } # check that ssh services are running
define service { hostgroup_name nagios-servers service_description SSH check_command check_ssh use generic-service notification_interval 0 ; set > 0 if you want to be renotified } 8、vi /etc/nagios3/conf.d/extinfo_nagios2.cfg 将hostgroup_name改为nagios-servers 增加： define hostextinfo{ hostgroup_name nagios-servers notes nagios-servers # notes_url http://webserver.localhost.localdomain/hostinfo.pl?host=netware1 icon_image base/debian.png icon_image_alt Debian GNU/Linux vrml_image debian.png statusmap_image base/debian.gd2 } 9、sudo /etc/init.d/nagios3 restart 10、访问http://devcluster01/nagios3/ 用户名：nagiosadmin密码：nagiosadmin 监控端：
1、apt-get install nagios-nrpe-server 2、vi /etc/nagios/nrpe.cfg 替换： g/127.0.0.1/s//192.168.1.1/g 3、sudo /etc/init.d/nagios-nrpe-server restart 十九、配置Splunk
1、wget http://download.splunk.com/releases/5.0.2/splunk/linux/splunk-5.0.2-149561-Linux-x86_64.tgz 2、tar -zxvf splunk-5.0.2-149561-Linux-x86_64.tgz 3、cd splunk 4、bin/splunk start --answer-yes --no-prompt --accept-license 5、访问http://devcluster01:8000 用户名：admin 密码：changeme 6、添加数据 -> 从 UDP 端口 -> UDP 端口 *: 1688 -> 来源类型 从列表 log4j -> 保存 7、配置hadoop vi /home/ysc/hadoop-1.1.1/conf/log4j.properties 修改： log4j.rootLogger=${hadoop.root.logger}, EventCounter, SYSLOG 增加： log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.facility=local1 log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n log4j.appender.SYSLOG.SyslogHost=host6:1688 log4j.appender.SYSLOG.threshold=INFO log4j.appender.SYSLOG.Header=true log4j.appender.SYSLOG.FacilityPrinting=true 8、配置hbase vi /home/ysc/hbase-0.92.2/conf/log4j.properties 修改： log4j.rootLogger=${hbase.root.logger},SYSLOG 增加： log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.facility=local1 log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n log4j.appender.SYSLOG.SyslogHost=host6:1688 log4j.appender.SYSLOG.threshold=INFO log4j.appender.SYSLOG.Header=true log4j.appender.SYSLOG.FacilityPrinting=true 9、配置nutch vi /home/lanke/ysc/nutch-2.1-hbase/conf/log4j.properties 修改： log4j.rootLogger=INFO,DRFA,SYSLOG 增加： log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.facility=local1 log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n log4j.appender.SYSLOG.SyslogHost=host6:1688 log4j.appender.SYSLOG.threshold=INFO log4j.appender.SYSLOG.Header=true log4j.appender.SYSLOG.FacilityPrinting=true 10、启动hadoop和hbase start-all.sh start-hbase.sh 二十、配置Pig
1、wget http://labs.mop.com/apache-mirror/pig/pig-0.11.0/pig-0.11.0.tar.gz 2、tar -xzvf pig-0.11.0.tar.gz 3、cd pig-0.11.0 4、vi /etc/profile 增加： export PIG_HOME=/home/ysc/pig-0.11.0 export PATH=$PIG_HOME/bin:$PATH 5、source /etc/profile 6、cp conf/log4j.properties.template conf/log4j.properties 7、vi conf/log4j.properties 8、pig 二十一、配置Hive
1、wget http://mirrors.cnnic.cn/apache/hive/hive-0.10.0/hive-0.10.0.tar.gz 2、tar -xzvf hive-0.10.0.tar.gz 3、cd hive-0.10.0 4、vi /etc/profile 增加： export HIVE_HOME=/home/ysc/hive-0.10.0 export PATH=$HIVE_HOME/bin:$PATH 5、source /etc/profile 6、cp conf/hive-log4j.properties.template conf/hive-log4j.properties 7、vi conf/hive-log4j.properties 替换： log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter 为： log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter 二十二、配置Hadoop2.x集群 1、wget http://labs.mop.com/apache-mirror/hadoop/common/hadoop-2.0.2-alpha/hadoop-2.0.2-alpha.tar.gz 2、tar -xzvf hadoop-2.0.2-alpha.tar.gz 3、cd hadoop-2.0.2-alpha 4、vi etc/hadoop/hadoop-env.sh 追加： export JAVA_HOME=/home/ysc/jdk1.7.0_05 export HADOOP_HEAPSIZE=2000 5、vi etc/hadoop/core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://devcluster01:9000</value> <description> Where to find the Hadoop Filesystem through the network. Note 9000 is not the default port. (This is slightly changed from previous versions which didnt have "hdfs") </description> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> <description>The size of buffer for use in sequence files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.</description> </property> 6、vi etc/hadoop/mapred-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property>
<name>mapred.job.reduce.input.buffer.percent</name> <value>1</value> <description>The percentage of memory- relative to the maximum heap size- to retain map outputs during the reduce. When the shuffle is concluded, any remaining map outputs in memory must consume less than this threshold before the reduce can begin. </description> </property> <property>
<name>mapred.job.shuffle.input.buffer.percent</name> <value>1</value> <description>The percentage of memory to be allocated from the maximum heap size to storing map outputs during the shuffle. </description> </property> <property>
<name>mapred.inmem.merge.threshold</name> <value>0</value> <description>The threshold, in terms of the number of files for the in-memory merge process. When we accumulate threshold number of files we initiate the in-memory merge and spill to disk. A value of 0 or less than 0 indicates we want to DON'T have any threshold and instead depend only on the ramfs's memory consumption to trigger the merge. </description> </property> <property>
<name>io.sort.factor</name> <value>100</value> <description>The number of streams to merge at once while sorting files. This determines the number of open file handles.</description> </property> <property>
<name>io.sort.mb</name> <value>240</value> <description>The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1MB, which should minimize seeks.</description> </property> <property> <name>mapred.map.output.compression.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> <description>If the map outputs are compressed, how should they be compressed? </description> </property> <property>
<name>mapred.output.compression.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> <description>If the job outputs are compressed, how should they be compressed? </description> </property> <property> <name>mapred.output.compression.type</name> <value>BLOCK</value> <description>If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. </description> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx2000m</value> </property> <property>
<name>mapred.output.compress</name> <value>true</value> <description>Should the job outputs be compressed? </description> </property> <property>
<name>mapred.compress.map.output</name> <value>true</value> <description>Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. </description> </property> <property>
<name>mapred.tasktracker.map.tasks.maximum</name> <value>5</value> </property> <property>
<name>mapred.map.tasks</name> <value>15</value> </property> <property>
<name>mapred.tasktracker.reduce.tasks.maximum</name> <value>5</value> <description> define mapred.map tasks to be number of slave hosts.the best number is the number of slave hosts plus the core numbers of per host </description> </property> <property>
<name>mapred.reduce.tasks</name> <value>15</value> <description> define mapred.reduce tasks to be number of slave hosts.the best number is the number of slave hosts plus the core numbers of per host </description> </property> <property> <name>mapred.system.dir</name> <value>/home/ysc/mapreduce/system</value> </property> <property>
<name>mapred.local.dir</name> <value>/home/ysc/mapreduce/local</value> </property> <property>
<name>mapreduce.job.counters.max</name> <value>12000</value> <description>Limit on the number of counters allowed per job. </description> </property> 7、vi etc/hadoop/yarn-site.xml <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>devcluster01:8031</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>devcluster01:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>devcluster01:8030</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>devcluster01:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>devcluster01:8088</value> </property> <property> <description>Classpath for typical applications.</description> <name>yarn.application.classpath</name> <value> $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $YARN_HOME/*,$YARN_HOME/lib/* </value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/home/ysc/h2/data/1/yarn/local,/home/ysc/h2/data/2/yarn/local,/home/ysc/h2/data/3/yarn/local</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>/home/ysc/h2/data/1/yarn/logs,/home/ysc/h2/data/2/yarn/logs,/home/ysc/h2/data/3/yarn/logs</value> </property> <property> <description>Where to aggregate logs</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/home/ysc/h2/var/log/hadoop-yarn/apps</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>devcluster01:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>devcluster01:19888</value> </property> 8、vi etc/hadoop/hdfs-site.xml <property> <name>dfs.permissions.superusergroup</name> <value>root</value> </property> <property> <name>dfs.name.dir</name> <value>/home/ysc/dfs/filesystem/name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/ysc/dfs/filesystem/data</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.block.size</name> <value>6710886400</value> <description>The default block size for new files.</description> </property> 9、启动hadoop bin/hdfs namenode -format sbin/start-dfs.sh sbin/start-yarn.sh 10、访问管理页面 http://devcluster01:8088 http://devcluster01:50070 |
Nutch安装指南
转自：http://user.qzone.qq.com/281032878/blog/1362131478#!app=2&via=QZ.HashRefresh&pos=1362131478