hadoop2.6.5安装（二）是完全分布式安装的基本配置 ，该文属于前期学习的踩雷笔记。
系统：CentOS 6.5
软件包：hadoop2.6.5 、jdk8
一、ssh-keygen生成密钥，并加入授权：
1 [hadoop@hadoop26 ~]$ cd ~/.ssh/ 2 [hadoop@hadoop26 .ssh]$ 3 [hadoop@hadoop26 .ssh]$ ssh-keygen -t rsa 4 Generating public/private rsa key pair. 5 Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 6 Enter passphrase (empty for no passphrase): 7 Enter same passphrase again: 8 Your identification has been saved in /home/hadoop/.ssh/id_rsa. 9 Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. 10 The key fingerprint is: 11 d6:aa:79:d8:9c:65:0e:e9:4a:27:ab:4b:dd:a4:d8:d2 hadoop@hadoop26.example.com 12 The key's randomart image is: 13 +--[ RSA 2048]----+ 14 | | 15 | | 16 | | 17 | . | 18 | S.. | 19 | = =o.o | 20 | + E=+* | 21 | . o.** . | 22 | oo=o | 23 +-----------------+ 24 [hadoop@hadoop26 .ssh]$ 25 [hadoop@hadoop26 .ssh]$ cat id_rsa.pub >> authorized_keys 26 [hadoop@hadoop26 .ssh]$ chmod 600 ./authorized_keys
二、安装jdk
1.由于该centos是最简化安装，缺少wget命令，先安装wget。
1 [hadoop@hadoop26 ~]$ java -version 2 -bash: java: command not found 3 [hadoop@hadoop26 ~]$ rpm -qa | grep java 4 [hadoop@hadoop26 ~]$ wget --no-check-certificate --no-cookies --header "Cookies: oraclelicense=accept-securebackup-cookies" http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz 5 -bash: wget: command not found 6 [hadoop@hadoop26 ~]$ yum install -y wget* 7 You need to be root to perform this command. 8 [hadoop@hadoop26 ~]$ su - 9 Password: 10 [root@hadoop26 ~]# yum install -y wget 11 Running Transaction 12 Installing : wget-1.12-10.el6.x86_64 1/1 13 Verifying : wget-1.12-10.el6.x86_64 1/1 14 Installed: 15 wget.x86_64 0:1.12-10.el6 16 Complete!
2.wget安装完成之后，获取jdk，该hadoop2.6.5安装包对应的一般是jdk8（见文末链接）
1 [root@hadoop26 ~]# wget --no-check-certificate --no-cookies --header "Cookies: oraclelicense=accept-securebackup-cookies" http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz 2 --2018-02-01 17:18:36-- http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz 3 Resolving download.oracle.com... 23.56.28.195 4 Connecting to download.oracle.com|23.56.28.195|:80... connected. 5 HTTP request sent, awaiting response... 302 Moved Temporarily 6 Location: https://edelivery.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz [following] 7 --2018-02-01 17:18:36-- https://edelivery.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz 8 Resolving edelivery.oracle.com... 184.85.123.124, 2600:1417:a000:2a7::2d3e, 2600:1417:a000:2b7::2d3e 9 Connecting to edelivery.oracle.com|184.85.123.124|:443... connected. 10 HTTP request sent, awaiting response... 302 Moved Temporarily 11 Location: http://download.oracle.com/errors/download-fail-1505220.html [following] 12 --2018-02-01 17:18:40-- http://download.oracle.com/errors/download-fail-1505220.html 13 Connecting to download.oracle.com|23.56.28.195|:80... connected. 14 HTTP request sent, awaiting response... 200 OK 15 Length: 5307 (5.2K) [text/html] 16 Saving to: “jdk-8u141-linux-x64.tar.gz” 17 100%[===============================================================================>] 5,307 --.-K/s in 0s 18 2018-02-01 17:18:41 (99.7 MB/s) - “jdk-8u141-linux-x64.tar.gz” saved [5307/5307] 19 [root@hadoop26 java]# tar zxvf jdk-8u141-linux-x64.tar.gz 20 gzip: stdin: not in gzip format 21 tar: Child returned status 1 22 tar: Error is not recoverable: exiting now 23 [root@hadoop26 java]# tar -xvf jdk-8u141-linux-x64.tar.gz 24 gzip: stdin: not in gzip format 25 tar: Child returned status 1 26 tar: Error is not recoverable: exiting now 27 [root@hadoop26 java]# file jdk-8u141-linux-x64.tar.gz 28 jdk-8u141-linux-x64.tar.gz: HTML document text
====================================================
[root@hadoop26 java]# rm -f jdk-8u141-linux-x64.tar.gz
===重新找了对应的jdk，很多专业的网站上都有提供jdk的下载的===
ps：解压jdk的压缩包时，出现如上错误，然后file一下，发现这个文件根本不是tar压缩文件，估计是wget的时候错误了。
然后去打开那个jdk下载的网页查一下 ，结果发现是oracle网页出现故障，所以wget的是个报错网页。
于是将错误的jdk删除掉，重新获取一个
三、安装hadoop
1.jdk已安装成功，需要下载hadoop tar包了（hadoop-2.6.5.tar.gz见文末链接）
2.解压hadoop的tar包并且配置相关文件
1 [root@hadoop26 local]# tar -zxvf hadoop-2.6.5.tar.gz 2 [root@hadoop26 local]# mv hadoop-2.6.5 hadoop 3 [root@hadoop26 local]# ll hadoop* 4 -rw-r--r-- 1 root root 199635269 Feb 5 2018 hadoop-2.6.5.tar.gz 5 6 hadoop: 7 total 132 8 drwxrwxr-x 2 1000 1000 4096 Oct 3 2016 bin 9 drwxrwxr-x 3 1000 1000 4096 Oct 3 2016 etc 10 drwxrwxr-x 2 1000 1000 4096 Oct 3 2016 include 11 drwxrwxr-x 3 1000 1000 4096 Oct 3 2016 lib 12 drwxrwxr-x 2 1000 1000 4096 Oct 3 2016 libexec 13 -rw-rw-r-- 1 1000 1000 84853 Oct 3 2016 LICENSE.txt 14 -rw-rw-r-- 1 1000 1000 14978 Oct 3 2016 NOTICE.txt 15 -rw-rw-r-- 1 1000 1000 1366 Oct 3 2016 README.txt 16 drwxrwxr-x 2 1000 1000 4096 Oct 3 2016 sbin 17 drwxrwxr-x 4 1000 1000 4096 Oct 3 2016 share 18 [root@hadoop26 local]# chown -R hadoop:hadoop ./hadoop 19 [root@hadoop26 local]# ./hadoop/bin/hadoop version 20 Error: JAVA_HOME is not set and could not be found. 21 [root@hadoop26 local]# cd hadoop 22 [root@hadoop26 hadoop]# cd etc/ 23 [root@hadoop26 etc]# ls 24 hadoop 25 [root@hadoop26 etc]# cd hadoop/ 26 [root@hadoop26 hadoop]# ls 27 capacity-scheduler.xml hadoop-policy.xml kms-log4j.properties ssl-client.xml.example 28 configuration.xsl hdfs-site.xml kms-site.xml ssl-server.xml.example 29 container-executor.cfg httpfs-env.sh log4j.properties yarn-env.cmd 30 core-site.xml httpfs-log4j.properties mapred-env.cmd yarn-env.sh 31 hadoop-env.cmd httpfs-signature.secret mapred-env.sh yarn-site.xml 32 hadoop-env.sh httpfs-site.xml mapred-queues.xml.template 33 hadoop-metrics2.properties kms-acls.xml mapred-site.xml.template 34 hadoop-metrics.properties kms-env.sh slaves 35 [root@hadoop26 hadoop]# vim ./hadoop-env.sh 36 [root@hadoop26 hadoop]# cd ../ 37 [root@hadoop26 etc]# cd ../ 38 [root@hadoop26 hadoop]# bin/ 39 container-executor hdfs mapred.cmd yarn 40 hadoop hdfs.cmd rcc yarn.cmd 41 hadoop.cmd mapred test-container-executor 42 [root@hadoop26 hadoop]# bin/hadoop version 43 Hadoop 2.6.5 44 Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997 45 Compiled by sjlee on 2016-10-02T23:43Z 46 Compiled with protoc 2.5.0 47 From source with checksum f05c9fa095a395faa9db9f7ba5d754 48 This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.6.5.jar 49 [root@hadoop26 hadoop]# ls 50 bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share 51 [root@hadoop26 hadoop]# cd etc/ 52 [root@hadoop26 etc]# ls 53 hadoop 54 [root@hadoop26 etc]# cd hadoop/ 55 [root@hadoop26 hadoop]# ls 56 capacity-scheduler.xml hadoop-policy.xml kms-log4j.properties ssl-client.xml.example 57 configuration.xsl hdfs-site.xml kms-site.xml ssl-server.xml.example 58 container-executor.cfg httpfs-env.sh log4j.properties yarn-env.cmd 59 core-site.xml httpfs-log4j.properties mapred-env.cmd yarn-env.sh 60 hadoop-env.cmd httpfs-signature.secret mapred-env.sh yarn-site.xml 61 hadoop-env.sh httpfs-site.xml mapred-queues.xml.template 62 hadoop-metrics2.properties kms-acls.xml mapred-site.xml.template 63 hadoop-metrics.properties kms-env.sh slaves 64 [root@hadoop26 hadoop]# vim core-site.xml 65 [root@hadoop26 hadoop]# vim hdfs-site.xml 66 [root@hadoop26 hadoop]# ls *.xml 67 capacity-scheduler.xml hadoop-policy.xml httpfs-site.xml kms-site.xml 68 core-site.xml hdfs-site.xml kms-acls.xml yarn-site.xml 69 [root@hadoop26 hadoop]# vim mapred-site.xml.template
注意：
配置文件中，如果是伪分布式的话，则使用localhost则ok；
但是如果是完全分布式模式的安装和配置，步骤如下：
1.配置hosts文件
2.建立hadoop运行账号
3.配置ssh免密码连入
4.下载并解压hadoop安装包
5.配置namenode，修改site文件
6.配置hadoop-env.sh
7.配置masters 和slaves文件
8.向各节点复制hadoop
9.格式化namenode
10.启动hadoop，用jps检验各后台进程是否成功启动
附：==软件下载地址==