Cloudera Manager安装前准备
（一）机器准备
192.168.80.148 clouderamanager01 （部署ClouderaManager-server和Mirror server） （部署Agent） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
192.168.80.149 clouderamanager02 （部署Agent） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
192.168.80.150 clouderamanager03 （部署Agent） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
注意。我这里，因为安装了两种情况。所以在写博客时，后面全部是用 clouderamanager01、 clouderamanager02和 clouderamanager03（注意是小写）！！！
下面这种，也是可以的
192.168.80.148 clouderamanager01 （部署ClouderaManager-server和Mirror server） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
192.168.80.149 clouderamanager02 （部署Agent） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
192.168.80.150 clouderamanager03 （部署Agent） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
192.168.80.151 clouderamanager04 （部署Agent） （分配1G，我这里只是体验过程，当然你可以分配更多或更小)
说白了，就是在clouderamanager01上安装了ClouderaManger。
就是在clouderamanager01上安装了cdh单节点集群。
就是在clouderamanager01、clouderamanager02和clouderamanager03上安装了cdh的3节点集群。
（二）机器的部署配置
这里我采用克隆的方式，当然大家也可以单独搭建linux虚拟机。
VMware workstation 11 的下载
VMWare Workstation 11的安装
CentOS 6.5的安装详解
CentOS 6.5安装之后的网络配置
CentOS 6.5静态IP的设置（NAT和桥接联网方式都适用）
CentOS常用命令、快照、克隆大揭秘
（1）修改主机名
现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02、clouderaManager03和clouderaManager04是一样的。当然我也可以输入命令来查看一下主机名，我们输入hostname,我们发现主机名就是clouderaManager01，那么接下来我们就来修改一下主机名。
常见的有三种方法，一种是临时性的修改，但是当机器重启之后就会恢复原来的值，比如我们试一下。
还有一种方法就是永久修改，我们直接修改配置文件，我们直接修改/etc/sysconfig/network
实际上还有一种方法，修改/etc/hosts,他对网络中的主机名起作用，但是对系统本身（也就是本机）不起作用。他和第一种的效果是一样的。实际上我们一般用这个来配置主机名和IP之间的对应关系。
我这里采用第二种方法。改完之后没有变化，我们使用reboot命令重启一下机器就可以了。
（2）Linux虚拟机的网络配置和系统配置
我这里，仅拿clouderaManager01来说，其他的clouderaManager02、clouderaManager03和clouderaManager04是一样的。
（1）ping回环地址：能ping通，说明虚拟机的网络协议是好的。
(2)ping网关
点击“编辑”——》“虚拟网络编辑器”
但是这里边要注意一个问题，因为我们是克隆的。所以我们还要改一下MAC地址（就是我们硬件的地址，用来定义网络设备的位置），因为我们新的MAC地址和我们网卡配置里的MAC地址不一致，所以会导致网络连接不上。
所以我们先来查看一下我们新的MAC地址
我们输入vi /etc/udev/rules.d/70-persistent-net.rules(如果你是完全克隆，这个文件里应该包含eth0和eth1，我们将eth0配置删除，只保留eth1,记录该配置对应的MAC地址，并将eth1改名为eth0
接下来修改网卡vi /etc/sysconfig/network-scripts/ifcfg-eth0
你要是新搭建的Linux虚拟机的时候只需要把ONBOOT 的no改为yes就行，（开启我们的网卡，我们才能够看到我们的IP地址，BOOTPROTO=dhcp,指定获取IP的方法，dhcp(动态主机配置协议)还有一种是static ，静态的）
改回dhcp之后要重启一下，否则还是查看不到IP地址。(或者用service network restart命令)
OK，前面的修改完之后，下面我们就输入ifconfig来查看一下IP，然后我们ping IP
接下来我们再来ping www.baidu.com
OK,都能ping通，这就说明我们已经和外部网络打通了。
（3）X-shell远程连接
Xmanager Enterprise *安装步骤
我这里，仅拿clouderaManager01来说，其他的clouderaManager02和clouderaManager03是一样的。
（4）用户设置：hadoop用户
新建用户组、用户、用户密码、删除用户组、用户（适合CentOS、Ubuntu）
我这里，是新建的hadoop用户。
我这里，仅拿clouderaManager01来说，其他的clouderaManager02、clouderaManager03和clouderaManager04是一样的。
（5）为hadoop用户设置sudo权限
在root用户下，输入visudo打开文件
在文件末尾（按快捷键，Shift + g）添加hadoop ALL=(ALL) NOPASSWD:ALL
hadoop ALL=(ALL) NOPASSWD:ALL
我这里，仅拿clouderaManager01来说，其他的clouderaManager02、clouderaManager03和clouderaManager04是一样的。
（6）配置静态IP
现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
vi /etc/sysconfig/network-scripts/ifcfg-eth0
（7）查看并关闭防火墙
CentOS下的防火墙关闭
先查看一下防火墙状态
service iptables status
如果防火墙在运行，就关闭，也有两种方法。
一种是临时关闭
service iptables stop
一种是永久关闭
chkconfig iptables off
这里我采用永久性关闭。
同理，现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
（8）关闭SELINUX
hadoop 50070 无法访问问题解决汇总
sudo vi /etc/selinux/config 把SELINUX改为disabled
然后重启reboot。
同理，现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
（9）相关服务的安装
如果没有安装的话可以使用下面的命令安装一下。
[root@clouderaManager01 hadoop]# yum -y install lrzsz [root@clouderaManager01 hadoop]# yum install -y openssh-clients
同理，现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
（10）时钟同步
[root@clouderaManager01 hadoop]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime cp: overwrite `/etc/localtime'? y [root@clouderaManager01 hadoop]# yum -y install ntp
[root@clouderaManager01 hadoop]# cd /usr/share/zoneinfo/Asia/ [root@clouderaManager01 Asia]# ntpdate pool.ntp.org 6 Apr 20:30:19 ntpdate[2938]: step time server 173.255.215.209 offset -86417.188942 sec [root@clouderaManager01 Asia]# date Thu Apr 6 20:30:23 CST 2017 [root@clouderaManager01 Asia]#
同理，现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
（11）修改hosts文件
vi /etc/hosts
在里面添加主机名和IP之间的对应关系
192.168.80.148 clouderaManager01 192.168.80.149 clouderaManager02 192.168.80.150 clouderaManager03
同理，现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
（12）配置ssh免密码登录 （对于这一步，其实可以不做，因为，在CM时会自动安装，大家可以跳过）
关于这一步，请移步，先看看再说，细加小心。
hadoop-2.6.0.tar.gz的集群搭建（5节点）
这里我以clouderaManager01为例来配置ssh。拿这个来说，其他的clouderaManager02、clouderaManager03和clouderaManager04是一样的。
[hadoop@clouderaManager01 ~]$ pwd /home/hadoop [hadoop@clouderaManager01 ~]$ mkdir .ssh [hadoop@clouderaManager01 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: cd:51:91:1e:95:47:b0:44:2e:fe:75:a5:ac:44:76:86 hadoop@clouderaManager01 The key's randomart image is: +--[ RSA 2048]----+ | o==+.| | .o+...| | ..E.=..| | o =.= ..| | S o o o o| | . o ..| | . . | | | | | +-----------------+ [hadoop@clouderaManager01 ~]$ cd .ssh [hadoop@clouderaManager01 .ssh]$ ls id_rsa id_rsa.pub [hadoop@clouderaManager01 .ssh]$ cat id_rsa.pub >> authorized_keys [hadoop@clouderaManager01 .ssh]$ ls authorized_keys id_rsa id_rsa.pub [hadoop@clouderaManager01 .ssh]$
[hadoop@clouderaManager01 .ssh]$ ls
authorized_keys id_rsa id_rsa.pub
[hadoop@clouderaManager01 .ssh]$ cd ..
[hadoop@clouderaManager01 ~]$ chmod 700 .ssh
[hadoop@clouderaManager01 ~]$ chmod 600 .ssh/*
[hadoop@clouderaManager01 ~]$ ssh clouderaManager01
The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established.
RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts.
[hadoop@clouderaManager01 ~]$ exit;
logout
Connection to clouderaManager01 closed.
[hadoop@clouderaManager01 ~]$ ssh clouderaManager01
Last login: Thu Apr 6 20:37:16 2017 from clouderamanager01
[hadoop@clouderaManager01 ~]$ exit;
logout
Connection to clouderaManager01 closed.
[hadoop@clouderaManager01 ~]$
将所有节点（clouderaManager02、clouderaManager03和clouderaManager04）中的共钥id_ras.pub拷贝到clouderaManager01中的authorized_keys文件中。
[hadoop@clouderaManager02 ~]$ cat ~/.ssh/id_rsa.pub | ssh hadoop@clouderaManager01 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established. RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts. hadoop@clouderamanager01's password: [hadoop@clouderaManager02 ~]$
[hadoop@clouderaManager03 ~]$ cat ~/.ssh/id_rsa.pub | ssh hadoop@clouderaManager01 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established. RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts. hadoop@clouderamanager01's password: [hadoop@clouderaManager03 ~]$
[hadoop@clouderaManager04 ~]$ cat ~/.ssh/id_rsa.pub | ssh hadoop@clouderaManager01 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established. RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts. hadoop@clouderamanager01's password: [hadoop@clouderaManager04 ~]$
得到
[hadoop@clouderaManager01 ~]$ cd .ssh [hadoop@clouderaManager01 .ssh]$ ls authorized_keys id_rsa id_rsa.pub known_hosts [hadoop@clouderaManager01 .ssh]$ cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAswjj3hTe9kqHOIA0vBu567OWLvEASOlum5NceSXrTFbL/9oG4mWZPcHJrgfLMEkHeJt+ZzT/7PtnDk2VjZS8cRBTbAxv3FQoqVLZtfZh9h+NfDr7H9RA8H8ksZL8qWB3zmhZ2nGCI+4ZVn9YtTKMgOoTiEzTWfv41k7WA4KWJYX7NqziOGzxqegpKOLK2Ec7vQuXq8zlSRW1iWMwyMdQtJK2mP66nWKrgVx5SQjipcJ+fDqcqghGYBj1kWfRV4xSeCQrdk/qIMO6PaLWKqUXSUnBP9HD/0+pDRrCCS/EaKpAs0YiiPu8MfGe3nLEqny4vbUtv61+zOlQ7OeKYWxq1Q== hadoop@clouderaManager01 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx2qIKmR564eQK+Lnp59oEvbB4HrBtxRxbTBNViO5tC4yDst1vwuPo23t9BlmMqsbCWSsAkEo6Vj3UA/qwdqKDFK1ssLRgzVG3OsqC7WrlEuSy7V27EpnfEFMkbXBiTrnRcbKNu3wdzgJZ7Mc8Yd0CqZ5c6UyPEzrgBou489fLyzKNSov+uzkahw1l7S77/lh03rbq7YQ/7gEK23tZgofsYFf+k8w6pmDpvl/mSv2156PkWMFQr9yLGen/q6kKsKqg6KT7B06a6zlB6khom+ou5p+dZDiR5AZkI3aNF0brZ7+OTckWjuh2xxgy9ZbRxy/mb6YNwjEl5mTlfFQkkQasQ== hadoop@clouderaManager02 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxYvoxWFWM9J/eyoe+O9EfA3wBe4oHQs6PZf7e5sQSoQJV+0gVufezZ+iWxGcL92uTNDE6PcrrfMWPv68Fl5mXsXrhVX8gJA/RKfeVnfLdNYI5AKmqniVq0h+qYv1wz/hOcaq/mNp3dHR3cyf+xjUc+gTcOIvJsJmBEVr5YtsfMCqVzjZZYjChEJ+jiRlf1fIwiogE0M8ncrqGDjS6aPDV3vsceHOspuuXD0APEdxb9YUVlAUXpXNADpmz3P/2V1aHbzNqFZWHcdyUtob7fABWQnuaCJPvaUXNGDBXt7Kv18d0Kbix4q1+o67nkx7B2nAQZT4D3EV3XnZLh6Gvym4Mw== hadoop@clouderaManager03 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwGScqrw5oMaAj3MLS9wZLkFFLZH8+xXG2f5jM7SNFNcWTO4LAefvrZGS2uaWFPk0PTS+f2dfPX/r0etOFvIBobOmBfrcMByLj954sGFvlD/lU7sPjUFh/OwcXo3UxJyCHnwE4pkVpcmpmyubfzYqKyo8eRMBKLQQx9KlV9MXj1ADe/iwckGXMCG8bzz0/S92S9Y8FxhbTdXhUnVshtn1oJYPr+SxUk4lFed+k6hQah3lXfjd3iJ+uhRfSBkpV9PkSeDyD1h+8ymXNCkZcgcYRXclWziv8FR5f+J2NM5ogvDlGkkcjYKSa4NvCmNsrAPvDEKHnbhV2FbaWgcIC6Tgpw== hadoop@clouderaManager04 [hadoop@clouderaManager01 .ssh]$
然后将中clouderaManager01的authorized_keys文件分发到所有节点上面。
[hadoop@clouderaManager01 .ssh]$ scp -r authorized_keys hadoop@clouderaManager02:~/.ssh/ The authenticity of host 'clouderamanager02 (192.168.80.149)' can't be established. RSA key fingerprint is 1c:56:a2:f4:6e:5b:bc:50:cf:0c:c9:56:59:e6:63:0b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager02,192.168.80.149' (RSA) to the list of known hosts. hadoop@clouderamanager02's password: authorized_keys 100% 1624 1.6KB/s 00:00 [hadoop@clouderaManager01 .ssh]$ scp -r authorized_keys hadoop@clouderaManager03:~/.ssh/ The authenticity of host 'clouderamanager03 (192.168.80.150)' can't be established. RSA key fingerprint is 59:0d:16:18:d0:2c:39:ff:8e:87:3a:31:92:55:86:f4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager03,192.168.80.150' (RSA) to the list of known hosts. hadoop@clouderamanager03's password: authorized_keys 100% 1624 1.6KB/s 00:00 [hadoop@clouderaManager01 .ssh]$ scp -r authorized_keys hadoop@clouderaManager04:~/.ssh/ The authenticity of host 'clouderamanager04 (192.168.80.151)' can't be established. RSA key fingerprint is 02:48:fd:f0:15:25:71:a4:ae:89:5d:3d:18:e9:64:b9. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager04,192.168.80.151' (RSA) to the list of known hosts. hadoop@clouderamanager04's password: authorized_keys 100% 1624 1.6KB/s 00:00 [hadoop@clouderaManager01 .ssh]$
大家通过ssh 相互访问，如果都能无密码访问，代表ssh配置成功。这里很简单，我就不多赘述了。
（13）JDK的安装（建议1.8）（对于这一步，其实可以不做，因为，在CM时会自动安装，大家可以跳过）（但是要先卸载linux系统自带的openjdk，因为自动也是安装的是orcal公司的）
用文件上传工具FileZila把jdk1.8的tar包上传到我们指定的软件安装目录下（比如：/home/hadoop/app或者/usr/local/jdk）,或者使用rz命令上传也行，但是要切换到root用户下，或者在hadoop用户下使用sudo rz
jdk-8u60-linux-x64.tar.gz
注意，先记得。
Centos 6.5下的OPENJDK卸载和SUN的JDK安装、环境变量配置
[root@clouderaManager01 local]# mkdir jdk [root@clouderaManager01 local]# chown -R hadoop:hadoop jdk
[hadoop@clouderaManager01 jdk]$ pwd /usr/local/jdk [hadoop@clouderaManager01 jdk]$ ls [hadoop@clouderaManager01 jdk]$ rz [hadoop@clouderaManager01 jdk]$ ll total 176996 -rw-r--r-- 1 hadoop hadoop 181238643 Apr 7 09:17 jdk-8u60-linux-x64.tar.gz [hadoop@clouderaManager01 jdk]$
解压jdk
删除jdk压缩包，并赋予用户和用户组。
[hadoop@clouderaManager01 jdk]$ pwd /usr/local/jdk [hadoop@clouderaManager01 jdk]$ ll total 177000 drwxr-xr-x 8 hadoop hadoop 4096 Aug 5 2015 jdk1.8.0_60 -rw-r--r-- 1 hadoop hadoop 181238643 Apr 7 09:17 jdk-8u60-linux-x64.tar.gz [hadoop@clouderaManager01 jdk]$ rm jdk-8u60-linux-x64.tar.gz [hadoop@clouderaManager01 jdk]$ ll total 4 drwxr-xr-x 8 hadoop hadoop 4096 Aug 5 2015 jdk1.8.0_60 [hadoop@clouderaManager01 jdk]$
配置jdk环境变量并使用source命令环境变量生效
#java export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib export PATH=$PATH:$JAVA_HOME/bin
使用java -version命令查看JDK是否安装成功
[root@clouderaManager01 jdk]# source /etc/profile [root@clouderaManager01 jdk]# su hadoop
注意上面操作要在所有节点都执行
（14）以及，我们还可以设置文件打开数量和用于最大进程数
文件打开数量
ulimit -a
[hadoop@clouderaManager01 ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7685 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [hadoop@clouderaManager01 ~]$
默认是1024，为此，我们需要修改
用户最大进程数
[hadoop@clouderaManager01 ~]$ ulimit -u 1024 [hadoop@clouderaManager01 ~]$
[hadoop@clouderaManager01 ~]$ sudo vi /etc/security/limits.conf
# /etc/security/limits.conf # #Each line describes a limit for a user in the form: # #<domain> <type> <item> <value> # #Where: #<domain> can be: # - an user name # - a group name, with @group syntax # - the wildcard *, for default entry # - the wildcard %, can be also used with %group syntax, # for maxlogin limit # #<type> can have the two values: # - "soft" for enforcing the soft limits # - "hard" for enforcing hard limits # #<item> can be one of the following: # - core - limits the core file size (KB) # - data - max data size (KB) # - fsize - maximum filesize (KB) # - memlock - max locked-in-memory address space (KB) # - nofile - max number of open files # - rss - max resident set size (KB) # - stack - max stack size (KB) # - cpu - max CPU time (MIN) # - nproc - max number of processes # - as - address space limit (KB) # - maxlogins - max number of logins for this user # - maxsyslogins - max number of logins on the system # - priority - the priority to run user process with # - locks - max number of file locks the user can hold # - sigpending - max number of pending signals # - msgqueue - max memory used by POSIX message queues (bytes) # - nice - max nice priority allowed to raise to values: [-20, 19] # - rtprio - max realtime priority # #<domain> <type> <item> <value> # #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 # End of file
* soft nofile 65535 * hard nofile 65535 * soft nproc 32000 * hard nproc 32000
同理，现在的主机名是clouderaManager01，拿这个来说，其他的clouderaManager02和clouderaManager03是一样的。
OK，到此为止我们的准备工作基本就都做好了，接下来我们就先来部署时间服务器和时间客户端工作。请移步，