hadoop ——完全分布式环境搭建

hadoop 完全分布式环境搭建

1、虚拟机角色分配：

192.168.44.184 hadoop02 NameNode/DataNode ResourceManager/NodeManager
192.168.44.185 hadoop03 DataNode NodeManager
192.168.44.186 hadoop04 DataNode NodeManager

每台机器配置好hostname,hosts：/etc/hostname 中配置好hostname

/etc/hosts 中配置好hosts：

192.168.44.184 hadoop02
192.168.44.185 hadoop03
192.168.44.186 hadoop04

2、sskey免登录设置

1）ssh-keygen -t rsa （会在.ssh目录生成公钥和私钥,每台机器都要生成）

2）以hadoop02为主，将公钥copy到各个机器

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop02
    ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop03
    ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop04

3、jdk的安装(在hadoop02上配置，另外两台机器稍后复制过去)

1）直接下载jdk包放到安装目录即可（jdk-8u131-linux-x64.tar.gz解压到安装目录 tar zxvf ./jdk-8u131-linux-x64.tar.gz -C /home/hadoop/app/）

2）配置环境变量直接在 .bash_profile配置

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131
    export PATH=$JAVA_HOME/bin:$PATH

4.下载安装hadoop-2.6.0-cdh5.7.0(在hadoop02上配置，另外两台机器稍后复制过去)

cdh下载地址：（http://archive.cloudera.com/cdh5/cdh/5/)

1）直接下载hadoop-2.6.0-cdh5.7.0.tar.gz包放到安装目录即可

2）配置环境变量

 export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
    export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH

5、配置修改hadoop相关配置文件(在hadoop02上配置，另外两台机器稍后复制过去)

1）hadoop_home/etc/hadoop/hadoop-env.sh 配置jdk

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131

2）etc/hadoop/core-site.xml

<property>
            <name>fs.defaultFS</name>
            <value>hdfs://hadoop02:8020</value>
        </property>
         
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/home/hadoop/app/tmp</value>
        </property>

3）etc/hadoop/hdfs-site.xml 设置副本系数和name、data的存放路径

<property>
            <name>dfs.replication</name>
            <value>3</value>
        </property>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>/home/hadoop/app/tmp/dfs/name</value>
        </property>
        <property>
           <name>dfs.datanode.data.dir</name>
           <value>/home/hadoop/app/tmp/dfs/data</value>
        </property>

4）etc/hadoop/yarn-site.xml

<property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>hadoop02</value>
        </property>

5）etc/hadoop/mapred-site.xml

<property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>

6）/etc/hadoop/slaves(标明从节点)

hadoop02
        hadoop03
        hadoop04

6、分发安装包到 hadoop03、hadoop03节点上

scp -r ~/app hadoop@hadoop03:~/    
        scp -r ~/app hadoop@hadoop04:~/
        scp  ~/.bash_profile hadoop@hadoop03:~/
        scp  ~/.bash_profile hadoop@hadoop04:~/

在hadoop03,hadoop04上分别source下.bash_profile文件

7、NameNode格式化：只要在hadoop02上格式化

bin/hdfs namenode -format

8、启动hadoop集群。在NameNode启动

sbin/start-all.sh

启动日志：

[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ ./sbin/start-all.sh
    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
    18/08/19 23:20:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Starting namenodes on [hadoop02]
    hadoop02: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop02.out
    hadoop02: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop02.out
    hadoop04: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop04.out
    hadoop03: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop03.out
    Starting secondary namenodes [0.0.0.0]
    The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
    ECDSA key fingerprint is 78:05:94:d6:0b:e2:78:7d:93:3c:90:d0:ab:7c:ed:e2.
    Are you sure you want to continue connecting (yes/no)? yes
    0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
    0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop02.out
    18/08/19 23:21:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    starting yarn daemons
    starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop02.out
    hadoop03: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop03.out
    hadoop04: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop04.out
    hadoop02: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop02.out

9、验证

1）jps验证：

[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ jps
    3862 SecondaryNameNode
    4086 NodeManager
    3591 NameNode
    3991 ResourceManager
    3678 DataNode
    4399 Jps

[hadoop@hadoop03 ~]$ jps
    3460 NodeManager
    3576 Jps
    3372 DataNode

[hadoop@hadoop04 ~]$ jps
    3284 NodeManager
    3400 Jps
    3196 DataNode

2）web页面验证：http://hadoop02:50070 （hdfs）

http://hadoop02:8088 （yarn）

10、停止

sbin/stop-all.sh