Hadoop完全分布式模式

集群配置

1. 集群部署规划

表2-3

hadoop002

hadoop003

hadoop004

HDFS

NameNode

DataNode

SecondaryNameNode

DataNode

YARN

NodeManager

ResourceManager

NodeManager

2. 配置集群

（1）核心配置文件

配置core-site.xml

[root@hadoop002 hadoop]# vim core-site.xml
[root@hadoop002 hadooop]#

<configuration>
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
    <value>hdfs://hadoop002:9000</value>
</property>

<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop-2.7.2/data/tmp</value>
</property>


</configuration>

（2）HDFS配置文件

配置hdfs-site.xml

[root@hadoop002 hadoop]# vim  hdfs-site.xml
[root@hadoop002 hadoop]#

<configuration>
<property>
        <name>dfs.replication</name>
        <value>3</value>
</property>
<!-- 指定Hadoop辅助名称节点主机配置 -->
<property>
      <name>dfs.namenode.secondary.http-address</name>
      <value>hadoop004:50090</value>
</property>

</configuration>

（3）YARN配置文件

配置yarn-site.xml

[root@hadoop002 hadoop]#  vim yarn-site.xml
[root@hadoop002 hadoop]#

在该文件中增加如下配置

<configuration>

<!-- Reducer获取数据的方式 -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop003</value>
</property>

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<!-- 日志保留时间设置7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

</configuration>

（4）MapReduce配置文件

配置mapred-site.xml

[root@hadoop102 hadoop]$ cp mapred-site.xml.template mapred-site.xml

<configuration>
<!-- 指定MR运行在YARN上 -->
      <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
       </property>

<!-- 历史服务器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop001:10020</value>
</property>
<!-- 历史服务器web端地址 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop001:19888</value>
</property>

</configuration>

3．在集群上分发配置好的Hadoop配置文件

[root@hadoop102 hadoop]$ xsync /opt/module/hadoop-2.7.2/

4．查看文件分发情况

[root@hadoop103 hadoop]$ cat /opt/module/hadoop-2.7.2/etc/hadoop/core-site.xml

4.3.4 集群单点启动

（1）如果集群是第一次启动，需要格式化NameNode

[root@hadoop102 hadoop-2.7.2]$ hadoop namenode -format

（2）在hadoop102上启动NameNode

sbin/hadoop-daemon.sh start datanode

[root@hadoop102 hadoop-2.7.2]$ jps

3461 NameNode

（3）在hadoop002、hadoop003以及hadoop004上分别启动DataNode

[root@hadoop002 hadoop-2.7.2]$ hadoop-daemon.sh start datanode
[root@hadoop002 hadoop-2.7.2]$ jps
3461 NameNode
3608 Jps
3561 DataNode

[root@hadoop003 hadoop-2.7.2]$ hadoop-daemon.sh start datanode
[root@hadoop003 hadoop-2.7.2]$ jps
3190 DataNode
3279 Jps

[root@hadoop004 hadoop-2.7.2]$ hadoop-daemon.sh start datanode
[root@hadoop004 hadoop-2.7.2]$ jps
3237 Jps
3163 DataNode

其中要群起集群的话，可以看这边博客，要设置一下SSH:https://www.cnblogs.com/dalianpai/p/12194726.html

群起集群

1. 配置slaves

/opt/module/hadoop-2.7.2/etc/hadoop/slaves

[root@hadoop002 hadoop]# cat slaves
hadoop002
hadoop003
hadoop004
[root@hadoop002 hadoop]#

注意：该文件中添加的内容结尾不允许有空格，文件中不允许有空行。

同步所有节点配置文件

[root@hadoop002 hadoop]# xsync slaves
fname=slaves
pdir=/opt/module/hadoop-2.7.2/etc/hadoop
------------------- hadoop3 --------------
sending incremental file list
slaves

sent 119 bytes  received 41 bytes  106.67 bytes/sec
total size is 30  speedup is 0.19
------------------- hadoop4 --------------
sending incremental file list
slaves

sent 119 bytes  received 41 bytes  320.00 bytes/sec
total size is 30  speedup is 0.19
[root@hadoop002 hadoop]# jps
24551 NameNode
24475 DataNode
25868 Jps

2. 启动集群

（1）如果集群是第一次启动，需要格式化NameNode（注意格式化之前，一定要先停止上次启动的所有namenode和datanode进程，然后再删除data和log数据）

[root@hadoop002 hadoop]# cd ../../
[root@hadoop002 hadoop-2.7.2]# sbin/hadoop-daemon.sh stop datanode
stopping datanode
[root@hadoop002 hadoop-2.7.2]# sbin/hadoop-daemon.sh stop namenode
stopping namenode

[root@hadoop003 hadoop-2.7.2]# sbin/hadoop-daemon.sh stop datanode
stopping datanode
[root@hadoop003 hadoop-2.7.2]# jps
26702 Jps

[root@hadoop004 hadoop-2.7.2]# sbin/hadoop-daemon.sh stop datanode
stopping datanode
[root@hadoop004 hadoop-2.7.2]# jps
20994 Jps

[atguigu@hadoop102 hadoop-2.7.2]$ bin/hdfs namenode -format

（2）启动HDFS

root@hadoop002 hadoop-2.7.2]# sbin/start-dfs.sh
Starting namenodes on [hadoop002]
hadoop002: starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-namenode-hadoop002.out
hadoop002: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop002.out
hadoop003: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop003.out
hadoop004: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop004.out
Starting secondary namenodes [hadoop004]
hadoop004: starting secondarynamenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-hadoop004.out
[root@hadoop002 hadoop-2.7.2]# jps
26023 NameNode
26152 DataNode
26361 Jps

[root@hadoop004 hadoop-2.7.2]# jps
21122 SecondaryNameNode
21029 DataNode
21163 Jps

（3）启动YARN

[root@hadoop003 hadoop-2.7.2]#  sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-root-resourcemanager-hadoop003.out
hadoop004: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-root-nodemanager-hadoop004.out
hadoop002: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-root-nodemanager-hadoop002.out
hadoop003: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-root-nodemanager-hadoop003.out
[root@hadoop003 hadoop-2.7.2]# jps
26963 NodeManager
26739 DataNode
27256 Jps
26862 ResourceManager

注意：NameNode和ResourceManger如果不是同一台机器，不能在NameNode上启动 YARN，应该在ResouceManager所在的机器上启动YARN。

（4）Web端查看SecondaryNameNode

（a）浏览器中输入：http://hadoop004:50090/status.html

（b）查看SecondaryNameNode信息，如图

集群基本测试

（1）上传文件到集群

上传小文件

[root@hadoop002 hadoop-2.7.2]# bin/hdfs dfs -put wcinput/wc.input /
[root@hadoop002 hadoop-2.7.2]# ll

上传大文件

[root@hadoop002 hadoop-2.7.2]# bin/hdfs dfs -put /opt/module/jdk1.8.0_144/lib/tools.jar /
20/01/15 12:11:08 WARN hdfs.DFSClient: Slow waitForAckedSeqno took 39284ms (threshold=30000ms)

（2）上传文件后查看文件存放在什么位置

[root@hadoop002 hadoop-2.7.2]# cd data/tmp/dfs/data/current/BP-1442086029-172.16.25.74-1579016149783/current/
[root@hadoop002 current]# ll
total 16
-rw-r--r-- 1 root root   19 Jan 15 11:55 dfsUsed
drwxr-xr-x 3 root root 4096 Jan 15 12:05 finalized
drwxr-xr-x 2 root root 4096 Jan 15 12:11 rbw
-rw-r--r-- 1 root root  132 Jan 15 11:58 VERSION
[root@hadoop002 current]# cd finalized/subdir0/
[root@hadoop002 subdir0]# ll
total 4
drwxr-xr-x 2 root root 4096 Jan 15 12:11 subdir0
[root@hadoop002 subdir0]# cd subdir0/
[root@hadoop002 subdir0]# ll
total 18008
-rw-r--r-- 1 root root       48 Jan 15 12:05 blk_1073741825
-rw-r--r-- 1 root root       11 Jan 15 12:05 blk_1073741825_1001.meta
-rw-r--r-- 1 root root 18284613 Jan 15 12:11 blk_1073741826
-rw-r--r-- 1 root root   142859 Jan 15 12:11 blk_1073741826_1002.meta

假如文件大于128M，存储的时候会进行切分。

4.3.7 集群启动/停止方式总结

1. 各个服务组件逐一启动/停止

（1）分别启动/停止HDFS组件

hadoop-daemon.sh start / stop namenode / datanode / secondarynamenode

（2）启动/停止YARN

yarn-daemon.sh start / stop resourcemanager / nodemanager

2. 各个模块分开启动/停止（配置ssh是前提）常用

（1）整体启动/停止HDFS

start-dfs.sh / stop-dfs.sh

（2）整体启动/停止YARN

start-yarn.sh / stop-yarn.sh