Hadoop 3.1.0 在 Ubuntu 16.04 上的安装过程

安装过程主要参考官方文档：

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

目标：

Set up and configure a single-node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

部署安装一个单节点的 Hadoop ，以便使用 Hadoop MapReduce、HDFS 完成一些简单操作。

软件准备：

Ubuntu 16.04

这里使用 VMware 搭建 Ubuntu 16.04 的虚拟机系统

Hadoop 3.1.0

Hadoop 安装包下载镜像站：Apache Download Mirrors

这里下载的版本是： hadoop-3.1.0.tar.gz

解压下载的 Hadoop 压缩包：

$ tar -zxvf hadoop-3.0.0.tar.gz

Openjdk 8.0

Version 2.7 and later of Apache Hadoop requires Java 7. It is built and tested on both OpenJDK and Oracle (HotSpot)'s JDK/JRE.

Earlier versions (2.6 and earlier) support Java 6.

Here are the known JDKs in use or which have been tested:

Version	Status	Reported By
oracle 1.7.0_15	Good	Cloudera
oracle 1.7.0_21	Good (4)	Hortonworks
oracle 1.7.0_45	Good	Pivotal
openjdk 1.7.0_09-icedtea	Good (5)	Hortonworks
oracle 1.6.0_16	Avoid (1)	Cloudera
oracle 1.6.0_18	Avoid	Many
oracle 1.6.0_19	Avoid	Many
oracle 1.6.0_20	Good (2)	LinkedIn, Cloudera
oracle 1.6.0_21	Good (2)	Yahoo!, Cloudera
oracle 1.6.0_24	Good	Cloudera
oracle 1.6.0_26	Good(2)	Hortonworks, Cloudera
oracle 1.6.0_28	Good	LinkedIn
oracle 1.6.0_31	Good(3, 4)	Cloudera, Hortonworks

$ sudo apt upgrade

$ sudo apt install openjdk-8-jre openjdk-8-jdk

JAVA 可以使用 Oracle 的JDK 或者直接安装 OpenJDK，这里安装的是 OpenJDK 8。

可以使用 java -version 来查看 java 版本。

安装 java 之后编辑 /etc/profile ，配置 Java、Hadoop 环境变量：

$ sudo vi /etc/profile

# set oracle jdk 、hadoop environment
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export HADOOP_HOME=/home/wu/hadoop-3.1.0

$ source /etc/profile

使用以下命令检查环境值：

$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-amd64

$ echo $HADOOP_HOME
/home/wu/hadoop-3.1.0

ssh

$ sudo apt install ssh

安装过程：

Prepare to Start the Hadoop Cluster

在 hadoop 安装目录下，编辑 etc/hadoop/hadoop-env.sh 文件：

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

可以测试一下 hadoop：

$ bin/hadoop

或者：

$ bin/hadoop version

Hadoop 3.1.0
Source code repository https://github.com/apache/hadoop -r 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
Compiled by centos on 2018-03-30T00:00Z
Compiled with protoc 2.5.0
From source with checksum 14182d20c972b3e2105580a1ad6990
This command was run using /home/wu/hadoop-3.1.0/share/hadoop/common/hadoop-common-3.1.0.jar

启动 Hadoop

启动hadoop集群有三种模式：

本地（独立）模式，Local (Standalone) Mode
伪分布式模式，Pseudo-Distributed Mode
完全分布式模式、Fully-Distributed Mode

1、针对本地模式 Standalone Operation ：

By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.

The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.

可以使用下面的一个例子简单使用 hadoop ：

  $ mkdir input
  $ cp etc/hadoop/*.xml input
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.1.jar grep input output 'dfs[a-z.]+'
  $ cat output/*

2、针对伪分布式模式 Pseudo-Distributed Mode ：

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

配置环境：

  $ sudo vi etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

$ sudo vi etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

设置 SSH 免密码登陆（Setup passphraseless ssh）：

先测试是否能免密码登陆（Now check that you can ssh to the localhost without a passphrase）：

  $ ssh localhost

如果需要密码才能登陆，则执行下面的命令（If you cannot ssh to localhost without a passphrase, execute the following commands）：

  $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  $ chmod 0600 ~/.ssh/authorized_keys

执行（Execution）：

The following instructions are to run a MapReduce job locally.

格式化文件系统：

  $ bin/hdfs namenode -format

启动 namenode、datanode 守护进程：

  $ sbin/start-dfs.sh

　　Starting namenodes on [localhost]
　　Starting datanodes
　　Starting secondary namenodes [ubuntu]

The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).

此时可以访问namenode的web服务：http://localhost:9870/，查看namenode健康状况，可以观察到有一个存活的datanode节点。

执行mapreduce任务

# 在分布式文件系统中创建用户目录
$ bin/hdfs dfs -mkdir /user  
$ bin/hdfs dfs -mkdir /user/root  
  
# 拷贝数据到分布式文件系统中  
$ bin/hdfs dfs -mkdir -p input  
$ bin/hdfs dfs -put etc/hadoop/*.xml input  
  
# 运行hadoop提供的mapreduce任务  
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.0.jar grep /input output 'dfs[a-z.]+'  
  
# 拷贝任务执行结果到本地文件系统中

　$ sudo bin/hdfs dfs -mkdir -p output

　$ bin/hdfs dfs -get output output

　$ cat output/*

  
# 或直接从分布式文件系统中查看计算结果  
# $ bin/hdfs dfs -cat output/*

最后，结束守护进程：

 $ sbin/stop-dfs.sh