hadoop 2.5.1 、Hadoop 2.7 Hadoop 2.6

1 rpm 安装

yum install rpm

2 JDK安装

《《一定要先删除JDK！！！！！！》》

rpm -qa | grep java

rpm -e --nodeps

rpm -qa | grep java

下载 JDK

tar -zxvf or xvf java

环境变量更改

/usr/local/opt/jdk1.7.0_67

jdk环境变量

8.1) vi /etc/profile

8.2)

export JAVA_HOME=/usr/local/opt/jdk1.7.0_67

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

1 下载Hadoop

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.1/hadoop-2.5.1.tar.gz

wget http://mirror.bit.edu.cn/apache/hbase/hbase-1.0.2/hbase-1.0.2-bin.tar.gz

wget http://mirror.bit.edu.cn/apache/hbase/1.1.2/hbase-1.1.2-bin.tar.gz

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

如果环境变量崩了就子机whereis vim 应该是这样 /usr/bin/vim /etc/profile

hadoop 环境变量

vi /etc/profile

/usr/local/opt/hadoop-2.5.1

/home/hadoop/hadoop/hadoop-2.6.0

export HADOOP_HOME=/usr/local/opt/hadoop-2.7.2

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$HIVE_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin

hive 环境变量

export HIVE_HOME=/usr/hive-0.11.0

4 hdfs 初始化 yarn 初始化

进入hadoop下面的etc/hadoop/不是系统的etc，是hadoop下面的），ls，看到很多配置文件。

修改core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://t1:9000/</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-data/tmp/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
</configuration>

修改hdfs-site.xml

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value></value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hdfs/data</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.federation.nameservices</name>
<value>t1</value>
</property>
<property>
<name>dfs.namenode.rpc-address.master</name>
<value>t1:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.master</name>
<value>t1:23001</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address.master</name>
<value>t1:23002</value>
</property>

</configuration>

修改mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

</property>

</configuration>

修改yarn-site.xml

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager

</description>

<name>yarn.resourcemanager.resource-tracker.address</name>

</property>

the hostsis the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager.

</description>

<name>yarn.resourcemanager.address</name>

</property>

host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager.

</description>

<name>yarn.resourcemanager.scheduler.address</name>

</property>

<description>The address of the RM admin interface.</description>

<name>yarn.resourcemanager.admin.address</name>

</property>

<description>Resource Manager的Web访问地址</description>

<name>yarn.resourcemanager.webapp.address</name>

</property>

<name>yarn.nodemanager.local-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/local</value>

<description>the local directories used by the nodemanager to store the temp file</description>

</property>

<name>yarn.nodemanager.log-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/logs</value>

<description>the directories used by Nodemanagers as log directories</description>

</property>

</configuration>

修改 hadoop-env.sh、yarn-env.sh

export JAVA_HOME=/usr/local/jdk

5 配置 slave

vi etc/hadoop/slaves

加入

6 HDFS 初始化

./bin/hdfs namenode -format

如果 format 出异常 JAVA环境变量监察一下。

7 将 hadoop 目录分发至其余节点

整文件夹拷贝

scp -r file root@slave1:/path

8 启动所有。

./sbin/start-all.sh

http://master:50070/ hdfs

http://master:50030/ mapReduce

成功后可以运行个例子

hadoop-2.7.2/share/hadoop/mapreduce 这里有一坨。

**************************** HDFS ***************************

yum install gcc

查看系统的libc版本
# ll /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Apr 24 16:49 /lib64/libc.so.6 -> libc-2.5.so
系统中的版本为2.5
将系统中的glibc升级为2.9
下载glibc
wget http://ftp.gnu.org/gnu/glibc/glibc-2.9.tar.bz2
下载glibc-linuxthreads
wget http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.5.tar.bz2
解压
$tar -jxvf glibc-2.9.tar.bz2
$cd glibc-2.9
$tar -jxvf ../glibc-linuxthreads-2.5.tar.bz2
$cd ..
$export CFLAGS="-g -O2"
$./glibc-2.9/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
$make
#make install
安装编译过程中需要注意三点：
1、要将glibc-linuxthreads解压到glibc目录下。
2、不能在glibc当前目录下运行configure。
3、加上优化开关，export CFLAGS="-g -O2"，否则会出现错误

***************************** HIVE *****************************

http://apache.fayea.com/hive/

God has given me a gift. Only one. I am the most complete fighter in the world. My whole life, I have trained. I must prove I am worthy of someting. rocky_24