Hadoop 3.0 安装

1.      下载Hadoop 3.0

http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz

从零开始的 Win7 64Bit 编译Hadoop3 (此步无需执行,直接下载第二步的链接库)

https://my.oschina.net/alexnine/blog/779794

2.      下载对应的链接库

https://github.com/steveloughran/winutils

将对应hadoop.dll和winutils.exe放置到hadoop-3.0.0in

3.      配置hadoop环境变量

创建HADOOP_HOME,另外在Path下添加 %HADOOP_HOME%in

4.      配置

http://blog.csdn.net/dream_an/article/details/51862640

(1)    到hadoop根目录,建立workplace

 

datanode

namenode

tmp

(2)    修改hadoop-3.0.0etchadoophadoop-env.cmd

修改JAVA_HOME为你自己jdk路径

注意:如果你的JDK安装在ProgramFiles目录下,名称用PROGRA~1Java 否则中间的空格可能会识别失败。

怎么跑到hadoop上不识别了呢?

在网上搜索了下资料发现原来是路径上包含了一个空格

所以有以下2个解决办法:

1.用路径替代符

C:PROGRA~1Javajdk1.8.0_91

PROGRA~1  ===== C:Program Files 目录的dos文件名模式下的缩写

长于8个字符的文件名和文件夹名,都被简化成前面6个有效字符,后面~1,有重名的就 ~2,~3,

2.用引号括起来

"C:ProgramFiles"Javajdk1.8.0_91

 

                  

(3)    hdfs-site.xml

<configuration>

<!-- 这个参数设置为1,因为是单机版hadoop -->

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>/D:/Tools/Hadoop/hadoop-3.0.0/workplace/datanode</value>

</property>

</configuration>

(4)    mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapred.job.tracker</name>

<value>hdfs://localhost:9001</value>

</property>

</configuration>

(5)    core-site.xml

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>/D:/Tools/Hadoop/hadoop-3.0.0/workplace/tmp</value>

</property>

        

<property>

<name>dfs.name.dir</name>

<value>/D:/Tools/Hadoop/hadoop-3.0.0/workplace/namenode</value>

</property>

         <property>

                   <name>fs.default.name</name>

                   <value>hdfs://localhost:9000</value>

         </property>

</configuration>

(6)    yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

</configuration>

5.      格式化namenode

%HADOOP_HOME%inhdfsnamenode –format

 

6.      启动namenode和datanode

%HADOOP_HOME%sbinstart-dfs.cmd

 

7.      启动YARN

%HADOOP_HOME%sbinstart-yarn.cmd

8.      上传文件到文件系统

%HADOOP_HOME%inhdfs dfs -put D:\myfile.txt /

查看

%HADOOP_HOME%inhdfs dfs -ls /

 

9.      运行例子

%HADOOP_HOME%inyarn jar%HADOOP_HOME%sharehadoopmapreducehadoop-mapreduce-examples-3.0.0.jarwordcount /myfile.txt /out

(1)    CreateSymbolicLink error(1314): A required privilege is not held by the client

以管理员权限启动cmd àHadoop

(2)    运行

 

(3)    跑第二遍org.apache.hadoop.mapred.FileAlreadyExistsException: Outputdirectory hdfs://localhost:9000/out already exists

【Mapreduce】从代码上解决Output directory already exists错误,避免每次调试都要手动删除输出文件夹

http://blog.csdn.net/yongh701/article/details/50601811

http://dblab.xmu.edu.cn/blog/hadoop-build-project-using-eclipse/

1)      在Eclipse中操作HDFS 中的文件

http://dblab.xmu.edu.cn/blog/hadoop-build-project-using-eclipse/

 

 

2)      查看HDFS文件系统数据的三种方法

http://blog.csdn.net/flyfish111222/article/details/51995523

3)      在eclipse中删除out


4)      重新跑

 

10.  结束Hadoop

%HADOOP_HOME%sbinstop-yarn.cmd

%HADOOP_HOME%sbinstop-dfs.cmd

原文地址:https://www.cnblogs.com/xiang--liu/p/9710395.html