想学习运行一个mapreduce程序试试，记录如下
本地运行word count
新建maven项目，添加hadoop-client，版本比如3.1.2
官方的wordcount直接拿来用就可以
需要把winutils.exe和hadoop.dll放到环境变量HADOOP_HOME中，这两个文件夹需要在bin子文件夹中，下载链接
添加两个运行参数，一个输入文件名，一个输出文件名，直接就可以运行了
单服务器配置（伪分布式
服务器使用centos安装，主机名c1，静态ip地址
使用rpm安装了oracle的jdk8，解压hadoop-3.1.2到/hadoop/文件夹下
ssh-keygen
ssh-copy-id localhost
core-site.xml如下
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://c1:8020</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>c1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
这时候运行还是会出错，找不到MRAppMaster类，类似如下：
Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.reduce.e nv</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
修改如下
mapred-site.xml
<configuration>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2</value>
</property>
</configuration>
修改workers或slaves文件，不同版本的hadoop文件名不一样
把localhost改成主机名，比如c1
要在/etc/hosts加入c1的静态地址，否则，就可能连不上nodemanager，比如
192.168.1.111 c1
启动dfs/yarn服务
格式化只需要第一次运行一下。
bin/hadoop namenode -format
bin/start-dfs.sh
bin/start-yarn.sh
stop之后想再start可能需要等待文件列表同步，正常要等30秒，可以在网页startup信息看。
开发机可以看到c1的50070（hadoop2）或9870（hadoop3）端口就说明hdfs起来了
可以看到8088端口说明yarn起来了
修改程序，加入YARN配置
wordcount程序需要服务端的xml配置，可以放到conf文件夹，把这个文件夹在IDE里点右键设置为Resources，不然程序写相对路径也访问不到
修改wordcount程序，加入xml配置
Configuration conf = new Configuration();
conf.addResource("core-site.xml");
conf.addResource("yarn-site.xml");
conf.addResource("hdfs-site.xml");
conf.addResource("mapred-site.xml");
conf.set("mapreduce.app-submission.cross-platform","true");
conf.set("mapreduce.framework.name","yarn");
conf.set("mapreduce.job.jar","target\mr1-1.0-SNAPSHOT.jar");
在maven工具中调用package，生成的jar文件名和路径，写到上面程序最后一行
在hdfs中放一个input.txt文件作为输入，如果有权限问题可以参考命令
hdfs dfs -chown user:group input.txt
hdfs dfs -chmod 777 input.txt
提交到yarn运行程序
结果如下
2019-03-26 16:49:33,942 INFO [main] client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at c1/192.168.1.111:8032
2019-03-26 16:49:34,363 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadResourcesInternal(147)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-03-26 16:49:34,372 INFO [main] mapreduce.JobResourceUploader (JobResourceUploader.java:disableErasureCodingForPath(883)) - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/cdarling/.staging/job_1553590163847_0001
2019-03-26 16:49:34,500 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(292)) - Total input files to process : 1
2019-03-26 16:49:35,349 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:1
2019-03-26 16:49:35,824 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_1553590163847_0001
2019-03-26 16:49:35,826 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: []
2019-03-26 16:49:35,932 INFO [main] conf.Configuration (Configuration.java:getConfResourceAsInputStream(2752)) - resource