马士兵hadoop第一课：虚拟机搭建和安装hadoop及启动
马士兵hadoop第二课：hdfs集群集中管理和hadoop文件操作
马士兵hadoop第四课：Yarn和Map/Reduce配置启动和原理讲解
(1)关于hdfs小结
hadoop由hdfs + yarn + map/reduce组成，
hdfs是数据库存储模块，主要由1台namenode和n台datanode组成的一个集群系统，
datanode可以动态扩展，文件根据固定大小分块（默认为128M），
每一块数据默认存储到3台datanode，故意冗余存储，防止某一台datanode挂掉，数据不会丢失。
HDFS = NameNode + SecondaryNameNode + journalNode + DataNode
hdfs的典型应用就是：百度云盘
（2）修改hadoop.tmp.dir默认值
hadoop.tmp.dir默认值为/tmp/hadoop-${user.name}，由于/tmp目录是系统重启时候会被删除，所以应该修改目录位置。
修改core-site.xml（在所有节点上都修改）
[root@master ~]# vim core-site.xml
修改完namenode和datanode上的hadoop.tmp.dir参数后，需要格式化namenode，在master上执行：
[root@master ~]# hdfs namenode -format
（4）测试期间关闭权限检查
为了简单起见，需要关闭权限检查，需要在namenode的hdfs-site.xml上，添加配置：
<property> <name>dfs.permissions.enabled</name> <value>false</value> </property>
重新启动namenode:
[root@master ~]# hadoop-daemon.sh stop namenode
[root@master ~]# hadoop-daemon.sh start namenode
(5) 使用FileSyste类来读写hdfs
package com.hadoop.hdfs; import java.io.FileInputStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path; public class HelloHDFS {</span><span style="color: #0000ff">public</span> <span style="color: #0000ff">static</span> Log log = LogFactory.getLog(HelloHDFS.<span style="color: #0000ff">class</span><span style="color: #000000">); </span><span style="color: #0000ff">public</span> <span style="color: #0000ff">static</span> <span style="color: #0000ff">void</span> main(String[] args) <span style="color: #0000ff">throws</span><span style="color: #000000"> Exception { Configuration conf </span>= <span style="color: #0000ff">new</span><span style="color: #000000"> Configuration(); conf.set(</span>"fs.defaultFS", "hdfs://192.168.56.100:9000"<span style="color: #000000">); conf.set(</span>"dfs.replication", "2"<span style="color: #000000">);//默认为3 FileSystem fileSystem </span>=<span style="color: #000000"> FileSystem.get(conf); </span><span style="color: #0000ff">boolean</span> success = fileSystem.mkdirs(<span style="color: #0000ff">new</span> Path("/yucong"<span style="color: #000000">)); log.info(</span>"创建文件是否成功:" +<span style="color: #000000"> success); </span><span style="color: #0000ff">success</span> = fileSystem.exists(<span style="color: #0000ff">new</span> Path("/yucong"<span style="color: #000000">)); log.info(</span>"文件是否存在:" +<span style="color: #000000"> success); success </span>= fileSystem.delete(<span style="color: #0000ff">new</span> Path("/yucong"), <span style="color: #0000ff">true</span><span style="color: #000000">); log.info(</span>"删除文件是否成功：" +<span style="color: #000000"> success); </span><span style="color: #008000">/*</span><span style="color: #008000">FSDataOutputStream out = fileSystem.create(new Path("/test.data"), true); FileInputStream fis = new FileInputStream("c:/test.txt"); IOUtils.copyBytes(fis, out, 4096, true);</span><span style="color: #008000">*/</span><span style="color: #000000"> FSDataOutputStream out </span>= fileSystem.create(<span style="color: #0000ff">new</span> Path("/test2.data"<span style="color: #000000">)); FileInputStream in </span>= <span style="color: #0000ff">new</span> FileInputStream("c:/test.txt"<span style="color: #000000">); </span><span style="color: #0000ff">byte</span>[] buf = <span style="color: #0000ff">new</span> <span style="color: #0000ff">byte</span>[4096<span style="color: #000000">]; </span><span style="color: #0000ff">int</span> len =<span style="color: #000000"> in.read(buf); </span><span style="color: #0000ff">while</span>(len != -1<span style="color: #000000">) { out.write(buf,</span>0<span style="color: #000000">,len); len </span>=<span style="color: #000000"> in.read(buf); } in.close(); out.close(); FileStatus[] statuses </span>= fileSystem.listStatus(<span style="color: #0000ff">new</span> Path("/"<span style="color: #000000">)); log.info(statuses.length); </span><span style="color: #0000ff">for</span><span style="color: #000000">(FileStatus status : statuses) { log.info(status.getPath()); log.info(status.getPermission()); log.info(status.getReplication()); } }}
这是一个maven项目，pom.xml文件为：
<dependencies><span style="color: #0000ff"><</span><span style="color: #800000">dependency</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">groupId</span><span style="color: #0000ff">></span>org.apache.hadoop<span style="color: #0000ff"></</span><span style="color: #800000">groupId</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">artifactId</span><span style="color: #0000ff">></span>hadoop-common<span style="color: #0000ff"></</span><span style="color: #800000">artifactId</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">version</span><span style="color: #0000ff">></span>2.7.3<span style="color: #0000ff"></</span><span style="color: #800000">version</span><span style="color: #0000ff">></span> <span style="color: #0000ff"></</span><span style="color: #800000">dependency</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">dependency</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">groupId</span><span style="color: #0000ff">></span>org.apache.hadoop<span style="color: #0000ff"></</span><span style="color: #800000">groupId</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">artifactId</span><span style="color: #0000ff">></span>hadoop-hdfs<span style="color: #0000ff"></</span><span style="color: #800000">artifactId</span><span style="color: #0000ff">></span> <span style="color: #0000ff"><</span><span style="color: #800000">version</span><span style="color: #0000ff">></span>2.7.3<span style="color: #0000ff"></</span><span style="color: #800000">version</span><span style="color: #0000ff">></span> <span style="color: #0000ff"></</span><span style="color: #800000">dependency</span><span style="color: #0000ff">></span></dependencies>
马士兵视频课程百度云盘下载：http://pan.baidu.com/s/1kVSbxS7