Centos 虚拟机 Hadoop个人主机命令简记（10.5~10.11每周总结）

一丶脚本命令

1. xsync: 同步三台主机的文件修改一台文件后使用该命令可以实现多台的同步修改

使用方法 xsync +文件路径 +目标文件路径

#!/bin/bash
#1. 判断参数
if [ $# -lt 1  ]
then
  echo Not Enough Arguement!
  exit;
fi
#2. 遍历集群所有机器
for host in hadoop102 hadoop103 hadoop104
do
echo =================  $host  ================
#3. 遍历所有目录，挨个发送
for file in $@
do
  #4 判断文件是否存在
  if [ -e $file  ]
  then
      #5. 获取父目录
      pdir=$(cd -P $(dirname $file); pwd)
      #6. 获取当前文件名称
      fname=$(basename $file)
      ssh $host "mkdir -p $pdir"
      rsync -av $pdir/$fname $host:$pdir 
  else
        echo $file does not exists!    
  fi
 done
done

2. mycluster.sh: 群起或者群停 Hadoop 结点

使用： mycluster.sh start/stop

#!/bin/bash
if [ $# -lt 1 ]
then
   echo "Input Args Error!!!!"
   exit
fi

case $1 in
start)
     echo "========== START HDFS =========="
     ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/start-dfs.sh
     echo "========== START YARN =========="
     ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/start-yarn.sh    
;;
stop)
     echo "========== STOP YARN =========="
     ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/stop-yarn.sh 
     echo "========== STOP HDFS =========="
     ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/stop-dfs.sh
;;
*)
    echo "Input Args Error!!!!"
;;
esac

3. myjps: 查看三个结点的数据

执行操作：myjps.sh

#!/bin/bash
if [ $# -lt 1 ]
then
   echo "Input Args Error!!!!"
   exit
fi

case $1 in
start)
     echo "========== START HDFS =========="
     ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/start-dfs.sh
     echo "========== START YARN =========="
     ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/start-yarn.sh    
;;
stop)
     echo "========== STOP YARN =========="
     ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/stop-yarn.sh 
     echo "========== STOP HDFS =========="
     ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/stop-dfs.sh
;;
*)
    echo "Input Args Error!!!!"
;;
esac
[atguigu@hadoop102 bin]$ cat myjps.sh
#!/bin/bash
for i in hadoop102 hadoop103 hadoop104
do
  echo "========== $i jps =========="
  ssh $i /opt/module/jdk1.8.0_212/bin/jps
done

二丶HDFS 操作简记

1. hadoop fs -help 帮助命令: 列出所有命令帮助信息

　hadoop fs -help （具体命令）可以看到具体命令帮助信息

2.hadoop fs -ls [hdfs当中的目录]: 显示目录下的所有文件和文件的权限

　hadoop fs -ls -R [hdfs当中的目录]: 显示目录下的所有文件

3. hadoop fs -cat [hdfs 文件路径] ：查看文件内容

4.hadoop fs -mkdir [hdfs 存储路径] ：创建目录

hadoop fs -mkdir -p [hdfs 存储路径] ：创建多级目录

5.hadoop fs -rm [文件路径] ：删除文件

hadoop fs -rm -r [文件路径] ：删除文件夹

6.hadoop fs -put [文件路径]: 复制文件

7. hadoop fs -cp [URI] [URI]：将文件从源路径复制到目标路径

三丶IDEA 连接Hadoop 需要的Moven项目

 <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>

        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>2.12.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.1.3</version>
        </dependency>
     <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-yarn-common</artifactId>
         <version>3.1.3</version>
     </dependency>

 </dependencies>