Hadoop生态圈-Oozie实战之逻辑调度执行多个Job
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
1>.启动hadoop集群
[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]# sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [s101] s101: starting namenode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-namenode-yinzhengjie.out s102: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-datanode-s102.out s104: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-datanode-s104.out s103: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-datanode-s103.out Starting secondary namenodes [s101] s101: starting secondarynamenode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-secondarynamenode-yinzhengjie.out starting yarn daemons starting resourcemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-yinzhengjie-resourcemanager-s101.out s104: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-root-nodemanager-s104.out s102: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-root-nodemanager-s102.out s103: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-root-nodemanager-s103.out [root@yinzhengjie hadoop-2.5.0-cdh5.3.6]#
[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]# sbin/mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/mapred-yinzhengjie-historyserver-s101.out [root@yinzhengjie hadoop-2.5.0-cdh5.3.6]#
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# bin/oozied.sh start Setting OOZIE_HOME: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6 Setting OOZIE_CONFIG: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/conf Sourcing: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/conf/oozie-env.sh setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" Setting OOZIE_CONFIG_FILE: oozie-site.xml Setting OOZIE_DATA: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/data Setting OOZIE_LOG: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs Setting OOZIE_LOG4J_FILE: oozie-log4j.properties Setting OOZIE_LOG4J_RELOAD: 10 hostname: Name or service not known Setting OOZIE_HTTP_HOSTNAME: Setting OOZIE_HTTP_PORT: 11000 Setting OOZIE_ADMIN_PORT: 11001 Setting OOZIE_HTTPS_PORT: 11443 Setting OOZIE_BASE_URL: http://:11000/oozie Setting CATALINA_BASE: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server Setting OOZIE_HTTPS_KEYSTORE_FILE: /root/.keystore Setting OOZIE_HTTPS_KEYSTORE_PASS: password Setting OOZIE_INSTANCE_ID: Setting CATALINA_OUT: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs/catalina.out Setting CATALINA_PID: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp/oozie.pid Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs/derby.log Adding to CATALINA_OPTS: -Doozie.home.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6 -Doozie.config.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/conf -Doozie.log.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs -Doozie.data.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/data -Doozie.instance.id= -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname= -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path= Using CATALINA_BASE: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server Using CATALINA_HOME: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server Using CATALINA_TMPDIR: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp Using JRE_HOME: /soft/jdk Using CLASSPATH: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/bin/bootstrap.jar Using CATALINA_PID: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp/oozie.pid Existing PID file found during start. Removing/clearing stale PID file. [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# xcall.sh jps ============= s101 jps ============ 3376 ResourceManager 3223 SecondaryNameNode 3753 JobHistoryServer 4028 Jps 3853 Bootstrap 3071 NameNode 命令执行成功 ============= s102 jps ============ 3330 Jps 3052 NodeManager 2975 DataNode 命令执行成功 ============= s103 jps ============ 2978 DataNode 3333 Jps 3055 NodeManager 命令执行成功 ============= s104 jps ============ 2528 NodeManager 2458 DataNode 2795 Jps 命令执行成功 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
查看oozie界面是否启动成功：
2>.解压官方案例模板
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# tar -zxf oozie-examples.tar.gz [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
3>.编写脚本
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/test-1.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com /bin/date -d today +"%Y-%m-%d %T" > /home/yinzhengjie/data/access-1.log [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/test-2.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com /bin/date -d today +"%Y-%m-%d %T" > /home/yinzhengjie/data/access-2.log [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
4>.编辑job.properties配置文件
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# more yinzhengjie-oozie-jobs/shell/job.properties # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # #HDFS地址 nameNode=hdfs://s101:8020 #ResourceManager地址 jobTracker=s101:8021 #队列名称 queueName=default examplesRoot=yinzhengjie-oozie-jobs #指定oozie的shell脚本存放路径 oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/shell #指定执行的脚本名称 EXEC1=test-1.sh EXEC2=test-2.sh [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
5>.编辑workflow.xml 配置文件
1 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/workflow.xml 2 <workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf"> 3 <start to="yinzhengjie-shell-node1"/> 4 <action name="yinzhengjie-shell-node1"> 5 <shell xmlns="uri:oozie:shell-action:0.2"> 6 <job-tracker>${jobTracker}</job-tracker> 7 <name-node>${nameNode}</name-node> 8 <configuration> 9 <property> 10 <name>mapred.job.queue.name</name> 11 <value>${queueName}</value> 12 </property> 13 </configuration> 14 <exec>${EXEC1}</exec> 15 <file>/user/root/yinzhengjie-oozie-jobs/shell/${EXEC1}#${EXEC1}</file> 16 <!-- <argument>my_output=Hello Oozie</argument>--> 17 <capture-output/> 18 </shell> 19 <ok to="yinzhengjie-shell-node2"/> 20 <error to="fail"/> 21 </action> 22 23 <action name="yinzhengjie-shell-node2"> 24 <shell xmlns="uri:oozie:shell-action:0.2"> 25 <job-tracker>${jobTracker}</job-tracker> 26 <name-node>${nameNode}</name-node> 27 <configuration> 28 <property> 29 <name>mapred.job.queue.name</name> 30 <value>${queueName}</value> 31 </property> 32 </configuration> 33 <exec>${EXEC2}</exec> 34 <file>/user/root/yinzhengjie-oozie-jobs/shell/${EXEC2}#${EXEC2}</file> 35 <!-- <argument>my_output=Hello Oozie</argument>--> 36 <capture-output/> 37 </shell> 38 <ok to="end"/> 39 <error to="fail"/> 40 </action> 41 <kill name="fail"> 42 <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 43 </kill> 44 <end name="end"/> 45 </workflow-app> 46 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
6>.上传任务配置到hdfs
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put yinzhengjie-oozie-jobs/shell/ /user/root/yinzhengjie-oozie-jobs/shell [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -ls -R /user/root/yinzhengjie-oozie-jobs/shell -rw-r--r-- 1 root supergroup 234 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/blog.sh -rw-r--r-- 1 root supergroup 1121 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/job.properties -rw-r--r-- 1 root supergroup 175 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/test-1.sh -rw-r--r-- 1 root supergroup 175 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/test-2.sh -rw-r--r-- 1 root supergroup 1702 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/workflow.xml [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
7>.执行任务
[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# bin/oozie job -oozie http://s101:11000/oozie -config yinzhengjie-oozie-jobs/shell/job.properties -run job: 0000000-180907200401168-oozie-root-W [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#
8>.
9>.