BTrace追踪Hadoop部署

Hadoop集成BTrace

1.BTrace

1.1BTrace部署

1.下载BTrace工具包，官网地址：https://kenai.com/projects/btrace/downloads/directory/releases

2.解压btrace-bin.tar.gz，命令如下：

tar -zxvf btrace-bin.tar.gz

3.配置环境变量

BTRACE_HOME=/usr/btrace-bin

PATH=$PATH:$BTRACE_HOME/bin

4.执行btrace（注意：$BTRACE_HOME/bin下btrace、btracec、btracer文件权限）命令如下，表示部署成功：

1.2BTrace脚本

BTrace测试脚本代码：

 1 importcom.sun.btrace.annotations.*; 
 2  
 3 import static com.sun.btrace.BTraceUtils.*;
 4  
 5 @BTrace
 6 public class TraceMR
 7 {
 8        @TLS privatestaticlongmapperSetupStartTime = 0l;
 9  
10        @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper",
11                        method="run",
12                        location=@Location(where=Where.BEFORE, value=Kind.CALL,clazz="/.*/", method="setup"))
13        public static void onMapper_run_Before_Call_setup() {
14               mapperSetupStartTime =timeNanos();
15               println(strcat("MAP	STARTUP_MEM	",str(used(heapUsage()))));
16        }
17  
18        @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper",
19                        method="run",
20                        location=@Location(where=Where.AFTER, value=Kind.CALL,clazz="/.*/", method="setup"))
21        public static void onMapper_run_After_Call_setup() {
22               println(strcat("MAP	SETUP	",str(timeNanos() -mapperSetupStartTime)));
23               println(strcat("MAP	SETUP_MEM	",str(used(heapUsage()))));
24        }
25 }

1.3编译BTrace脚本

编译命令：btracec TraceMR.java

2.Hadoop配置

2.1分发文件

将BTrace文件：$BTRACE_HOME/build目录下的btrace-agent.jar和btrace-boot.jar，以及编译Btrace脚本后生成的class文件分发到各个子节点上。建议使用脚本分发，代码如下：

 1 #!/usr/bin/envbash
 2 #Usage:
 3 #  ./install_btrace.sh <slaves_file>
 4 # 
 5 # where:
 6 #    slaves_file = File containing a list of slavemachines
 7 #
 8 #Example:
 9 #  ./install_btrace.sh /root/SLAVE_NAMES.txt
10 #Make sure we have all the arguments
11 if [$# -ne 1 ]; then
12    echo "Usage: $0<slaves_file>"
13    echo " slaves_file = File containing a list of slave machines"
14    echo ""
15    exit -1
16 fi
17  
18 #Get the slaves file
19 declareSLAVES_FILE=$1;
20 iftest ! -e $SLAVES_FILE; then
21    echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting"
22    exit -1
23 fi
24 #Get the slaves file
25 declareSLAVES_FILE=$1;
26 iftest ! -e $SLAVES_FILE; then
27    echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting"
28    exit -1
29 fi
30  
31 MASTER_BTRACE_DIR=/home/wangpeng/zhiming/usr/btrace/build
32 SLAVES_BTRACE_DIR=/home/wangpeng/zhiming/usr
33  
34 forslave in `cat "$SLAVES_FILE"`; do
35 {
36    printf "Installing on host:$slave
"
37    ssh $slave "mkdir -p$SLAVES_BTRACE_DIR"
38    scp ${MASTER_BTRACE_DIR}/btrace-agent.jar$slave:$SLAVES_BTRACE_DIR/.
39    scp ${MASTER_BTRACE_DIR}/btrace-boot.jar$slave:$SLAVES_BTRACE_DIR/.
40    scp ${MASTER_BTRACE_DIR}/TraceMR.class$slave:$SLAVES_BTRACE_DIR/.
41 }
42 done

2.2配置Hadoop文件

在所有子节点的$HADOOP_HOME安装目录下找到mapred-site.xml文件，进行如下设置：

代码：

1 <property>
2          <name>mapred.map.child.java.opts</name>
3          <value>
4      -Xmx1024m -javaagent:/home/wangpeng/zhiming/usr/btrace/build/btrace-agent.jar     =script=/home/wangpeng/zhiming/usr//TraceMR.class,stdout=true,noServer=true
5          </value>
6 </property>

2.3运行MR程序

1.准备一个写好的MR程序，如WordCount.java程序；

2.执行hadoop jar WordCount.jar WordCount /user/wzm/input /user/wzm/output。

2.4查看输出文件

在所有子节点的$HADOOP_HOME安装目录下找到logs目录中的stdout文件，执行命令：cat stdout，且有如下输出表示配置成功：