Hadoop调试记录(2)

自从上次调通hbase后很久没有碰hadoop了，今日想写一个mapreduce的小程序。于是先运行了下自带的wordcount示例程序，却报错了。

信息如下：

kevin@ubuntu:~/usr/hadoop/hadoop$ ./bin/hadoop jar hadoop-examples-1.2.1.jar wordcount readme.txt output
15/05/11 08:20:04 INFO input.FileInputFormat: Total input paths to process : 1
15/05/11 08:20:04 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/05/11 08:20:04 WARN snappy.LoadSnappy: Snappy native library not loaded
15/05/11 08:20:05 INFO mapred.JobClient: Running job: job_201505110806_0003
15/05/11 08:20:06 INFO mapred.JobClient:  map 0% reduce 0%
15/05/11 08:20:06 INFO mapred.JobClient: Task Id : attempt_201505110806_0003_m_000002_0, Status : FAILED
Error initializing attempt_201505110806_0003_m_000002_0:
java.io.IOException: Exception reading file:/~/usr/hadoop/hadoop/tmp/mapred/local/ttprivate/taskTracker/kevin/jobcache/job_201505110806_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:178)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1289)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
    at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File file:/~/usr/hadoop/hadoop/tmp/mapred/local/ttprivate/taskTracker/kevin/jobcache/job_201505110806_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:255)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:436)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

错误是无法读取jobToken的文件，由于程序不会出错，推测是上次在配置hbase时修改hadoop的core-site.xml文件出错了。在stackoverflow上有一个同样的问题，有人建议删除该配置文件中关于缓存文件目录的配置，因为这一部分是会被系统默认创建的，不需要手动配置。即下面这一部分：

1 <!-- In: conf/core-site.xml --> 
2 <property> 
3   <name>hadoop.tmp.dir</name> 
4   <value>/app/hadoop/tmp</value> 
5   <description>A base for other temporary directories.</description> 
6 </property>

删掉这个问题就解决了。关闭hadoop，重启hadoop。出现新问题 JobTracker is in safe mode：

15/05/11 08:47:40 ERROR security.UserGroupInformation: PriviledgedActionException as:kevin cause:org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.SafeModeException: JobTracker is in safe mode
    at org.apache.hadoop.mapred.JobTracker.checkSafeMode(JobTracker.java:5188)
    at org.apache.hadoop.mapred.JobTracker.getStagingAreaDir(JobTracker.java:3677)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

碰巧stackoverflow上也有相关的问题，应该是权限的问题，运行以下命令即可：

bin/hadoop dfsadmin -safemode leave

问题均解决，wordcount成功运行。