hadoop运行常见问题FAQ

问题1：Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out
Answer：
程序里面需要打开多个文件，进行分析，系统一般默认数量是1024，（用ulimit -a可以看到）对于正常使用是够了，但是对于程序来讲，就太少了。
修改办法：
修改2个文件。
/etc/security/limits.conf
vi /etc/security/limits.conf
加上：
* soft nofile 102400
* hard nofile 409600

$cd /etc/pam.d/
$sudo vi login
添加 session required /lib/security/pam_limits.so

针对第一个问题我纠正下答案：
这是reduce预处理阶段shuffle时获取已完成的map的输出失败次数超过上限造成的，上限默认为5。引起此问题的方式可能会有很多种，比如网络连接不正常，连接超时，带宽较差以及端口阻塞等。。。通常框架内网络情况较好是不会出现此错误的。

问题2：Too many fetch-failures

Answer:
出现这个问题主要是结点间的连通不够全面。
1) 检查、/etc/hosts
要求本机ip 对应服务器名
要求要包含所有的服务器ip + 服务器名
2) 检查 .ssh/authorized_keys
要求包含所有服务器（包括其自身）的public key

3：处理速度特别的慢出现map很快但是reduce很慢而且反复出现 reduce=0%
Answer:
结合第二点，然后
修改 conf/hadoop-env.sh 中的export HADOOP_HEAPSIZE=4000

4：能够启动datanode，但无法访问，也无法结束的错误
在重新格式化一个新的分布式文件时，需要将你NameNode上所配置的dfs.name.dir这一namenode用来存放NameNode 持久存储名字空间及事务日志的本地文件系统路径删除，同时将各DataNode上的dfs.data.dir的路径 DataNode 存放块数据的本地文件系统路径的目录也删除。如本此配置就是在NameNode上删除/home/hadoop/NameData，在DataNode上删除/home/hadoop/DataNode1和/home/hadoop/DataNode2。这是因为Hadoop在格式化一个新的分布式文件系统时，每个存储的名字空间都对应了建立时间的那个版本（可以查看/home/hadoop /NameData/current目录下的VERSION文件，上面记录了版本信息），在重新格式化新的分布式系统文件时，最好先删除NameData 目录。必须删除各DataNode的dfs.data.dir。这样才可以使namedode和datanode记录的信息版本对应。
注意：删除是个很危险的动作，不能确认的情况下不能删除！！做好删除的文件等通通备份！！

5：java.io.IOException: Could not obtain block: blk_194219614024901469_1100 file=/user/hive/warehouse/src_20090724_log/src_20090724_log
出现这种情况大多是结点断了，没有连接上。

6：java.lang.OutOfMemoryError: Java heap space
出现这种异常，明显是jvm内存不够得原因，要修改所有的datanode的jvm内存大小。
Java -Xms1024m -Xmx4096m

一般jvm的最大内存使用应该为总内存大小的一半，我们使用的8G内存，所以设置为4096m，这一值可能依旧不是最优的值。

7: Namenode in safe mode
解决方法
bin/hadoop dfsadmin -safemode leave

8:java.net.NoRouteToHostException: No route to host
j解决方法：
sudo /etc/init.d/iptables stop

9：更改namenode后，在hive中运行select 依旧指向之前的namenode地址
这是因为：When youcreate a table, hive actually stores the location of the table (e.g.
hdfs://ip:port/user/root/...) in the SDS and DBS tables in the metastore . So when I bring up a new cluster the master has a new IP, but hive's metastore is still pointing to the locations within the old
cluster. I could modify the metastore to update with the new IP everytime I bring up a cluster. But the easier and simpler solution was to just use an elastic IP for the master
所以要将metastore中的之前出现的namenode地址全部更换为现有的namenode地址

10：Your DataNode is started and you can create directories with bin/hadoop dfs -mkdir, but you get an error message when you try to put files into the HDFS (e.g., when you run a command like bin/hadoop dfs -put).
解决方法：
Go to the HDFS info web page (open your web browser and go to http://namenode:dfs_info_port where namenode is the hostname of your NameNode and dfs_info_port is the port you chose dfs.info.port; if followed the QuickStart on your personal computer then this URL will be http://localhost:50070). Once at that page click on the number where it tells you how many DataNodes you have to look at a list of the DataNodes in your cluster.
If it says you have used 100% of your space, then you need to free up room on local disk(s) of the DataNode(s).
If you are on Windows then this number will not be accurate (there is some kind of bug either in Cygwin's df.exe or in Windows). Just free up some more space and you should be okay. On one Windows machine we tried the disk had 1GB free but Hadoop reported that it was 100% full. Then we freed up another 1GB and then it said that the disk was 99.15% full and started writing data into the HDFS again. We encountered this bug on Windows XP SP2.
11：Your DataNodes won't start, and you see something like this in logs/*datanode*:
Incompatible namespaceIDs in /tmp/hadoop-ross/dfs/data
原因：
Your Hadoop namespaceID became corrupted. Unfortunately the easiest thing to do reformat the HDFS.
解决方法：
You need to do something like this:
bin/stop-all.sh
rm -Rf /tmp/hadoop-your-username/*
bin/hadoop namenode -format
12：You can run Hadoop jobs written in Java (like the grep example), but your HadoopStreaming jobs (such as the Python example that fetches web page titles) won't work.
原因：
You might have given only a relative path to the mapper and reducer programs. The tutorial originally just specified relative paths, but absolute paths are required if you are running in a real cluster.
解决方法：
Use absolute paths like this from the tutorial:
bin/hadoop jar contrib/hadoop-0.15.2-streaming.jar
  -mapper  $HOME/proj/hadoop/multifetch.py
  -reducer $HOME/proj/hadoop/reducer.py
  -input urls/*
  -output  titles

13： 2009-01-08 10:02:40,709 ERROR metadata.Hive (Hive.java:getPartitions(499)) - javax.jdo.JDODataStoreException: Required table missing : ""PARTITIONS"" in Catalog "" Schema "". JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables"
原因：就是因为在 hive-default.xml 里把 org.jpox.fixedDatastore 设置成 true 了

问题14：

09/08/31 18:25:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:Bad connect ack with firstBadLink 192.168.1.11:50010
> 09/08/31 18:25:45 INFO hdfs.DFSClient: Abandoning block blk_-8575812198227241296_1001
> 09/08/31 18:25:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:
Bad connect ack with firstBadLink 192.168.1.16:50010
> 09/08/31 18:25:51 INFO hdfs.DFSClient: Abandoning block blk_-2932256218448902464_1001
> 09/08/31 18:25:57 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:
Bad connect ack with firstBadLink 192.168.1.11:50010
> 09/08/31 18:25:57 INFO hdfs.DFSClient: Abandoning block blk_-1014449966480421244_1001
> 09/08/31 18:26:03 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:
Bad connect ack with firstBadLink 192.168.1.16:50010
> 09/08/31 18:26:03 INFO hdfs.DFSClient: Abandoning block blk_7193173823538206978_1001
> 09/08/31 18:26:09 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable
to create new block.
>       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2731)
>       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996)
>       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2182)
>
> 09/08/31 18:26:09 WARN hdfs.DFSClient: Error Recovery for block blk_7193173823538206978_1001
bad datanode[2] nodes == null
> 09/08/31 18:26:09 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/umer/8GB_input"
- Aborting...
> put: Bad connect ack with firstBadLink 192.168.1.16:50010

解决方法：
I have resolved the issue:
What i did:

1) '/etc/init.d/iptables stop' -->stopped firewall
2) SELINUX=disabled in '/etc/selinux/config' file.-->disabled selinux
I worked for me after these two changes

问题15：

hadoop datanode 问题 INFO org.apache.hadoop.ipc.RPC: Server at /:9000 not available yet, Zzzzz..

用bin/hadoop dfsadmin -report 查看hadoop的情况，现实的信息如下;

Configured Capacity: 0(0KB)
Present Capacity: 0(0KB)
DFS Remaining: 0(0KB)
DFS Used: 0(0KB)
DSF Used%:?%
Under Replicated blocks:0
Blocks with corrupt replicas: 0
Missing blocks: 0
----------------------------------------------------
Databodes available: 0(0 total, 0 dead)

总是出现datanode连接不上namenode的问题。

在datanode也就是slave上面查看datanode的日志时，的错误为：

2011-10-26 17:57:05,231 INFO org.apache.hadoop.ipc.RPC: Server at /192.168.0.100:9000 not available yet, Zzzzz...
2011-10-26 17:57:07,235 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.0.100:9000. Already tried 0 time(s).
2011-10-26 17:57:08,236 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.0.100:9000. Already tried 1 time(s).
2011-10-26 17:57:09,237 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.0.100:9000. Already tried 2 time(s).
2011-10-26 17:57:10,239 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.0.100:9000. Already tried 3 time(s).
2011-10-26 17:57:11,240 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.0.100:9000. Already tried 4 time(s).
2011-10-26 17:57:12,241 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.0.100:9000. Already tried 5 time(s).

也是datanode连接不上namenode。

而在namenode上面显示：

2011-10-26 14:18:49,686 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000, call addBlock(/root/hadoop/tmp/mapred/system/jobtracker.info, DFSClient_-1928560478, null, null) from 127.0.0.1:32817: error: java.io.IOException: File /root/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /root/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:690)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)

也就是说namenode也在试图将jobtracker.info存入hdfs文件系统中，而又存不进去。然后查了一下网上的说法，之后发现原来是/etc/hosts中的ip映射的问题。由于在master中/etc/hosts的配置为：

127.0.0.1 master

127.0.1.1 server.ubuntu-domain server

192.168.0.100 server

192.168.0.111 hdfs1因此可能存在一个优先匹配第一个碰见的问题，之后是将前两行注释掉(后来又将第一行改为了127.0.0.1 localhost)。然后在进行正常的hadoop format和启动，就可以连接上了。

最后可能还有时候出现错误datanode自动关闭的问题。解决这个问题的方法是删除所有masters，slaves中的tmp文件。然后format，重启就可以了。