报错背景
hadoop启动报错。
报错现象
Number of suppressed write-lock reports: 0 Longest write-lock held interval: 10734 2020-12-15 10:48:07,720 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage java.io.IOException: There appears to be a gap in the edit log. We expected txid 927, but got txid 1265. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:890) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:321) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:978) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:685) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566) 2020-12-15 10:48:07,722 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@bigdata01:50070 2020-12-15 10:48:07,823 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system... 2020-12-15 10:48:07,823 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped. 2020-12-15 10:48:07,823 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete. 2020-12-15 10:48:07,823 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. java.io.IOException: There appears to be a gap in the edit log. We expected txid 927, but got txid 1265. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:890) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:321) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:978) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:685) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566) 2020-12-15 10:48:07,825 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2020-12-15 10:48:07,825 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
报错原因
上面日志提示 We expected txid 927, but got txid 1265 也就是Namenode想要获取927这个edit，但是获取到的是1265，查看 jn 目录下，发现的确只有1265 即：edits_0000000000000001265-0000000000000001265
报错解决
（1）方法一：命令修复元数据
# hadoop namenode -recover
遇到 (Y/N) 选Y
遇到 (c/s/q/a) 选c
重启hadoop即可。
（2）方法二：复制缺失文件
hadoop集群下会有两个JournalNode，可以将另一个节点存在，而此节点不存在的文件使用scp命令复制过来，
文件所在目录是自定义的，例如：/data/hadoop/hdfs/journal/cluster/current
# scp edits_0000000000000000041-0000000000000000043 root@bigdata01:/data/hadoop/hdfs/journal/cluster/current
# hdfs zkfc -formatZK
重启hadoop即可。