HRegionServer无法启动，启动hbase后马上挂掉的问题
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server bd004,60020,1561549217082 has been rejected; Reported time is too far out of sync with master. Time difference of 66357ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:360)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:253)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1397)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2155)
... 2 more
2019-06-26 19:40:21,307 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server bd004,60020,1561549217082: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server bd004,60020,1561549217082 has been rejected; Reported time is too far out of sync with master. Time difference of 66357ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:360)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:253)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1397)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server bd004,60020,1561549217082 has been rejected; Reported time is too far out of sync with master. Time difference of 66357ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:360)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:253)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1397)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:287)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2157)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:894)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server bd004,60020,1561549217082 has been rejected; Reported time is too far out of sync with master. Time difference of 66357ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:360)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:253)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1397)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2155)
... 2 more
2019-06-26 19:40:21,310 FATAL [regionserver60020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2019-06-26 19:40:21,311 INFO [regionserver60020] regionserver.HRegionServer: STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server bd004,60020,1561549217082 has been rejected; Reported time is too far out of sync with master. Time difference of 66357ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:360)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:253)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1397)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-06-26 19:40:21,311 INFO [regionserver60020] ipc.RpcServer: Stopping server on 60020
2019-06-26 19:40:21,313 INFO [regionserver60020] regionserver.HRegionServer: Stopping infoServer
2019-06-26 19:40:21,322 INFO [regionserver60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2019-06-26 19:40:21,425 INFO [regionserver60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2019-06-26 19:40:21,425 INFO [regionserver60020] regionserver.HRegionServer: aborting server null
2019-06-26 19:40:21,425 DEBUG [regionserver60020] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@494c377c
2019-06-26 19:40:21,426 INFO [regionserver60020] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16b92f67b770006
2019-06-26 19:40:21,431 INFO [regionserver60020] zookeeper.ZooKeeper: Session: 0x16b92f67b770006 closed
2019-06-26 19:40:21,431 INFO [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2019-06-26 19:40:21,439 INFO [regionserver60020] regionserver.HRegionServer: stopping server null; all regions closed.
2019-06-26 19:40:21,540 INFO [regionserver60020] regionserver.Leases: regionserver60020 closing leases
2019-06-26 19:40:21,540 INFO [regionserver60020] regionserver.Leases: regionserver60020 closed leases
2019-06-26 19:40:21,547 INFO [regionserver60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2019-06-26 19:40:21,547 INFO [regionserver60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2019-06-26 19:40:21,547 INFO [regionserver60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2019-06-26 19:40:21,547 INFO [regionserver60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2019-06-26 19:40:21,553 WARN [regionserver60020] zookeeper.RecoverableZooKeeper: Node /hbase/rs/bd004,60020,1561549217082 already deleted, retry=false
2019-06-26 19:40:21,553 WARN [regionserver60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/bd004,60020,1561549217082
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:179)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1290)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1279)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1377)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1065)
at java.lang.Thread.run(Thread.java:748)
2019-06-26 19:40:21,572 INFO [regionserver60020] zookeeper.ZooKeeper: Session: 0x36b92f577540011 closed
2019-06-26 19:40:21,572 INFO [regionserver60020] regionserver.HRegionServer: stopping server null; zookeeper connection closed.
2019-06-26 19:40:21,572 INFO [regionserver60020] regionserver.HRegionServer: regionserver60020 exiting
2019-06-26 19:40:21,573 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2543)
2019-06-26 19:40:21,573 INFO [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2019-06-26 19:40:21,578 INFO [Thread-9] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1205bd62
2019-06-26 19:40:21,580 INFO [Thread-9] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2019-06-26 19:40:21,585 INFO [Thread-9] regionserver.ShutdownHook: Shutdown hook finished.
HRegionServer 起来之后又停了，只起来了两个，应该是三个，4停了
经检查是因为服务器时间不同步导致的，时间相差大概1分钟左右
同步时间
ntpdate pool.ntp.org
之后重启服务
stop-hbase.sh
start-hbase.sh
也有人说是当加载错误的协处理器之后，会导致regionserver挂掉，在配置文件中加以下配置：
（我没修改配置）
<property>
<name>hbase.coprocessor.abortonerror</name>
<value>false</value>
</property>
多试，多想，多参考