尝试在CentOS 7上启动cosbench遭遇的一些问题之三

当cosbench的测试莫名其妙的terminated了，而且时而发生，时而不发生，mission log里也看不出什么信息，记得看一眼system log.

如果发现这个call stack, 那么请注意，很可能这次测试的失败是由于controller和drivers与storage cluster 之间的时间不同步引起的。

2020-08-13 02:19:28,977 [ERROR] [AbstractAgent] - unexpected exception

java.lang.ArrayIndexOutOfBoundsException: -9626

at com.intel.cosbench.bench.Counter.doAdd(Counter.java:65)

at com.intel.cosbench.driver.model.OperatorContext.doAddSample(OperatorContext.java:76)

at com.intel.cosbench.driver.model.OperatorContext.addSample(OperatorContext.java:70)

at com.intel.cosbench.driver.agent.WorkAgent.onSampleCreated(WorkAgent.java:211)

at com.intel.cosbench.driver.operator.Preparer.operate(Preparer.java:99)

at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)

at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)

at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)

at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)

at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)

at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

2020-08-13 02:19:28,977 [ERROR] [MissionHandler] - detected workers [19, 20, 21, 22, 23, 24] have encountered errors

2020-08-13 02:19:28,979 [INFO] [MissionHandler] - mission M2E66EA747D has been terminated

当你在controller的system.log中发现如下的记录，那么说明这次测试的terminate很可能是由于controller与drivers之间的时间不同步引起的。

2020-08-20 10:44:59,277 [WARN] [PingDriverRunner] - The driver driver1 at http://10.246.21.82:18088/driver is not reachable at the 1 time, with error message: Connection refused (Connection refused)
2020-08-20 11:22:57,348 [WARN] [AbstractCommandTasklet] - time drift is still longer than tolerable time drift 300 mSec after 3 times of synchronization

2020-08-20 17:47:37,351 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 400 - no such key defined: sizes

2020-08-20 17:47:37,359 [ERROR] [StageRunner] - detected tasks [t7, t8, t9, t10, t11, t12] have encountered errors
2020-08-20 17:47:37,365 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 400 - unrecognized request: org.apache.catalina.connector.RequestFacade@28481cc4

2020-08-20 17:47:37,366 [ERROR] [Aborter] - fail to abort driver
com.intel.cosbench.controller.tasklet.TaskletException
     at com.intel.cosbench.controller.tasklet.AbstractCommandTasklet.issueCommand(AbstractCommandTasklet.java:81)
     at com.intel.cosbench.controller.tasklet.Aborter.executeAbort(Aborter.java:53)
     at com.intel.cosbench.controller.tasklet.Aborter.execute(Aborter.java:42)
     at com.intel.cosbench.controller.tasklet.AbstractTasklet.call(AbstractTasklet.java:47)
     at com.intel.cosbench.controller.tasklet.AbstractTasklet.call(AbstractTasklet.java:1)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:748)

进一步的排查，可以使用下面的命令，来让controller和drivers同时返回本地时间，一边让时间的差距一目了然。如果不这么做，则很难明确几台机器上的的时间差距是不是输入间隔命令的那几秒造成的。