hadoop 2012.07.24 读写测试

namenode 独立虚拟机:1G内存

datanode1、datanode2 虚拟于同一塔式服务器,内存各1G

 

----- TestDFSIO ----- : write
Date & time: Tue Jul 24 17:15:06 CST 2012
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 16.458439970308973
Average IO rate mb/sec: 17.915302276611328
IO rate std deviation: 5.502547439324805
Test exec time sec: 702.77

关闭数据库后测试

Date & time: Tue Jul 24 18:16:42 CST 2012
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 22.132315836999922
Average IO rate mb/sec: 24.76445960998535
IO rate std deviation: 8.614986952186385
Test exec time sec: 543.453


两台虚拟机作为datanode部署在一个节点上,在做写入测试时报Too many fetch-failures错误,这是因为读写过于频繁导致datanode与namenode通信异常导致的,移除其中一个datanode后再进行读写测试,就不再报联系不上的错误了。

移除一台datanode后测试结果:

----- TestDFSIO ----- : write
Date & time: Tue Jul 24 18:31:06 CST 2012
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 14.386874566595404
Average IO rate mb/sec: 14.926310539245605
IO rate std deviation: 2.5935396939344066
Test exec time sec: 456.993

两台建立与独立机器上的虚拟机,其中有一个datanode没有配置hosts文件,导致DiskErrorException: Could not find taskTracker

----- TestDFSIO ----- : write
Date & time: Thu Jul 26 01:00:34 PDT 2012
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 54.977376809442916
Average IO rate mb/sec: 64.9064712524414

IO rate std deviation: 27.569772802264755
Test exec time sec: 346.016

两台建立与独立机器上的虚拟机,正常配置datanode的hosts文件后

----- TestDFSIO ----- : write
Date & time: Thu Jul 26 16:28:38 CST 2012
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 69.33322240017749
Average IO rate mb/sec: 80.34981536865234
IO rate std deviation: 31.796870275012598
Test exec time sec: 333.343

原文地址:https://www.cnblogs.com/MorZe/p/2606889.html