一.测试分类
ceph的性能测试包含rados性能测试和rbd性能测试
二.测试工具
rados性能测试工具：使用ceph自带的rados bench工具、rados losd-gen工具
rbd性能测试工具：rbd bench-write进行块设备写性能测试，fio+rbd ioengine测试、fio+libaio测试
三.测试示例
3.1 rados性能测试之rados bench
3.1.1 工具的命令行语法
rados bench -p <pool_name> <seconds> <write|seq|rand> -b <block size> -t --no-cleanup
- pool_name: 测试的存储池名称
- seconds: 测试时间，以秒为单位
- <write|seq|rand>：操作的模式，write:写 seq:顺序读 rand:随机读
- -b：block size即块大小，默认为4M
- -t：读/写并行数，默认为16
- --no-cleanup：表示测试完成后不删除测试用数据，在做读测试之前，需要使用该参数运行一遍写测试来产生数据，在全部测试完毕后可以运行rados -p <pool_name> cleanup 来清理数据
3.1.2 执行写测试
- 创建一个测试使用的存储池test
[root@openstack01 ~]# docker exec ceph_mon ceph osd pool create test 512 512
- 执行写测试，测试时间设置为60s
(ceph-mon)[root@openstack01 ~]# rados bench -p test 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_openstack01_603
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 16 0 0 0 - 0
2 16 16 0 0 0 - 0
3 16 24 8 10.6653 10.6667 2.66058 2.22005
4 16 27 11 10.9986 12 3.19831 2.45947
5 16 29 13 10.3986 8 2.54069 2.61219
6 16 38 22 14.6647 36 2.52673 3.23277
7 16 43 27 15.4265 20 4.06122 3.28708
8 16 46 30 14.998 12 3.23905 3.27569
9 16 47 31 13.7759 4 2.94027 3.26487
10 16 50 34 13.5982 12 3.41596 3.31326
11 16 55 39 14.1799 20 5.3298 3.45411
12 16 58 42 13.9981 12 5.90129 3.60335
13 16 59 43 13.229 4 6.06364 3.66057
14 16 61 45 12.8554 8 5.25054 3.74845
15 16 64 48 12.7982 12 8.46896 3.84977
16 16 65 49 12.2483 4 7.87838 3.93199
17 16 68 52 12.2336 12 7.36673 4.06689
18 16 72 56 12.4427 16 7.06 4.2234
19 16 75 59 12.4193 12 7.78165 4.30382
2018-10-11 11:57:52.537579 min lat: 1.89714 max lat: 8.54175 avg lat: 4.35334
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 79 63 12.5982 16 6.90524 4.35334
21 16 83 67 12.7601 16 5.97492 4.47472
22 16 85 69 12.5437 8 5.49733 4.50754
23 16 86 70 12.1722 4 4.1921 4.50304
24 16 87 71 11.8317 4 6.14773 4.5262
25 16 90 74 11.8383 12 5.11989 4.5118
26 16 91 75 11.5368 4 5.1742 4.52063
27 16 95 79 11.7021 16 8.31085 4.62613
28 16 96 80 11.427 4 7.6217 4.66358
29 16 96 80 11.0329 0 - 4.66358
30 16 96 80 10.6652 0 - 4.66358
31 16 96 80 10.3211 0 - 4.66358
32 16 96 80 9.99859 0 - 4.66358
33 16 96 80 9.69558 0 - 4.66358
34 16 96 80 9.41042 0 - 4.66358
35 16 101 85 9.71291 2.85714 8.7633 4.95709
36 16 101 85 9.44309 0 - 4.95709
37 16 103 87 9.40406 4 17.9027 5.1727
38 16 105 89 9.36708 8 18.0327 5.37878
39 16 105 89 9.12691 0 - 5.37878
2018-10-11 11:58:12.540426 min lat: 1.89714 max lat: 18.0327 avg lat: 5.37878
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 105 89 8.89874 0 - 5.37878
41 16 105 89 8.68167 0 - 5.37878
42 16 106 90 8.57019 1 6.43809 5.39055
43 16 114 98 9.11497 32 5.11341 6.40932
44 16 115 99 8.99871 4 7.16406 6.41694
45 16 115 99 8.79874 0 - 6.41694
46 16 116 100 8.69442 2 11.4983 6.46775
47 16 118 102 8.67962 8 9.64869 6.54381
48 16 119 103 8.58212 4 5.13036 6.53009
49 16 119 103 8.40697 0 - 6.53009
50 16 120 104 8.31882 2 7.10649 6.53563
51 16 125 109 8.54781 20 8.44788 6.79488
52 16 130 114 8.768 20 9.57393 6.83936
53 16 134 118 8.9044 16 10.3648 6.79498
54 16 136 120 8.88764 8 9.32118 6.81407
55 16 140 124 9.01692 16 2.17708 6.69443
56 16 140 124 8.8559 0 - 6.69443
57 16 143 127 8.91103 6 4.81556 6.68976
58 16 146 130 8.96426 12 10.3827 6.68146
59 16 146 130 8.81232 0 - 6.68146
2018-10-11 11:58:32.543137 min lat: 1.63302 max lat: 23.564 avg lat: 6.68146
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 146 130 8.66545 0 - 6.68146
61 16 146 130 8.52339 0 - 6.68146
62 16 146 130 8.38592 0 - 6.68146
63 16 146 130 8.25281 0 - 6.68146
64 16 146 130 8.12387 0 - 6.68146
65 16 146 130 7.99889 0 - 6.68146
66 16 146 130 7.87769 0 - 6.68146
67 11 147 136 8.11828 2.66667 15.7406 6.94076
68 11 147 136 7.99889 0 - 6.94076
69 11 147 136 7.88297 0 - 6.94076
Total time run: 69.107454
Total writes made: 147
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 8.50849
Stddev Band 7.95831
Max bandwidth (MB/sec): 36
Min bandwidth (MB/sec): 0
Average IOPS: 2
Stddev IOPS: 2
Max IOPS: 9
Min IOPS: 0
Average Latency(s): 7.44131
Stddev Latency(s): 4.9394
Max latency(s): 23.564
Min latency(s): 1.63302
上面数据Bandwidth为8.50849MB/sec，Average Latency为：7.44131s
3.1.3 执行随机读测试
(ceph-mon)[root@openstack01 ~]# rados bench -p test 60 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 152 136 543.849 544 0.0743183 0.0995761
2 16 306 290 579.818 616 0.00970808 0.102821
3 16 444 428 570.514 552 0.0405567 0.104487
4 15 568 553 552.862 500 0.00818665 0.109911
5 16 700 684 546.952 524 0.043003 0.111776
6 16 836 820 546.446 544 0.104833 0.113024
7 16 959 943 538.658 492 0.0117524 0.114411
8 16 1079 1063 531.318 480 0.0381715 0.116316
9 16 1199 1183 525.608 480 0.0552762 0.118574
10 16 1293 1277 510.644 376 0.0362924 0.117607
11 16 1423 1407 511.476 520 0.0142599 0.122025
12 16 1566 1550 516.511 572 0.008037 0.12061
13 16 1686 1670 513.694 480 0.00940001 0.119639
14 16 1793 1777 507.568 428 0.0111657 0.123314
15 16 1923 1907 508.39 520 0.0120602 0.123097
16 16 2047 2031 507.611 496 0.0106408 0.12326
17 16 2164 2148 505.277 468 0.00817543 0.123239
18 16 2264 2248 499.426 400 0.0130886 0.125529
19 15 2391 2376 500.083 512 0.00987636 0.125445
2018-10-11 14:27:20.997424 min lat: 0.00644293 max lat: 1.0492 avg lat: 0.125502
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 2516 2500 499.875 496 0.00979231 0.125502
21 16 2642 2626 500.067 504 0.271557 0.125655
22 16 2769 2753 500.424 508 0.00853595 0.12471
23 16 2891 2875 499.88 488 0.358424 0.125615
24 16 3009 2993 498.716 472 0.356571 0.125675
25 16 3127 3111 497.645 472 0.452718 0.125959
26 15 3269 3254 500.5 572 0.286575 0.125597
27 16 3393 3377 500.182 492 0.0994856 0.125185
28 16 3512 3496 499.316 476 0.0637264 0.125534
29 16 3648 3632 500.853 544 0.307832 0.125653
30 15 3791 3776 503.354 576 0.00929097 0.124856
31 16 3919 3903 503.501 508 0.00955382 0.124689
32 16 4045 4029 503.514 504 0.180285 0.124946
33 15 4150 4135 501.103 424 0.0448866 0.125427
34 16 4287 4271 502.361 544 0.363722 0.125243
35 16 4394 4378 500.233 428 0.00944837 0.125241
36 16 4512 4496 499.447 472 0.0106443 0.126
37 16 4629 4613 498.591 468 0.0995126 0.126007
38 16 4738 4722 496.941 436 0.267176 0.126746
39 16 4869 4853 497.626 524 0.303769 0.126441
2018-10-11 14:27:41.001727 min lat: 0.00644293 max lat: 1.0492 avg lat: 0.126309
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 4999 4983 498.184 520 0.0542142 0.126309
41 16 5119 5103 497.739 480 0.00888314 0.126283
42 15 5245 5230 497.976 508 0.0360334 0.126255
43 16 5380 5364 498.857 536 0.015746 0.126103
44 16 5508 5492 499.152 512 0.22606 0.126146
45 16 5626 5610 498.547 472 0.0110425 0.126163
46 16 5759 5743 499.273 532 0.378206 0.126063
47 16 5885 5869 499.372 504 0.0112606 0.125814
48 15 6000 5985 498.634 464 0.42884 0.126265
49 16 6127 6111 498.74 504 0.611502 0.126078
50 16 6255 6239 499.004 512 0.100672 0.1262
51 16 6370 6354 498.238 460 0.428983 0.12635
52 16 6495 6479 498.271 500 0.0130692 0.126363
53 16 6620 6604 498.302 500 0.160325 0.126399
54 16 6748 6732 498.554 512 0.0384334 0.126252
55 16 6885 6869 499.448 548 0.00900053 0.12599
56 15 7016 7001 499.956 528 0.0435343 0.125842
57 15 7136 7121 499.603 480 0.0516195 0.125927
58 16 7243 7227 498.299 424 0.224335 0.126219
59 16 7368 7352 498.326 500 0.0679909 0.126414
2018-10-11 14:28:01.006114 min lat: 0.00644293 max lat: 1.0492 avg lat: 0.126286
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 15 7502 7487 499.019 540 0.0106773 0.126286
Total time run: 60.227155
Total reads made: 7502
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 498.247
Average IOPS: 124
Stddev IOPS: 10
Max IOPS: 154
Min IOPS: 94
Average Latency(s): 0.126668
Max latency(s): 1.0492
Min latency(s): 0.00644293
上面数据Bandwidth为498.247MB/sec，Average Latency为：0.126668s
3.1.4 执行顺序读测试
(ceph-mon)[root@openstack01 ~]# rados bench -p test 60 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 147 131 523.861 524 0.0094258 0.0971213
Total time run: 1.199266
Total reads made: 147
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 490.3
Average IOPS: 122
Stddev IOPS: 0
Max IOPS: 131
Min IOPS: 131
Average Latency(s): 0.126695
Max latency(s): 0.612257
Min latency(s): 0.00736062
上面数据Bandwidth为490.3MB/sec，Average Latency为：0.126695s