利用雅虎ycsb对cassandra做性能测试

准备：

环境：

两台虚拟机：ip:192.168.138.128/129;配置：2核4G；

版本：apache-cassandra-3.10

　　 ycsb-cassandra-binding-0.13.0-SNAPSHOT

系统参数优化：

1 时钟同步-----非常重要，涉及到集群内数据同步问题

在crontab中加上：

*/1 * * * * /usr/sbin/ntpdate time.nist.gov

2 关闭cpu变频控制

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governordo

[ -f $CPUFREQ ] || continue

echo -n performance > $CPUFREQdone

3 在/etc/sysctl.conf增加配置文件：

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.core.rmem_default = 16777216

net.core.wmem_default = 16777216

net.core.optmem_max = 40960

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

vm.max_map_count = 1048575

vm.swappiness=0

4 关闭zone_reclaim_mode

$ echo 0 > /proc/sys/vm/zone_reclaim_mode

5 增加cassandra用户控制：/etc/security/limits.d/cassandra.conf

<cassandra_user> - memlock unlimited

<cassandra_user> - nofile 100000

<cassandra_user> - nproc 32768

<cassandra_user> - as unlimited

6 redhat下需要修改/etc/security/limits.d/90-nproc.conf:

cassandra_user - nproc 32768

确认参数是否优化可以看cassandra进程号相关文件：

$ cat /proc/pid/limits

关闭使用swap分区：

swapoff --all

安装配置

不再赘述，直接解压放到指定地点就好了。需要注意的是在cassandra.yaml中需要更改以下配置：

cluster_name: 'mycluster'

data_file_directories:

- /data/cassandra/data

commitlog_directory: /data/cassandra/commitlog

saved_caches_directory: /data/cassandra/saved_caches

- seeds: "127.0.0.1,192.168.138.128,192.168.138.129"

listen_address: 192.168.138.128

rpc_address: 192.168.138.128

安装配置ycsb:

从github中下载最新版本的ycsb进行编译：

mvn -pl com.yahoo.ycsb:cassandra-binding -am clean package

（注意：mvn要用3版本，同时需要在编译本机环境开启一个127.0.0.1的cassandra并执行以下脚本：

CREATE KEYSPACE ycsb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;

CREATE TABLE ycsb.usertable (

y_id text PRIMARY KEY,

field0 text,

field1 text,

field2 text,

field3 text,

field4 text,

field5 text,

field6 text,

field7 text,

field8 text,

field9 text

) WITH bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND comment = ''

AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}

AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

AND crc_check_chance = 1.0

AND dclocal_read_repair_chance = 0.1

AND default_time_to_live = 0

AND gc_grace_seconds = 864000

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair_chance = 0.0

AND speculative_retry = '99PERCENTILE';

）

然后解压包：cp /root/YCSB-master/cassandra/target/ycsb-cassandra-binding-0.13.0-SNAPSHOT.tar.gz ~/

如果报相关日志错误则下载：slf4j-1.7.25.tar.gz将相关的Log4j包放置到对应的lib目录下。

配置cassandra.properties:

hosts = 192.168.138.128,192.168.138.129

port = 9042

cassandra.keyspace = usertable

cassandra.username = dd

cassandra.password = 111111

cassandra.readconsistencylevel = ANY

cassandra.writeconsistencylevel = ANY

cassandra.maxconnections = 100

cassandra.connecttimeoutmillis = 1000000000

cassandra.readtimeoutmillis = 1000000000

配置workload:

#vim workloads/workloada
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=false
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
requestdistribution=zipfian
fieldcount 表示每条数据中的字段数,默认为 10;
fieldlength 表示每个字段的值的长度,默认为 100;
readallfields 域用来标识是否读取所有的所有的字段,取值有 ture 或 false;
readproportion,
updateproportion,
scanproportion,
insertproportion 分别表示该 workload中读、更新、扫描和插入操作占总操作的百分比,这四个值的和为 1;
requestdistribution 表示数据的分布情况,当前支持 uniform,zipfian 和 latest,默认为 uniform;
maxscanlength 域主要为扫描操作定义,定义了最大扫描的记录数量,默认为 1000;
scanlengthdistribution 域也是为扫描操作定义的,为每次扫描的长度定义相应的分布,默认是 uniform;
insertorder 域主要分两种 ordered 和 hashed,默认为 hashed;
operationcount 总共的 operation 数量;
maxexecutiontime 为该 workload 定义了最长的执行时间,单位为 s。
AverageLatency(平均潜伏期)平均潜伏期（average latency）：指当磁头移动到数据所在的磁道后，然后等待所要的数据块继续转动（半圈或多些、少些）到磁头下的时间，单位为毫秒（ms）。平均潜伏期是越小越好，潜伏期小代表硬盘的读取数据的等待时间短，这就等于具有更高的硬盘数据传输率。

创建用户结构：

CREATE KEYSPACE usertable WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;

create table usertable (y_id varchar primary key,field0 varchar,field1 varchar,field2 varchar,field3 varchar,field4 varchar,field5 varchar,field6 varchar,field7 varchar,field8 varchar,field9 varchar);

执行命令./bin/ycsb load cassandra-cql -P workloads/workloada -P cassandra.properties -p columnfamily=usertable -s -threads 20 > ./writeread-log.log

结果如下：

Datacenter: datacenter1; Host: /192.168.138.128; Rack: rack1

[OVERALL], RunTime(ms), 4974.0

[OVERALL], Throughput(ops/sec), 201.04543626859672

[TOTAL_GCS_Copy], Count, 13.0

[TOTAL_GC_TIME_Copy], Time(ms), 51.0

[TOTAL_GC_TIME_%_Copy], Time(%), 1.0253317249698433

[TOTAL_GCS_MarkSweepCompact], Count, 0.0

[TOTAL_GC_TIME_MarkSweepCompact], Time(ms), 0.0

[TOTAL_GC_TIME_%_MarkSweepCompact], Time(%), 0.0

[TOTAL_GCs], Count, 13.0

[TOTAL_GC_TIME], Time(ms), 51.0

[TOTAL_GC_TIME_%], Time(%), 1.0253317249698433

[CLEANUP], Operations, 200.0

[CLEANUP], AverageLatency(us), 11310.105

[CLEANUP], MinLatency(us), 0.0

[CLEANUP], MaxLatency(us), 2263039.0

[CLEANUP], 95thPercentileLatency(us), 0.0

[CLEANUP], 99thPercentileLatency(us), 1.0

[INSERT], Operations, 1000.0

[INSERT], AverageLatency(us), 230845.253

[INSERT], MinLatency(us), 2180.0

[INSERT], MaxLatency(us), 807423.0

[INSERT], 95thPercentileLatency(us), 659455.0

[INSERT], 99thPercentileLatency(us), 699903.0

[INSERT], Return=OK, 1000

参考资料：http://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettings.html

http://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsCStress.html

http://blog.csdn.net/qianlong4526888/article/details/53747617