需求
搭建一个redis集群,支持故障恢复,。
Redis内置的一个集群需要至少3个主节点,所以搭建规模为3主3个从的集群用于测试。节点信息列表:
role | ip | port | bus port | remark |
master1 | 192.168.201.209 | 7000 | 17000 | 主节点1 |
master2 | 192.168.201.209 | 7001 | 17001 | 主节点2 |
master3 | 192.168.201.209 | 7002 | 17002 | 主节点3 |
replica1 | 192.168.201.209 | 7003 | 17003 | 从节点1 |
replica1 | 192.168.201.209 | 7004 | 17004 | 从节点2 |
replica1 | 192.168.201.209 | 7005 | 17005 | 从节点3 |
sentinel1 | 192.168.201.209 | 6000 | - | 哨兵1 |
sentinel2 | 192.168.201.209 | 6001 | - | 哨兵2 |
sentinel3 | 192.168.201.209 | 6002 | - | 哨兵3 |
环境:
- 镜像redis:alpine,redis的版本为5.0.3
- macOS 10.13.4
- Docker 18.03.1-ce
docker中用到的配置文件命名
➜ config pwd /Users/cymin/redis/config ➜ config tree . . ├── m1.conf ├── m2.conf ├── m3.conf ├── r1.conf ├── r2.conf ├── r3.conf ├── s1.conf ├── s2.conf └── s3.conf 0 directories, 9 files
Note that:
- Masters are called M1, M2, M3, ..., Mn.
- Slaves are called R1, R2, R3, ..., Rn (R stands for replica).
- Sentinels are called S1, S2, S3, ..., Sn.
3个redis master实例配置
➜ config cat m1.conf cluster-announce-ip 192.168.201.209 cluster-announce-port 7000 cluster-announce-bus-port 17000 cluster-enabled yes cluster-config-file "/tmp/nodes.conf" cluster-node-timeout 5000 appendonly yes
剩余的master端口依次+1, cluster-announce-bus-port的配置+10000.
说明:
- cluster-announce-ip 告诉redis实例外部映射地址
- cluster-announce-port 告诉redis实例外部映射端口
-
cluster-announce-bus-port 开启集群总线端口,值=cluster-announce-port +10000
- masterauth | requirepass 密码,master和replica需要配置为一样的密码
-
其他都是基本配置,相关文档:https://redis.io/topics/cluster-tutorial
3个redis replica(slave)实例配置
➜ config cat r1.conf cluster-announce-ip 192.168.201.209 cluster-announce-port 7003 cluster-announce-bus-port 17003 cluster-enabled yes cluster-config-file "/tmp/nodes.conf" cluster-node-timeout 5000 appendonly yes
剩余的cluster-announce-port端口依次+1,cluster-announce-bus-port的配置+10000。
sentinel实例之间的通讯端口默认是26379.
3个哨兵实例配置
➜ config cat s1.conf sentinel announce-ip 192.168.201.209 sentinel announce-port 6000 sentinel monitor master1 192.168.201.209 7000 2 sentinel down-after-milliseconds master1 60000 sentinel failover-timeout master1 180000 sentinel parallel-syncs master1 1 sentinel monitor master2 192.168.201.209 7001 2 sentinel down-after-milliseconds master2 60000 sentinel failover-timeout master2 180000 sentinel parallel-syncs master2 1 sentinel monitor master3 192.168.201.209 7002 2 sentinel down-after-milliseconds master3 60000 sentinel failover-timeout master3 180000 sentinel parallel-syncs master3 1
剩余的announce-port配置依次加1.
说明:每个哨兵只需要监控master节点即可,配置项说明见文档:https://redis.io/topics/sentinel
启动redis实例和哨兵
run.sh
# 3个master docker run --name master1 -tid --network service -v $HOME/redis/config:/opt/config -p 7000:6379 -p 17000:16379 redis:alpine redis-server /opt/config/m1.conf docker run --name master2 -tid --network service -v $HOME/redis/config:/opt/config -p 7001:6379 -p 17001:16379 redis:alpine redis-server /opt/config/m2.conf docker run --name master3 -tid --network service -v $HOME/redis/config:/opt/config -p 7002:6379 -p 17002:16379 redis:alpine redis-server /opt/config/m3.conf # 3个replica docker run --name replica1 -tid --network service -v $HOME/redis/config:/opt/config -p 7003:6379 -p 17003:16379 redis:alpine redis-server /opt/config/r1.conf docker run --name replica2 -tid --network service -v $HOME/redis/config:/opt/config -p 7004:6379 -p 17004:16379 redis:alpine redis-server /opt/config/r2.conf docker run --name replica3 -tid --network service -v $HOME/redis/config:/opt/config -p 7005:6379 -p 17005:16379 redis:alpine redis-server /opt/config/r3.conf # 3个sentinel docker run --name sentinel1 -tid --network service -v $HOME/redis/config:/opt/config -p 6000:26379 redis:alpine redis-sentinel /opt/config/s1.conf docker run --name sentinel2 -tid --network service -v $HOME/redis/config:/opt/config -p 6001:26379 redis:alpine redis-sentinel /opt/config/s2.conf docker run --name sentinel3 -tid --network service -v $HOME/redis/config:/opt/config -p 6002:26379 redis:alpine redis-sentinel /opt/config/s3.conf
创建集群
docker exec -it master1 sh /data # redis-cli --cluster create 192.168.201.209:7000 192.168.201.209:7001 192.168.201.209:7002 192.168.201.209:7003 192.168.201.209:7004 192.168.201.209:7005 --cluster-replicas 1
最后收到"[OK] All 16384 slots covered.”的日志时,集群搭建成功。
连接集群中某个redis实例
redis-cli -c -h 192.168.201.209 -p 7000
查看集群状态
redis-cli cluster nodes
查看某个节点的主从状态
redis-cli -h 192.168.201.209 -p 7000 info Replication
查看某个哨兵的状态
redis-cli -h 192.168.201.209 -p 6000 info Sentinel
故障自动恢复测试
# 集群节点状态 /data # redis-cli cluster nodes b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547716358000 4 connected da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547716357525 5 connected 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547716357000 2 connected 5461-10922 e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 slave 7d0d00d7710fc416ac2c3f44ea86b306b498038e 0 1547716357014 6 connected 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547716358535 3 connected 10923-16383 7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 master - 0 1547716356511 1 connected 0-5460 # 停止master1 docker stop master1 # 集群节点状态 /data # redis-cli cluster nodes b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547716754542 4 connected da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547716754542 5 connected 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547716753000 2 connected 5461-10922 e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547716754000 7 connected 0-5460 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547716754946 3 connected 10923-16383 7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 master,fail - 1547716421213 1547716420405 1 disconnected # 重新启动master1 docker restart master1 # 集群节点状态 /data # redis-cli cluster nodes b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547716837000 4 connected da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547716837000 5 connected 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547716836000 2 connected 5461-10922 e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547716838053 7 connected 0-5460 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547716837138 3 connected 10923-16383 7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 slave e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 0 1547716838153 7 connected
观察当id为7d0d00d7710fc416ac2c3f44ea86b306b498038e的master节点挂掉的时候,其下id为e9f1294a466d3c2e4c3b40d59c8237c346c7fed7的slave的节点自动升级为master节点,当之前的master又重新启动的时候自动变成slave节点。
增加一个节点
# 增加一个redis实例,replica4端口映射到本地为7006 docker run --name replica4 -tid --network service -v $HOME/redis/config:/opt/config -p 7006:6379 -p 17006:16379 redis:alpine redis-server /opt/config/r4.conf # 给id为4de60b987e84de81a6fce10d68f4b1ba0d2ac28a增加一个slave redis-cli --cluster add-node 192.168.201.209:7006 192.168.201.209:7002 --cluster-slave --cluster-master-id 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a # 查看集群节点状态 /data # redis-cli cluster nodes b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547718761000 4 connected da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547718761820 5 connected 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547718761000 2 connected 5461-10922 e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547718761000 7 connected 0-5460 435a5e171bd4ffc02e862d174609fa0d07045925 192.168.201.209:7006@17006 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547718761312 3 connected 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547718760000 3 connected 10923-16383 7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 slave e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 0 1547718761517 7 connected
删除一个节点
/data # redis-cli --cluster del-node 192.168.201.209:7000 435a5e171bd4ffc02e862d174609fa0d07045925 >>> Removing node 435a5e171bd4ffc02e862d174609fa0d07045925 from cluster 192.168.201.209:7000 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. # 查看集群节点状态 /data # redis-cli cluster nodes b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547719609601 4 connected da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547719610512 5 connected 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547719611000 2 connected 5461-10922 e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547719611626 7 connected 0-5460 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547719610512 3 connected 10923-16383 7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 slave e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 0 1547719610613 7 connected
数据同步测试
/data # redis-cli -c -h 192.168.201.209 -p 7000 192.168.201.209:7000> keys * (empty list or set) 192.168.201.209:7000> set name cymin -> Redirected to slot [5798] located at 192.168.201.209:7001 OK 192.168.201.209:7001> /data # redis-cli -c -h 192.168.201.209 -p 7001 192.168.201.209:7001> keys * 1) "name" 192.168.201.209:7001> /data # redis-cli -c -h 192.168.201.209 -p 7002 192.168.201.209:7002> keys * (empty list or set) 192.168.201.209:7002> get name -> Redirected to slot [5798] located at 192.168.201.209:7001 “cymin” 192.168.201.209:7001> 192.168.201.209:7001> get afawfgawfgw -> Redirected to slot [15081] located at 192.168.201.209:7002 (nil) 192.168.201.209:7002> get feawgwegewgew -> Redirected to slot [7926] located at 192.168.201.209:7001 (nil) 192.168.201.209:7001>
观察数据读写均在master节点,如果在replica节点中读写key,则会自动重定向到对应的master节点中。
集群密码设置
进入各个(6个)实例进行依次设置:
redis-cli -c -h 192.168.201.209 -p 7000 127.0.0.1:7000> config set masterauth passwd123 127.0.0.1:7000> config set requirepass passwd123
# 重新进入实例,写入配置
redis-cli -a abc123 -c -h 192.168.201.209 -p 7000 127.0.0.1:7000> config rewrite
java连接集群事例
Jar:
compile group: 'redis.clients', name: 'jedis', version: '3.0.1'
使用jedis在代码中连接redis集群,因为是集群,可以自动发现,也同样只要一个地址就行了
import org.apache.commons.pool2.impl.GenericObjectPoolConfig; import redis.clients.jedis.HostAndPort; import redis.clients.jedis.JedisCluster; import java.util.HashSet; import java.util.Set; public class RedisTest { public static void main(String[] args) { testWithPassword(); } public static void test() { Set<HostAndPort> jedisClusterNodes = new HashSet<>(); jedisClusterNodes.add(new HostAndPort("192.168.201.208", 7000)); JedisCluster jc = new JedisCluster(jedisClusterNodes); String value = jc.get("name"); System.out.println(value); } public static void testWithPassword() { Set<HostAndPort> jedisClusterNodes = new HashSet<>(); jedisClusterNodes.add(new HostAndPort("192.168.201.208", 7000)); GenericObjectPoolConfig gopc = new GenericObjectPoolConfig(); gopc.setMaxTotal(32); gopc.setMaxIdle(4); JedisCluster jc = new JedisCluster(jedisClusterNodes, 10000, 10000, 3,"abc123", gopc); String value = jc.get("name"); System.out.println(value); } }
The end.