Redis Cluster 强制kill某一个节点和shutdown某一个节点后修复过程

redis cluster 命令行,执行以下命令需登录cluster,是集群所独有的
集群(cluster)
CLUSTER INFO 打印集群的信息
CLUSTER NODES 列出集群当前已知的所有节点(node),以及这些节点的相关信息

节点(node)
CLUSTER MEET <ip> <port> 将 ip 和 port 所指定的节点添加到集群当中,让它成为集群的一份子
CLUSTER FORGET <node_id> 从集群中移除 node_id 指定的节点
CLUSTER REPLICATE <node_id> 将当前节点设置为 node_id 指定的节点的从节点
CLUSTER SAVECONFIG 将节点的配置文件保存到硬盘里面

槽(slot)
CLUSTER ADDSLOTS <slot> [slot ...] 将一个或多个槽(slot)指派(assign)给当前节点
CLUSTER DELSLOTS <slot> [slot ...] 移除一个或多个槽对当前节点的指派
CLUSTER FLUSHSLOTS 移除指派给当前节点的所有槽,让当前节点变成一个没有指派任何槽的节点
CLUSTER SETSLOT <slot> NODE <node_id> 将槽 slot 指派给 node_id 指定的节点,如果槽已经指派给另一个节点,那么先让另一个节点删除该槽>,然后再进行指派
CLUSTER SETSLOT <slot> MIGRATING <node_id> 将本节点的槽 slot 迁移到 node_id 指定的节点中
CLUSTER SETSLOT <slot> IMPORTING <node_id> 从 node_id 指定的节点中导入槽 slot 到本节点
CLUSTER SETSLOT <slot> STABLE 取消对槽 slot 的导入(import)或者迁移(migrate)

键 (key)
CLUSTER KEYSLOT <key> 计算键 key 应该被放置在哪个槽上
CLUSTER COUNTKEYSINSLOT <slot> 返回槽 slot 目前包含的键值对数量
CLUSTER GETKEYSINSLOT <slot> <count> 返回 count 个 slot 槽中的键

验证将集群某一个节点强行kill后是否可以自动加入集群:
查看redis cluster关系:

[root@192-13-168-77 ~]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes
06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575439453000 3 connected 10923-16383
d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575439454000 1 connected 0-5460
532ef94c81188111827fef599ee73c0996a04e5e 192.13.168.77:7007@17007 slave 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 0 1575439455000 6 connected
46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575439455742 5 connected
ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575439454743 2 connected 5461-10922
b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575439454000 4 connected

查看redis cluster进程:

[root@192-13-168-77 ~]# ps -ef | grep redis
root 2039 1 0 14:00 ? 00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster] 
root 2236 1 0 14:00 ? 00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster] 
root 2335 1 0 14:00 ? 00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster] 
root 2365 1 0 14:00 ? 00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster] 
root 2391 1 0 14:00 ? 00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster] 
root 2418 1 0 14:00 ? 00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:7007 [cluster]
[root@192-13-168-77 ~]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes
06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575439453000 3 connected 10923-16383
d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575439454000 1 connected 0-5460
532ef94c81188111827fef599ee73c0996a04e5e 192.13.168.77:7007@17007 slave 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 0 1575439455000 6 connected
46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575439455742 5 connected
ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575439454743 2 connected 5461-10922
b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575439454000 4 connected

通过redis cluster slots查看主从关系:

[root@192-13-168-77 ~]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster slots | xargs -n8 | awk '{print $3":"$4"->"$6":"$7}' | sort -nk2 -t ':' | uniq
192.13.168.77:2002->192.13.168.77:5005
192.13.168.77:3003->192.13.168.77:6006
192.13.168.77:7007->192.13.168.77:4004

强行kill:

[root@192-13-168-77 ~]# kill -9 2418
[root@192-13-168-77 ~]# ps -ef | grep redis
root 2039 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster] 
root 2236 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster] 
root 2335 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster] 
root 2365 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster] 
root 2391 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster]

查看7007从4004的日志,4004升级为新master:

[root@192-13-168-77 log]# cat redis_4004.log 
23377:C 04 Dec 13:40:40.502 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
23377:C 04 Dec 13:40:40.502 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=23377, just started
23377:C 04 Dec 13:40:40.502 # Configuration loaded
23378:M 04 Dec 13:40:40.517 # Server initialized
23378:M 04 Dec 13:40:40.517 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
23378:M 04 Dec 13:40:40.517 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
23378:M 04 Dec 13:57:15.782 # User requested shutdown...
23378:M 04 Dec 13:57:15.783 # Redis is now ready to exit, bye bye...
2334:C 04 Dec 14:00:25.888 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2334:C 04 Dec 14:00:25.888 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=2334, just started
2334:C 04 Dec 14:00:25.889 # Configuration loaded
2335:M 04 Dec 14:00:25.896 # Server initialized
2335:M 04 Dec 14:00:25.896 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2335:M 04 Dec 14:00:25.896 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2335:M 04 Dec 14:01:24.003 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH
2335:M 04 Dec 14:01:24.180 # IP address for this node updated to 192.13.168.77
2335:M 04 Dec 14:01:29.005 # Cluster state changed: ok
2335:M 04 Dec 14:06:56.069 # Manual failover requested by slave 532ef94c81188111827fef599ee73c0996a04e5e.
2335:M 04 Dec 14:06:56.258 # Failover auth granted to 532ef94c81188111827fef599ee73c0996a04e5e for epoch 7
2335:M 04 Dec 14:06:56.263 # Connection with slave 192.13.168.77:7007 lost.
2335:M 04 Dec 14:06:56.266 # Configuration change detected. Reconfiguring myself as a replica of 532ef94c81188111827fef599ee73c0996a04e5e
2335:S 04 Dec 14:06:56.538 # Master replication ID changed to 880dfbbc29acda08aa0f997ec0d3d9238f987cca
2335:S 04 Dec 14:46:13.970 # Connection with master lost.
2335:S 04 Dec 14:46:14.427 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:15.427 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:16.430 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:17.433 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:18.436 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:19.438 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:20.441 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:21.443 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:22.445 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:23.448 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:24.449 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:25.451 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:26.453 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:27.455 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:28.457 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:29.459 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:29.960 # Start of election delayed for 658 milliseconds (rank #0, offset 3738).
2335:S 04 Dec 14:46:30.461 # Error condition on socket for SYNC: Connection refused
2335:S 04 Dec 14:46:30.662 # Starting a failover election for epoch 8.
2335:S 04 Dec 14:46:30.667 # Failover election won: I'm the new master.
2335:S 04 Dec 14:46:30.667 # configEpoch set to 8 after successful failover
2335:M 04 Dec 14:46:30.667 # Setting secondary replication ID to 880dfbbc29acda08aa0f997ec0d3d9238f987cca, valid up to offset: 3739. New replication ID is 8ae5eb556fbfd6c13ec33b1123c87de1fbe4db05

到7007data目录下删除nodes-7007.conf:
[root@192-13-168-77 ~]# cd /u02/redis/7007/

[root@192-13-168-77 7007]# ls
conf data log pid
[root@192-13-168-77 7007]# cd data/
[root@192-13-168-77 data]# ls
nodes-7007.conf redis_7007_dump.rdb

[root@192-13-168-77 data]# rm -rf nodes-7007.conf

[root@192-13-168-77 data]# ps -ef | grep redis
root 2039 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster]
root 2236 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster]
root 2335 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster]
root 2365 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster]
root 2391 1 0 14:00 ? 00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster]


启动7007进程:

[root@192-13-168-77 data]# /usr/local/redis-4.0.9/bin/redis-server /u02/redis/7007/conf/redis_7007.conf

因强制kill无法自动加入集群:

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes
06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575442290000 8 connected 10923-16383
d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575442289000 1 connected 0-5460
532ef94c81188111827fef599ee73c0996a04e5e :0@0 master,fail,noaddr - 1575441974024 1575441972822 7 disconnected
46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575442290483 5 connected
ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575442291483 2 connected 5461-10922
b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575442289481 4 connected

在每个节点执行:
CLUSTER FORGET 06031e33797ef0aa6427bddb1ff958f7af0f1a4a

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB"
192.13.168.77:2002> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e
OK

192.13.168.77:3003> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e
OK

192.13.168.77:5005> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e
OK

192.13.168.77:6006> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e
OK

重新启动7007 redis进程:

[root@192-13-168-77 data]# /usr/local/redis-4.0.9/bin/redis-server /u02/redis/7007/conf/redis_7007.conf

[root@192-13-168-77 data]# ps -ef | grep redis
root 2039 1 0 14:00 ? 00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster] 
root 2236 1 0 14:00 ? 00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster] 
root 2335 1 0 14:00 ? 00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster] 
root 2365 1 0 14:00 ? 00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster] 
root 2391 1 0 14:00 ? 00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster] 
root 2931 1 0 15:07 ? 00:00:00 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:7007 [cluster]

将7007重新加入为4004的slave

[root@192-13-168-77 data]# redis-trib.rb add-node --slave --master-id 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:7007 192.13.168.77:2002
>>> Adding node 192.13.168.77:7007 to cluster 192.13.168.77:2002
>>> Performing Cluster Check (using node 192.13.168.77:2002)
M: d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: 46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006
slots: (0 slots) slave
replicates ef76f232efb578249e8d0ec8fef8ec02b3524010
M: ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005
slots: (0 slots) slave
replicates d49ebf2a5f3605487ea4c8deee7e2aa2782667e6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.13.168.77:7007 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.13.168.77:4004.
[OK] New node added correctly.


检查redis cluster状态:

[root@192-13-168-77 data]# redis-trib.rb check 192.13.168.77:2002
>>> Performing Cluster Check (using node 192.13.168.77:2002)
M: d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006
slots: (0 slots) slave
replicates ef76f232efb578249e8d0ec8fef8ec02b3524010
M: ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007
slots: (0 slots) slave
replicates 06031e33797ef0aa6427bddb1ff958f7af0f1a4a
S: b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005
slots: (0 slots) slave
replicates d49ebf2a5f3605487ea4c8deee7e2aa2782667e6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


重新查看集群状态:

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 3003 -c -a "ysBhqkYHDifB" cluster nodes
b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575452066959 4 connected
1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007@17007 slave 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 0 1575452066000 8 connected
46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575452065958 5 connected
06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575452067960 8 connected 10923-16383
ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 myself,master - 0 1575452065000 2 connected 5461-10922
d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 master - 0 1575452067000 1 connected 0-5460


正常shtudown某一个节点后重新加入集群:

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 4004 -c -a "ysBhqkYHDifB" shutdown
[root@192-13-168-77 log]# cat redis_7007.log
4540:S 04 Dec 18:03:32.106 # Connection with master lost.
4540:S 04 Dec 18:03:32.921 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:33.924 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:34.925 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:35.928 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:36.930 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:37.932 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:38.934 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:39.935 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:40.936 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:41.938 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:42.940 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:43.942 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:44.943 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:45.945 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:46.948 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:47.550 # Start of election delayed for 554 milliseconds (rank #0, offset 17724).
4540:S 04 Dec 18:03:47.950 # Error condition on socket for SYNC: Connection refused
4540:S 04 Dec 18:03:48.150 # Starting a failover election for epoch 9.
4540:S 04 Dec 18:03:48.155 # Failover election won: I'm the new master.
4540:S 04 Dec 18:03:48.155 # configEpoch set to 9 after successful failover
4540:M 04 Dec 18:03:48.155 # Setting secondary replication ID to 8ae5eb556fbfd6c13ec33b1123c87de1fbe4db05, valid up to offset: 17725. New replication ID is e44a29dcbf3dce8e1d29c06bd310ba2ba3d0c41b

查看集群状态:

redis-cli -h 192.13.168.77 -p 4004 -c -a "ysBhqkYHDifB" cluster nodes
Could not connect to Redis at 192.13.168.77:4004: Connection refused
[root@192-13-168-77 log]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes 
06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master,fail - 1575453812190 1575453808482 8 disconnected
d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575453939000 1 connected 0-5460
46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575453940773 5 connected
ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575453939000 2 connected 5461-10922
1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007@17007 master - 0 1575453939770 9 connected 10923-16383
b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575453940000 4 connected

重新启动4004进程:

[root@192-13-168-77 log]# /usr/local/redis-4.0.9/bin/redis-server /u02/redis/4004/conf/redis_4004.conf

查看集群状态:(4004自动添加进来并变成slave)

[root@192.13.168.77 log]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes 
06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 slave 1dba8274e36bde79a215a77d1f241ae6fc81c03e 0 1575454019000 9 connected
d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575454018000 1 connected 0-5460
46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575454019942 5 connected
ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575454020944 2 connected 5461-10922
1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007@17007 master - 0 1575454018000 9 connected 10923-16383
b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575454020000 4 connected
原文地址:https://www.cnblogs.com/hankyoon/p/11984620.html