上一章，我们通过实践和其他文章的帮助，在k8s的环境安装了redis-ha，并且对其进行了一些实验来验证他的主从切换是否有效。本篇中将会分析，究竟是如何实现了redis-ha的主从切换，以及其与K8S平面进行的交互。
1.如何实现的redis的搭建
我曾经以为是在helm/chart中写入了脚本来完成这件事，但是仔细看过代码之后，并未发现明显的内容，关于搭建redis-ha和主从切换的脚本。
地址：https://github.com/helm/charts/tree/master/stable/redis-ha
后来，通过查看redis镜像的日志发现了一些内容，
地址：https://quay.io/repository/smile/redis/manifest/sha256:8948a952920d4495859c984546838d4c9b4c71e0036eef86570922d91cacb3df?tab=layers
可以看到，在这个镜像构建日志中，有几个疑似相关内容的文件，/usr/local/bin目录下的promte.sh,redis-launcher.sh,label-updater.sh
进入到pod中，我们可以看到redis-launcher是作为启动时就运行的脚本，所以我就推测这一切都是这个文件引起的。
2.脚本内容
redis-launcher.sh:
bash-4.4# cat /usr/local/bin/redis-launcher.sh
#!/bin/bash
# Copyright 2017 Ismail KABOUBI
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This script determines whether the pod that executes it will be a Redis Sentinel, Master, or Slave
# The redis-ha Helm chart signals Sentinel status with environment variables. If they are not set, the newly
# launched pod will scan K8S to see if there is an active master. If not, it uses a deterministic means of
# sensing whether it should launch as master then writes master or slave to the label called redis-role
# appropriately. It's this label that determines which LB a pod can be seen through.
#
# The redis-role=master pod is the key for the cluster to get started. Sentinels will wait for it to appear
# in the LB before they finish launching. All other pods wait for the Sentinels to ID the master.
#
# Pods also set the labels podIP and runID. RunID is the first few characters of the unique run_id value
# generated by each Redis sever.
#
# During normal operation, there should be only one redis-role=master pod. If it fails, the Sentinels
# will nominate a new master and change all the redis-role values appropriately.
echo "Starting redis launcher"
echo "Setting labels"
label-updater.sh & plabeler=$!
echo "Selecting proper service to execute"
# Define config file locations
SENTINEL_CONF=/etc/redis/sentinel.conf
MASTER_CONF=/etc/redis/master.conf
SLAVE_CONF=/etc/redis/slave.conf
# Adapt to dynamically named env vars
ENV_VAR_PREFIX=`echo $REDIS_CHART_PREFIX|awk '{print toupper($0)}'|sed 's/-/_/g'`
PORTVAR="${ENV_VAR_PREFIX}MASTER_SVC_SERVICE_PORT"
HOSTVAR="${ENV_VAR_PREFIX}MASTER_SVC_SERVICE_HOST"
MASTER_LB_PORT="${!PORTVAR}"
MASTER_LB_HOST="${!HOSTVAR}"
QUORUM=${QUORUM:-2}
# Only sets AUTH if the ENV var REDIS_PASS is set.
REDISAUTH=""
[ -n "$REDIS_PASS" ] && REDISAUTH="-a $REDIS_PASS" || REDISAUTH=""
# Launch master when `MASTER` environment variable is set
function launchmaster() {
# If we know we're a master, update the labels right away
kubectl label --overwrite pod $HOSTNAME redis-role="master"
echo "Using config file $MASTER_CONF"
if [[ ! -e /redis-master-data ]]; then
echo "Redis master data doesn't exist, data won't be persistent!"
mkdir /redis-master-data
fi
if [ -n "$REDIS_PASS" ]; then
sed -i "s/# requirepass/requirepass ${REDIS_PASS}
#/" $MASTER_CONF
fi
redis-server $MASTER_CONF --protected-mode no $@
}
# Launch sentinel when `SENTINEL` environment variable is set
function launchsentinel() {
# If we know we're a sentinel, update the labels right away
kubectl label --overwrite pod $HOSTNAME redis-role="sentinel"
echo "Using config file $SENTINEL_CONF"
while true; do
# The sentinels must wait for a load-balanced master to appear then ask it for its actual IP.
MASTER_IP=$(kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"
"}{end}' -l redis-role=master|grep running|grep $REDIS_CHART_PREFIX|awk '{print $2}'|xargs)
echo "Current master is $MASTER_IP"
if [[ -z ${MASTER_IP} ]]; then
continue
fi
timeout -t 3 redis-cli ${REDISAUTH} -h ${MASTER_IP} -p ${MASTER_LB_PORT} INFO
if [[ "$?" == "0" ]]; then
break
fi
echo "Connecting to master failed. Waiting..."
sleep 10
done
echo "sentinel monitor mymaster ${MASTER_IP} ${MASTER_LB_PORT} ${QUORUM}" > ${SENTINEL_CONF}
echo "sentinel down-after-milliseconds mymaster 15000" >> ${SENTINEL_CONF}
echo "sentinel failover-timeout mymaster 30000" >> ${SENTINEL_CONF}
echo "sentinel parallel-syncs mymaster 10" >> ${SENTINEL_CONF}
echo "bind 0.0.0.0" >> ${SENTINEL_CONF}
echo "sentinel client-reconfig-script mymaster /usr/local/bin/promote.sh" >> ${SENTINEL_CONF}
if [ -n "$REDIS_PASS" ]; then
echo "sentinel auth-pass mymaster ${REDIS_PASS}" >> ${SENTINEL_CONF}
fi
redis-sentinel ${SENTINEL_CONF} --protected-mode no $@
}
# Launch slave when `SLAVE` environment variable is set
function launchslave() {
kubectl label --overwrite pod $HOSTNAME redis-role="slave"
echo "Using config file $SLAVE_CONF"
if [[ ! -e /redis-master-data ]]; then
echo "Redis master data doesn't exist, data won't be persistent!"
mkdir /redis-master-data
fi
i=0
while true; do
master=${MASTER_LB_HOST}
timeout -t 3 redis-cli ${REDISAUTH} -h ${master} -p ${MASTER_LB_PORT} INFO
if [[ "$?" == "0" ]]; then
break
fi
i=$((i+1))
if [[ "$i" -gt "30" ]]; then
echo "Exiting after too many attempts"
kill $plabeler
exit 1
fi
echo "Connecting to master failed. Waiting..."
sleep 1
done
if [ -n "$REDIS_PASS" ]; then
sed -i "s/# masterauth/masterauth ${REDIS_PASS}
#/" $SLAVE_CONF
sed -i "s/# requirepass/requirepass ${REDIS_PASS}
#/" $SLAVE_CONF
fi
sed -i "s/%master-ip%/${MASTER_LB_HOST}/" $SLAVE_CONF
sed -i "s/%master-port%/${MASTER_LB_PORT}/" $SLAVE_CONF
redis-server $SLAVE_CONF --protected-mode no $@
}
#Check if MASTER environment variable is set
if [[ "${MASTER}" == "true" ]]; then
echo "Launching Redis in Master mode"
launchmaster
exit 0
fi
# Check if SENTINEL environment variable is set
if [[ "${SENTINEL}" == "true" ]]; then
echo "Launching Redis Sentinel"
launchsentinel
echo "Launcsentinel action completed"
exit 0
fi
# Determine whether this should be a master or slave instance
echo "Looking for pods running as master"
MASTERS=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"
"}{end}' -l redis-role=master|grep running|grep $REDIS_CHART_PREFIX`
if [[ "$MASTERS" == "" ]]; then
echo "No masters found: "$MASTERS" Electing first master..."
SLAVE1=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.creationTimestamp} {.metadata.name} {.status.containerStatuses[0].state} {"
"} {end}' -l redis-node=true |grep running|sort|awk '{print $2}'|grep $REDIS_CHART_PREFIX|head -n1`
if [[ "$SLAVE1" == "$HOSTNAME" ]] || [[ "$SLAVE1" == "" ]]; then
echo "Taking master role"
launchmaster
else
echo "Electing $SLAVE1 master"
launchslave
fi
exit 0
else
echo "Found $MASTERS"
echo "Launching Redis in Slave mode"
launchslave
exit 0
fi
echo "Launching Redis in Slave mode"
launchslave
echo "Launchslave action completed"
label-updater.sh
bash-4.4# cat /usr/local/bin/label-updater.sh
# Push some helpful vars into labels
PODIP=`hostname -i`
echo podIP $PODIP
kubectl label --overwrite pod $HOSTNAME podIP="$PODIP"
if [ "$SENTINEL" ]; then
exit
fi
RUNID=""
# Only sets AUTH if the ENV var REDIS_PASS is set.
REDISAUTH=""
[ -n "$REDIS_PASS" ] && REDISAUTH="-a $REDIS_PASS" || REDISAUTH=""
while true; do
RUNID=`redis-cli $REDISAUTH info server |grep run_id|awk -F: '{print $2}'|head -c6`
if [ -n "$RUNID" ]; then
kubectl label --overwrite pod $HOSTNAME runID="$RUNID"
break
else
sleep 1
fi
done
promote.sh
bash-4.4# cat /usr/local/bin/promote.sh
#!/usr/bin/env bash
MASTERIP=$6
# Convert the IP of the promoted pod to a hostname
MASTERPOD=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"
"}{end}' -l redis-role=slave --sort-by=.metadata.name|grep running|grep $MASTERIP|awk '{print $1}'`
echo "PROMO ARGS: $@"
echo "PROMOTING $MASTERPOD ($MASTERIP) TO MASTER"
kubectl label --overwrite pod $MASTERPOD redis-role="master"
# Demote anyone else who jumped to master
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[0].state}{"
"}{end}' -l redis-role=master --sort-by=.metadata.name|grep running|awk '{print $1}'|grep $REDIS_CHART_PREFIX|grep -v $MASTERPOD|xargs -n1 -I% kubectl label --overwrite pod % redis-role="slave"
echo "OTHER MASTERS $MASTERS"
3.大致原理
详细的内容我还没有开始看，但是可以讲一下大致的原理，就是在每个pod在启动的时候都会起这样的一个redis-launcher的进程，这个就像一个agent一样，主要完成redis的master，slave和sentinel的配置，同时，他会将各个pod的角色反向通过kubectl命令传给K8S平面。
如果是发生了主从结构已经起来了，但是中途因为某种原因挂掉了，则会通过监控sentinel的状态来触发更改K8S平面pod的Role的过程。这个设定是在启动sentinel完成的，代码在这里：
echo "sentinel client-reconfig-script mymaster /usr/local/bin/promote.sh" >> ${SENTINEL_CONF}
他会监控，如果对于mymaster这个集群中的sentinel发生了reconfig的事件的时候，就会去触发/usr/local/bin/promote.sh这个脚本。
所以可以看到，是redis的pod在控制，而不是K8S平面在进行控制。之后有时间，我会详细的读一下这个脚本，然后加上一些注释。
更多openstack/trove的文章：http://www.cnblogs.com/S-tec-songjian/
此文章属博客园用户S-tec原创作品，受国家《著作权法》保护，未经许可，任何单位及个人不得做营利性使用；若仅做个人学习、交流等非营利性使用，应当指明作者姓名、作品名称，原文地址，并且不得侵犯作者依法享有的其他权利。