k8s master节点的高可用 (haproxy + keepalived)

准备

Apiserver	所有服务的总入口
ETCD	存储数据
controller-manager	控制器
scheduler	调度服务
kubelet	维持容器的生命周期, 运行cli
proxy	实现负载方案



172.16.240.100 k8s-master01
172.16.240.101 k8s-master02
172.16.240.102 k8s-master03
172.16.240.103 k8s-nodetest
172.16.240.103 k8s-vip
172.16.240.200 registry

设置hosts文件

echo '172.16.240.100 k8s-master01' >> /etc/hosts
echo '172.16.240.101 k8s-master02' >> /etc/hosts
echo '172.16.240.102 k8s-master03' >> /etc/hosts
echo '172.16.240.103 k8s-nodetest' >> /etc/hosts
echo '172.16.240.120 k8s-vip' >> /etc/hosts

安装依赖包

yum -y install conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat 
yum -y install bash-com*

设置防火墙为Iptables并设置规则

systemctl stop firewalld && systemctl disable firewalld
yum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save

关闭SELINUX 和SWAP

swapoff -a && sed -i '/swap/ s/^(.*)$/#1/g' /etc/fstab
setenforce 0 && sed -i ' s/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

调整内核参数

cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

调整系统时区

timedatectl set-timezone Asia/Shanghai
timedatectl set-local-rtc 0

重启依赖于系统事件的服务

systemctl restart rsyslog.service 
systemctl restart crond.service

关闭系统不需要的服务

systemctl stop postfix.service && systemctl disable postfix.service

设置rsyslogd和systemd journald

mkdir /var/log/journal
mkdir /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/99-prophet.cof <<EOF
[Journal]
# 持久化保存到磁盘
Storage=persistent

# 压缩历史日志
Compress=yes

SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000

# 最大占用空间10G
SystemMaxUse=10G

# 单日志文件最大 200M
SystemMaxFileSize=200M

# 日志保存时间2周
MaxRetentionSec=2week

# 不将日志转发到syslog
ForwardToSyslog=no
EOF

systemctl restart systemd-journald.service

Kube-proxy开启ipvs的前置条件

modprobe br_netfilter
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash
/etc/sysconfig/modules/ipvs.modules && lsmod |grep -e ip_vs -e nf_conntrack_ipv4

安装docker

yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum update -y 
yum -y install docker-ce-18.06.3.ce

配置daemon

mkdir /etc/docker
cat >> /etc/docker/daemon.json<<EOF
{
       "registry-mirrors": [
        "https://1nj0zren.mirror.aliyuncs.com",
        "https://docker.mirrors.ustc.edu.cn",
        "http://f1361db2.m.daocloud.io",
        "https://registry.docker-cn.com"
    ],
     "insecure-registries" : [ "172.16.240.200:5000", "172.16.240.110:8999" ], 
     "exec-opts": ["native.cgroupdriver=systemd"],
     "log-driver": "json-file",
    "log-opts": {"max-size": "100m"}
}
EOF

systemctl start docker
systemctl enable docker

升级内核

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install -y kernel-lt
grub2-set-default 'CentoS Linux(4.4.214-1.el7.elrepo.×86_64) 7 (Core)'

关闭NUMA

cp /etc/default/grub{,.bak}
sed -i 's#GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap net.ifnames=0 rhgb quiet"#GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap net.ifnames=0 rhgb quiet numa=off"#g' /etc/default/grub

默认配置文件备份, 生成新的grub配置文件

cp /boot/grub2/grub.cfg{,.bak}
grub2-mkconfig -o /boot/grub2/grub.cfg

下载高可用镜像

wise2c/keepalived-k8s:latest
wise2c/haproxy-k8s:latest

配置keepalived

参考文档 https://www.cnblogs.com/keep-live/p/11543871.html

  • 安装 keepalived
yum install -y keepalived

  • 备份keepalived的配置文件
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg-back

  • 修改 keepalived的配置文件
vim etc/haproxy/haproxy.cfg

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.240.120/24
    }
}

说明:

  1. global_defs 只保留 router_id（每个节点都不同）；
  2. 修改 interface（vip绑定的网卡），及 virtual_ipaddress（vip地址及掩码长度）；
  3. 删除后面的示例
  4. 其他节点只需修改 state 为 BACKUP，优先级 priority 低于100即可。

  • 启动服务
systemctl start keepalived && systemctl enable keepalived && systemctl status keepalived

  • 查看状态
ip a s eth0

172.16.240.100 k8s-master01


172.16.240.101 k8s-master02


172.16.240.102 k8s-master03


结论: vip只会在一台机器上；如果两个机器都有vip，可能是防火墙拦截了vrrp协议。


配置haproxy


  • 下载镜像 到 172.16.240.200 registry , 在三台master上从镜像仓库下载镜像

172.16.240.200 registry 操作

docker pull haproxy:2.1.3
docker tag haproxy:2.1.3 172.16.240.200:5000/haproxy:2.1.3
docker push 172.16.240.200:5000/haproxy:2.1.3

172.16.240.100 k8s-master01 / 172.16.240.101 k8s-master02/ 172.16.240.102 k8s-master03 上操作 

docker pull 172.16.240.200:5000/haproxy:2.1.3

  • 准备haproxy的配置文件haproxy.cfg
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
#chroot /usr/share/haproxy
#user haproxy
#group haproxy
daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    timeout connect  5000
    timeout client  50000
    timeout server  50000

frontend stats-front
  bind *:8081
  mode http
  default_backend stats-back

frontend fe_k8s_6444
  bind *:6444
  mode tcp
  timeout client 1h
  log global
  option tcplog
  default_backend be_k8s_6443
  acl is_websocket hdr(Upgrade) -i WebSocket
  acl is_websocket hdr_beg(Host) -i ws

backend stats-back
  mode http
  balance roundrobin
  stats uri /haproxy/stats
  stats auth pxcstats:secret

backend be_k8s_6443
  mode tcp
  timeout queue 1h
  timeout server 1h
  timeout connect 1h
  log global
  balance roundrobin
  server k8s-master01 172.16.240.100:6443
  server k8s-master02 172.16.240.101:6443
  server k8s-master03 172.16.240.102:6443

  • 启动镜像
docker run -d --restart=always --name haproxy-k8s -p 6444:6444 -e MasterIP1=172.16.240.100 -e MasterIP2=172.16.240.101 -e MasterIP3=172.16.240.102 -v /root/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg 172.16.240.200:5000/haproxy:2.1.3

  • 安装kubeadm, kubectl, kubelet
cat <<EOF >/etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum -y install kubeadm-1.17.4 kubectl-1.17.4 kubelet-1.17.4
systemctl enable kubelet

k8s 初始化主节点

kubeadm config print init-defaults > kubeadm-config.yaml


apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.17.3
imageRepository: 172.16.240.200:5000 
controlPlaneEndpoint: 172.16.240.120:6444
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs:
  - 172.16.240.100
  - 172.16.240.101
  - 172.16.240.102
networking:
  serviceSubnet: 192.168.0.0/16
  podSubnet: "10.244.0.0/16"
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

说明:

  1. controlPlaneEndpoint 填写vip地址, 端口必须和 haproxy中的 frontend fe_k8s_6444 下定义的 bind的端口号一样
  2. certSANs 填写所有的master的ip地址

kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.log



mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

  • 查看server ip



  • 加入master节点
kubeadm join 172.16.240.103:6444 --token abcdef.0123456789abcdef     --discovery-token-ca-cert-hash sha256:a25921ab01da166ae537d348aa6cb8b134e257108d9756d1de249831c5aa4bdd     --control-plane --certificate-key 0bda80f7c5869b70c6092a6c50968ed73159825fc1135c8c31d18762f35e5ef4

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

  • 在172.16.240.100 k8s-master01节点上查看
kubectl get node

部署网络

如果不部署网络, 所有的节点都not ready

在 172.16.240.100 k8s-master01 上部署

wget https://docs.projectcalico.org/v3.10/manifests/calico.yaml
kubectl create -f calico.yaml

  • 查看节点状态
kubectl get pod -n kube-system -owide


部署dashboard

文档：https://www.jianshu.com/p/40c0405811ee


nodeport方式


将recommended.yaml 中镜像的地址改为本地镜像的地址

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc6/aio/deploy/recommended.yaml
mv recommended.yaml dashboard.yaml

修改yaml文件

  • 将service type字段设置为nodeport


  • 注释掉Dashboard Secret ，不然后面访问显示网页不安全，证书过期，我们自己生成证书


  • 将镜像修改为镜像仓库地址


kubectl apply -f dashboard.yaml


设置权限文件

  • admin-user.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

  • admin-user-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system


kubectl create -f admin-user.yaml 
kubectl create -f admin-user-role-binding.yaml

生成证书

mkdir key && cd key
openssl genrsa -out dashboard.key 2048
openssl req -new -out dashboard.csr -key dashboard.key -subj '/CN=172.16.240.100'

  • 查看secret
openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt


  • 删除默认的secret
kubectl delete secret kubernetes-dashboard-certs -n kube-system


  • 生成新的secret

    这里的secret必须在kubernetes-dashboard 名称空间生成, 否则dashboard会起不来, dashboard是启动在kubernetes-dashboard 这个名称空间, 所以secret 也必须在这个空间生成

kubectl create secret generic kubernetes-dashboard-certs --from-file=dashboard.key --from-file=dashboard.crt -n kubernetes-dashboard

  • 删除dashboard的pod, 相当于重启dashboard, 再次查看dashboard是否启动了


网页登录dashboard

  • 查看 dashboard暴露的接口
kubectl get svc -n kubernetes-dashboard


  • 浏览器输入 https://172.16.240.100:30748/

    出现页面就没问题


  • 获取token
kubectl -n kube-system get secret



kubectl describe secret admin-user-token-4xfsp -n kube-system


k8s的一些命令


  • 查看当前节点的健康状况
kubectl -n kube-system exec etcd-k8s-master01  -- etcdctl --endpoints=https://172.16.240.100:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health

  • 查看当前哪个节点活跃
 kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
 kubectl get endpoints kube-scheduler -n kube-system -o yaml

  • 查看集群信息
kubectl cluster-info


  • 查看权限
kubectl config current-context


  • 查看集群的角色
kubectl get clusterrole


  • 删除 serviceaccounts
kubectl delete serviceaccount -n kube-system tiller


  • 删除clusterrolebindings
kubectl delete clusterrolebindings  -n kube-system tiller

【推广】 免费学中医，健康全家人
原文地址：https://www.cnblogs.com/cjwnb/p/12532402.html