写在前面：
1. 该文章部署方式为二进制部署。
2. 版本信息 k8s 1.7.6，etcd 3.2.9
3. 高可用部分 etcd做高可用集群、kube-apiserver 为无状态服务使用haproxy做负载均衡，kube-controller-manager和kube-scheduler使用自身的选举功能，无需考虑高可用问题。
环境说明：
本环境中网络说明，宿主机及node网段为192.168.1.x/24,service cluster网段为172.16.x.x/16 ,pod网段为172.17.x.x/16,下面会用到。
| 主机名 | ip地址 | 服务 | 备注 |
| master1 | 192.168.1.18 | etcd flanneld kube-apiserver kube-controller-manager kube-scheduler haproxy keepalived | VIP 192.168.1.24作为apiserver的浮动ip |
| master2 | 192.168.1.19 | etcd flanneld kube-apiserver kube-controller-manager kube-scheduler haproxy keepalived | |
| master3 | 192.168.1.20 | etcd flanneld kube-apiserver kube-controller-manager kube-scheduler | |
| node1 | 192.168.1.21 | flanneld docker kube-proxy kubelet harbor | |
| node2 | 192.168.1.22 | flanneld docker kube-proxy kubelet harbor | |
| node3 | 192.168.1.23 | flanneld docker kube-proxy kubelet harbor |
步骤：
1. 证书及kubeconfig文件生成(该操作在任何一台master上执行即可)
kubernetes 系统的各组件需要使用 TLS 证书对通信进行加密，本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 和其它证书；
生成的 CA 证书和秘钥文件如下：
- ca-key.pem
- ca.pem
- kubernetes-key.pem
- kubernetes.pem
- kube-proxy.pem
- kube-proxy-key.pem
- admin.pem
- admin-key.pem
使用证书的组件如下：
- etcd：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
- kube-apiserver：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
- kubelet：使用 ca.pem；
- kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem；
- kubectl：使用 ca.pem、admin-key.pem、admin.pem；
证书生成需要使用cfssl，下面安装cfssl：
[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 [root@k8s-master01 ~]# chmod +x cfssl_linux-amd64 [root@k8s-master01 ~]# mv cfssl_linux-amd64 /usr/bin/cfssl
[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 [root@k8s-master01 ~]# chmod +x cfssljson_linux-amd64 [root@k8s-master01 ~]# mv cfssljson_linux-amd64 /usr/bin/cfssljson
[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 [root@k8s-master01 ~]# chmod +x cfssl-certinfo_linux-amd64 [root@k8s-master01 ~]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
创建 CA (Certificate Authority)
创建 CA 配置文件
[root@k8s-master01 ~]# mkdir /opt/ssl [root@k8s-master01 ~]# cd /opt/ssl [root@k8s-master01 ~]# cfssl print-defaults config > config.json [root@k8s-master01 ~]# cfssl print-defaults csr > csr.json [root@k8s-master01 ~]# cat ca-config.json { "signing": { "default": { "expiry": "8760h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } } }
创建 CA 证书签名请求
[root@k8s-master01 ~]# cat ca-csr.json { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
生成 CA 证书和私钥
[root@k8s-master01 ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca [root@k8s-master01 ~]# ls ca* ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
创建 kubernetes 证书
创建 kubernetes 证书签名请求
[root@k8s-master01 ~]# cat kubernetes-csr.json { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.18", "192.168.1.19", "192.168.1.20", "192.168.1.21", "192.168.1.22", "192.168.1.23", "172.16.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
- 如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表，由于该证书后续被
etcd集群和kubernetes master集群使用，所以上面分别指定了etcd集群、kubernetes master集群的主机 IP 和kubernetes服务的服务 IP（一般是kue-apiserver指定的service-cluster-ip-range网段的第一个IP，如 本环境中的172.16.0.1。
生成 kubernetes 证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes $ ls kuberntes* kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem
创建 admin 证书
创建 admin 证书签名请求
$ cat admin-csr.json { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] }
生成 admin 证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin $ ls admin* admin.csr admin-csr.json admin-key.pem admin.pem
创建 kube-proxy 证书
创建 kube-proxy 证书签名请求
$ cat kube-proxy-csr.json { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
生成 kube-proxy 客户端证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy $ ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
分发证书
将生成的证书和秘钥文件（后缀名为.pem）拷贝到所有机器的 /etc/kubernetes/ssl 目录下备用；
[root@k8s-master01 ~]# cd /opt/ssl/ [root@k8s-master01 ssl]# mkdir -p /etc/kubernetes/ssl/[root@k8s-master01 ssl]# cp * /etc/kubernetes/ssl/ [root@k8s-master01 ssl]# for i in `seq 19 23`; do scp -r /etc/kubernetes/ 192.168.1.$i:/etc/;done
创建 kubeconfig 文件
配置kubectl的kubeconfig文件
文件会生产在 /root/.kube/config
#配置 kubernetes 集群 [root@k8s-master01 ~]# kubectl config set-cluster kubernetes > --certificate-authority=/etc/kubernetes/ssl/ca.pem > --embed-certs=true > --server=https://192.168.1.24:6444 Cluster "kubernetes" set. #配置 客户端认证 [root@k8s-master01 ~]# kubectl config set-credentials admin > --client-certificate=/etc/kubernetes/ssl/admin.pem > --embed-certs=true > --client-key=/etc/kubernetes/ssl/admin-key.pem User "admin" set. [root@k8s-master01 ~]# kubectl config set-context kubernetes > --cluster=kubernetes > --user=admin Context "kubernetes" created. [root@k8s-master01 ~]# kubectl config use-context kubernetes Switched to context "kubernetes". #分发文件 [root@k8s-master01 ~]# for i in `seq 19 23`;do scp -r /root/.kube 192.168.1.$i:/root/;done config 100% 6260 6.1KB/s 00:00 config 100% 6260 6.1KB/s 00:00 config 100% 6260 6.1KB/s 00:00 config 100% 6260 6.1KB/s 00:00 config 100% 6260 6.1KB/s 00:00 [root@k8s-master01 ~]#
kubelet、kube-proxy 等 Node 机器上的进程与 Master 机器的 kube-apiserver 进程通信时需要认证和授权；
kubernetes 1.4 开始支持由 kube-apiserver 为客户端生成 TLS 证书的 TLS Bootstrapping 功能，这样就不需要为每个客户端生成证书了；该功能当前仅支持为 kubelet 生成证书；
创建 TLS Bootstrapping Token
Token auth file
Token可以是任意的包涵128 bit的字符串，可以使用安全的随机数发生器生成。
[root@k8s-master01 ssl]# cd /etc/kubernetes/ [root@k8s-master01 kubernetes]# export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') [root@k8s-master01 kubernetes]# cat > token.csv <<EOF > ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" > EOF [root@k8s-master01 kubernetes]# ls ssl token.csv [root@k8s-master01 kubernetes]# cat token.csv bd962dfaa4b87d896c4e944f113428d3,kubelet-bootstrap,10001,"system:kubelet-bootstrap" [root@k8s-master01 kubernetes]#
将token.csv发到所有机器（Master 和 Node）的 /etc/kubernetes/ 目录。
[root@k8s-master01 kubernetes]# for i in `seq 19 23`; do scp token.csv 192.168.1.$i:/etc/kubernetes/;done token.csv 100% 84 0.1KB/s 00:00 token.csv 100% 84 0.1KB/s 00:00 token.csv 100% 84 0.1KB/s 00:00 token.csv 100% 84 0.1KB/s 00:00 token.csv 100% 84 0.1KB/s 00:00 [root@k8s-master01 kubernetes]#
创建 kubelet bootstrapping kubeconfig 文件
kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求，需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper 角色，然后 kubelet 才有权限创建认证请求(certificatesigningrequests)。
先创建认证请求 user 为 master 中 token.csv 文件里配置的用户 只需在一个node中创建一次就可以
Master节点执行
[root@k8s-master01 bin]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
创建kubelet kubeconfig文件
拷贝kubectl二进制文件
[root@k8s-master01 bin]# cp kubectl /usr/bin/
[root@k8s-master01 bin]# cd /etc/kubernetes/
#配置集群 [root@k8s-master01 kubernetes]# kubectl config set-cluster kubernetes > --certificate-authority=/etc/kubernetes/ssl/ca.pem > --embed-certs=true > --server=https://192.168.1.24:6444 > --kubeconfig=bootstrap.kubeconfig Cluster "kubernetes" set.
#配置客户端认证 [root@k8s-master01 kubernetes]# kubectl config set-credentials kubelet-bootstrap > --token=bd962dfaa4b87d896c4e944f113428d3 > --kubeconfig=bootstrap.kubeconfig User "kubelet-bootstrap" set.
#配置关联 [root@k8s-master01 kubernetes]# kubectl config set-context default > --cluster=kubernetes > --user=kubelet-bootstrap > --kubeconfig=bootstrap.kubeconfig Context "default" created.
#配置默认关联 [root@k8s-master01 kubernetes]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig Switched to context "default". [root@k8s-master01 kubernetes]# ls bootstrap.kubeconfig ssl token.csv
#分发文件 [root@k8s-master01 kubernetes]# for i in `seq 19 23`; do scp bootstrap.kubeconfig 192.168.1.$i:/etc/kubernetes/;done bootstrap.kubeconfig 100% 2166 2.1KB/s 00:00 bootstrap.kubeconfig 100% 2166 2.1KB/s 00:00 bootstrap.kubeconfig 100% 2166 2.1KB/s 00:00 bootstrap.kubeconfig 100% 2166 2.1KB/s 00:00 bootstrap.kubeconfig 100% 2166 2.1KB/s 00:00 [root@k8s-master01 kubernetes]#
创建kube-proxy kubeconfig文件
[root@k8s-master01 ~]# cd /etc/kubernetes/ #Node节点 配置集群 [root@k8s-master01 kubernetes]# kubectl config set-cluster kubernetes > --certificate-authority=/etc/kubernetes/ssl/ca.pem > --embed-certs=true > --server=https://192.168.1.24:6444 > --kubeconfig=kube-proxy.kubeconfig Cluster "kubernetes" set. #配置客户端认证 [root@k8s-master01 kubernetes]# kubectl config set-credentials kube-proxy > --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem > --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem > --embed-certs=true > --kubeconfig=kube-proxy.kubeconfig User "kube-proxy" set. #配置关联 [root@k8s-master01 kubernetes]# kubectl config set-context default > --cluster=kubernetes > --user=kube-proxy > --kubeconfig=kube-proxy.kubeconfig Context "default" created. #配置默认关联 [root@k8s-master01 kubernetes]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig Switched to context "default". [root@k8s-master01 kubernetes]# ls bootstrap.kubeconfig kube-proxy.kubeconfig ssl token.csv [root@k8s-master01 kubernetes]#
#分发文件到所有node节点即可
[root@k8s-master01 kubernetes]# for i in `seq 19 23`; do scp kube-proxy.kubeconfig 192.168.1.$i:/etc/kubernetes/;done
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
2. etcd高可用部署
3. master节点配置
安装
[root@k8s-master01 src]# tar zxvf flannel-v0.9.0-linux-amd64.tar.gz flanneld mk-docker-opts.sh README.md [root@k8s-master01 src]# mv flanneld /usr/bin/ [root@k8s-master01 src]# mv mk-docker-opts.sh /usr/bin/ [root@k8s-master01 src]# for i in `seq 19 23`;do scp /usr/bin/flanneld /usr/bin/mk-docker-opts.sh 192.168.1.$i:/usr/bin/ ;done flanneld 100% 33MB 32.9MB/s 00:00 mk-docker-opts.sh 100% 2139 2.1KB/s 00:00 flanneld 100% 33MB 32.9MB/s 00:00 mk-docker-opts.sh 100% 2139 2.1KB/s 00:00 flanneld 100% 33MB 32.9MB/s 00:01 mk-docker-opts.sh 100% 2139 2.1KB/s 00:00 flanneld 100% 33MB 32.9MB/s 00:00 mk-docker-opts.sh 100% 2139 2.1KB/s 00:00 flanneld 100% 33MB 32.9MB/s 00:01 mk-docker-opts.sh 100% 2139 2.1KB/s 00:00 [root@k8s-master01 src]#
所有master节点分发二进制程序
[root@k8s-master01 bin]# for i in `seq 18 20`;do scp kube-apiserver kube-controller-manager kube-scheduler 192.168.1.$i:/usr/bin/;donekube-apiserver 100% 176MB 88.2MB/s 00:02 kube-controller-manager 100% 131MB 65.3MB/s 00:02 kube-scheduler 100% 73MB 72.6MB/s 00:01 kube-apiserver 100% 176MB 58.8MB/s 00:03 kube-controller-manager 100% 131MB 65.3MB/s 00:02 kube-scheduler 100% 73MB 72.6MB/s 00:01 kube-apiserver 100% 176MB 58.8MB/s 00:03 kube-controller-manager 100% 131MB 65.3MB/s 00:02 kube-scheduler 100% 73MB 72.6MB/s 00:01
添加CA证书到系统信任库
使用动态CA配置
update-ca-trust force-enable
拷贝ca根证书到指定目录
cp /etc/kubernetes/ssl/ca.pem /etc/pki/ca-trust/source/anchors/
生效
update-ca-trust extract
5.3 配置flannel的ip段
etcd节点执行 此网段为上面提到的pod网段
[root@k8s-master01 src]# etcdctl --endpoint https://192.168.1.18:2379 set /flannel/network/config '{"Network":"172.17.0.0/16"}' {"Network":"172.17.0.0/16"} [root@k8s-master01 src]#
5.4 配置flannel
设置flanneld.service
[root@k8s-master01 system]# cat /usr/lib/systemd/system/flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify EnvironmentFile=/etc/sysconfig/flanneld EnvironmentFile=-/etc/sysconfig/docker-network ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS ExecStartPost=/usr/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service [root@k8s-master01 system]#
配置文件：
[root@k8s-master01 system]# cat /etc/sysconfig/flanneld FLANNEL_ETCD_ENDPOINTS="https://192.168.1.18:2379,https://192.168.1.19:2379,https://192.168.1.20:2379" FLANNEL_ETCD_PREFIX="/flannel/network" FLANNEL_OPTIONS="--iface=eth0"
#iface为物理网卡名
[root@k8s-master01 system]# cat /etc/sysconfig/docker-network
CKER_NETWORK_OPTIONS=
#可以为空
[root@k8s-master01 system]# cat /usr/bin/flanneld-start
#!/bin/sh
exec /usr/bin/flanneld
-etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS:-${FLANNEL_ETCD}}
-etcd-prefix=${FLANNEL_ETCD_PREFIX:-${FLANNEL_ETCD_KEY}}
"$@"
[root@k8s-master01 system]# chmod +x /usr/bin/flanneld-start
确保docker已停止
systemctl stop docker
启动flanneld服务
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
验证
[root@k8s-master01 system]# ifconfig flannel0 flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1472 inet 172.17.2.0 netmask 255.255.0.0 destination 172.17.2.0 unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
配置kube-apiserver
创建日志目录
[root@k8s-master01 ~]# mkdir /var/log/kubernetes
配置service文件
[root@k8s-master01 system]# cat /usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] User=root ExecStart=/usr/bin/kube-apiserver --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota --advertise-address=192.168.1.18 --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --authorization-mode=RBAC --bind-address=192.168.1.18 --client-ca-file=/etc/kubernetes/ssl/ca.pem --enable-swagger-ui=true --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --etcd-servers=https://192.168.1.18:2379,https://192.168.1.19:2379,https://192.168.1.20:2379 --event-ttl=1h --kubelet-https=true --insecure-bind-address=192.168.1.18 --runtime-config=rbac.authorization.k8s.io/v1alpha1 --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=172.16.0.0/16 --service-node-port-range=30000-32000 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --logtostderr=false --log-dir=/var/log/kubernetes --v=2 Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target [root@k8s-master01 system]#
#其中--service-cluster-ip-range=172.16.0.0/16 就是上面提到的service 网段，这里面要注意的是 --service-node-port-range=30000-32000
这个地方是 映射外部端口时 的端口范围，随机映射也在这个范围内映射，指定映射端口必须也在这个范围内。
#启动服务
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver配置kube-controller-manager
配置service文件
[root@k8s-master01 kubernetes]# cat /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/usr/bin/kube-controller-manager --address=0.0.0.0 --master=http://192.168.1.24:8081 --allocate-node-cidrs=true --service-cluster-ip-range=172.16.0.0/16 --cluster-cidr=172.17.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target [root@k8s-master01 kubernetes]# #启动服务 systemctl daemon-reload systemctl enable kube-controller-manager systemctl start kube-controller-manager systemctl status kube-controller-manager
配置kube-controller-scheduler
4. node节点配置
5. 安装私有仓库harbor并高可用配置
6. 安装dns插件
7. 安装 dashboard
8. 安装监控插件