节点信息：
| 主机名 | IP | 角色 |
| k8s-master | 10.10.0.10 | master节点 |
| k8s-node01 | 10.10.0.11 | 集群worke节点 |
| k8s-node02 | 10.10.0.12 | 集群worke节点 |
一、初始化系统环境
系统初始化环境配置，需要在三台服务器都执行，以下命令请看对应的命令参数，在不同服务器执行。
1）主机名
分别在三台机器执行对应命令，设置主机名
[root@centos01 ~]# hostnamectl set-hostname k8s-master [root@centos02 ~]# hostnamectl set-hostname k8s-node01 [root@centos03 ~]# hostnamectl set-hostname k8s-node02
2）关闭防火墙、selinux
[root@centos01 ~]# systemctl disable firewalld [root@centos01 ~]# systemctl stop firewalld [root@centos01 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
3）设置内核
[root@centos01 ~]# vim /etc/sysctl.d/k8s.conf
## 添加如下内容
net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 ## 执行命令生效 [root@centos01 ~]# modprobe br_netfilter [root@centos01 ~]# sysctl -p /etc/sysctl.d/k8s.conf
4）免密钥
在master节点进行对2台node节点进行免密钥登陆
[root@k8s-master ~]# ssh-keygen [root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.11
[root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.12
二、部署kubernetes集群
1）配置yum源
yum源三台机器都需要配置，这里我们以master主机为例，node节点也按照此yum配置即可
[root@k8s-master ~]# yum -y install wget [root@k8s-master ~]# cd /etc/yum.repos.d ## 配置docker-ce源 [root@k8s-master yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ## 配置kubernetes源 [root@k8s-master yum.repos.d]# vim /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes Repo baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg enabled=1 ## 下载校验文件 [root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg [root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg ## 导入校验文件 [root@k8s-master ~]# rpm --import rpm-package-key.gpg [root@k8s-master ~]# rpm --import yum-key.gpg
[root@k8s-master ~]# yum clean all && yum makecache fast
2）安装docker、kubelet、kubeadm等工具
[root@k8s-master ~]# yum install kubelet-1.14.1 kubeadm-1.14.1 kubectl-1.14.1 docker-ce -y
## node节点无需配置kubctl组件
[root@k8s-node01 ~]# yum install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y
[root@k8s-node02 ~]# yum install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y
3）配置docker kubelet
2台node节点也需要修改对应配置
## 设置kubelet启动时忽略swap报错 [root@k8s-master ~]# vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false"
## 设置开机自启动
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable docker && systemctl restart docker
[root@k8s-master ~]# systemctl enable kubelet && systemctl restart kubelet
4）初始化master节点
kubernetes从1.13版本开始，可以指定镜像仓库进行集群初始化操作，所以我们直接指定阿里云镜像仓库进行集群初始化，这样无需再关心国内网络环境是否可以下载到对应的官方原始pod镜像
[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# kubeadm init --apiserver-advertise-address=10.10.0.10 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.14.1 --pod-network-cidr=10.244.0.0/16
参数解释：
- apiserver-advertise-address：apiserver地址及master节点地址
- image-repository：镜像仓库地址
- pod-network-cidr：pod网络
以上初始化，需要等待一段时间，因为需要下载对应组件镜像，master节点初始化完成后，可看到以下提示信息：
........ [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
注意：
token是node节点加入时需要用到的信息，需要记录下来，tocken值24小时后失效，若果超过24小时你再进行node节点加入集群，需要重新生成tocken。
5）配置集群环境变量
用户想要使用kubectl操作集群，则需要配置kubectl环境变量，这些命令也是上面kubeadm init后输出的内容
[root@k8s-master ~]# mkdir -p $HOME/.kube [root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config
## 查看集群信息
[root@k8s-master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
## 查看node就绪状态
### 由于还没有安装网络插件，以及node节点未加入集群，所以只显示一个master节点信息
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 7m33s v1.14.1
三、部署网络插件
集群的运行依赖于网络，k8s本身并不支持网络，需要额外部署对应的网络插件，才可实现集群的个组件网络通信。我们这里采用flannel作为集群网络插件。
项目地址：https://github.com/coreos/flannel
## 在线部署 [root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ## 或者把清单配置文件下载本地再执行清单文件应用 [root@k8s-master ~]# mkdir /opt/k8s/flannel [root@k8s-master ~]# cd /opt/k8s/flannel [root@k8s-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml [root@k8s-master ~]# kubectl apply -f kube-flannel.yml
四、node节点加入集群
1）加入集群
node节点加入集群操作基本一致，这里以其中一台为例。kubeadm join命令为master节点初始化成功后显示的命令，上文已经提到过。
[root@k8s-node01 ~]# swapoff -a
[root@k8s-node01 ~]# kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Hostname]: hostname "k8s-node01" could not be reached
[WARNING Hostname]: hostname "k8s-node01": lookup k8s-node01 on 114.114.114.114:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Hostname]: hostname "k8s-node01" could not be reached
[WARNING Hostname]: hostname "k8s-node01": lookup k8s-node01 on 114.114.114.114:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
注意：
node节点初始化加入集群，会看到对应的初始化成功信息，初始化时由于需要下载对应pod镜像，需要等待一段时间，node才会成功加入。
2）查看节点就绪状态
[root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready master 139m v1.14.1 k8s-node01 Ready <none> 2m8s v1.14.1 k8s-node02 Ready <none> 64s v1.14.1
3）集群删除
以上就是kubeadm部署集群过程，过程其实并不复杂，如果在集群部署过程中，出现问题，想要删除重新初始化集群，使用以下命令：
kubeadm reset ifconfig cni0 down && ip link delete cni0 ifconfig flannel.1 down && ip link delete flannel.1 rm -rf /var/lib/cni/
特别说明：
1）关于token失效
上面说过，master集群初始化后，token24小时后就会失效，如果到了token失效时间，node再加入集群，需要重新生产token：
## 查看token状态
### TTL值 就是token生于时间 [root@k8s-master ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 5ti5kd.o32bm9lofv6zej94 21h 2019-05-22T11:16:31+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token ## 重新生产token [root@k8s-master ~]# kubeadm token create hb0mhv.ckb79uumxh06br8e ## 获取--discovery-token-ca-cert-hash值 [root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
## 加入集群命令
[root@k8s-master ~]# kubeadm join 10.10.0.10:6443 --token hb0mhv.ckb79uumxh06br8e
--discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f