# Kubernetes Deploy
本文记载Kubernetes通过kubeadm方式安装的详细过程及遇到的问题与解决方法。
更新于2020-08-02
## Kubernetes源
考虑到国内的网络环境，采用 *阿里云* mirrors.aliyun.com 的源.
> https://developer.aliyun.com/mirror/kubernetes
> 下载地址：https://mirrors.aliyun.com/kubernetes/
> 官方主页：https://kubernetes.io/
aliyun-kubernetes.repo
aliyun-docker-ce.repo
yum install docker-ce
yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2
(1)借助NTP服务设置节点时间精确同步;
(2)通过DNS完成各节点的主机名解析，测试环境主机数量较少时也可以使用hosts文件进行;
(3)关闭各节点的iptables或firewalld服务,并确保它们被禁止随系统引导过程启动;
(4)各节点禁用Selinux(否则在运行容器时可以会遇到各种奇葩报错);
(5)各节点禁用所有的swap设备(生产环境中强烈建议禁用,虽说K8S也支持配置参数来启用swap但这样会降低集群性能，使用"swapoff -a"只是临时关闭交换分区使用,永久关闭需要编辑"/etc/fstab"文件将挂载swap哪一行前面加一个"#"进行注释);
(6)若要使用ipvs模型的proxy，各节点还需要载入ipvs相关的各模块
## 1. Kubernetes Repo
> /etc/yum.repos.d/kubernetes-aliyun.repo
```shell
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
```
## 2. Yum install Kubeadm
```shell
setenforce 0
yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2
yum install -y --nogpgcheck kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
[root@master01 ~]# kubeadm config images list
W0727 15:53:36.203783 21851 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0727 15:53:36.203925 21851 version.go:103] falling back to the local client version: v1.18.2
W0727 15:53:36.204122 21851 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.18.2
k8s.gcr.io/kube-controller-manager:v1.18.2
k8s.gcr.io/kube-scheduler:v1.18.2
k8s.gcr.io/kube-proxy:v1.18.2
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.7
```
## 3. 调整docker
```
docker info
cat >/etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://2325obkh.mirror.aliyuncs.com"]
}
EOF
```
docker自1.13版起会自动设置iptable的FORWARD默认策略为DROP，这可能会影响kubernetes集群依赖的报文转发功能,因此，需要在docker服务启动后，重新将FORWARD链的默认策略设置为ACCEPT。
vim /usr/lib/systemd/system/docker.service"
在"ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"之后
新增一行"ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT"(意思是docker服务器启动成功后会执行执行该命令)
```shell
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
```
systemctl daemon-reload
## 4. 调整系统参数
```shell
yum install chrony
chronyc sourcestats -v
swapoff -a
setenforce=0
systemctl stop firewalld.service
```
> 加载ipvs模块
```shell
vim /etc/sysconfig/modules/ipvs.modules
#!/usr/bin/env bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for mod in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do
/usr/sbin/modinfo -F filename $mod &> /dev/null
if [ $? -eq 0 ];then
/sbin/modprobe $mod
fi
done
# master与node上均执行
chmod +x /etc/sysconfig/modules/ipvs.modules
lsmod | grep ip_vs
```
kubeadm config print init-defaults
kubeadm config images list
kubeadm config images pull
docker images
docker image ls
docker image tag old_image:version new_image:version
docker image pull
docker image rm
registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 =
registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 = k8s.gcr.io/pause:3.2
registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar = registry.aliyuncs.com/google_containers/etcd:3.4.3-0 = k8s.gcr.io/etcd:3.4.3-0
registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 = k8s.gcr.io/coredns:1.6.7
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-apiserver__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-controller-manager__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-scheduler__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/pause -o registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar
docker save registry.aliyuncs.com/google_containers/etcd -o registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/coredns -o registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 k8s.gcr.io/kube-proxy:v1.18.2
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.2 k8s.gcr.io/kube-apiserver:v1.18.2
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.2 k8s.gcr.io/kube-controller-manager:v1.18.2
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2 k8s.gcr.io/kube-scheduler:v1.18.2
docker image tag registry.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
docker image tag registry.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 k8s.gcr.io/coredns:1.6.7
[root@master01 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.18.2 0d40868643c6 3 months ago 117MB
k8s.gcr.io/kube-controller-manager v1.18.2 ace0a8c17ba9 3 months ago 162MB
k8s.gcr.io/kube-scheduler v1.18.2 a3099161e137 3 months ago 95.3MB
k8s.gcr.io/kube-apiserver v1.18.2 6ed75ad404bd 3 months ago 173MB
k8s.gcr.io/pause 3.2 80d28bedfe5d 5 months ago 683kB
k8s.gcr.io/coredns 1.6.7 67da37a9a360 6 months ago 43.8MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 9 months ago 288MB
> 初始化CLuster
>> 初始测试dry-run
kubeadm init
--apiserver-advertise-address=192.168.221.134
--image-repository registry.aliyuncs.com/google_containers
--kubernetes-version v1.18.2
--pod-network-cidr=10.244.0.0/16
--dry-run
>> init cluster
kubeadm init
--apiserver-advertise-address=192.168.221.134
--image-repository registry.aliyuncs.com/google_containers
--kubernetes-version v1.18.2
--pod-network-cidr=10.244.0.0/16
```shell
[root@master01 tmp]# kubeadm init --apiserver-advertise-address=192.168.221.134 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.2 --pod-network-cidr=10.244.0.0/16
W0727 16:40:13.862004 32012 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.221.134]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master01 localhost] and IPs [192.168.221.134 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master01 localhost] and IPs [192.168.221.134 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0727 16:40:22.595129 32012 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0727 16:40:22.596079 32012 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 24.507077 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: a4imyy.ttddd23g8ddh7dv7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.221.134:6443 --token a4imyy.ttddd23g8ddh7dv7
--discovery-token-ca-cert-hash sha256:4f784226e480a6de709b1f21e6e67ec02050942e634968002c1b86e670db9b0c
[root@master01 tmp]#
```
> 检查kubelet.service 状态
```shell
[root@master01 ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2020-07-27 16:40:53 CST; 1min 25s ago
Docs: https://kubernetes.io/docs/
Main PID: 33413 (kubelet)
Tasks: 16
Memory: 74.7M
CGroup: /system.slice/kubelet.service
└─33413 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 --fail-swap-on=false
Jul 27 16:41:54 master01 kubelet[33413]: E0727 16:41:54.116034 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:41:55 master01 kubelet[33413]: W0727 16:41:55.025914 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:41:59 master01 kubelet[33413]: E0727 16:41:59.233911 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:00 master01 kubelet[33413]: W0727 16:42:00.028419 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:42:04 master01 kubelet[33413]: E0727 16:42:04.354686 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:05 master01 kubelet[33413]: W0727 16:42:05.029246 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:42:09 master01 kubelet[33413]: E0727 16:42:09.444976 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:10 master01 kubelet[33413]: W0727 16:42:10.048128 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:42:14 master01 kubelet[33413]: E0727 16:42:14.551861 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:15 master01 kubelet[33413]: W0727 16:42:15.049274 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
[root@master01 ~]#
## 部署Network Plugin
mkdir .kube
cp /etc/kubernetes/admin.conf .kube/config
```shell
[root@master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01 NotReady master 11m v1.18.2
# 添加一个node后
[root@master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos77minimal NotReady <none> 41s v1.18.2
master01 NotReady master 22m v1.18.2
```
node 添加ipvs.modules
node 导入k8s-image.tar
cp master:/etc/sysconfig/kubelet node
kubeadm reset
kubeadm token create --print-join-command
kubectl get pods -n kube-system
# 查看pod产生的日志
kubectl logs -f calico-node-5rgfg -n kube-system
kubectl describe pod calico-node-5rgfg -n kube-system 查看pod的具体描述
kubectl -n kube-system get ev 查看集群环境输出
## flannel
sysctl net.bridge.bridge-nf-call-iptables=1