Kubernetes集群添加节点

Kubernetes Cluster add ndoe

加入过程

kubeadm 从 API 服务器下载需要的集群信息。默认情况下，使用引导token和 CA 密钥哈希来验证数据的真实性。也可以通过文件或 URL 直接发现根 CA
拿到集群验证信息后，kubelet进入TLS bootstrapping 过程TLS bootstrap 使用共享token临时向Kubernetes API服务器进行身份验证以提交证书签名请求（CSR）；默认情况下control-pane会自动签署此CSR请求
最后，kubeadm 配置本地 kubelet 使用分配给节点的确定标识连接到 API 服务器

How to do

默认情况下token保留时间是24小时，如果超出该时间，token将会自动删除，则需要手动创建token及hash值

创建token

<root@HK-K8S-CP ~># kubeadm token create
W0815 14:54:38.564119   10867 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
hnno8a.rnijnbrhejz72t7w

生成token的hash值

<root@HK-K8S-CP ~># openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | 
      openssl dgst -sha256 -hex | sed 's/^.* //'
bcf878114948a608e3f47f2a3824bae94b1f3f9ce9bd529d21a19f0d8af4c6cb

获取control-plane-host:port，在加入集群之前，前提条件需要获取Kubernetes集群的control-plane-host:port，具体查看方法如下

<root@HK-K8S-CP ~># kubectl describe configmaps -n kube-system  kubeadm-config
Name:         kubeadm-config
Namespace:    kube-system
Labels:       <none>
Annotations:  
Data
====
ClusterConfiguration:
----
apiServer:
  certSANs:
  - 47.57.234.123
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 172.19.0.203:6443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.18.5
networking:
  dnsDomain: nflow.so
  podSubnet: 172.20.0.0/20
  serviceSubnet: 10.10.0.0/24
scheduler: {}

ClusterStatus:
----
apiEndpoints:
  hk-k8s-cp:
    advertiseAddress: 172.19.1.119
    bindPort: 6443
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterStatus

Events:  <none>

加入集群（注意下面的Contraol Plane的地址与上文获取的不同，这是由于阿里云SLB自身问题，如果在创建的集群的时候，将CP节点作为SLB的后端服务器，需要创建一个公网地址并且指定certSANs参数即可绕过，但是如果只是添加节点到集群中，即可使用CP节点的内网地址）

<root@HK-K8S-WN4 ~># kubeadm join 172.19.1.119:6443 --token hnno8a.rnijnbrhejz72t7w --discovery-token-ca-cert-hash sha256:bcf878114948a608e3f47f2a3824bae94b1f3f9ce9bd529d21a19f0d8af4c6cb --v=5
W0815 15:03:45.687465    3589 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
I0815 15:03:45.687523    3589 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName
I0815 15:03:45.687568    3589 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I0815 15:03:45.687648    3589 preflight.go:90] [preflight] Running general checks
I0815 15:03:45.687704    3589 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0815 15:03:45.687757    3589 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf
I0815 15:03:45.687773    3589 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0815 15:03:45.687783    3589 checks.go:102] validating the container runtime
I0815 15:03:45.770908    3589 checks.go:128] validating if the service is enabled and active
I0815 15:03:45.862041    3589 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0815 15:03:45.862110    3589 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0815 15:03:45.862136    3589 checks.go:649] validating whether swap is enabled or not
I0815 15:03:45.862166    3589 checks.go:376] validating the presence of executable conntrack
I0815 15:03:45.862191    3589 checks.go:376] validating the presence of executable ip
I0815 15:03:45.862217    3589 checks.go:376] validating the presence of executable iptables
I0815 15:03:45.862237    3589 checks.go:376] validating the presence of executable mount
I0815 15:03:45.862260    3589 checks.go:376] validating the presence of executable nsenter
I0815 15:03:45.862280    3589 checks.go:376] validating the presence of executable ebtables
I0815 15:03:45.862298    3589 checks.go:376] validating the presence of executable ethtool
I0815 15:03:45.862318    3589 checks.go:376] validating the presence of executable socat
I0815 15:03:45.862337    3589 checks.go:376] validating the presence of executable tc
I0815 15:03:45.862361    3589 checks.go:376] validating the presence of executable touch
I0815 15:03:45.862381    3589 checks.go:520] running all checks
I0815 15:03:45.955690    3589 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I0815 15:03:45.955912    3589 checks.go:618] validating kubelet version
I0815 15:03:46.009037    3589 checks.go:128] validating if the service is enabled and active
I0815 15:03:46.017134    3589 checks.go:201] validating availability of port 10250
I0815 15:03:46.017322    3589 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt
I0815 15:03:46.017334    3589 checks.go:432] validating if the connectivity type is via proxy or direct
I0815 15:03:46.017366    3589 join.go:441] [preflight] Discovering cluster-info
I0815 15:03:46.017388    3589 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "172.19.1.119:6443"
I0815 15:03:46.025145    3589 token.go:116] [discovery] Requesting info from "172.19.1.119:6443" again to validate TLS against the pinned public key
I0815 15:03:46.031503    3589 token.go:133] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.19.1.119:6443"
I0815 15:03:46.031521    3589 discovery.go:51] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I0815 15:03:46.031535    3589 join.go:455] [preflight] Fetching init configuration
I0815 15:03:46.031541    3589 join.go:493] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W0815 15:03:46.361210    3589 configset.go:76] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" is forbidden: User "system:bootstrap:hnno8a" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
I0815 15:03:46.364438    3589 interface.go:400] Looking for default routes with IPv4 addresses
I0815 15:03:46.364456    3589 interface.go:405] Default route transits interface "eth0"
I0815 15:03:46.364566    3589 interface.go:208] Interface eth0 is up
I0815 15:03:46.364624    3589 interface.go:256] Interface "eth0" has 1 addresses :[172.19.1.147/24].
I0815 15:03:46.364645    3589 interface.go:223] Checking addr  172.19.1.147/24.
I0815 15:03:46.364656    3589 interface.go:230] IP found 172.19.1.147
I0815 15:03:46.364671    3589 interface.go:262] Found valid IPv4 address 172.19.1.147 for interface "eth0".
I0815 15:03:46.364683    3589 interface.go:411] Found active IP 172.19.1.147 
I0815 15:03:46.364730    3589 preflight.go:101] [preflight] Running configuration dependant checks
I0815 15:03:46.364744    3589 controlplaneprepare.go:211] [download-certs] Skipping certs download
I0815 15:03:46.364763    3589 kubelet.go:111] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I0815 15:03:46.365705    3589 kubelet.go:119] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I0815 15:03:46.367273    3589 kubelet.go:145] [kubelet-start] Checking for an existing Node in the cluster with name "hk-k8s-wn4" and status "Ready"
I0815 15:03:46.368707    3589 kubelet.go:159] [kubelet-start] Stopping the kubelet
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I0815 15:03:51.547974    3589 cert_rotation.go:137] Starting client certificate rotation controller
I0815 15:03:51.552665    3589 kubelet.go:194] [kubelet-start] preserving the crisocket information for the node
I0815 15:03:51.552684    3589 patchnode.go:30] [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "hk-k8s-wn4" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

切换至CP节点查看节点是否加入成功

<root@HK-K8S-CP ~># kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
hk-k8s-cp    Ready    master   152d    v1.18.5
hk-k8s-wn1   Ready    worker   152d    v1.18.5
hk-k8s-wn2   Ready    worker   150d    v1.18.5
hk-k8s-wn3   Ready    worker   150d    v1.18.5
hk-k8s-wn4   Ready    <none>   6m27s   v1.18.5

修改加入节点的角色，如下

<root@HK-K8S-CP ~># kubectl label node hk-k8s-wn4  node-role.kubernetes.io/worker=worker
node/hk-k8s-wn4 labeled
<root@HK-K8S-CP ~># kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
hk-k8s-cp    Ready    master   152d    v1.18.5
hk-k8s-wn1   Ready    worker   152d    v1.18.5
hk-k8s-wn2   Ready    worker   150d    v1.18.5
hk-k8s-wn3   Ready    worker   150d    v1.18.5
hk-k8s-wn4   Ready    worker   7m48s   v1.18.5