https://www.kubernetes.org.cn/doc-16
https://blog.csdn.net/zjysource/article/details/52086835
https://blog.csdn.net/wucong60/article/details/81911859
https://www.kubernetes.org.cn/tags/kubeadm
https://www.katacoda.com/courses/kubernetes
Kubernetes包提供了一些服务：
kube-apiserver,
kube-scheduler,
kube-controller-manager,
kubelet,
kube-proxy。
这些服务通过systemd进行管理，配置信息都集中存放在一个地方：/etc/kubernetes。
我们将会把这些服务运行到不同的主机上。第一台主机，centosmaster，将是Kubernetes 集群的master主机。这台机器上将运行kube-apiserver, kubecontroller-manager和kube-scheduler这几个服务，此外，master主机上还将运行etcd。其余的主机，fed-minion，将是从节点，将会运行kubelet, proxy和docker。
systemctl stop firewalld systemctl disable firewalld yum -y install ntp systemctl start ntpd systemctl enable ntpd
配置master
yum -y install etcd docker kubernetes
对etcd进行配置，编辑/etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD存储目录 ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #修改项，表示etcd在2379端口上监听所有网络接口。 ETCD_NAME="default" #ETCD名称 ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379" #对外提供服务的地址
对Master节点上的Kubernetes进行配置，编辑配置文件/etc/kubernetes/config
KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=0" KUBE_ALLOW_PRIV="--allow-privileged=false" KUBE_MASTER="--master=http://192.168.0.201:8080" #修改为master的地址。含义：将apiserver进程的服务地址告诉controller-manager, scheduler和proxy进程。
/etc/kubernetes/apiserver
这些配置让apiserver进程在8080端口上监听所有网络接口，并告诉apiserver进程etcd服务的地址。
KUBE_API_ADDRESS="--address=0.0.0.0" #修改项 KUBE_API_PORT="--port=8080" #添加项 KUBELET_PORT="--kubelet-port=10250" KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379" KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota" KUBE_API_ARGS=""
这台机器的linux的内核中的SELinux不支持 overlay2 graph driver 。
解决方法有两个，要么启动一个新内核，要么就在docker配置文件里面里禁用selinux，--selinux-enabled=false
将“--selinux-enabled”改成“--selinux-enabled=false”
[root@registry lib]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
# Modify these options if you want to change the way the docker daemon runs
#OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
OPTIONS='--selinux-enabled=false --log-driver=journald --signature-verification=false --registry-mirror=https://fzhifedh.mirror.aliyuncs.com --insecure-registry=registry.sese.com' #修改这里的"--selinux-enabled"，改成"--selinux-enabled=false"
if [ -z "${DOCKER_CERT_PATH}" ]; then
DOCKER_CERT_PATH=/etc/docker
fi
启动服务：
for SERVICES in etcd docker kube-apiserver kube-controller-manager kube-scheduler; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
现在我们可以使用kubectl get nodes命令来查看，当然，目前还没有Node节点加入到该Kubernetes集群，所以命令的执行结果是空的：
# kubectl get nodes NAME STATUS AGE
etcdctl mk /atomic.io/network/config '{"Network":"172.17.0.0/16"}'
#节点上的flannel用master上的etcd里的/atomic.io/network/config来创建网络的表
配置节点
yum -y install flannel docker kubernetes
/etc/sysconfig/flanneld #对flannel进行配置
FLANNEL_ETCD_ENDPOINTS="http://192.168.0.201:2379" #修改成master的地址，告诉flannel进程etcd服务的位置以及在etcd上网络配置信息的节点位置 FLANNEL_ETCD_PREFIX="/atomic.io/network"
/etc/kubernetes/config：
KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=0" KUBE_ALLOW_PRIV="--allow-privileged=false" KUBE_MASTER="--master=http://192.168.0.201:8080" #修改成master的地址，将Kubernetes的apiserver进程的服务地址告诉Kubernetes的controller-manager, scheduler和proxy进程。
/etc/kubernetes/kubelet
KUBELET_ADDRESS="--address=0.0.0.0" KUBELET_PORT="--port=10250" KUBELET_HOSTNAME="--hostname-override=192.168.0.200" #节点服务器的IP KUBELET_API_SERVER="--api-servers=http://192.168.0.201:8080" #master服务器的IP KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest" KUBELET_ARGS=""
节点上启动kube-proxy kubelet docker和flanneld进程并查看其状态
for SERVICES in kube-proxy kubelet docker flanneld; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICES done
Master节点上使用kubectl get nodes命令来查看，可以看到加入的Node节点
[root@localhost ~]# kubectl get nodes NAME STATUS AGE 192.168.0.200 Ready 1h
搭建私有仓库：
vim /etc/pki/tls/openssl.cnf
[ v3_ca ] subjectAltName = IP:192.168.169.125 #增加一行
cd /etc/pki/tls
[root@localhost tls]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt
Generating a 4096 bit RSA private key ..................................................................................................................++ writing new private key to 'certs/domain.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:86 #国家名（2位代码） State or Province Name (full name) []:jiangsu~^H^Hxvzhou #省/市/自治区名称（全名） Locality Name (eg, city) [Default City]:xv^Hzhou #地点名称（例如，城市） Organization Name (eg, company) [Default Company Ltd]:ni^Hhao #组织名称（例如，公司） Organizational Unit Name (eg, section) []:jishu #组织单位名称（例如，部门） Common Name (eg, your name or your server's hostname) []:hostname #公用名（例如，您的名称或服务器的主机名） Email Address []:123@qq.com #邮箱地址
证书创建完毕后，在certs目录下出现了两个文件：证书文件domain.crt和私钥文件domain.key
在192.168.169.125上安装docker
yum -y install docker
将前面生成的domain.crt文件复制到/etc/docker/certs.d/192.168.169.125:5000目录下，然后重启docker进程：
mkdir -p /etc/docker/certs.d/192.168.0.205:5000 cp certs/domain.crt /etc/docker/certs.d/192.168.0.205:5000/ca.crt systemctl restart docker
[root@localhost tls]# docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2
Unable to find image 'registry:2' locally Trying to pull repository docker.io/library/registry ... 2: Pulling from docker.io/library/registry c87736221ed0: Pull complete 1cc8e0bb44df: Pull complete 54d33bcb37f5: Pull complete e8afc091c171: Pull complete b4541f6d3db6: Pull complete Digest: sha256:3b00e5438ebd8835bcfa7bf5246445a6b57b9a50473e89c02ecc8e575be3ebb5 Status: Downloaded newer image for docker.io/registry:2 /usr/bin/docker-current: Error response from daemon: Invalid container name (registry ), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed. See '/usr/bin/docker-current run --help'.
最后，将domain.crt文件复制到Kubernetes集群里的所有节点的/etc/docker/certs.d/192.168.169.125:5000目录下，并重启各节点的docker进程，例如在192.168.169.121节点上运行：
mkdir -p /etc/docker/certs.d/192.168.169.125:5000 scp root@192.168.0.205:~/certs/domain.crt /etc/docker/certs.d/192.168.0.205:5000/ca.crt systemctl restart docker
容器的简单命令：
1. 先查看所有的容器 # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e3274a72e8d6 tomcat "catalina.sh run" 2 weeks ago Exited (130) 19 minutes ago tomcat8080 看到了这个名为 “tomcat8080” 的容器，并且这个容器是非运行（Exited）状态。 注：“docker ps” 是查看当前运行的容器，“docker ps -a” 是查看所有容器（包括停止的）。 2. 移除这个“tomcat8080”容器 # docker rm e3274a72e8d6 e3274a72e8d6 3. 然后再创建新容器 # docker run --name tomcat8080 -d -p 8080:8080 tomcat 新容器创建成功，并且是运行状态： # docker ps -a
Kubernetes Web UI搭建
这节我以搭建Kubernetes Web UI（kubernetes-dashboard）来简要演示如何使用Docker私有库。
由于我的Kubernetes集群无法直接从gcr.io拉取kubernetes-dashboard的镜像，我事先下载了镜像文件并使用docker load命令加载镜像：
# docker load < kubernetes-dashboard-amd64_v1.1.0.tar.gz
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry 2 c6c14b3960bd 3 days ago 33.28 MB
ubuntu latest 42118e3df429 9 days ago 124.8 MB
hello-world latest c54a2cc56cbb 4 weeks ago 1.848 kB
172.28.80.11:5000/kubernetes-dashboard-amd64 v1.1.0 20b7531358be 5 weeks ago 58.52 MB
registry 2 8ff6a4aae657 7 weeks ago 171.5 MB
我为加载的kubernetes-dashboard镜像打上私有库的标签并推送到私有库：
# docker tag 20b7531358be 192.168.169.125:5000/kubernetes-dashboard-amd64
# docker push 192.168.169.125:5000/kubernetes-dashboard-amd64
从Kubernetes官网获取了kubernetes-dashboard的配置文件https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml，对其进行编辑如下：
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configuration to deploy release version of the Dashboard UI.
#
# Example usage: kubectl create -f <this_file>
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
labels:
app: kubernetes-dashboard
version: v1.1.0
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: kubernetes-dashboard
template:
metadata:
labels:
app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: 192.168.169.125:5000/kubernetes-dashboard-amd64
imagePullPolicy: Always
ports:
- containerPort: 9090
protocol: TCP
args:
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
- --apiserver-host=192.168.169.120:8080
livenessProbe:
httpGet:
path: /
port: 9090
initialDelaySeconds: 30
timeoutSeconds: 30
---
kind: Service
apiVersion: v1
metadata:
labels:
app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 80
targetPort: 9090
selector:
app: kubernetes-dashboard
尤其要注意：1 创建的Pods所要拉取的镜像是Docker私有库的192.168.169.125：5000/kubernetes-dashboard-adm64； 2 apiserver-host参数是192.168.169.120:8080，即Kubernetes Master节点的aipserver服务地址。
修改完kubernetes-dashboard.yaml后保存到Kubernetes Master节点192.168.169.120节点上，在该节点上用kubectl create命令创建kubernetes-dashboard：
# kubectl create -f kubernetes-dashboard.yaml
创建完成后，查看Pods和Service的详细信息：
# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx 1/1 Running 0 3h
kube-system kubernetes-dashboard-4164430742-lqhcg 1/1 Running 0 2h
# kubectl describe pods/kubernetes-dashboard-4164430742-lqhcg --namespace="kube-system"
Name: kubernetes-dashboard-4164430742-lqhcg
Namespace: kube-system
Node: 192.168.169.124/192.168.169.124
Start Time: Mon, 01 Aug 2016 16:12:02 +0800
Labels: app=kubernetes-dashboard,pod-template-hash=4164430742
Status: Running
IP: 172.17.17.3
Controllers: ReplicaSet/kubernetes-dashboard-4164430742
Containers:
kubernetes-dashboard:
Container ID: docker://40ab377c5b8a333487f251547e5de51af63570c31f9ba05fe3030a02cbb3660c
Image: 192.168.169.125:5000/kubernetes-dashboard-amd64
Image ID: docker://sha256:20b7531358be693a34eafdedee2954f381a95db469457667afd4ceeb7146cd1f
Port: 9090/TCP
Args:
--apiserver-host=192.168.169.120:8080
QoS Tier:
cpu: BestEffort
memory: BestEffort
State: Running
Started: Mon, 01 Aug 2016 16:12:03 +0800
Ready: True
Restart Count: 0
Liveness: http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready True
No volumes.
No events.
# kubectl describe service/kubernetes-dashboard --namespace="kube-system"
Name: kubernetes-dashboard
Namespace: kube-system
Labels: app=kubernetes-dashboard
Selector: app=kubernetes-dashboard
Type: NodePort
IP: 10.254.213.209
Port: <unset> 80/TCP
NodePort: <unset> 31482/TCP
Endpoints: 172.17.17.3:9090
Session Affinity: None
No events.
7.发布nginx服务
7.1 创建pod : nginx-pod.yaml
kubectl create -f nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
labels:
name: nginx-pod
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
7.2 查看pod的状态
[root@localhost ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-pod 0/1 ContainerCreating 0 2h
等10分钟再试
NAME READY STATUS RESTARTS AGE nginx-pod 1/1 Running 0 13m
PS：这里经常会因为网络问题失败，可以先使用docker手动pull镜像后再使用kubectl来create pod，如果还是不行，就delete pod之后再create pod，实在不行，可以重启机器试试，还是不行，那就是配置出问题了
7.3 创建replicationController : nginx-rc.yaml
kubectl create -f nginx-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-rc
spec:
replicas: 1
selector:
name: nginx-pod
template:
metadata:
labels:
name: nginx-pod
spec:
containers:
- name: nginx-pod
image: nginx
ports:
- containerPort: 80
7.4 查看ReplicationController状况
[root@localhost ~]# kubectl get rc NAME DESIRED CURRENT READY AGE nginx-rc 1 1 0 1h
7.5 创建service : nginx-service.yaml
kubectl create -f nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: NodePort
ports:
- port: 80
nodePort: 30001
selector:
name: nginx-pod
7.6 查看service状态
PS:其中Kubernetes服务为Kube系统自带服务，无需理会
7.7 测试发布的nginx服务
使用其他机器的浏览器访问node1机器的30001端口
[root@localhost ~]# kubectl describe svc nginx Name: nginx-service Namespace: default Labels: <none> Selector: name=nginx-pod Type: NodePort IP: 10.254.98.243 Port: <unset> 80/TCP NodePort: <unset> 30001/TCP Endpoints: <none> Session Affinity: None No events. [root@localhost ~]# kubectl describe svc nginx-service Name: nginx-service Namespace: default Labels: <none> Selector: name=nginx-pod Type: NodePort IP: 10.254.98.243 Port: <unset> 80/TCP NodePort: <unset> 30001/TCP Endpoints: <none> Session Affinity: None No events.
[root@localhost build]# sh postinstall.sh
postinstall.sh: line 19: cd: ./node_modules/wiredep: No such file or directory
postinstall.sh: line 20: ../../build/patch/wiredep/wiredep.patch: No such file or directory
postinstall.sh: line 21: cd: lib: No such file or directory
postinstall.sh: line 22: ../../../build/patch/wiredep/detect-dependencies.patch: No such file or directory
postinstall.sh: line 29: go: command not found
postinstall.sh: line 33: cd: ./.tools/: No such file or directory
Cloning into 'xtbgenerator'...
remote: Enumerating objects: 229, done.
remote: Total 229 (delta 0), reused 0 (delta 0), pack-reused 229
Receiving objects: 100% (229/229), 27.78 MiB | 45.00 KiB/s, done.
Resolving deltas: 100% (73/73), done.
Note: checking out 'd6a6c9ed0833f461508351a80bc36854bc5509b2'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at d6a6c9e... fix empty --js param, recompile bin/XtbGenerator.jar
[root@localhost build]# sh run-gulp-in-docker.sh
ERRO[0000] Can't add file /dashboard-1.8.3/build/xtbgenerator/.git/objects/pack/tmp_pack_eZzn8x to tar: archive/tar: write too long
Sending build context to Docker daemon 43.7 MB
Step 1/6 : FROM golang
Trying to pull repository docker.io/library/golang ...
latest: Pulling from docker.io/library/golang
e79bb959ec00: Already exists
d4b7902036fe: Already exists
1b2a72d4e030: Already exists
d54db43011fd: Pull complete
963c818ebafc: Pull complete
2c6333e9b74a: Pull complete
3b0c71504fac: Pull complete
Digest: sha256:62538d25400afa368551fdeebbeed63f37a388327037438199cdf60b7f465639
Status: Downloaded newer image for docker.io/golang:latest
---> 213fe73a3852
Step 2/6 : RUN curl -sL https://deb.nodesource.com/setup_9.x | bash - && apt-get install -y --no-install-recommends openjdk-8-jre nodejs patch && rm -rf /var/lib/apt/lists/* && apt-get clean
---> Running in ef39b4003488
nsenter: could not ensure we are a cloned binary: Invalid argument
container_linux.go:247: starting container process caused "write parent: broken pipe"
oci runtime error: container_linux.go:247: starting container process caused "write parent: broken pipe"
Unable to find image 'kubernetes-dashboard-build-image:latest' locally
Trying to pull repository docker.io/library/kubernetes-dashboard-build-image ...
/usr/bin/docker-current: repository docker.io/kubernetes-dashboard-build-image not found: does not exist or no pull access.
See '/usr/bin/docker-current run --help'.