Kubernetes的安全框架（总的来说就是各种形式访问API Server）
访问K8S集群的资源需要过三关：认证、鉴权、准入控制
普通用户若要安全访问集群API Server，往往需要证书、Token或者用户名+密码；Pod访问API Server，需要ServiceAccount
K8S安全控制框架主要由下面3个阶段进行控制，每一个阶段都支持插件方式，通过API Server配置来启用插件。
1.Authentication
2.Authorization
3.Admission Control
[root@master01 yaml_doc]# kubectl get sa NAME SECRETS AGE default 1 11d [root@master01 yaml_doc]# [root@master01 yaml_doc]# kubectl get secrets NAME TYPE DATA AGE default-token-sj2lw kubernetes.io/service-account-token 3 15d #token是用secret资源类型存储的 会生成一个跟ServiceAccount：pod-reader同名的secret （在同一个命名空间下） [root@master01 yaml_doc]# kubectl describe secret default-token-sj2lw Name: default-token-sj2lw Namespace: default Labels: <none> Annotations: kubernetes.io/service-account.name: default kubernetes.io/service-account.uid: 1c1e6043-0430-11ea-9860-1866dafb2f54 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1359 bytes namespace: 7 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tc2oybHciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFjMWU2MDQzLTA0MzAtMTFlYS05ODYwLTE4NjZkYWZiMmY1NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.b1CuohIrZi_ubpkoYQAQKzCELgWeGSbLAEiCTyN7mIZj1CVro4SkftoaTYisyJnhmMKw8PQyvDunKOLa-tkE1C-XeAVh_rcno80NXOD9Uz33CijUPNZlHNwx0CimjGS_AA5yHoUXmLxAsm-cOq-SkWaawtUfxnRAQVv2TKEjfmha9AaLPw6Gj4-DgBPPhV8pWpBNnuuwDR-tKyyWUTAIE8wGDDhIYyIgKgzo_q7MCOupQpYUiUzwu83B5DTmimHXVcbZ3ffkR1X198KIS3lUu_fR997Pn2ZoxK5mjZ1PmzK6I0gUhSfU7wI8_AUy67sKmRUEJ2ORcAVkk-KQ_PaXUw
1、Authentication #传输安全，认证 相当于进公司大门
传输安全：告别8080，迎接6443 [root@master01 yaml_doc]# netstat -anput | grep 8080 | grep LISTEN #不对外 tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 169560/kube-apiserv [root@master01 yaml_doc]# [root@master01 yaml_doc]# netstat -anput | grep 6443 | grep LISTEN tcp 0 0 10.192.27.100:6443 0.0.0.0:* LISTEN 169560/kube-apiserv [root@master01 yaml_doc]#
认证 三种客户端身份认证： •HTTPS 证书认证：基于CA证书签名的数字证书认证 [root@master01 yaml_doc]# ls /root/k8s/k8s-cert/ #不同的组件通过证书访问API Server admin.csr admin-key.pem ca-config.json ca-csr.json ca.pem kube-proxy.csr kube-proxy-key.pem server.csr server-key.pem admin-csr.json admin.pem ca.csr ca-key.pem k8s-cert.sh kube-proxy-csr.json kube-proxy.pem server-csr.json server.pem [root@master01 yaml_doc]# •HTTP Token认证：通过一个Token来识别用户 [root@master01 yaml_doc]# cat /opt/kubernetes/cfg/token.csv #kubelet-bootstrap认证使用 0fb61c46f8991b718eb38d27b605b008,kubelet-bootstrap,10001,"system:kubelet-bootstrap" [root@master01 yaml_doc]# •HTTP Base认证：用户名+密码的方式认证
2、Authorization #授权 相当于授权进入那个办公区
RBAC（Role-Based Access Control，基于角色的访问控制）：负责完成授权（Authorization）工作。
3、Admission Control #准入控制 相当于验证你的权限
AdminssionControl实际上是一个准入控制器插件列表，发送到APIServer的请求都需要经过这个列表中的每个准入控制器插件的检查，检查不通过，则拒绝请求。 1.11版本以上推荐使用的插件： --enable-admission-plugins= NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds, ResourceQuota [root@master01 yaml_doc]# ps -ef | grep apiserver root 85972 18645 0 10:36 pts/0 00:00:00 grep --color=auto apiserver root 169560 1 3 11月11 ? 10:17:24 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.192.27.100:2379,https://10.192.27.115:2379,https://10.192.27.116:2379 --bind-address=10.192.27.100 --secure-port=6443 --advertise-address=10.192.27.100 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 准入控制字段设置：--enable-admission-plugins=NamespaceLifecycle（命令控制）,LimitRanger（限制pod使用的资源CPU 内存）, ServiceAccount（会在每个pod中导入默认的serviceaccount，使pod有权限访问API server）,ResourceQuota（高级的配额管理）,NodeRestriction（node加入进来以最小的权限运行） --authorization-mode=RBAC,Node --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem [root@master01 yaml_doc]#
着重介绍第二个授权阶段
•角色
Role：授权特定命名空间的访问权限
ClusterRole：授权所有命名空间的访问权限 #集群层面
•角色绑定
RoleBinding：将角色绑定到主体（即subject）
ClusterRoleBinding：将集群角色绑定到主体
•主体（subject）
User：用户 #给使用人自己用的
Group：用户组
ServiceAccount：服务账号 #给程序用的
k8s实现多租户不同部门不同项目之间的隔离一般通过命名空间 [root@master01 yaml_doc]# kubectl get ns #默认命名空间 NAME STATUS AGE default Active 11d kube-public Active 11d kube-system Active 11d [root@master01 yaml_doc]# kubectl create ns ctnrs #创建命名空间 namespace/ctnrs created [root@master01 yaml_doc]# kubectl get ns NAME STATUS AGE ctnrs Active 2s default Active 11d kube-public Active 11d kube-system Active 11d [root@master01 yaml_doc]# 两个node节点配置一下 --pod-infra-container-image=10.192.27.111/library/pause-amd64:3.0 然后重启服务 [root@node01 cfg]# ps -ef | grep kubelet root 166978 1 9 11:10 ? 00:00:00 /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.192.27.115 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=10.192.27.111/library/pause-amd64:3.0 [root@master01 yaml_doc]# kubectl run nginx --image=10.192.27.111/library/nginx:latest -n ctnrs kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/nginx created [root@master01 yaml_doc]# kubectl get all -n ctnrs NAME READY STATUS RESTARTS AGE pod/nginx-644d56598-n4hnv 1/1 Running 0 3m19s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx 1/1 1 1 3m19s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-644d56598 1 1 1 3m19s [root@master01 yaml_doc]# [root@master01 yaml_doc]# kubectl scale deploy/nginx --replicas=3 -n ctnrs #调整副本数 deployment.extensions/nginx scaled [root@master01 yaml_doc]# kubectl get pods -n ctnrs NAME READY STATUS RESTARTS AGE nginx-644d56598-bqkg4 0/1 ContainerCreating 0 2s nginx-644d56598-dtcn9 0/1 ContainerCreating 0 2s nginx-644d56598-n4hnv 1/1 Running 0 4m47s [root@master01 yaml_doc]#
来一个例子吧
#创建一个role角色 [root@master01 yaml_doc]# vim rbac-role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: ctnrs name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] #只能操作pod资源 verbs: ["get", "watch", "list"] #只读 [root@master01 yaml_doc]# kubectl create -f rbac-role.yaml role.rbac.authorization.k8s.io/pod-reader created [root@master01 yaml_doc]# kubectl get role -n ctnrs NAME AGE pod-reader 16s [root@master01 yaml_doc]#
######################测试方式一 用户测试########################### [root@master01 yaml_doc]# vim rbac-rolebinding.yaml #绑定角色 kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-pods namespace: ctnrs subjects: #指定用户 - kind: User name: jane #用户名 apiGroup: rbac.authorization.k8s.io roleRef: #绑定哪个角色 kind: Role name: pod-reader #角色名 apiGroup: rbac.authorization.k8s.io [root@master01 yaml_doc]# kubectl create -f rbac-rolebinding.yaml rolebinding.rbac.authorization.k8s.io/read-pods created [root@master01 yaml_doc]# kubectl get role,rolebinding -n ctnrs NAME AGE role.rbac.authorization.k8s.io/pod-reader 9m1s NAME AGE rolebinding.rbac.authorization.k8s.io/read-pods 2m19s [root@master01 yaml_doc]# #生成证书 以上用户如何识别身份 [root@master01 yaml_doc]# mkdir jane [root@master01 yaml_doc]# cd jane/ [root@master01 jane]# vim rabc-user.sh cat > jane-csr.json <<EOF { "CN": "jane", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes jane-csr.json | cfssljson -bare jane #设置集群IP 、证书、上下文等配置 生成用户jane的访问集群APIserver的配置文件 kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.192.27.117:6443 --kubeconfig=jane-kubeconfig kubectl config set-credentials jane --client-key=jane-key.pem --client-certificate=jane.pem --embed-certs=true --kubeconfig=jane-kubeconfig kubectl config set-context default --cluster=kubernetes --user=jane --kubeconfig=jane-kubeconfig #使用默认上下文 kubectl config use-context default --kubeconfig=jane-kubeconfig [root@master01 jane]# cp /root/k8s/k8s-cert/ca* . #用到根证书（kube-apiserver是的证书） 生成用户客户端证书 [root@master01 jane]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem rabc-user.sh [root@master01 jane]# bash rabc-user.sh 2019/09/17 20:54:24 [INFO] generate received request 2019/09/17 20:54:24 [INFO] received CSR 2019/09/17 20:54:24 [INFO] generating key: rsa-2048 2019/09/17 20:54:25 [INFO] encoded CSR 2019/09/17 20:54:25 [INFO] signed certificate with serial number 30365589649039563082592558185944057795826217362 2019/09/17 20:54:25 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). Cluster "kubernetes" set. User "jane" set. Context "default" created. Switched to context "default". [root@master01 jane]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem jane.csr jane-csr.json jane-key.pem jane-kubeconfig jane.pem rabc-user.sh [root@master01 jane]# [root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get pods -n ctnrs #根据上面的权限（只有ctnrs命令空间的pod只读权限） 查看指定的内容 NAME READY STATUS RESTARTS AGE nginx-7cdbd8cdc9-7rc5x 1/1 Running 0 29m nginx-7cdbd8cdc9-l4zlj 1/1 Running 0 27m nginx-7cdbd8cdc9-s8ggw 1/1 Running 0 27m [root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get svc -n ctnrs Error from server (Forbidden): services is forbidden: User "jane" cannot list resource "services" in API group "" in the namespace "ctnrs" [root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get svc #default空间没有权限 Error from server (Forbidden): services is forbidden: User "jane" cannot list resource "services" in API group "" in the namespace "default" [root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get pods #default空间没有权限 Error from server (Forbidden): pods is forbidden: User "jane" cannot list resource "pods" in API group "" in the namespace "default" [root@master01 jane]#
#########################测试方式二 ServiceAccount：服务账号测试 #给程序用的 ########### #使用web UI https://10.192.27.115:30001/ 或者116 [root@master01 jane]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP 4d11h kubernetes-dashboard NodePort 10.0.0.141 <none> 443:30001/TCP 13d [root@master01 jane]# kubectl get secret -n kube-system NAME TYPE DATA AGE coredns-token-bkjq9 kubernetes.io/service-account-token 3 4d11h dashboard-admin-token-x2fmd kubernetes.io/service-account-token 3 13d #超级管理员 default-token-gkwdg kubernetes.io/service-account-token 3 14d kubernetes-dashboard-certs Opaque 11 13d kubernetes-dashboard-key-holder Opaque 2 13d kubernetes-dashboard-token-2qxt6 kubernetes.io/service-account-token 3 13d [root@master01 jane]# vim sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: pod-reader #创建sa对应下 这个可以改 namespace: ctnrs --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sa-read-pods namespace: ctnrs subjects: - kind: ServiceAccount name: pod-reader #对应上SA名字 这个可以改 roleRef: kind: Role name: pod-reader #上面创建的的角色 apiGroup: rbac.authorization.k8s.io [root@master01 jane]# kubectl apply -f sa.yaml serviceaccount/pod-reader created rolebinding.rbac.authorization.k8s.io/sa-read-pods created [root@master01 jane]# kubectl get sa -n ctnrs NAME SECRETS AGE default 1 42m pod-reader 1 23s [root@master01 jane]# [root@master01 jane]# kubectl describe sa pod-reader -n ctnrs Name: pod-reader Namespace: ctnrs Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"pod-reader","namespace":"ctnrs"}} Image pull secrets: <none> Mountable secrets: pod-reader-token-5vmrr Tokens: pod-reader-token-5vmrr Events: <none> [root@master01 jane]# kubectl describe secret pod-reader -n ctnrs #token是用secret资源类型存储的 会生成一个跟ServiceAccount：pod-reader同名的secret （在同一个命名空间下） Name: pod-reader-token-5vmrr Namespace: ctnrs Labels: <none> Annotations: kubernetes.io/service-account.name: pod-reader kubernetes.io/service-account.uid: 7b1a812f-d94c-11e9-8a2e-000c29586be2 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1359 bytes namespace: 5 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjdG5ycyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwb2QtcmVhZGVyLXRva2VuLTV2bXJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InBvZC1yZWFkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3YjFhODEyZi1kOTRjLTExZTktOGEyZS0wMDBjMjk1ODZiZTIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Y3RucnM6cG9kLXJlYWRlciJ9.JyiOaiL79AVAs59QRQY7fyPn2Ukc3b52NejeuUZK5JlHJN7rE6mLqZnJtEm2ef5Yp07rVpf0IAqWzEOcPRC6uKzYKT544BxQxoEHOP1GNaWFDh6JHIeMHP1Ak-4FqmI9q7a7YyCYrvR7RtMuTdUr8G0Fo2gyNzlIkp1UlJH1-LpoAR0BG5MBtFzj4trxlGSIzVHtAHoIXXcG5Ke0mEHWEt0zWEFUJ5r-y-X99HDR3n5yER43djywjpncxzJ426Rm-R3azjpHUZ3LP4urkIwSWocIxuVcGAay5EgX9sHcZb5a-1fT3IZPX77ATK9LDcZfG0UoPYgDjbUPw4BqLq9HGg [root@master01 jane]#
使用web UI https://10.192.27.115:30001/ 或者116
