StatefulSet 使用NFS配置动态持久化存储

https://www.cnblogs.com/kevingrace/p/13969995.html

StatefulSet是为了解决有状态服务的容器问题而设计的，对应的Deployment和ReplicaSet是为了无状态服务而设计的。

StatefulSet应用场景包括：

稳定的持久化存储，即Pod重新调度后还是能访问到相同的持久化数据，基于PVC来实现
稳定的网络标志，即Pod重新调度后其PodName和HostName不变，基于Headless Service（即没有Cluster IP的Service）来实现
有序部署，有序扩展，即Pod是有顺序的。在部署或者扩展的时候要依据定义的顺序依次依次进行（即从0到N-1，在下一个Pod运行之前所有之前的Pod必须都是Running和Ready状态），基于init containers来实现
有序收缩，有序删除（即从N-1到0）
StatefulSet要求Pod的名称是有顺序的，每一个Pod都不能被随意取代，即使Pod重建之后，名称依然不变。

StatefulSet由以下几部分组成：

用于定义网络标志的Headless Service（headless-svc：无头服务。因为没有IP地址，所以它不具备负载均衡的功能了）
用于创建PersistentVolumes的volumeClaimTemplates。
定义具体应用的StatefulSet。

StatefulSet配置NFS动态持久化存储的流程：

创建 NFS 服务器。
创建 Service Account。用来管控 NFS provisioner 在k8s集群中运行的权限。
创建 StorageClass。负责创建 PVC 并调用 NFS provisioner 进行预定的工作，并关联 PV 和 PVC。
创建 NFS provisioner。有两个功能,一个是在NFS共享目录下创建挂载点(volume)；二是建立 PV 并将 PV 与 NFS 挂载点建立关联。

StatefulSet有状态应用pod访问地址
pod_name.service_name.namespace_name.svc.cluster.local:pod_app_port

一、NFS部署
可以参考：NFS双机热备高可用环境
NFS服务器为：172.16.60.194
NFS挂载目录为：/data/storage/mobile-decision-server_data

NFS的配置：

[root@k8s-storage01 ~]# cat /etc/exports
/data/storage 172.16.60.197(rw,sync,no_root_squash)
/data/storage 172.16.60.193(rw,sync,no_root_squash)
/data/storage 172.16.60.48(rw,sync,no_root_squash)
/data/storage 172.16.60.120(rw,sync,no_root_squash)

二、StatefulSet使用NFS配置动态持久化存储
下面是使用NFS作为StatefulSet持久化存储的操作记录，分别需要创建nfs-provisioner的rbac、storageclass、nfs-client-provisioner和statefulset的pod。

1）创建nfs-rbac.yaml

[root@k8s-master01 mobile-decision-server]# cat nfs-rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
  namespace: kevin
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: nfs-provisioner-runner
   namespace: kevin
rules:
   -  apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "delete"]
   -  apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
   -  apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses"]
      verbs: ["get", "list", "watch"]
   -  apiGroups: [""]
      resources: ["events"]
      verbs: ["watch", "create", "update", "patch"]
   -  apiGroups: [""]
      resources: ["services", "endpoints"]
      verbs: ["get","create","list", "watch","update"]
   -  apiGroups: ["extensions"]
      resources: ["podsecuritypolicies"]
      resourceNames: ["nfs-provisioner"]
      verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: kevin
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

创建并查看

[root@k8s-master01 mobile-decision-server]# kubectl apply -f nfs-rbac.yaml
  
[root@k8s-master01 mobile-decision-server]# kubectl get sa -n kevin|grep nfs
nfs-provisioner                1         7h7m
[root@k8s-master01 mobile-decision-server]# kubectl get clusterrole -n kevin|grep nfs
nfs-provisioner-runner                                                 7h7m
[root@k8s-master01 mobile-decision-server]# kubectl get clusterrolebinding -n kevin|grep nfs
run-nfs-provisioner                                    7h7m

2）创建nfs-class.yaml

[root@k8s-master01 mobile-decision-server]# cat nfs-class.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: managed-nfs-storage
  namespace: kevin
provisioner: mobile-decision-server/nfs
reclaimPolicy: Retain

创建

[root@k8s-master01 mobile-decision-server]# kubectl apply -f nfs-class.yaml

3）创建mobile-decision-server-nfs.yml
PROVISIONER_NAME的值一定要和StorageClass中的provisioner相等！
/persistentvolumes的mountPath 挂载点不能修改！

[root@k8s-master01 mobile-decision-server]# cat mobile-decision-server-nfs.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  namespace: kevin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath:  /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: mobile-decision-server/nfs
            - name: NFS_SERVER
              value: 172.16.60.194
            - name: NFS_PATH
              value: /data/storage/mobile-decision-server_data
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.16.60.194
            path: /data/storage/mobile-decision-server_data

创建并查看

[root@k8s-master01 mobile-decision-server]# kubectl apply -f mobile-decision-server-nfs.yml
  
[root@k8s-master01 mobile-decision-server]# kubectl get pods -n kevin|grep nfs
nfs-client-provisioner-6d78cd7874-krpvj   1/1     Running   0          5h3m

4）创建statefulset

[root@k8s-master01 mobile-decision-server]# cat mobile-decision-server.yml
apiVersion: v1
kind: Service
metadata:
  name: mobile-decision-server
  namespace: kevin
  labels:
    app: mobile-decision-server
spec:
  clusterIP: None
  selector:
    app: mobile-decision-server
  ports:
    - port: 8080
      name: server
      targetPort: 8080
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mobile-decision-server
  namespace: kevin
spec:
  serviceName: mobile-decision-server
  replicas: 2
  selector:
    matchLabels:
      app: mobile-decision-server
  template:
    metadata:
      labels:
        app: mobile-decision-server
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodetype
                operator: In
                values:
                - mobile-decision-server
      serviceAccount: nfs-provisioner
      containers:
      - name: mobile-decision-server
        image: 172.16.60.196/finhub/mobile-decision-server__20
        imagePullPolicy: Always
        ports:
        - name: dserverport
          containerPort: 8080
        resources:
          requests:
            cpu: 200m
            memory: 600Mi
          limits:
            cpu: 200m
            memory: 1024Mi
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
          preStop:
            exec:
              command: ["/bin/sh","-c","kill 1"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: dserverport
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
        volumeMounts:
        - name: data
          mountPath: /onestop/app
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: managed-nfs-storage
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 20Gi

创建并查看

[root@k8s-master01 mobile-decision-server]# kubectl apply -f mobile-decision-server.yml
  
[root@k8s-master01 mobile-decision-server]# kubectl get pods -n kevin|grep mobile-decision-server
mobile-decision-server-0                  1/1     Running   0          5h4m
mobile-decision-server-1                  1/1     Running   0          5h4m

查看pv和pvc

[root@k8s-master01 mobile-decision-server]# kubectl get pvc -n kevin
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
data-mobile-decision-server-0   Bound    pvc-9433d725-bc0a-4fae-a2dc-5c22b2c3928a   20Gi       RWX            managed-nfs-storage   5h10m
data-mobile-decision-server-1   Bound    pvc-3615cf99-31ea-491b-8808-8800057d0a21   20Gi       RWX            managed-nfs-storage   5h9m
  
[root@k8s-master01 mobile-decision-server]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                  STORAGECLASS          REASON   AGE
pvc-3615cf99-31ea-491b-8808-8800057d0a21   20Gi       RWX            Delete           Bound    kevin/data-mobile-decision-server-1   managed-nfs-storage            5h9m
pvc-9433d725-bc0a-4fae-a2dc-5c22b2c3928a   20Gi       RWX            Delete

5）查看NFS共享目录

只要pvc不删除，NFS共享目录下的文件夹名称不会改变。

[root@k8s-storage01 ~]# cd /data/storage/mobile-decision-server_data/
[root@k8s-storage01 mobile-decision-server_data]# ll
total 8
drwxrwxrwx 4 root root 4096 Oct 10 11:53 kevin-data-mobile-decision-server-0-pvc-9433d725-bc0a-4fae-a2dc-5c22b2c3928a
drwxrwxrwx 4 root root 4096 Oct 10 11:53 kevin-data-mobile-decision-server-1-pvc-3615cf99-31ea-491b-8808-8800057d0a21

可以手动删除pvc和pv

# kubectl delete pvc pvc-name -n namespace
# kubectl delete pv pv-name