k8s 部署 log-pilot 收集容器标准输出日志和指定路径应用日志

日志系统要求：
1.因开发项目要求，一个pod 内有多个日志路径，需要收集
2.同时需要收集pod 容器的标准输出日志

环境：
本次环境es、kibana 均部署在k8s 集群外，在物理机部署，只需要log-pilot 指定es 地址

具体步骤：
创建 daemonset log-pilot
kubectl get daemonsets.apps      log-pilot    -o yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: log-pilot
  name: log-pilot
  namespace: default
spec:
  selector:
    matchLabels:
      k8s-app: log-es
  template:
    metadata:
      labels:
        k8s-app: log-es
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: PILOT_LOG_PREFIX
          value: mytest         #收集容器日志前缀标识，容器日志必须指定同一标识 mytest
        - name: FILEBEAT_OUTPUT
          value: elasticsearch  #日志指定输出位置为 es 
        - name: ELASTICSEARCH_HOST
          value: 10.16.0.178    # es 地址
        - name: ELASTICSEARCH_PORT
          value: "9200" # es 端口
        image: yzsjhl-evdc-node03.opi.com/tj/log-pilot:0.9.5-filebeatv01
        imagePullPolicy: IfNotPresent
        name: log-pilot
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: sock
        - mountPath: /host
          name: root
          readOnly: true
        - mountPath: /var/lib/filebeat
          name: varlib
        - mountPath: /var/log/filebeat
          name: varlog
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/run/docker.sock
          type: ""
        name: sock
      - hostPath:
          path: /
          type: ""
        name: root
      - hostPath:
          path: /var/lib/filebeat
          type: DirectoryOrCreate
        name: varlib
      - hostPath:
          path: /var/log/filebeat
          type: DirectoryOrCreate
        name: varlog
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate


创建应用容器deployments  (java 项目)，有三个日志路径需要收集

$ kubectl get deployments.apps   gateway-deploy    -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gateway
      release: stabel
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: gateway
        env: test
        release: stabel
    spec:
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai   
        - name: mytest_logs_gatewaystdout  # 容器的标准输出日志
          value: stdout
        - name: mytest_logs_gatewayaccesslogdir  #第一个应用日志 
          value: /data/logs/service-gateway/access/*.log
        - name: mytest_logs_gatewayauthlogdir #第二个应用日志 
          value: /data/logs/service-gateway/auth/*.log
        - name: mytest_logs_gatewayrootlogdir #第三个应用日志 
          value: /data/logs/service-gateway/root/*.log
        image: yzsjhl-evdc-node03.opi.com/renren-backend/gateway:20200519161529
        imagePullPolicy: IfNotPresent
        name: gateway
        ports:
        - containerPort: 8089
          name: http1
          protocol: TCP
        - containerPort: 20007
          name: http2
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts: # 每个日志 emptyDir都需要挂载在容器日志的指定位置
        - mountPath: /data/logs/service-gateway/access
          name: gatewayaccess-logs
        - mountPath: /data/logs/service-gateway/auth
          name: gatewayauth-logs
        - mountPath: /data/logs/service-gateway/root
          name: gatewayroot-logs
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}   # 每个日志路径都需要指定一个 emptyDir
        name: gatewayaccess-logs
      - emptyDir: {}
        name: gatewayauth-logs
      - emptyDir: {}
        name: gatewayroot-logs

查看kibana，看是否收集到日志：