mac单机 k8s minikube ELK yaml 详细配置踩坑

1.准备工作

安装minikube, 安装kubectl(不再赘述, 自行查找)
启动minikube
minikube start image-mirror-country='cn' --registry-mirror=https://registry.docker-cn.com --memory=4096 --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
新建k8s namespace, 用于逻辑隔离整套elk服务
kubectl create namespace elkspace

2.编写k8s yaml文件(###标记都是注释, copy请删除)

elasticsearch.yaml

- apiVersion: apps/v1
  kind: Deployment  ###启动es容器, 级别可以是Pod等,有不同定义的
  metadata:
    name: es-single
    namespace: elkspace  ###第一步已定义, 可以全部不使用, 走默认default namespace
    labels:
      app: es-single
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: es-single  ###定义k:v格式标签, 用于Service通过标签查询并连接
    template:
      metadata:
        name: es-single
        labels:
          app: es-single
      spec:
        containers:
        - image: elasticsearch:7.9.1  ###docker镜像, 注意ELK三者的版本号请统一
          imagePullPolicy: IfNotPresent
          name: es
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:  ###环境变量, 相当于docker run中的-e参数
          - name: "discovery.type"  ###单节点es
            value: "single-node"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx2g"
          ports:  ###设置容器暴露端口, 相当于设置Pod端口9200,9300 tcp可用
          - containerPort: 9200
            protocol: TCP
          - containerPort: 9300
            protocol: TCP
- apiVersion: v1
  kind: Service  ###启动服务, 用于对外暴露Pod信息, 因为k8s外无法直连Pod, 必须通过Service暴露出去端口
  metadata:
    name: es-single-service
    namespace: elkspace
  spec:
    type: NodePort  ###相当于Pod端口映射Node端口, 外部访问通过NodeIP+NodePort就可以连到服务
    ports:
    - name: esport
      port: 9200  ###Service连接Pod端口
      targetPort: 9200  ###上面暴露的es Pod端口
      protocol: TCP
      nodePort: 30011  ###k8s外部访问端口, 自定义, 但是有范围
    - name: esportlink
      port: 9300
      targetPort: 9300
      protocol: TCP
      nodePort: 30021
    selector:  ###很重要, 服务绑定到指定K:V形式的Pod, 上面配置好的
      app: es-single

kibana.yaml

- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: kb-single
    namespace: elkspace
    labels:
      app: kb-single
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: kb-single
    template:
      metadata:
        name: kb-single
        labels:
          app: kb-single
      spec:
        containers:
        - name: kb
          image: kibana:7.9.1
          imagePullPolicy: IfNotPresent
          env:  ###非常重要!!!服务之间ip动态变化, 配置连接最好用域名, Service域名规则是:service.namespace
          - name: ELASTICSEARCH_HOSTS  ###非常重要!!!注意是ELASTICSEARCH_HOSTS还是ELASTICSEARCH_URL, 得和当前版本kibana中/usr/share/kibana/config/kibana.yml的参数名称一样, 可以docker启动镜像进去看看
            value: "http://es-single-service.elkspace:9200"
          - name: XPACK_SECURITY_ENABLED
            value: "true"
          ports:
          - name: ui
            containerPort: 5601
            protocol: TCP
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 0.5
              memory: 500Mi
- apiVersion: v1
  kind: Service
  metadata:
    name: kb-single-service
    namespace: elkspace
  spec:
    type: NodePort
    ports:
    - port: 5601
      protocol: TCP
      targetPort: 5601
      nodePort: 30031
    selector:
      app: kb-single

logstash.yaml

- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: log-single
    namespace: elkspace
    labels:
      app: log-single
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: log-single
    template:
      metadata:
        name: log-single
        labels:
          app: log-single
      spec:
        containers:
        - name: log
          image: logstash:7.9.1
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 5044
            protocol: TCP
          volumeMounts:  ###该容器需要挂载的地址,也就是logstash.conf,因为logstash日志接收和转发信息都在此文件配置
          - name: log-config
            mountPath: /usr/share/logstash/pipeline
          env:  ###非常重要!!!和上一步的kibana此处配置一样的效果,参数名称需要和logstash/config/中配置文件一样
          - name: XPACK_MONITORING_ELASTICSEARCH_HOSTS
            value: "http://es-single-service.elkspace:9200"
          securityContext:
            privileged: true
        volumes:  ###指定从哪里挂载目录
        - name: log-config  ###对应到volumeMounts中的name
          hostPath:  ###从当前Node本地路径加载,直接写我自己电脑绝对路径不对,此时的Node是minikube容器,所以这个地址也是minikube的目录,所以首先得把电脑本地路径挂载到minikube路径, 然后这里才能取到配置文件等信息(下一步说明)
            path: /data/pipeline
- apiVersion: v1
  kind: Service
  metadata:
    name: lg-single-service
    namespace: elkspace
  spec:
    type: NodePort
    ports:
    - port: 5044
      protocol: TCP
      targetPort: 5044
      nodePort: 30041
    selector:
      app: log-single

本地随便找个地方新建logstash.conf文件, 就是上一步中hostPath挂载的logstash.conf配置文件, 文件信息如下, 注意es 服务的地址, 和上面配置的一样

input {
  tcp {
    port => 5044
    mode => "server"
  }
}

output {
  elasticsearch {
    hosts => ["es-single-service.elkspace:9200"]
    index => "apidemo"
  }

  stdout{
    codec => rubydebug
  }
}

然后本地路径挂载minikube指定地址, &表示后台操作

/Users/grahamliu/elk/pipeline 电脑本地logstash.conf所在目录

/data/pipeline minikube指定地址, 也就是第3步中配置的hostpath, minikube文件目录详细信息可自行查阅
```
minikube mount /Users/grahamliu/elk/pipeline:/data/pipeline &
```

3.启动上面的各项配置文件(自行查询)

4.外部如何打开k8s中的Kibana? 如何连接写入日志到k8s logstash?

咱们部署是是单Node服务, 所有外部访问都是通过NodeIP+NodePort访问

查询minikube node的命令, InternalIP: 192.168.xx.x

pro-2:~ grahamliu$ kubectl get node
NAME       STATUS   ROLES    AGE    VERSION
minikube   Ready    master   117d   v1.17.3
pro-2:~ grahamliu$ kubectl describe node minikube
Name:               minikube
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=minikube
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 03 Jan 2021 11:57:08 +0800
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 30 Apr 2021 16:45:08 +0800   Sun, 03 Jan 2021 11:57:02 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 30 Apr 2021 16:45:08 +0800   Sun, 03 Jan 2021 11:57:02 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 30 Apr 2021 16:45:08 +0800   Sun, 03 Jan 2021 11:57:02 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 30 Apr 2021 16:45:08 +0800   Sun, 03 Jan 2021 11:57:11 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.xx.x
  Hostname:    minikube
Capacity:

此时只要用以上ip加上之前Service中暴露的端口就能成功访问各个服务了, 可以用命令查询所有端口信息

pro-2:~ grahamliu$ kubectl get svc -n elkspace
NAME                TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
es-single-service   NodePort   10.100.233.92    <none>        9200:30011/TCP,9300:30021/TCP   22h
kb-single-service   NodePort   10.104.129.185   <none>        5601:30031/TCP                  3h22m
lg-single-service   NodePort   10.111.174.220   <none>        5044:30041/TCP                  176m

5.整个流程都通了以后, 其实elk三个yaml可以整合到一个yaml一次性启动, elk.yaml编排如下

kind: List
apiVersion: v1
items:
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: es-single
    namespace: elkspace
    labels:
      app: es-single
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: es-single
    template:
      metadata:
        name: es-single
        labels:
          app: es-single
      spec:
        containers:
        - image: elasticsearch:7.9.1
          imagePullPolicy: IfNotPresent
          name: es
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
          - name: "discovery.type"
            value: "single-node"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx2g"
          ports:
          - containerPort: 9200
            protocol: TCP
          - containerPort: 9300
            protocol: TCP
- apiVersion: v1
  kind: Service
  metadata:
    name: es-single-service
    namespace: elkspace
  spec:
    type: NodePort
    ports:
    - name: esport
      port: 9200
      targetPort: 9200
      protocol: TCP
      nodePort: 30011
    - name: esportlink
      port: 9300
      targetPort: 9300
      protocol: TCP
      nodePort: 30021
    selector:
      app: es-single
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: kb-single
    namespace: elkspace
    labels:
      app: kb-single
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: kb-single
    template:
      metadata:
        name: kb-single
        labels:
          app: kb-single
      spec:
        containers:
        - name: kb
          image: kibana:7.9.1
          imagePullPolicy: IfNotPresent
          env:
          - name: ELASTICSEARCH_HOSTS
            value: "http://es-single-service.elkspace:9200"
          - name: XPACK_SECURITY_ENABLED
            value: "true"
          ports:
          - name: ui
            containerPort: 5601
            protocol: TCP
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 0.5
              memory: 500Mi
- apiVersion: v1
  kind: Service
  metadata:
    name: kb-single-service
    namespace: elkspace
  spec:
    type: NodePort
    ports:
    - port: 5601
      protocol: TCP
      targetPort: 5601
      nodePort: 30031
    selector:
      app: kb-single
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: log-single
    namespace: elkspace
    labels:
      app: log-single
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: log-single
    template:
      metadata:
        name: log-single
        labels:
          app: log-single
      spec:
        containers:
        - name: log
          image: logstash:7.9.1
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 5044
            protocol: TCP
          volumeMounts:
          - name: log-config
            mountPath: /usr/share/logstash/pipeline
          env:
          - name: XPACK_MONITORING_ELASTICSEARCH_HOSTS
            value: "http://es-single-service.elkspace:9200"
          securityContext:
            privileged: true
        volumes:
        - name: log-config
          hostPath:
            path: /data/pipeline
- apiVersion: v1
  kind: Service
  metadata:
    name: lg-single-service
    namespace: elkspace
  spec:
    type: NodePort
    ports:
    - port: 5044
      protocol: TCP
      targetPort: 5044
      nodePort: 30041
    selector:
      app: log-single