[摘要] kubernetes 可能会产生垃圾或者僵尸pod，在删除rc的时候，相应的pod没有被删除，手动删除pod后会自动重新创建，这时一般需要先删除掉相关联的resources，实际中还要具体情况具体分析。
在使用Helm部署gitlab的时候发现有垃圾的pod存在，一直删除不掉
# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE gitlab-migrations.1-2km-4rwnm 1/1 Running 158 21h gitlab-migrations.1-30d-gx9nz 1/1 Running 164 21h gitlab-migrations.1-dsw-m5qbc 1/1 Running 167 21h
一般来说是要删除pod关联的rc，rs以及deployment等等，可以执行如下命令查询
kubectl get ClusterRole -n kube-system kubectl get ClusterRoleBinding -n kube-system kubectl get Deployments -n kube-system kubectl get Pods -n kube-system kubectl get ServiceAccounts -n kube-system kubectl get CustomResourceDefinition -n kube-system kubectl get replicasets -n kube-system kubectl get replicationcontrollers -n kube-system kubectl get services -n kube-system
找到对应的标签进行删除，对应的pod一般就会被删除掉，如果pod还在的话再手动删除pod。
实际中我这三个pod还是无法删除掉，删除之后仍然会重新启动，所以仔细看了下pod的具体描述内容。
# kubectl describe pod -n kube-system gitlab-migrations.1-30d-gx9nz
Name: gitlab-migrations.1-30d-gx9nz
Namespace: kube-system
Node: 172.30.0.169/172.30.0.169
Start Time: Mon, 26 Mar 2018 11:53:45 +0800
Labels: app=migrations
controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
job-name=gitlab-migrations.1-30d
release=gitlab
Annotations: <none>
Status: Running
IP: 172.100.14.7
Controlled By: Job/gitlab-migrations.1-30d
Init Containers:
configure:
Container ID: docker://a3051d24e65267489aa97ae368bcbe5819e4ae4f3b5ddc4d277b0b25ee585716
Image: busybox
Image ID: docker-pullable://docker.io/busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21
Port: <none>
Command:
sh
/config/configure
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 26 Mar 2018 11:54:24 +0800
Finished: Mon, 26 Mar 2018 11:54:24 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/config from migrations-config (ro)
/init-config from init-migrations-secrets (ro)
/init-secrets from migrations-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)
Containers:
migrations:
Container ID: docker://c33e809de76ba44da330063d9e3c6f58631b30e3c805ea561dc68af38ffe3926
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-rails:latest
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-rails@sha256:7dca5eb3b686342a9cff798f8349ad07459a1ef1e9d47a23b787b284d874cc17
Port: <none>
Args:
/scripts/wait-for-deps
/scripts/db-migrate
State: Running
Started: Tue, 27 Mar 2018 09:50:09 +0800
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 27 Mar 2018 09:41:52 +0800
Finished: Tue, 27 Mar 2018 09:45:02 +0800
Ready: True
Restart Count: 164
Environment:
CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates
CONFIG_DIRECTORY: /var/opt/gitlab/config/gitlab/
GITLAB_ROOT_PASSWORD: Vi4ree2R
Mounts:
/etc/gitlab from migrations-secrets (ro)
/home/git/gitlab/config/secrets.yml from migrations-secrets (rw)
/var/opt/gitlab/templates from migrations-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
migrations-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-migrations
Optional: false
init-migrations-secrets:
<unknown>
migrations-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
default-token-6rdp2:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6rdp2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 6m (x655 over 21h) kubelet, 172.30.0.169 MountVolume.SetUp failed for volume "migrations-config" : configmaps "gitlab-migrations" not found
Warning BackOff 3m (x3516 over 21h) kubelet, 172.30.0.169 Back-off restarting failed container
需要关注的地方是Labels标签
Labels: app=migrations
controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
job-name=gitlab-migrations.1-30d
release=gitlab
发现有个job-name=gitlab-migrations.xxx
于是继续查看kubernetes是否有job存在
# kubectl get jobs -n kube-system NAME DESIRED SUCCESSFUL AGE gitlab-migrations.1-2km 1 0 23h gitlab-migrations.1-30d 1 0 23h gitlab-migrations.1-dsw 1 0 23h
先执行删除job看下：
# kubectl delete job gitlab-migrations.1-2km -n kube-system job "gitlab-migrations.1-2km" deleted # kubectl delete job gitlab-migrations.1-30d -n kube-system job "gitlab-migrations.1-30d" deleted # kubectl delete job gitlab-migrations.1-dsw -n kube-system job "gitlab-migrations.1-dsw" deleted
再看pod情况：
# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE gitlab-migrations.1-30d-gx9nz 1/1 Terminating 173 23h gitlab-migrations.1-dsw-m5qbc 0/1 Terminating 176 23h
附图为kubernets 的一些对象资源：
再看pod已经被删除了，还有处于Terminating状态的，一会就会消失掉。至此pod删除成功，所以删除与pods相关联的依赖资源是解决问题的最终方法！