一、集群安装Jenkins
1、准备jenkins镜像文件
docker pull jenkins/jenkins:2.204.1
docker tag a3f949e5ebfd harbor地址/infra/jenkins:v2.204.1
docker push harbor地址/infra/jenkins:v2.204.1
2、YAML资源文件
创建一个文件夹 在这个文件下创建3个 yaml 文件 pvc.yaml rbac.yaml jenkins.yaml
Jenkins master的安装，需要将数据持久化。可以利用NFS作为存储资源，创建PVC对象来挂载。PV/PVC配置文件pvc.yaml如下：
apiVersion: v1
kind: PersistentVolume
metadata:
name: opspv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
nfs:
server: 172.29.207.78 #注意：此处为NFS服务器的地址
path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: opspvc
namespace: kube-ops
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
同时，对于即将创建的Jenkins master资源对象，需要授予其一些权限，比如增删改查等。相应的配置文件rbac.yaml如下：
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
然后，基于jenkins/jenkins:lts 镜像创建jenkins master镜像，配置文件jenkins.yaml 如下：
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: kube-ops
spec:
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccount: jenkins
containers:
- name: jenkins
image: harbor.zzx/infra/jenkins:v2.204.1
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
subPath: jenkins
mountPath: /var/jenkins_home
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
imagePullSecrets:
- name: docker-harbor-registry
securityContext:
fsGroup: 1000
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: kube-ops
labels:
app: jenkins
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
nodePort: 30080 #注意：外部访问端口
- name: agent
port: 50000
targetPort: agent
最后，在目录内执行命令如下：
kubectl create namespace kube-ops
kubectl create -f ./
此时，我们通过命令kubectl -n kube-ops get pod可以查看到jenkins已成功创建。
二、jenkins动态配置slave
1、初始化jenkins配置
浏览器打开masterIP:30080
其中的管理员密码，我们既可以进入容器内对应的目录查看，也可以在nfs server服务器上执行命令：
cat /data/k8s/jenkins/secrets/initialAdminPassword 来查看；然后选择安装推荐的插件，如下：
然后添加管理员账户即可进入Jenkins界面。
2、配置jenkins slave
(1) 安装Kubernetes插件
进入 Manage Jenkins—>Manage Plugins—>可选插件(Available)—>Kubernetes plugin勾选，直接安装即可。
(2) 配置Kubernetes插件
点击Manage Jenkins—>Configure System—>云—>新增一个云—>Kubernetes，如下：
然后配置如下：
先注意 名称默认为kubernetes，然后 Kubernetes地址 填写https://kubernetes.default.svc.cluster.local，命名空间为kube-ops；接着点击右边的 连接测试 按钮，如果显示Connection test successful，表示Jenkins可以和Kubernetes集群正常通信了。最后，在Jenkins地址，填入：http://服务名.kube-ops.svc.cluster.local:8080，如下所示：
(3) 创建Pipeline动态构建测试
创建一个类型为Pipeline类型Job命名为test-slave，然后在Pipeline脚本填写下面一个简单的测试脚本
def label = "jenkins-slave-${UUID.randomUUID().toString()}"
podTemplate(label: label, cloud: 'kubernetes') {
node(label) {
stage('Run shell') {
sh 'sleep 10s'
sh 'echo hello world.'
}
}
}
(4) 点击构建流水线
可以看到在jenkins的namespaces下自动创建了对应的agent pod 相当于就是一个jenkins 的node 当任务执行完成这个pod会自动退出这个pod默认会去pull一个jenkins/jnlp-slave:x.xx-xx-alpine的镜像
[root@hdss7-21 ~]# kubectl get pods -n k8s-ops
NAME READY STATUS RESTARTS AGE
jenkins-77b9c47874-qjgfd 1/1 Running 1 13h
jenkins-slave-c07daa7b-31ef-41ea-825e-05c9c721edad-sb7h6-lpgwv 1/1 Running 0 18s