run gitlab-runner in k8s

run gitlab-runner in k8s

主要成果

runner运行在k8s内

• 衍生的job同步运行在k8s同名namespace中
• job执行时，可以直接通过命令访问到k8s其它的services
• 使用gcr.io/kaniko-project/executor构建docker image(替换原dind[docker in docker]模型)

versions

• Kubernetes 1.17
• helm 3.1.2
• gitlab 13.1.2
• gitlab-runner 13.2.2/13.3.0

构建方式

• install gitlab-runner with helm
  • add gitlab charts repo
  • pull gitlab-runner
  • tweak values.yaml
  • install gitlab-runner
• tweak docker daemon
• config git project .gitlab-ci.yml

install gitlab-runner with helm

add gitlab charts repo

helm repo add gitlab https://charts.gitlab.io/

pull gitlab-runner

helm repo update
helm pull gitlab/gitlab-runner  --untar

tweak values.yaml

imagePullPolicy: IfNotPresent
gitlabUrl: https://git.somewhere.com      ## 修改点
runnerRegistrationToken: "xxxxxxxxxxxxxx" ## 修改点
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
  create: true
  clusterWideAccess: false
  podSecurityPolicy:
    enabled: false
    resourceNames:
    - gitlab-runner
metrics:
  enabled: true
runners:
  image: ubuntu:16.04
  locked: false
  tags: "in-k8s-env001"    ## 修改点
  privileged: true
  pollTimeout: 1800        ## 修改点，默认值180，但是拉取官方镜像会很慢
  outputLimit: 4096
  cache: {}
  builds: {}
  services: {}
  helpers: {}
securityContext:
  fsGroup: 65533
  runAsUser: 100
resources: {}
affinity: {}
nodeSelector: {}
tolerations: []
hostAliases: []
podAnnotations: {}
podLabels: {}

install gitlab-runner

kubectl create ns gitlab
helm -n gitlab install runner gitlab/gitlab-runner --values=values.yaml

tweak docker daemon

为加速docker拉取官方镜像速度，可以调整k8s node docker daemon的配置 /etc/docker/daemon.json

{
  "registry-mirrors": ["https://xxxxxxx.mirror.aliyuncs.com"]
}

需要去阿里云自己申请加速域名

config git project .gitlab-ci.yml

stages:
  - build
  - buildImg


variables:
  GOPROXY: https://goproxy.cn
  GO111MODULE: "on"
  APP_IMAGE_ID: ${DOCKER_REGISTRY}/${CI_PROJECT_NAME}-${CI_PROJECT_ID}:$CI_COMMIT_REF_NAME-${CI_PIPELINE_ID}

build:
  stage: build
  image:
    name: "golang:1.15"
  tags:
    - in-k8s-env001
  artifacts:
    paths:
      - app
  script:
    - go build -o app


buildImg:
  stage: buildImg
  image:
    name: gcr.io/kaniko-project/executor:debug # 参考 https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
    entrypoint: [""]
  tags:
    - in-k8s-env001
  script:
    - mkdir -p /kaniko/.docker
    - echo "{"auths":{"$DOCKER_REGISTRY":{"username":"${DOCKER_USER}","password":"${DOCKER_PASSWORD}"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $APP_IMAGE_ID