K8s部署Jenkins主从节点并与Gitblit集成

V2AS问路

K8s部署Jenkins主从节点并与Gitblit集成

阅读原文时间：2021年04月22日阅读：1

一、制作Jenkins-master镜像

拉取base镜像

docker pull jenkinsci/jenkins:lts
运行base镜像，为便于构建，这里打算手动运行和安装好镜像内容后进行docker commit出一个新镜像。

docker run -tid -p 8080:8080 –name jenkins-master jenkinsci/jenkins:lts
访问jenkins-master镜像暴露的url: http://<ip>:8080
根据提示在容器内获取admin的初始秘钥。
安装推荐的插件（Suggested plugins），如果没有安装Kubernetes插件，则在

系统管理–>管理插件–>可选插件–>输入 Kubernetes
选择 Kubernetes plugin–>直接安装
复制kubectl命令到容器中

docker cp kubectl jenkins-master:/usr/bin
安装 libltdl7

docker exec -ti -u root jenkins-master /bin/sh
apt-get update
apt-get install libltdl7
apt-get clean
把Jenkins Home的文件拷贝出来，作为在k8s上持久化存放（Gluster或者NFS文件系统）。注意：主机上 .kube/config 文件也要放到这个目录下以供jenkins的k8s插件与kubernetes交互。

docker cp jenkins-master:/var/jenkins_home ./config/jenkins_home
mkdir -p ./config/jenkins_home/.kube && cp /root/.kube/config ./config/jenkins_home.kube/
如果后续需要在master上构建工程，安装所需要的组件（如nodejs、maven、JDK等）。然后提交镜像。

docker commit jenkins-master jenkins-k8s-master:v1

二、部署Jenkins-master

环境中master节点用Rancher部署，这里截取部分yaml内容以供参考：

jenkins-deploy.yaml

apiVersion: apps/v1beta2
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
labels:
workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
name: jenkins-master
namespace: ci-cd
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
annotations:
labels:
workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
spec:
affinity: {}
containers:
- image: jenkins-k8s-master:v1
imagePullPolicy: IfNotPresent
name: jenkins-master
ports:
- containerPort: 8080
hostPort: 8080
protocol: TCP
- containerPort: 50000
hostPort: 50000
protocol: TCP
resources: {}
volumeMounts:
- mountPath: /var/run/docker.sock
name: dockersock
- mountPath: /var/jenkins_home
name: jenkinshome
- mountPath: /root/.kube
name: kube-config
- mountPath: /usr/bin/docker
name: docker-binary
dnsPolicy: ClusterFirst
nodeName: izwz9ghgtf18pi19ytou0vz
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /var/run/docker.sock
type: ""
name: dockersock
- hostPath:
path: /mnt/jenkins/jenkins_home
type: ""
name: jenkinshome
- hostPath:
path: /root/.kube
type: ""
name: kube-config
- hostPath:
path: /usr/bin/docker
type: ""
name: docker-binary
jenkins-svc.yaml

apiVersion: v1
kind: Service
metadata:
annotations:
field.cattle.io/targetWorkloadIds: '["deployment:ci-cd:jenkins-master"]'
name: jenkins-master
namespace: ci-cd
spec:
ports:
- name: 8080tcp01-jenkins-master
  port: 8080
  protocol: TCP
  targetPort: 8080
- name: 50000tcp01-jenkins-master
  port: 50000
  protocol: TCP
  targetPort: 50000
  selector:
  workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
  sessionAffinity: None
  type: ClusterIP

三、制作Jenkins-slave镜像

在Dockerfile所在目录准备好以下内容

maven仓库的位置epel-apache-maven.repo

wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
kubectl可执行文件
jenkins-slave可执行文件
需要特殊配置的hosts文件
其它构建工具如node等
下载base镜像centos

docker pull centos

vi Dockerfile

FROM  centos
ARG AGENT_WORKDIR=/root/agent
RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/3.16/remoting-3.16.jar \
  && chmod 755 /usr/share/jenkins

COPY epel-apache-maven.repo /etc/yum.repos.d/epel-apache-maven.repo
RUN yum -y install git libtool-ltdl java-1.8.0-openjdk apache-maven
RUN rm -rf /var/cache/yum/x86_64/7/*

ENV AGENT_WORKDIR=${AGENT_WORKDIR}
COPY node /usr/local/
COPY kubectl /usr/bin/kubectl

COPY jenkins-slave /usr/local/bin/jenkins-slave
COPY .kube /root/.kube
COPY hosts /tmp/hosts
COPY ojdbc6.jar /root/ojdbc6.jar
RUN mkdir -p -- /lib-override && cp /usr/lib64/libnss_files.so.2 /lib-override
RUN sed -i 's:/etc/hosts:/tmp/hosts:g' /lib-override/libnss_files.so.2
ENV LD_LIBRARY_PATH /lib-override

RUN mkdir /root/.jenkins && mkdir -p ${AGENT_WORKDIR}
ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0
WORKDIR /root
ENTRYPOINT ["jenkins-slave"]

构建slave镜像

docker build -t jenkins-slave:v1 .

四、配置Jenkins与K8s协作

在Jenkins-master的UI界面上进入

系统管理 > 系统设置 > （最下方）新增一个云 > 选择Kubernetes

参考下图的配置

添加Pod template

五、配置Gitblit

Gitblit（这里使用版本是1.8.0）作为K8s外部系统，这里不考虑其部署。但为了和Jenkins交互，需要修改一些配置：

data目录下新建 gitblit.properties

include = defaults.properties

#

Define your overrides or custom settings below

#

web.canonicalUrl为gitblit自身的url，用于生成工程的访问路径等。

web.canonicalUrl = http://172.18.109.238:3000

groovy.jenkinsServer为Jenkins-master的访问路径，Gitblit会通过该url调用Jenkins-master上的hook来触发构建

groovy.jenkinsServer = http://172.18.63.25:8080
新建maven项目如下图所示

六、配置Jenkins-master的工程和代码

在Jenkins-master UI中，新建一个任务，配置：

构建触发器：轮询SCM，日程表留空
流水线：定义Pipeline Script from SCM
SCM选择Git，Repositories写Gitblit的代码库url和登陆用户
选择要build的代码线（branch）
脚本路径选择默认的Jenkinsfile

代码中新增Jenkinsfile
vi jenkinsfile

node('jnlp-slave') {
  stage('git clone'){
    checkout scm
  }

  stage 'build'
    sh "mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11.2.0.1.0 -Dpackaging=jar -Dfile=/root/ojdbc6.jar"
    sh "mvn package"
    sh "echo done.."
}

在git中提交代码