Kubernetes - Jenkins integration

Question

I've bootstrapped with kubeadm Kubernetes 1.9 RBAC cluster and I've started inside a POD Jenkins based on jenkins/jenkins:lts. I would like to try out https://github.com/jenkinsci/kubernetes-plugin . I have already created a serviceaccount based on the proposal in https://gist.github.com/lachie83/17c1fff4eb58cf75c5fb11a4957a64d2

> kubectl -n dev-infra create sa jenkins
> kubectl create clusterrolebinding jenkins --clusterrole cluster-admin --serviceaccount=dev-infra:jenkins
> kubectl -n dev-infra get sa jenkins -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: 2018-02-16T12:06:26Z
  name: jenkins
  namespace: dev-infra
  resourceVersion: "1295580"
  selfLink: /api/v1/namespaces/dev-infra/serviceaccounts/jenkins
  uid: d040041c-1311-11e8-a4f8-005056039a14
secrets:
- name: jenkins-token-vmt79

> kubectl -n dev-infra get secret jenkins-token-vmt79 -o yaml
apiVersion: v1
data:
  ca.crt: LS0tL...0tLQo=
  namespace: ZGV2LWluZnJh
  token: ZXlK...tdVE=
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: jenkins
    kubernetes.io/service-account.uid: d040041c-1311-11e8-a4f8-005056039a14
  creationTimestamp: 2018-02-16T12:06:26Z
  name: jenkins-token-vmt79
  namespace: dev-infra
  resourceVersion: "1295579"
  selfLink: /api/v1/namespaces/dev-infra/secrets/jenkins-token-vmt79
  uid: d041fa6c-1311-11e8-a4f8-005056039a14
type: kubernetes.io/service-account-token

After that I go to Manage Jenkins -> Configure System -> Cloud -> Kubernetes and set the Kubernetes URL to the Cluster API that I use also in my kubectl KUBECONFIG server: url:port.

When I hit test connection I get "Error testing connection https://url:port: Failure executing: GET at: https://url:port/api/v1/namespaces/dev-infra/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:dev-infra:default" cannot list pods in the namespace "dev-infra".

I don't want to give to the dev-infra:default user a cluster-admin role and I want to use the jenkins sa I created. I can't understand how to configure the credentials in Jenkins. When I hit add credentials on the https://github.com/jenkinsci/kubernetes-plugin/blob/master/configuration.png I get

<select class="setting-input dropdownList">
<option value="0">Username with password</option>
<option value="1">Docker Host Certificate Authentication</option>
<option value="2">Kubernetes Service Account</option>
<option value="3">OpenShift OAuth token</option>
<option value="4">OpenShift Username and Password</option>
<option value="5">SSH Username with private key</option>
<option value="6">Secret file</option>
<option value="7">Secret text</option>
<option value="8">Certificate</option></select>

I could not find a clear example how to configure Jenkins Kubernetes Cloud connector to use my Jenkins to authenticate with service account jenkins. Could you please help me to find step-by-step guide - what kind of of credentials I need?

Regards, Pavel

Solution 1

The best practice is to launch you Jenkins master pod with the serviceaccount you created, instead of creating credentials in Jenkins

See example yaml

Solution 2

The Kubernetes plugin for Jenkins reads this file /var/run/secrets/kubernetes.io/serviceaccount/token. Please see if your Jenkins pod has this. The service account should have permissions targeting pods in the appropriate namespace.

In fact, we are using Jenkins running outside kubernetes 1.9. We simply picked the default service account token (from default namespace), and put it in that file on the Jenkins master. Restarted ... and the kubernetes token credential type was visible.

We do have a role and rolebinding though:

kubectl create role jenkins --verb=get,list,watch,create,patch,delete --resource=pods
kubectl create rolebinding jenkins --role=jenkins --serviceaccount=default:default

In our case, Jenkins is configured to spin up slave pods in the default namespace. So this combination works.

More questions (similar): Can I use Jenkins kubernetes plugin when Jenkins server is outside of a kubernetes cluster?

Solution 3

After some digging it appears that the easiest way to go(without giving extra permissions to the default service account for the name space) is to

kubectl -n <your-namespace> create sa jenkins
kubectl create clusterrolebinding jenkins --clusterrole cluster-admin --serviceaccount=<your-namespace>:jenkins
kubectl get -n <your-namespace> sa/jenkins --template='{{range .secrets}}{{ .name }} {{end}}' | xargs -n 1 kubectl -n <your-namespace> get secret --template='{{ if .data.token }}{{ .data.token }}{{end}}' | head -n 1 | base64 -d -

Seems like you can store this token as type Secret text in Jenkins and the plugin is able to pick it up. Another advantage of this approach compared to overwriting the default service account, as mentioned earlier above is that you can have secret per cluster - meaning you can use one jenkins to connect to for example dev -> quality -> prod namespaces or clusters with separate accounts.

Please feel free to contribute, if you have a better way to go.

Regards, Pavel

For more details you can check: - https://gist.github.com/lachie83/17c1fff4eb58cf75c5fb11a4957a64d2 - https://github.com/openshift/origin/issues/6807

Comments

[ppavlov]

Hi Parag, thank you for your reply. My question was how do I tell Jenkins to use my custom token.I'm not sure you should be changing the token by hand. Specially if you run Jenkins inside your Kubernetes this is even ro share file-system for your jenkins container. Will share solution proposal below. Regards, Pavel

[ppavlov]

Hello, thank you for your reply and sorry for the late reply. I believe this approach is way more elegant than what I did. Thank you for sharing. Marking this as best answer. Only thing is your url is to a branch different then master . Maybe later this will not be there example yaml in master Regards, Pavel