Jenkins on Kubernetes¶

Goal¶

Run Jenkins as best as we can on Kubernetes, taking as much advantage of the platform and ecosystem as we can.

Steps to take¶

install LDAP
install and configure Apache Keycloak backed by LDAP
install Hashicorp Vault
install Jenkins
configure Jenkins with Jenkins Configuration as Code
verify we can use Kubernetes agents
verify we can use GitHub integration
expose Telemtry via OpenTelemetry
- collect telemtry via Prometheus/Grafana
- collect telemtry via Tanzu Observability

LDAP¶

helm repo add helm-openldap https://jp-gouin.github.io/helm-openldap/

helm repo update

helm upgrade --install ldap helm-openldap/openldap --namespace keycloak --values ldap-values.yaml --version 2.0.4

Keycloak¶

kubectl create namespace keycloak

kubectl apply -f keycloak-httpproxy.yaml

helm upgrade --install keycloak bitnami/keycloak --namespace keycloak --values keycloak-values.yaml --version 9.2.8

kubectl logs -f -n keycloak keycloak-0

Keycloak HTTPProxy¶

export LB_IP=$(kubectl get svc -n tanzu-system-ingress envoy -o jsonpath="{.status.loadBalancer.ingress[0].ip}")

export KEYCLOAK_HOSTNAME=keycloak.${LB_IP}.nip.io

cfssl gencert -ca ca.pem -ca-key ca-key.pem \
  -config cfssl.json \
  -profile=server \
  -cn="${KEYCLOAK_HOSTNAME}" \
  -hostname="${KEYCLOAK_HOSTNAME},keycloak.keycloak.svc.cluster.local,keycloak,localhost" \
   base-service-cert.json   | cfssljson -bare keycloak-server

Jenkins¶

helm repo add jenkins https://charts.jenkins.io
helm repo update

kubectl create namespace jenkins

DH_USER=
DH_EMAIL=
DH_PASS=
NS=jenkins

kubectl create secret docker-registry dockerhub-pull-secret \
  --docker-username=${DH_USER} \
  --docker-password=${DH_PASS} \
  --docker-email=${DH_EMAIL} \
  --namespace ${NS}

GH_USER=
GH_TOKEN=
NS=jenkins

kubectl create secret generic github-token \
  --from-literal=user="${GH_USER}" \
  --from-literal=token="${GH_TOKEN}" \
  --namespace ${NS}

JENKINS_ADMIN_USER=joostvdg
JENKINS_ADMIN_PASS=

kubectl create secret generic jenkins-admin \
  --from-literal=user="${JENKINS_ADMIN_USER}" \
  --from-literal=token="${JENKINS_ADMIN_PASS}" \
  --namespace ${NS}

Install Jenkins Helm¶

helm repo add jenkins https://charts.jenkins.io
helm repo update

helm upgrade --install jenkins jenkins/jenkins --namespace jenkins --values jenkins-values.yaml

Jenkins CasC¶

...

Jenkins HTTPProxy¶

export LB_IP=$(kubectl get svc -n tanzu-system-ingress envoy -o jsonpath="{.status.loadBalancer.ingress[0].ip}")

export JENKINS_HOSTNAME=jenkins.${LB_IP}.nip.io

cfssl gencert -ca ca.pem -ca-key ca-key.pem \
  -config cfssl.json \
  -profile=server \
  -cn="${JENKINS_HOSTNAME}" \
  -hostname="${JENKINS_HOSTNAME},jenkins.jenkins.svc.cluster.local,jenkins,localhost" \
   base-service-cert.json   | cfssljson -bare jenkins-server

Kaniko To Harbor¶

HARBOR_SERVER=harbor.10.220.7.70.nip.io
HARBOR_USER=
HARBOR_PASS=

kubectl create secret docker-registry harbor-registry-creds \
  --docker-username=${HARBOR_USER} \
  --docker-password=${HARBOR_PASS} \
  --docker-server=${HARBOR_SERVER} \
  --namespace jenkins

Checks API¶

create Jenkins credentials for GitHub API
ensure you have create a GitHub Server in Jenkins config

credentials:
  system:
    domainCredentials:
    - credentials:
      - string:
          description: "github token"
          id: "githubtoken"
          scope: GLOBAL
          secret: "{AQAAABAAAAAwDgut+8oOqmwh4qHohWO09AxFPsz78EXLrtoC4QEFr9nY8orwx/mcLaS11G831IDmO8Ftxs2QockuYPJveteneQ==}"
      - usernamePassword:
          description: "github-credentials"
          id: "github-credentials"
          password: "{AQAAABAAAAAwutoG/5mM+UqCYohxLLogZ7Dpd8lyfWh9MeAbNybMplhycx4Z17h1WgzQPQn0lHAM+mfHsufBMuRTwl79rnfk9g==}"
          scope: GLOBAL
          username: "joostvdg"

  gitHubPluginConfig:
    configs:
    - credentialsId: "githubtoken"
      name: "GitHub"
    hookUrl: "https://jenkins.10.220.7.70.nip.io/github-webhook/"

Pipeline¶

create credentials
- githubtoken of type secret text with the GitHub API Token
- github-credentials username and password -> github username and API Token
create GitHub Org Job
- add plugin GitHub Branch Source?

Webhooks behind a firewall¶

docker pull quay.io/schabrolles/smeeclient:stable --platform linux/amd64
docker tag quay.io/schabrolles/smeeclient:stable harbor.10.220.7.70.nip.io/test/smeeclient:stable
docker push harbor.10.220.7.70.nip.io/test/smeeclient:stable

SMEESOURCE
HTTPTARGET

kubectl run smee-client -n jenkins --rm -i --tty --image harbor.10.220.7.70.nip.io/test/harbor.10.220.7.70.nip.io/test/smeeclient:stable  \
     --env=SMEESOURCE=

apiVersion: v1
kind: Pod
metadata:
  name: smee-client
  namespace: jenkins
  labels:
    app: smee-client
spec:
  containers:
  - name: smee-client
    image: harbor.10.220.7.70.nip.io/test/smeeclient:stable
    env:
    - name: SMEESOURCE
      value: "https://smee.io/TVjVEDxMHHQeNPDl"
    - name: HTTPTARGET
      value: "http://jenkins:8080/github-webhook/"

Thanos¶

kubectl create namespace thanos

helm upgrade --install thanos bitnami/thanos --namespace thanos --values thanos-values.yaml

TO¶

kubectl create namespace wavefront

TO_API_TOKEN=
TO_NAMESPACE=
CLUSTER_NAME=

kubectl create secret generic to-api-token  --from-literal=api-token="${TO_API_TOKEN}" --namespace ${TO_NAMESPACE}

projectPacific:
  enabled: true
vspheretanzu:
  enabled: true
wavefront:
  url: https://vmware.wavefront.com
  existingSecret: to-api-token
collector:
  useDaemonset: true
  apiServerMetrics: true
  cadvisor:
    enabled: true
  logLevel: info
  discovery:
    annotationExcludes: []
  tags:
    datacenter: vbc-h20-62
    project: vbc-h20
    owner: joostvdg
kubeStateMetrics:
  enabled: true
proxy:
  replicaCount: 2
  zipkinPort: 9411
  args: --traceZipkinListenerPorts 9411 --otlpGrpcListenerPorts 4317 --otlpHttpListenerPorts 4318

kubectl -n wavefront patch svc wavefront-proxy --patch '{"spec": {"ports": [{"name":"oltphttp", "port": 4318, "protocol": "TCP"}, {"name":"oltpgrpc", "port": 4317, "protocol": "TCP"}]}}'

Vault¶

Vault Helm Install¶

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

kubectl create namespace vault

NS=vault
DH_USER=
DH_EMAIL=
DH_PASS=

kubectl create secret docker-registry dockerhub-pull-secret \
  --docker-username=${DH_USER} \
  --docker-password=${DH_PASS} \
  --docker-email=${DH_EMAIL} \
  --namespace ${NS}

KEYCLOAK_CLIENT_ID=
KEYCLOAK_CLIENT_SECRET=
KEYCLOAK_URL=

kubectl create secret generic oic-auth \
  --from-literal=clientID="${KEYCLOAK_CLIENT_ID}" \
  --from-literal=clientSecret="${KEYCLOAK_CLIENT_SECRET}" \
  --from-literal=keycloakUrl=${KEYCLOAK_URL} \
  --namespace jenkins

kubectl --namespace vault create secret tls tls-ca --cert ./tls-ca.cert --key ./tls-ca.key

helm upgrade --install vault hashicorp/vault --namespace vault --values vault-values.yaml

HTTPProxy¶

export LB_IP=$(kubectl get svc -n tanzu-system-ingress envoy -o jsonpath="{.status.loadBalancer.ingress[0].ip}")

export VAULT_HOSTNAME=vault.${LB_IP}.nip.io

cfssl gencert -ca ca.pem -ca-key ca-key.pem \
  -config cfssl.json \
  -profile=server \
  -cn="${VAULT_HOSTNAME}" \
  -hostname="${VAULT_HOSTNAME},vault.vault.svc.cluster.local,vault-ui.vault.svc.cluster.local,vault,vault-ui,localhost" \
   base-service-cert.json   | cfssljson -bare vault-server

cat vault-server-key.pem | base64

cat vault-server.pem | base64

kubectl apply -f vault-httpproxy.yaml

kubectl get httpproxy -n vault

Unsealing¶

kubectl get pods --selector='app.kubernetes.io/name=vault' --namespace=' vault'

kubectl exec --namespace vault --stdin=true --tty=true vault-0 -- vault operator init

TODO¶

Things to improve on.

make credentials come from Vault
make credentials come from Kubernetes
OpenTelemtry
- Prometheus/Grafana/?
- TO?
Jobs via SeedJob from JobDSL?

References¶

Last update: 2022-08-06 12:51:44