Create Cluster

Terraform

Get Kubeconfig

1
gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${REGION}

JX Boot

  • kubectl create secret generic external-dns-gcp-sa --from-file=credentials.json
  • download jx
    • brew install jx
  • configure cloud dns
  • run cjx boot to initialize
  • update jx-requirements.yaml
  • run cjx boot again to start the rest

JX Boot - CJXD

Clone Faillure

1
2
3
4
5
6
7
8
? Do you want to clone the Jenkins X Boot Git repository? Yes

Cloning https://github.com/cloudbees/cloudbees-jenkins-x-boot-config.git @ 1.0.36 to cloudbees-jenkins-x-boot-config
error: setting HEAD to origin/1.0.36: git output: fatal: ambiguous argument 'origin/1.0.36': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]': failed to run 'git reset --hard origin/1.0.36' command in directory 'cloudbees-jenkins-x-boot-config', output: 'fatal: ambiguous argument 'origin/1.0.36': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]''

Helm Apply Issue

1
2
3
4
5
6
Modified file /Users/joostvdg/Projects/Personal/Github/vasimos/jx/cjxd/gke/jxboot/cloudbees-jenkins-x-boot-config/env/Chart.yaml to set the chart to version 1
Ignoring templates/.gitignore
Applying the kubernetes overrides at ../kubeProviders/gke/values.tmpl.yaml
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/cloudbees/cloudbees-jenkins-x-versions.git and git ref: v0.0.15
error: failed to lint the chart '/var/folders/f_/mpwdv8s16r7_zt43r3r7k20w0000gn/T/jx-helm-apply-578543501/env': failed to run 'helm lint' command in directory '/var/folders/f_/mpwdv8s16r7_zt43r3r7k20w0000gn/T/jx-helm-apply-578543501/env', output: '==> Linting .
[ERROR] Chart.yaml: apiVersion is required

TLS - CloudDNS

For each environment we have to setup the issuer and certificate.

Easiest way I found, was to copy the yaml from the issuer and certificate in the jx namespace. You then remove the unnecesary elements, those generated by Kubernetes itself (such as creation date, status, etc).

You have to change the domain name and hosts values, as they should now point to the subdomain corresponding to this environment (unless its production). Once the files are good, you add them to your environment. You do so, by adding them to the templates folder -> env/templates.

1
kubectl -n jx get issuer letsencrypt-prod -o yaml

1
kubectl -n jx get certificate tls-<unique to your cluster>-p -o yaml

Add Support For Multiple Subdomains

If you want to support multiple subdomains, such as dev.cjxd.example.com and staging.cjxd.example.com you need multiple entries in the external dns controller for the domain.

ImagePullBackup

1
2
3
4
5
6
7
8
9
events:
  Type     Reason     Age                   From                                         Message
  ----     ------     ----                  ----                                         -------
  Normal   Scheduled  18m                   default-scheduler                            Successfully assigned jx-staging/jx-jx-qs-spring-boot-6-58b75446b4-pkd7x to gke-joost-cjxd-pool2-54e21b2f-hlhd
  Normal   Pulling    16m (x4 over 18m)     kubelet, gke-joost-cjxd-pool2-54e21b2f-hlhd  Pulling image "gcr.io/ps-dev-201405/jx-qs-spring-boot-6:0.0.1"
  Warning  Failed     16m (x4 over 18m)     kubelet, gke-joost-cjxd-pool2-54e21b2f-hlhd  Failed to pull image "gcr.io/ps-dev-201405/jx-qs-spring-boot-6:0.0.1": rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
  Warning  Failed     16m (x4 over 18m)     kubelet, gke-joost-cjxd-pool2-54e21b2f-hlhd  Error: ErrImagePull
  Normal   BackOff    8m24s (x42 over 18m)  kubelet, gke-joost-cjxd-pool2-54e21b2f-hlhd  Back-off pulling image "gcr.io/ps-dev-201405/jx-qs-spring-boot-6:0.0.1"
  Warning  Failed     3m18s (x64 over 18m)  kubelet, gke-joost-cjxd-pool2-54e21b2f-hlhd  Error: ImagePullBackOff

Then you're missing scopes in your GKE Node's.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
resource "google_container_node_pool" "nodepool2" {
  ... 
  node_config {
    machine_type = "n2-standard-2"
    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
  }
  ...
}

Add CJXD UI

1
jx add app jx-app-ui --version=0.1.2

This will make a PR, which, when merged launches a Master Promotion build.

1
jx get activities --watch --filter environment-joost-cjxd-dev

1
jx get build log

1
jx ui -p 8081

Unable To Enable DNS API

1
2
valid: there is a Secret: external-dns-gcp-sa in namespace: jx
error: unable to enable 'dns' api: failed to run 'gcloud services list --enabled --project XXXXXX' command in directory '', output: 'ERROR: (gcloud.services.list) User [XXXXXX857-compute@developer.gserviceaccount.com] does not have permission to access project [XXXXXX] (or it may not exist): Request had insufficient authentication scopes.'

1
2
3
4
valid: there is a Secret: external-dns-gcp-sa in namespace: jx
error: unable to enable 'dns' api: failed to run 'gcloud services list --enabled --project GCP PROJECT B' command in directory '', output: 'ERROR: (gcloud.services.list) User [389413650857-compute@developer.gserviceaccount.com] does not have permission to access project [GCP PROJECT B] (or it may not exist): Request had insufficient authentication scopes.'

Pipeline failed on stage 'release' : container 'step-create-install-values'. The execution of the pipeline has stopped.

1
2
3
oauth_scopes = [
    "https://www.googleapis.com/auth/cloud-platform"
]

Reset Installation

Remove Environment Repos

  • go to git and remove environment repo
  • clear jx boot config folder
  • clear cloud dns config (if used)
  • clear requirements from env repo's if re-used

Clean GCP Resources

Disks

Buckets

Others?

Clean Environment Information

1
./clear-clusters.sh

Clear Local Data

1
rm -rf ~/.jx/

Recreate Git Token

If for some reason the git token is invalid, you can recreate it with the commands below.

1
2
jx delete git token -n github <yourUserName>
jx create git token -n github <yourUserName>

Debug

TLS Configuration

Cert Manager

1
kubectl get pods -n cert-manager

1
kubectl logs -f -n cert-manager cm-cert-manager-885695c9f-bxhvk

External DNS Controller

kubectl logs -f exdns-external-dns-56948bff8-fq692

1
kubectl get ingress -A

1
2
NAMESPACE   NAME                  HOSTS                       ADDRESS         PORTS     AGE
jx          jx-vault-joost-cjxd   vault.dev.cjxd.kearos.net   35.204.54.193   80, 443   5m46s

The address from the ingress resource should match the address returned from the nslookup.

1
nslookup vault.dev.cjxd.kearos.net

1
2
3
4
5
6
Server:     192.168.178.1
Address:    192.168.178.1#53

Non-authoritative answer:
Name:   vault.dev.cjxd.kearos.net
Address: 35.204.54.193

Certificate Resources

  • certificate issuer
    • kubectl get issuer -n jx
  • certificate
    • kubectl get cert -n jx
  • certificate secret
    • kubectl get secret
  • cloud dns secret (credentials.json)

If the certificate is correct, it looks like this:

1
2
NAME                        READY   SECRET                      AGE
tls-dev-cjxd-kearos-net-p   True    tls-dev-cjxd-kearos-net-p   4m31s

Create new Environment

Create CloudBees Environment

  • copy existing resources into new (env/templates/)
  • update values to suit new environment
  • execute jx boot
  • configure cloud dns
  • cloud dns zone
  • external dns secret
  • dns domain forward
  • add cert and issuer to env repo
  • configure secrets
  • add requirements to repo
  • edit exdns-external-dns deployment
  • kubectl edit deployment -n jx exdns-external-dns

Get Resources

1
2
kubectl get env staging -oyaml > env/templates/cb.yaml
kubectl get sr joostvdg-env-cjxd-staging -oyaml > env/templates/cb-sr.yaml

Configure TLS

Create Cloud DNS Zone
1
2
3
ZONE_NAME=cb-cjxd-kearos-net
DESCRIPTION="joostvdg - cb env for cjxd"
DNS_NAME=cb.cjxd.kearos.net

1
gcloud dns managed-zones create ${ZONE_NAME} --description=${DESCRIPTION} --dns-name=${DNS_NAME}
Configure Certificate & Issuer
1
2
kubectl get issuer -n jx letsencrypt-prod -oyaml > env/templates/issuer.yaml
kubectl get cert -n jx tls-dev-cjxd-kearos-net-p -oyaml > env/templates/certificate.yaml
  • Rename the namespace to your namespace
  • remove status segment
  • remove kubernetes managed fields (uuid, timestamps, etc)
1
2
git add env/
git commit -m "add certs"

1
2
git push
jx get activities -w

1
kubectl get namespace
Configure DNS Secrets
1
2
kubectl get secret -n jx exdns-external-dns-token-cq5mv -oyaml > exdns-external-dns-token-env-cb.yaml
kubectl get secret -n jx external-dns-gcp-sa -oyaml > external-dns-gcp-sa-env-cb.yaml
  • Rename the namespace to your namespace
  • remove status segment
  • remove kubernetes managed fields (uuid, timestamps, etc)
1
2
kubectl apply -f exdns-external-dns-token-env-cb.yaml
kubectl apply -f external-dns-gcp-sa-env-cb.yaml

Confirm Certificate Works

1
kubectl get cert -n cloudbees

Add CloudBees Core

env/requirements.yaml

1
2
3
4
- name: cloudbees-core
  version: 3.6.0+4d2e34de1e86
  repository: https://charts.cloudbees.com/public/cloudbees
  alias: core

env/values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
core:
  OperationsCenter:
    HostName: core.cb.cjxd.kearos.net
    Ingress:
      Annotations:
        kubernetes.io/ingress.class: nginx
        kubernetes.io/tls-acme: "true"
        nginx.ingress.kubernetes.io/app-root: https://$best_http_host/cjoc/teams-check/
        nginx.ingress.kubernetes.io/proxy-body-size: 50m
        nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
      tls:
        Enable: true
        Host: core.cb.cjxd.kearos.net
        SecretName: tls-cb-cjxd-kearos-net-p
    ServiceType: ClusterIP
  nginx-ingress:
    Enabled: false

cb env

1

cb source repository

1

jx-requirements

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
- ingress:
    domain: cb.cjxd.kearos.net
    externalDNS: true
    namespaceSubDomain: .
    tls:
      email: joostvdg@gmail.com
      enabled: true
      production: true
  key: cb
  repository: env-cjxd-cb

TLS Unique DNS

If you want to make sure each environment has its own unique address, the external dns controller needs to filter on multiple domains.

Luckily, it is able to do so.

Unfortunately, it seems Jenkins X (with jx boot) doesn't seem to do this out of the box.

To do so, make sure each environment has its own domain, as shown in jx-requirements.

Then, edit the exdns deployment via kubectl edit exdn... and add an additional - --domain-filter= line at the args for each domain.

jx-requirements

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
- ingress:
    domain: staging.gke.kearos.net
    externalDNS: true
    namespaceSubDomain: ""
    tls:
      email: joostvdg@gmail.com
      enabled: true
      production: true
  key: staging
  repository: env-gke-staging
- ingress:
    domain: prod.gke.kearos.net
    externalDNS: true
    namespaceSubDomain: ""
    tls:
      email: joostvdg@gmail.com
      enabled: true
      production: true
  key: production
  repository: env-gke-production
gitops: true
ingress:
  cloud_dns_secret_name: external-dns-gcp-sa
  domain: jx.gke.kearos.net
  externalDNS: true
  namespaceSubDomain: -jx.
  tls:
    email: joostvdg@gmail.com
    enabled: true
    production: true
- ingress:
    domain: staging.gke.kearos.net
    externalDNS: true
    namespaceSubDomain: ""
    tls:
      email: joostvdg@gmail.com
      enabled: true
      production: true
  key: staging
  repository: env-gke-staging
- ingress:
    domain: prod.gke.kearos.net
    externalDNS: true
    namespaceSubDomain: ""
    tls:
      email: joostvdg@gmail.com
      enabled: true
      production: true
  key: production
  repository: env-gke-production
gitops: true
ingress:
  cloud_dns_secret_name: external-dns-gcp-sa
  domain: jx.gke.kearos.net
  externalDNS: true
  namespaceSubDomain: -jx.
  tls:
    email: joostvdg@gmail.com
    enabled: true
    production: true

exdns deployment config

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
    - args:
    - --log-level=info
    - --domain-filter=jx.gke.kearos.net
    - --domain-filter=staging.gke.kearos.net
    - --policy=upsert-only
    - --provider=google
    - --registry=txt
    - --interval=1m
    - --source=ingress
    - --google-project=kearos-gcp

Certmanager complaining about the wrong domain

In case Cert-Manager is complaining that while validating x.y.example.com it cannot find example.com.

See: https://github.com/jetstack/cert-manager/issues/1507

1
2
3
I1104 09:13:33.884549       1 base_controller.go:187] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="cloudbees/tls-cb-cjxd-kearos-net-p-2383487961-0"
I1104 09:13:33.884966       1 dns.go:104] cert-manager/controller/challenges/Present "level"=0 "msg"="presenting DNS01 challenge for domain" "dnsName"="cloudbees.cjxd.kearos.net" "domain"="cloudbees.cjxd.kearos.net" "resource_kind"="Challenge" "resource_name"="tls-cb-cjxd-kearos-net-p-2383487961-0" "resource_namespace"="cloudbees" "type"="dns-01"
E1104 09:13:34.141122       1 base_controller.go:189] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="No matching GoogleCloud domain found for domain kearos.net." "key"="cloudbees/tls-cb-cjxd-kearos-net-p-2383487961-0"

References