Jenkins X with Boot on AKS¶

This guide is about installing Jenkins X via jx boot on AKS, including leveraging Google's CloudDNS for dynamic DNS configuration (for ingress).

Create AKS Cluster¶

Either create a cluster via AKS Terraform (recommended) or via AKS CLI.

Install Jenkins X¶

Boot Config¶

Make a fork of the jenkins-x-boot-config repository and clone it.

GH_USER=

git clone https://github.com/${GH_USER}/jenkins-x-boot-config.git
cd jenkins-x-boot-config

Changes to make:

provider from gke to aks
set domain
set clustername
set external dns (see below)
set repository value for each environments (not dev) as below

- key: staging
  repository: environment-jx-aks-staging

External DNS¶

Using Google CloudDNS:

login to the GCP account you want to use
enable CloudDNS API by going to it
create a CloudDNS zone for your subdomain
- if the main domain is example.com -> aks.example.com
- once created, you get NS entries, copy these (usualy in the form ns-cloud-X{1-4}.googledomains.com
in your Domain's DNS configuration, map your subdomain to these NS entries
create a service account that can use CloudDNS API
add the Google Project to which the Service Account belongs to: jx-requirements.yaml and values.yaml
export the json configuration file
- rename the file to credentials.json
create secret a secret in Kubernetes
- kubectl create secret generic external-dns-gcp-sa --from-file=credentials.json
fix external dns values template -> systems/external-dns/values.tmpl.yaml
- add project: "{{ .Requirements.cluster.project }}" to external-dns.google

Important

You have to create the secret external-dns-gcp-sa in every namespace you set up TLS via the dns01 challenge.

jx-requirements.yaml

We're omitting the default values as much as possible, such as the dev and production environments.

cluster:
environmentGitOwner: <YOUR GITHUB ACCOUNT>
gitKind: github
gitName: github
gitServer: https://github.com
namespace: jx
project: your-google-project
provider: aks
environments:
- ingress:
    domain: staging.aks.example.com
    externalDNS: true
    namespaceSubDomain: ""
    tls:
    email: <YOUR EMAIL ADDRESS>
    enabled: true
    production: true
key: staging
repository: environment-jx-aks-staging
gitops: true
ingress:
domain: aks.example.com
externalDNS: true
namespaceSubDomain: -jx.
tls:
    email: <YOUR EMAIL ADDRESS>
    enabled: true
    production: true
kaniko: true
secretStorage: local

values.yaml

cluster:
    projectID: your-google-project

TLS Config¶

Update the jx-requirements.yaml, make sure ingress configuration is correct:

ingress:
  domain: aks.example.com
  externalDNS: true
  namespaceSubDomain: -jx.
  tls:
    email: admin@example.com
    enabled: true
    production: true

If all is done correctly with the CloudDNS configuration, the external dns will contain all the entries of the jx services (such as hook, chartmuseum) and certmanager will be able to verify the domain with Let's Encrypt.

Docker Registry Config¶

values.yaml

jenkins-x-platform:
    dockerRegistry: myacr.azurecr.io

This was not enough, added it to the values template: env/jenkins-x-platform/values.tmpl.yaml

dockerRegistry: myacr.azurecr.io

TLS For Application In Environment¶

create issuer
create certificate

Note

This implies you need to run jx boot at least once before working on your environment configuration!

Easiest way I found, was to copy the yaml from the issuer and certificate in the jx namespace. You then remove the unnecesary elements, those generated by Kubernetes itself (such as creation date, status, etc).

You have to change the domain name and hosts values, as they should now point to the subdomain corresponding to this environment (unless its production). Once the files are good, you add them to your environment. You do so, by adding them to the templates folder -> env/templates.

.
├── Jenkinsfile
├── LICENSE
├── Makefile
├── README.md
├── env
│   ├── Chart.yaml
│   ├── requirements.yaml
│   ├── templates
│   │   ├── certificate.yaml
│   │   └── issuer.yaml
│   └── values.yaml
└── jenkins-x.yml

kubectl -n jx get issuer letsencrypt-prod -o yaml

kubectl -n jx get certificate tls-<unique to your cluster>-p -o yaml

issuer.yaml

The end result should look like this:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        clouddns:
          project: your-google-project
          serviceAccountSecretRef:
            key: credentials.json
            name: external-dns-gcp-sa
      selector:
        dnsNames:
        - '*.staging.aks.example.com'
        - staging.aks.example.com

certificate.yaml

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
    name: tls-staging-aks-example-com-p
spec:
    commonName: '*.staging.aks.example.com'
    dnsNames:
    - '*.staging.aks.example.com'
    issuerRef:
        name: letsencrypt-prod
    secretName: tls-staging-aks-example-com-p

Last update: 2019-11-25 08:32:26