CloudBees CI - Multi Cluster EKS¶

• https://medium.com/@Joachim8675309/building-eks-with-eksctl-799eeb3b0efd

To create and configure cluster one, we execute the following steps:

1. Create Cluster via EKSCTL
2. Create Route53 Hosted Zone
3. Install Bootstrapping services (certmanager, ingress controller, external dns)
4. Install CloudBees CI

1. Create Cluster via EKSCTL¶

For some tasks, you need create IAM access for the workloads.

There are various ways to do this, I urge you to explore those that are suitable to you.

For simplicity, you can let EKSCTL generate them for you on the node instances. Be careful, as this means any Pod on these nodes can use the access.

For production clusters, you either want Service Account IAM access, or multiple node pools where workloads are restricted to node pools related to their level of required access.

Read here more information on eksctl and IAM policies.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster-jvandergriendt
  region: us-east-1
  version: "1.15"
  tags: 
    CreatedBy: jvandergriendt

nodeGroups:
  - name: ng1-masters
    instanceType: m5.large
    minSize: 3
    maxSize: 8
    volumeSize: 100
    volumeType: gp2
    labels:
      nodegroup-type: masters
    iam:
      withAddonPolicies:
      autoScaler: true
      externalDNS: true
      certManager: true
      ebs: true

availabilityZones: ["us-east-1d", "us-east-1f"]

AWS_PROFILE=cloudbees-iam
AWS_ROLE_ARN=

eksctl create cluster  --config-file cluster.yaml --profile cloudbees-eks

2. Create Route53 Hosted Zone¶

• via the UI: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingHostedZone.html
• Via CLI: https://docs.aws.amazon.com/cli/latest/reference/route53/create-hosted-zone.html

3. Install Bootstrapping services¶

Nginx¶

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

=== "Update Helm repositories

helm repo update

helm install ingress ingress-nginx/ingress-nginx -f ingress-values-c1.yaml

helm install ingress ingress-nginx/ingress-nginx -f ingress-values-c2.yaml

service:
    omitClusterIP: true
controller:
  tcp:
    configMapNamespace: default
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 7
  publishService:
    enabled: true
  metrics:
    enabled: true
tcp: 
  50000: "cloudbees-ci/cjoc:50000"

service:
    omitClusterIP: true
controller:
  tcp:
    configMapNamespace: default
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 7
  publishService:
    enabled: true
  metrics:
    enabled: true

External DNS¶

• https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md
• https://github.com/helm/charts/tree/master/stable/external-dns
• https://github.com/bitnami/charts/tree/master/bitnami/external-dns
• https://github.com/bitnami/charts/tree/master/bitnami/external-dns#tutorials

helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo update

helm install exdns bitnami/external-dns -f exdns-values.yaml

helm install my-release \
  --set provider=aws \
  --set aws.zoneType=public \
  --set txtOwnerId=HOSTED_ZONE_IDENTIFIER \
  --set domainFilters[0]=HOSTED_ZONE_NAME \
  bitnami/external-dns

provider: aws
domainFilters:
  - "eks.kearos.net"
metrics:
  enabled: true
aws:
  zoneType: public
txtOwnerId: HOSTED_ZONE_IDENTIFIER

Apply Policy¶

aws iam put-role-policy  --region us-east-1 --profile cloudbees-eks \
    --role-name ${ROLE} \
    --policy-name eks-jvandergriendt1-ExdnsPolicy \
    --policy-document file://exdns-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1591221999371",
      "Action": [
        "route53:ChangeResourceRecordSets"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Sid": "Stmt1591222034162",
      "Action": [
        "route53:ListHostedZones",
        "route53:ListResourceRecordSets",
        "route53:GetChange"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Certmanager¶

helm repo add jetstack https://charts.jetstack.io

helm repo update

kubectl apply \
   --validate=false \
   -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.crds.yaml

# certmanager expects to run in `cert-manager` namespace
kubectl create namespace cert-manager

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  -f certmanager-values.yaml

kubectl apply -f cluster-issuer.yaml

prometheus:
  enabled: true
ingressShim:
  defaultIssuerName: letsencrypt-prod
  defaultIssuerKind: ClusterIssuer

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: joostvdg@gmail.com
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    # example: cross-account zone management for example.com
    # this solver uses ambient credentials (i.e. inferred from the environment or EC2 Metadata Service)
    # to assume a role in a different account
    - selector:
        dnsZones:
          - "example.com"
      dns01:
        route53:
          region: us-east-1
          hostedZoneID: DIKER8JEXAMPLE 

4. Install CloudBees CI¶

kubectl create namespace cloudbees-ci

helm install cloudbees-ci cloudbees/cloudbees-core \
   -f cloudbees-ci-values.yaml \
   --namespace cloudbees-ci \
    --version 3.14.0+ebfb4625ad50

OperationsCenter:
  HostName: ci.eks.kearos.net
  Ingress:
    Annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    tls:
      Enable: true
      Host: ci.eks.kearos.net
      SecretName: tls-ci-eks-kearos-net-p
  ServiceType: ClusterIP

Prepare Cluster Two Access¶

First, follow step 1, 2 and 3 in the previous chapter but now for the second cluster.

Then, come back and continue with the next steps below.

Create & Prepare Namespace¶

kubectl create namespace masters

helm fetch cloudbees/cloudbees-core --untar

helm template cloudbees-core-masters-masters cloudbees-core \
   --namespace masters \
   -f namespace-masters-values.yaml \
   > namespace-masters.yaml

kubectl apply -f namespace-masters.yaml \
  --namespace masters

OperationsCenter:
    Enabled: false
Master:
    Enabled: true
    OperationsCenterNamespace: masters
Agents:
    Enabled: true

Create Managed Master in both clusters¶

1. Configure Operations Center with Access To EKS Cluster Two
2. Create second Kubernetes endpoint configuration
3. Create Managed Master in Cluster One
4. Create Managed Master in Cluster Two

Configure Operations Center with Access To EKS Cluster Two¶

This is complicated, there are various ways of getting access. These all depend on IAM controls and the policies that exist within your organization.

AWS IAM Authenticator¶

For more information, you can read this PR on the Jenkins Kubernetes Plugin. And here for downloading the aws-iam-authenticator binary for the target platform.

A set of steps that can work is the following:

curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.16.8/2020-04-16/bin/linux/amd64/aws-iam-authenticator

chmod +x ./aws-iam-authenticator

kubectl cp aws-iam-authenticator cjoc-0:/var/jenkins_home/bin

Then, you have to update the StatefulSet of Operations Center to include /var/jenkins_home/bin in the PATH.

You can add the snippet below to the cloudbees-ci.yaml and run a helm upgrade cloudbees-ci ... command (the same as the helm install but replace install with upgrade).

ContainerEnv:
  - name: PATH
    value: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/jenkins_home/bin"

Then, you create a kubeconfig file with the information for the EKS cluster, see below. And add this as a secret file credential to Operations Center's credentials. Use this credential in the Kubernetes Master Provision's Endpoint for Cluster Two.

clusters:
- cluster:
    certificate-authority-data: <redacted>
    server: <redacted>
  name: <redacted>
contexts:
- context:
    cluster: <redacted>
    user: <redacted>
  name: <redacted>
current-context: <redacted>
kind: Config
preferences: {}
users:
- name: <redacted>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      env:
        - name: AWS_ACCESS_KEY
          value: <redacted>
        - name: AWS_SECRET_ACCESS_KEY
          value: <redacted>
      args:
      - token
      - -i
      - eks-cluster-name
      command: aws-iam-authenticator

Create second Kubernetes endpoint configuration¶

Now we create a second Kubernetes Endpoint in the cluster.

The fields to fill in:

1. API endpoint URL: the cluster endpoint of Cluster Two
2. Display Name: display name so you know which cluster you create a Master in
3. Credentials: the credentials you created for IAM access to Cluster Two
4. Server Certificate: the server certificate of Cluster Two
5. Namespace: the namespace you have prepared, in our case masters
6. Master URL Pattern: the DNS pattern for Cluster Two, in our case https://*.two.domain.com
7. Ingress Class: unless you used a different Ingress than in this guide, set to nginx
8. Jenkins URL: the external URL of the Operations Center

echo "kjhKJSDH13123" | base64 -D

Update Master Provisioning¶

• Manage Jenkins -> Configure System -> Kubernetes Master Provisioning -> Advanced -> YAML

apiVersion: "extensions/v1beta1"
kind: "Ingress"
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - ${host}
    secretName: ${secretName}

Cleanup¶

• https://eksworkshop.com/920_cleanup/eksctl/

For each cluster, we can now run the eksctl cleanup.

eksctl delete cluster -f cluster.yaml --wait

AWS::IAM::Role/NodeInstanceRole: DELETE_FAILED – "Cannot delete entity, must delete policies first. (Service: AmazonIdentityManagement; Status Code: 409;

aws iam delete-role-policy --region ${REGION} --profile ${PROFILE} \
    --role-name ${ROLE_NAME} \
    --policy-name ${CLUSTER_NAME}-ExdnsPolicy