Jenkins X Workshop
Created by
Joost van der Griendt
Based on the work of
Jenkins X
and
Viktor Farcic
## whomai
### Joost van der Griendt * Work for CloudBees * Professional Services Consultant EMEA * Hail from Utrecht, The Netherlands * Call me `Jay`
### background * Java developer * CI/CD * Jenkins / Nexus / Sonar * Docker / Docker Swarm / Kubernetes
### Kenny Younger * Work for CloudBees * Technical Success Manager U.S.A. * Recently moved to Fort Worth, TX
### background * Software Engineer * Spring Boot / Apache Camel / BDD * OSS Jenkins (on Windows!)
This presentation
Workshop material
TL;DR
## It's become difficult * Maybe not rocket science * But no longer trivial * Yet, it should be simple...
> You do not need to understand Kubernetes to use Jenkins X. Jenkins X allows us to harness the power of Kubernetes without spending an eternity learning the ever-growing list of the things it does.
My Journey to JX
## The Beginning * Jenkins * Nexus * Sonar
## Evolution * VM's & Bash * VM's & Ansible * CoW * Fabric8 * Manatee & CIDC * Jenkins X
SDLC
## SDLC **S**oftware **D**evelopment **L**ife**C**ycle * People * Process * Technology > From inception till it is OUT of Production.
## SDLC * Self-Service Dev Platform * CI/CD, Agile, DevOps, DevSecOps * Observability * (Best) Practices * Developer Productivity
CICD
## CI/CD * Continous Integration * Continous Delivery * Continous Deployment * Continous everything * Progressive Delivery * GitOps...
### 10 Commandments * be agile * refactor * educate everyone * small teams / codebase * TDD
### 10 Commandments * everything-as-code * fast * pipeline = production * run build before commit * trunk based development
## Today's requirements * It needs to be cloud-native * It needs to be cloud-agnostic * It needs to scale from zero to how much I need * It needs to follow the everything-as-code principle * Everything needs to integrate with everything * It needs to be fast * It needs to "just work"
### How * adopt containerization * adopt schedulers * adopt serverless * re-evaluate tools * use tools for your workflow * batteries included but swappable
### How exactly? * Kubernetes * Jenkins X (Serverless) * GitOps (JX Or Prow) * Helm (or drop-in replacement) * Extend to suit your needs
## Pipelines
Kubernetes - Intro
## Kubernetes, what? > Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
## Kubernetes, why? * Built on ops experience * Modular * API first * Standardization * Container Orchestration * Proven
## Even if you don't need scaling * explicit resource reqs * auto-restarts * homogenous deployment * homogenous observability * homogenous extension
## Kubernetes, How? * Declarative instead of Imperative * Self-healing * High-Availability (HA) * Dynamic sizing & scaling * Immutable * Separate state from process * 12-factor app/container
## Kubernetes, How? * Standard but Extensible * Better utilization * Serverless even better * but more expensive for _predictable load_ * Dynamic Service Discovery * Batteries included but swappable * Layers of Abstraction
### Cluster thinking * Pets vs Cattle * Declarative * Infrastructure As Code++ * Immutable * Separate state from process * Observability = function of cluster
## Kubernetes Basics * Workloads * Services * Networking * ResourceManagement * Storage * LifeCycle * Scheduling
## Workloads * Pod * DaemonSet * ReplicaSet * Deployment * StatefulSet * (Cron)Jobs
Kubernetes Part I
## Pod > A Pod is the basic building block of Kubernetes–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents a running process on your cluster.
### Pod * Basic building block * Use (only) with other abstractions * Fungable * 1 Pod = 1 IP
## DaemonSet > A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them.
### DaemonSet - When * cluster storage, such as glusterd, ceph, on each node * logs collection, such as fluentd or logstash * monitoring...
## ReplicaSet > A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time
### ReplicaSet * Guarantees x-amount of pods running * Generally not used directly * Building block of Deployment
## Job > A Job creates one or more Pods and ensures that a specified number of them successfully terminate.
## CronJob > One CronJob object is like one line of a crontab (cron table) file. It runs a job periodically on a given schedule, written in Cron format
## Labels > Labels are key/value pairs that are attached to objects, such as pods.
## Labels > Labels can be used to organize and to select subsets of objects.
## Labels * Example: pod has label 'app=myapp' * `kubectl get pods -l app=myapp` * Services & Deployments map to Labels * In this case, we call it a: selector
## Labels ```YAML spec: selector: matchLabels: app: jenkins ```
## Service > A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is determined by a Label Selector.
### Service * Routes communication between pods * Routes communication between external and pod * Has its own DNS name * Registers its connected pods to an endpoint
## Deployment > You describe a desired state in a Deployment object, and the Deployment controller changes the actual state to the desired state at a controlled rate.
### Deployment * The workhorse of Kubernetes * Manages ReplicaSet (& Pod) * Generally combined with a Service * Abstraction of choice to build on-top
## Ingress > An API object that manages external access to the services in a cluster, typically HTTP. > Ingress can provide load balancing, SSL termination and name-based virtual hosting.
### Ingress * LoadBalancer for routing * Can create LB's via CloudController * Several ways of using * Several implementations
### Ingress - Ways of using * NodePort / LoadBalancer (Public Cloud Provider) * Path based * Host (Domain) based * Default
### Ingress - Implementations * Nginx Ingress * Traefik * Gloo * others...
Kubernetes Part II
## Kubernetes Architecture
## Controllers > Controller can create and manage multiple Pods for you, handling replication and rollout and providing self-healing capabilities at cluster scope.
## ConfigMaps > ConfigMaps bind configuration files, command-line arguments, environment variables, port numbers, and other configuration artifacts to your Pods' containers and system components at runtime.
### ConfigMaps * Separate config from Pods * Update config separately * Allows overriding container defaults * Do **not** use for confidential data!
### ConfigMaps * from files * from literals * from environment files
## Secrets > A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key.
### Secrets * from literal * usename & password * from file * docker pull credentials * certificates
## Namespaces > Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.
### Namespaces * there is always a 'default' * there are a few namespaces by default * helps separate users or environments * is part of K8S DNS structuring
### Namespaces * is part of your **context** * there are tools who help with this * including jx
## Securing Kubernetes * Users * ServiceAccounts * Roles * ClusterRoles * RoleBinding * ClusterRoleBinding
## ResourceManagement * Memory * CPU * Requests & Limits * QoS: best effort, guaranteed, burst
Kubernetes Part III
## Storage * Volumes * Persistent Volumes * Persistent Volume Claim * Persistent Volume Claim Template
### Volume * StorageClasses * Provisioners * Types of volumes
### Volume Types * emptyDir * configMap * local * nfs * [many more](https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes)
## StatefulSet > Like a Deployment , a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods.
### StatefulSet * used for stateful applications * wraps around ReplicaSet * uses volumes differently * manages pods differently
GitOps
## GitOps > *GitOps* is a way to do Continuous Delivery. It works by using Git as a single source of truth for declarative infrastructure and applications. With Git at the center of your delivery pipelines, developers can make pull requests to accelerate and simplify application deployments and operations tasks to Kubernetes. - WeaveNet
### Principles * entire system described declaratively * canonical desired system state versioned in Git * approved changes can be applied automatically * agents to ensure correctness now and over time
### What it comes down to * Git == Single Source of Truth * Changes are managed via Git * Effectuated by a pipeline listening to Git * Asynchronous, Decoupled and Explicit * Audit log & rights
Jenkins X - Intro
> You do not need to understand Kubernetes to use Jenkins X. Jenkins X allows us to harness the power of Kubernetes without spending eternity learning the ever-growing list of the things it does.
## Jenkins X - Introduction * What is Jenkins Jenkins X * Environments * Pipelines * Differences with Jenkins * Core components
### What is Jenkins X * Opinionated * Re-imagining how to do CI/CD * CLI * Kubernetes first * Based on standards
### Opinions * Use Git & Kubernetes * Trunk Based Development + PR's * You don't need a UI * Use Kubernetes, not understand it * Leverage existing tools
### CLI * Wrapper for Cloud CLI's * Wrapper for Kubernetes tools * Manage Jenkins X installation * Pipeline automation
### Kubernetes Native * Use Kubernetes primitives * Extended with CRD's * Git & Kubernetes is all you need * Following evolving practices/eco system
### Based on Standards * Use (defacto) standards * Use eco system's most used tools * Working with CDF to create new standards
### Environments * dev = your machine + jx namespace * staging * production * preview * on-demand-just-for-test
### Pipelines * as code -> Jenkinsfile * containerized * pipeline/multi-branch/github org * best practices
### Pipeline - Best Practices * use shell * able to run locally * declarative format * jx step
### Differences with Jenkins * Only runs on Kubernetes * Opinionated * Does not rely on plugins * CLI / API first * More than CI * End-to-End system * Can be serverless * Works out-of-the-box
### Core components * jx CLI * Helm * Kubernetes * CRD's & Namespaces * Docker Registry * Chartmuseum & Monocular * Nexus * Skaffold * Draft & BuildPacks * Jenkins
### Practices * Loosely-coupled Architectures * Self-service Configuration * Automated Provisioning * Continuous Build / Integration and Delivery * Automated Release Management * Incremental Testing * Infrastructure Configuration as Code * Comprehensive configuration management * Trunk based development and feature flags
Jenkins X - Install
## Jenkins X - Install * from scratch * via Terraform or cloud CLI * in existing cluster * GitOps for JX (experimental)
### Install Process * install missing CLI's * install cloud provider CLI * create Kubernets cluster * create `jx` namespace * install Tiller (optional) * setup ingress controller * configure Git repositories * configure secrets * install tools via Helm
### Install Configuration * disable Nexus * disable Chartmuseum * alternative Jenkins image * alternative Docker Registry * alternative Git provider
## Jenkins X - Go demo * create JX managed application * deploy to staging * update via PR * preview the PR * merge the PR and update staging * promote to production * Static Jenkins!
## Jenkins X - Import * add to git repo * create remote git repo * add any missing files from Build Pack * register webhook in remote repo * add git repo to your Jenkins * trigger the first pipeline
## Import gotchas * can ignore draft via `--no-draft` * can import GitHub org via `jx import --github...` * sets default branch patterns * override via `--branches "abc|xyz.*|.." * and configure via `jx edit branchpattern`
### Import demo * fork an existing application * import it into Jenkins X * make it work
## Jenkins X - Build Pack * there are standard packs * JX team packs * community packs * creating your own requires some knowledge beyond `using jx` * helm * draft * `jx` manages Build Packs for you * in `.~/.jx/draft/packs/`
### What's inside a Build Pack * Dockerfile * Jenkinsfile * Helm Chart (in `charts/${APP_NAME}` folder) * Preview Helm Chart (in `charts/preview`)
### When to create a Build Pack * only include hard requirements * leave optional things out * focus on the repeatable case * 1, 2, many
### Create a BuildPack * what will every micronaut require? * redis? maybe not * different dockerfile? yes * different health check endpoint? yes
### Create Build Pack demo * take lessons learned from Import Demo * look at standard packs * copy existing * adjust to fit our need * re-import our forked app * see if it works
## Explore .jx folder * cloud-environments * environments * jenkins-x-versions * jx.lock * adminSecrets.yaml * chartmuseumAuth.yaml * gitAuth.yaml * jenkinsAuth.yaml * draft * plugins
SDLC Continued
## SDLC Continued * Versioning * Package & Publish * Workflows
### Versioning * SemVer * Automation * jx-release-version * Explicit Dependencies
### SemVer - Old * Major * Minor * Patch
### SemVer - CICD * Major * Minor * Build
### Package & Publish * No rebuilding * Signify release candidate * Explicit Dependency Management * Inspection before deployment * Inspection later...t
### Workflow * commit * build * PR * preview * review * merge * build * promote
The X-Files
## Mulder * fork & clone * import into jx * fix redis config * fix health check endpoint * add unit tests to pipeline * add integration tests to pipeline * test PR * test staging & production
## Scully * fork & clone * jx import * fix Dockerfile * add Mulder as dependency * helm repo stuff * fix url to Mulder * click on Scully's "voice box" to get a quote...
## Jenkins X - Dev Workflow * ??
## Jenkins X - Build With Kaniko * why?
K8S Ecosystem
## Kubernetes Ecosystem * Docker * Helm * Knative * Skaffold * Chartmuseum * Tekton
## Docker
## Helm * package manager for Kubernetes * quivalent to `yum install
` * Kubernetes manifest template (Go templates) * packaged and versioned, reffered to as `Charts`
### Helm - Use Cases * manages packages, repo and installs * manage dependencies to other Charts * allows for customizing installations * used as building blocks in other tools * separate packaging (Chart) from runtime (application image) * there's the chart version * and the application version
## Knative
## Skaffold
## Chartmuseum * lightweight Helm chart repository * alternatives are Artifactory, Nexus 3 * supports different storage options * local (e.g., Kubernetes volume) * S3 / Minio * Google Cloud Storage * Azure Blob Store * Alibaba Cloud OSS storage * Openstack Object Storage
## Monocular * UI for Helm repositories * includes documentation and searching * insight into charts installed via helm
## Tekton
K8S Advanced
## Kubernetes Advanced * Scheduling * Policies * Admission Controllers * Customization (CRD's) * RBAC * Platformization
## ...
Serverless Jenkins
## Jenkins X - Serverless
## Jenkins X - Extensions
## Further steps
## Resources & Credits * [Viktor Farcic](https://vfarcic.github.io/#/cover) * [exoscale.com - CI](https://www.exoscale.com/static/syslog/2018-11-08-what-is-ci/what-is-continuous-integration.png) * [WeaveNet - What is GitOps](https://www.weave.works/technologies/gitops/#what-is-gitops)