This process has gotten even easier since this posting! Check out Rob's updated blog: Make a Kubernetes Operator in Helm in 5 steps.
We previously covered how to automate your existing Helm charts with Ansible. Today we’re going to also cover this topic, but using a technology called an Operator. An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling.
This post will walk through making an Operator based on an existing Helm chart to show the value you can get through an Operator, without having to write any Go code, using the Helm Operator kit from the Operator Framework. Afterwards, we are going to compare and contrast this Helm-powered Operator vs a Go-based Operator.
Our Desired User Experience
The goal of this post is to get to a state that is common within many companies: shipping an application or product into many environments in a stable, repeatable and auditable way. Let’s break down each one of these quickly.
A stable deployment process is one that is done in a fully automated fashion once all requirements have been met, whether that requires all tests to pass, or a release manager to give the go-head. It’s “hands off” after that.
A repeatable deployment process is done with a versioned artifact that is immutable once built. Building an application within a container is an easier way of doing this for a single component, but distributed systems are frequently built out of many components. Helm charts can tie these components together, but the chart itself now needs to be versioned. The Helm Operator can bake the chart and other required files into the Operator container image, so that you can have a single versioned representation of the complete distributed system. Using the versioned Operator container in multiple environments allows an engineer to run the same version of an application.
An auditable deployment process is one that fetches assets that it deploys and is executed on a controlled environment, instead of any individual engineer’s laptop. The Helm Operator allows you to use existing Helm charts without the need for a human to run any Helm CLI commands. This can reduce one-off mistakes and provides an audit trail of who/what was deployed.
At a high level view, the process looks as follows. We’ll dig into each below.
- A change to a chart is introduced. For example, a version bump in a component of our application.
- A continuous build process builds the Operator, copying in the new chart into the container.
- A test pipeline deploys the Operator into a new namespace on a test cluster. The test suite uses the new Operator to run the supported configurations of the app against the test suite. For example, if your app supports a regular and highly available configuration, the Operator can be used to spin up both for testing with the new chart. This namespace can be destroyed at the end.
- If the tests pass, a release manager can OK the release for deployment, or trigger it automatically. A continuous deployment system can use the Operator to do a rolling update of the application in place, or do a blue/green style deployment. This process can be repeated for other environments or namespaces where the application is running.
We are going to walk through this process manually, but you can use your imagination for where you’d plug in a service like Quay.io to build your containers and Jenkins/CircleCI/etc to run tests and communicate with your Kubernetes clusters.
Prerequisites: A Kubernetes Environment and Registry
The Helm Operator kit requires a Kubernetes 1.9+ cluster that contains enough resources to run one or two copies of your app. Locally on your machine, you should have
docker installed so we can run an image build, and
kubectl configured to point to your Kubernetes cluster. A familiarity with the usual Helm workflow should improve the clarity of this post.
How the Operator Works
The Helm Operator is designed to manage stateless applications that require very little logic when rolled out. If this doesn’t describe your app, we’ll cover alternative options at the end.
The main function of an Operator is to read from a custom object that represents your application instance and have it’s desired state match what is running. In the case of an Operator generated by the Helm Operator kit, the object’s
spec field is a list of configuration options that are typically described in Helm’s
values.yaml file. Instead of setting these values with flags (eg.
helm install -f values.yaml), we express them within the custom resource, which as a native Kubernetes object, enables the benefits of RBAC applied to it and an audit trail. Here’s an example of a simple custom resource:
replicaCount value, “2” in this case, is propagated into the Chart’s templates where following is used:
After an Operator is built and deployed, deploying a new instance of an app can become as simple as creating a new instance of a custom resource. Need to see the different instances running in all environments? List them with
<code>$ kubectl get Tomcats --all-namespaces</code>
There is no need to use the Helm command-line interface or install Tiller; Operators built with the Helm Operator kit import code from the Helm project. All you need to do is have an instance of the Operator running and register the custom resource with a Custom Resource Definition (CRD). And it obeys RBAC, so you can more easily prevent production changes.
Imagine having a product sign up process that creates a custom resource behind the scenes and a new instance is up and running shortly thereafter.
Building the Operator Image
We believe the best practice is to build a new Operator for each Chart. This can allow for more native feeling Kubernetes APIs (e.g. kubectl get Tomcats) and flexibility if you ever want to write a fully-fledged Operator in Go, migrating away from a Helm-based Operator. The first step is to embed your chart, Kubernetes API Group (e.g. apache.org/v1alpha1), and Kind (e.g. Tomcat) into the Operator’s container image. This can be rather simple, as the Dockerfile exposes these variables as Build Args. Build the Dockerfile, and push it to your registry:
$ export IMAGE=quay.io/<namespace>/tomcat-operator:v0.0.1
$ docker build \
--build-arg HELM_CHART=https://storage.googleapis.com/kubernetes-charts/tomcat-0.1.0.tgz \
--build-arg API_VERSION=apache.org/v1alpha1 \
--build-arg KIND=Tomcat \
-t $IMAGE .
$ docker push $IMAGE
Now we have a versioned artifact that contains our complete chart. We can use this to make repeatable instances of our app.
Deploy the Application
Before deploying an instance of our application, we need to register our CRD with Kubernetes and deploy our Operator.
If you’re developing your own Helm operator, use the Kubernetes manifests here as a starting point. These manifests must be customized in order to contain all the metadata about your new application. Some content is optional, such as an icon, but some content is required. Refer to the table found in the Helm Operator kit’s README for details.
If you want to follow along with the Tomcat example, you can find already customized manifests here. Using kubectl we can apply the customized CRD and Operator manifests:
$ cd examples/tomcat-operator
$ kubectl create -f crd.yaml
$ kubectl create -n <operator-namespace> -f rbac.yaml
$ sed “s|REPLACE_IMAGE|$IMAGE|” operator.yaml.template > operator.yaml
$ kubectl create -n <operator-namespace> -f operator.yaml
If your cluster has the Operator Lifecycle Manager installed, this process is simplified:
$ cd examples/tomcat-operator
$ kubectl create -f crd.yaml
$ sed “s|REPLACE_IMAGE|$IMAGE|” csv.yaml.template > csv.yaml
$ kubectl create -n <operator-namespace> -f csv.yaml
Now we can submit our custom resource, which the Operator is waiting for so that it can create our application:
<span>$ kubectl create -n <operator-namespace> -f cr.yaml</span>
It’s this Custom Resource (CR) YAML file that can be customized and used as a blueprint to stamp out additional instances of our application:
You can imagine how powerful this can be when you have a more complicated chart and have a fully automated deployment pipeline.
Comparing Helm versus a Helm Operator Running
Helm with an Operator is useful because any changes to your custom resources can be picked up immediately. No longer should you have to run Helm CLI commands to modify your applications due to the removal of Tiller from the cluster.
The Helm Operator is designed to excel at stateless applications because changes should be applied to the Kubernetes objects that are generated as part of the chart. This sounds limiting, but can be sufficient for a surprising amount of use-cases as shown by the proliferation of Helm charts built by the Kubernetes community.
Comparing a Helm Operator vs Go Operator
A powerful feature of an Operator is to enable the desired state to be reflected in your application. As your application gets more complicated and has more components, detecting these changes can be harder for a human, but easier for a computer.
The Helm Operator is designed to be simple: it listens for changes to your custom resources and pushes that configuration down to the objects via the chart and the templated values. Because this action is top down, the Operator is not taking a deep look at each individual object field, such as the labels on a specific Pod or a value within a ConfigMap. If one of these is changed manually, the Operator should not overwrite that value with the desired state until the next time the custom resource is changed. Most of the time this should not be an issue, and can be controlled with an RBAC policy.
Complex distributed systems use Kubernetes primitives and certain changes, such as removing labels or changing label selectors, can have disastrous effects on the operation of an app. A Go Operator, built using the Operator SDK, is written with a programming language at your disposal, to help power deeper introspection into not just the custom resource, but the Pods, Services, Deployments and ConfigMaps that make up your app. As an Operator author, you can check how any field from your desired state matches with the running configuration and reset it as part of the Operator’s reconciliation loop. A Go Operator can quickly revert change that is core to the operation of the application.
The Helm Operator kit is a a part of the Operator Framework: an open source toolkit to manage Kubernetes native applications in a more effective, automated, and scalable way. The Framework contains a maturity model for Operators, and provides tools to build Operators with Helm, Ansible and Go as required by your application.
We invite all that are interested in running or building Operators to join the Operator Framework community, including joining the Special Interest Group meetings. Join the mailing list to discuss questions, architectures and experiences with Operators.