Runtime Code profile for Kubernetes Operators

Abstract

The technique described in this document allows you to create instrumented operators and deploy them in a Kubernetes environment. It also allows you to collect the code coverage based on end-to-end tests on a composition of operators rather than on a single operator. For the simplicity of the example, we will apply this technique to the operator.

Introduction

Often, you are asked to have a code coverage above a given percentage as you are struggling to write unit tests. At the same time, you are asked to create functional tests and end-to-end tests, but you don’t have any coverage reports on them.

Unit tests are important, but at the end of the day, we are creating operators that provide functionalities tested by the implemented functional tests. All these operators are packaged together to build a product on which the end user will play a number of scenarios emulated by the end-to-end tests. These make functional tests and end-to-end tests more critical.

We can easily generate test profile reports on unit tests but not yet on functional tests and end-to-end tests as they are often disconnected from the operator. This means the tests will send CRs to the operator and not call some of its methods, which is what this presentation is about. This helps you generate profile reports on functional tests and end-to-end tests and so increases the test code coverage percentage using the same technology. Here we will focus on the code coverage profile as other profiles such as CPU and mem are provided by the operator metrics functionality.

Tests coverage is not all that is necessary, as the tests themselves must check if each result of each test is accurate, but it helps to focus the development of tests where it is most needed.

Operator-sdk: already has the capability to run the operator locally and with tests implemented in the same project. In this sample, we will also use Ginkgo to implement our end-to-end tests that will not have any dependencies on the operator.

Implementation Constraints and Challenges

The are challenges to be solved such as:

1. "Operator-sdk test" doesn't provide a way to generate instrumented binary: "go test" will be used.
2. “Go test” generates the reports only when the tests are done: Because the operator is a long-running process (listener), we will have to delete the tested pod or deployment at the end of the test.
3. The report is generated in the container: A volume will be used to store the file. We can not use kubectl cp because the pod will be dead by the time the kubectl cp runs. Also some images (like ubi-minimal) don’t have tar installed, and so kubectl cp can’t be used. If you use KiND, then you will have to provide a configuration file to map a volume in KiND with the host platform.
4. The operator is embedded in an image: We will modify the Dockerfile and entrypoint in order to instrument and run the operator with “go test.”

Process

The process is divided in three main phases:

1. Instrument and package the operator
2. Deploy and run the operator
3. Analyze the profile

1. Instrument and Package the Operator

“Go test” offers a way to generate code profile reports based on defined tests and often so called unit tests, but it can do more. In fact, it generates profile reports on all code visited during the test and we can launch a “go test” on the main() function.

“Go test” also offers the possibility to generate instrumented code, which will contain the functionality to accumulate the code profile data during the execution and generates the code profile report at the end.

1.1 Add the Main Test Method

You have to only add one file in your source code called main_test.go next to your current main.go:

cmd/main_test.go:

// +build testrunmain

package main

import (
   "testing"
)

func TestRunMain(t *testing.T) {
   main()
}

1.2 Build the Instrumented Binary:

Build instrumented binary: Usually the operator binary is build using operator-sdk build $IMAGE README.md Here, we will build the binary with the go test and so create a new Dockerfile Dockerfile-profile where the standard command:

COPY build/_output/bin/memcached-operator ${OPERATOR}`

will be replaced by:

go test -covermode=atomic -coverpkg-github.com/open-cluster-management/endpoint-operator/pkg/... -c -tags testrunmain ./cmd/manager -o build/_output/manager

The coverpkg parameter lists the packages for which the profile report must be done.

The -c requests the go test to create a binary instead of running the test.

The -tags mentions the packages that must be built for that operator.

The -o requests to generate a binary called manager since by default the generated binary name is the concatenation of the package name and .test.

The $IMAGE will be set with an extension -profile to avoid overwriting the production image.

2. Deploy and Run the Operator

2.1 Entrypoint

Usually the entrypoint is:

exec ${OPERATOR} $@

and a new entrypoint-profile will be created with:

exec ${OPERATOR} -test.run “^TestRunMain$” -test.coverprofile=/tmp/profile/$HOSTNAME=`date +%s%N`.out $@

P.S.: You can add more profiles such as cpu, mem, and block. Run go help test to see the parameters. Also check pprof to learn more about the available reports for these profiles.

The test.run specifies the test that needs to run and here: “^TestRunMain$”.

The test.coverprofile specifies the file where the profile output must be sent. The file name is built with the time in milliseconds to make it unique and so make sure we generate a new file at each pod restart.

2.2 Deployment

The operator.yaml must be updated to be customized to add the volume, securityContext... For that we will use the kustomize capability of kubectl.

An overlays/operator.yaml will be created, which will overlay the existing deploy/operator.yaml by:

• adding the securityContext
• adding the volumes and volumeMounts
• Emptying the commands to make sure the entrypoint will be used

Two extra files will be added to enusre the customization is working:

• overlays/kustomize.yaml
• deploy/kustomize.yaml

The deployment itself will be done using kubectl apply -k overlays instead of kubectl apply -f deploy/operator.yaml

In this example, we use KiND as cluster with this configuration file: build/kind-config/kind-config.yaml

2.3 Run the Operator

Use the following targets to run the operator:

• make create-cluster to create the KiND cluster.
• make install-profile to install the memcached

2.4 Run your Tests

Once the operator is deployed, run your tests.

2.5 Stop the Operator and Get the Profile

To get the profile, we must stop the pods. Here we will remove the memcached but stopping the pod will have the same effect and generate the profile file.

• make uninstall-profile to uninstall the memcached.
• make delete-cluster to delete the cluster.

The profile file will be created in the profile directory.

3. Analyze

The standard go tools can be used to generate the html or extract the profile percentage.

You can run make generate-profile for that.