SRE Life: Helpful Pointers for Debugging OpenShift

Red Hat is the only company to offer self-managed and fully-managed Kubernetes on all major public clouds, which enables our customers to bring their applications to a secure, open hybrid cloud regardless of geographic or staffing restrictions.

Red Hat OpenShift Dedicated is a fully-managed OpenShift service on AWS and Google Cloud, operated and supported by Red Hat, backed by a 99.9% SLA and 24x7 Red Hat Premium Support. OpenShift Dedicated is managed by Red Hat Site Reliability Engineers (SRE), who have years of security and operational experience working with OpenShift in development and production.

Introduction

A day in the life of a Red Hat Site Reliability Engineer (SRE) involves managing a large scale of Red Hat OpenShift Dedicated clusters which requires in-depth knowledge of several technologies and cloud platforms. This is further covered in great detail in Bringing Openshift Dedicated to Life at Red Hat blog.

We often come across complex issues that we need to debug and troubleshoot. With regard to this, one of the ways to debug an issue is to use the oc client itself, which would help us debug the pods and nodes in a OpenShift cluster.

This blog intends to cover certain implementation depth, differences in debugging the pod and node resources, and the auditing done by the command. We will start with understanding what the command is about and then dive more into the details further on.

What Is oc debug?

oc is a client utility that helps to develop, build, deploy, and run applications on any OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

Likewise, debug is a subcommand of oc which helps to launch a command shell to debug a running application or even a node in the cluster to debug an issue. So this command will create an exact replica for the pod that you want to debug and give you a command shell to debug the pod further.

For more details on using this command, you can refer the help manual using command:

$ oc debug --help

Suppose you would like to debug a pod named 'my-example-pod'. You can simply run the below command and start debugging:

$ oc debug pod/my-example-pod

The command can be used to debug a node as well by simply changing the resource to be debugged as a node type. The below command debugs a node named 'node.example.com':

$ oc debug node/node.example.com

Let us go a step further into what is exactly happening when you run the above commands to debug.

General Flow of Command

In order to understand what a command does in the background, we can simply increase the logging verbosity level of the command and go through its details. You can gradually increase the log level to understand what happens in the background, but for now, we will look at the command log with a verbosity level of 6.

For easy understanding and better readability, only the core API calls are referred to for now. For details, you can increase the verbosity level further and can refer to the code.

Let us start with only the core API events in debugging a normal unprivileged pod. The following debug command is run for a pod named 'my-example-pod':

$ oc debug pod/my-example-pod -v=6

1. Check if the pod that is to be debugged exists or not.

    GET /api/v1/namespaces/my-example-app/pods/my-example-pod
   

2. The new debug pod to be created needs to be an exact replica of the pod that is associated with the my-example-app deploymentconfig in this example. So the command will fetch further details from it.

    GET /apis/apps.openshift.io/v1/namespaces/my-example-app/deploymentconfigs/my-example-app
   

3. The deploymentconfig has the image defined as an imagestream so will fetch image details from it which will be used to create the new debug pod.

    GET /apis/image.openshift.io/v1/namespaces/my-example-app/imagestreamimages/my-example-app@sha256:7b0c152c8ef1753c64776bff4678cc201ca546699f6af0c2d816945c06f2591d
   

4. For now, enough details are gathered that are prerequisites to create a new debug pod. So a new debug pod is created with the same name of the pod but with a “-debug” suffix.

    POST /api/v1/namespaces/my-example-app/pods
   

5. Now that the new debug pod is created, we need to be notified of any changes to it, so will start to “watch” it.

    GET /api/v1/namespaces/my-example-app/pods?fieldSelector=metadata.name%3Dmy-example-pod-debug&resourceVersion=2157446&watch=true
   

6. Lastly, we need to have the shell of the debug pod to be available, so a tty with stdin and stdout is attached to the container named ‘my-example-app’ in the debug pod.

    POST /api/v1/namespaces/my-example-app/pods/my-example-pod-debug/attach?container=my-example-app&stdin=true&stdout=true&tty=true
   

7. Once we are done with the debugging and exit from the debug pod’s shell, the debug pod is automatically deleted by default.

    DELETE /api/v1/namespaces/my-example-app/pods/my-example-pod-debug
   

Now, let us go through the same for a node this time. For a node with name “node.example.com”, the following are the events:

$ oc debug node/node.example.com

1. Check if the node to be debugged exists or not.

    GET /api/v1/nodes/node.example.com
   

2. In order to create a debug pod to debug the node, the imagestreamtag ‘tools:latest’ is referred to from ‘openshift’ namespace by default, which will be used to create the debug pod. This defaults to ‘registry.redhat.io/rhel8/support-tools’ image with oc client version 4.6 and above or ‘registry.redhat.io/rhel7/support-tools’ with oc client version 4.5 and below.

    GET /apis/image.openshift.io/v1/namespaces/openshift/imagestreamtags/tools:latest
   

3. Create a debug pod using the respective support-tools image.

    POST /api/v1/namespaces/default/pods
   

4. Like any other pod, we would like to be notified of the changes happening to the pod, so will 'watch' it.

    GET /api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dnodeexamplecom-debug&resourceVersion=627459&watch=true
   

5. Attach a tty with stdin and stdout to the container inside the debug pod, which is named ‘container-00’ by default when debugging a node.

    POST /api/v1/namespaces/default/pods/nodeexamplecom-debug/attach?container=container-00&stdin=true&stdout=true&tty=true
   

6. Delete pod after debugging is done:

    DELETE /api/v1/namespaces/default/pods/nodeexamplecom-debug
   

So in general, the flow can be summarized as follows:

1. Check if the resource to be debugged exists or not.
2. Based on the type of resource to be debugged, evaluate the required image, or use a default image to create the debug pod.
3. Using this image, create a new debug pod with “-debug” suffix.
4. Once the debug pod is created, watch the pod so that any changes to it are notified.
5. Attach a tty with stdin and stdout such that interactive debugging can start.
6. Once the debugging is done, delete the pod.

The above default behavior can be changed and adjusted using any of the additional flags that the oc debug command offers. For details regarding the implementation, you can refer to the code.

Defaults and Comparison

Let us have a high level comparison of the core defaults of debugging a normal (unprivileged) pod and a node. It is to be noted that it is also possible to debug a privileged pod, but the scope here is to compare and debug pod and node which are different resource types.

Let us start with the simplest example to debug a pod called ‘my-example-pod’.

We see in the above console image that a new command shell started where one could start debugging further. As an example, a simple debug command such as id was run to check the user ID of the default user inside the pod. Note that the command output gives a random user ID as per Restricted SCC. This is default for the majority of the pods, which are not required to run in privileged mode.

Now let us debug a node called 'node.example.com'.

So we see that the node debugging requires to happen in 'privileged' mode as root. The table below summarizes the key differences between the debug of a pod and a node that we have seen so far:

For more details on Security Context Constraint(SCC) on OpenShift, you can also refer to this blog.

What SELinux Context Is Used by a Privileged Debug Pod?

When a debug is done for a privileged pod like SDN, OVS pods, and others, or a debug is done for a node (after running chroot /host), the SELinux context set for the root user is system_u:system_r:spc_t:s0.

So what does the SELinux context system_u:system_r:spc_t:s0 mean?

As per solution article, the SELinux context spc_t means “Super Privileged Container Type” (SPC). SPCs are the containers that contain software used to manage the host system that the container will be running on. Since these containers could do anything on the system and one does not want SELinux blocking any access, spc_t was made an unconfined domain.

What Happens If a File Is Changed on the Node?

By default in OpenShift 4, the node level configuration is done by Machine Config Operator (MCO). So if anyone were to use oc debug command and change a file on the node, the change will be reported and caught by machine-config-daemon pod if the file is managed by MCO as well. When this happens, the node will be annotated with following:

machineconfiguration.openshift.io/reason: unexpected on-disk state validating
machineconfiguration.openshift.io/state: Degraded

Additionally, the machine-config-daemon pod logs will show the existing and the expected changes for the file on the node and report "content mismatch for file" error message with details. You can check the logs for the respective node by finding out which machine-config-daemon pod is running on that node.

$ oc get pod -n openshift-machine-config-operator -owide -l k8s-app=machine-config-daemon

As an example, the following console output shows that a file /etc/crio/crio.conf.d/00-default was changed where the crio.metrics configuration was commented but that was not expected.

It is to be noted that MCO will not revert any file content but will only report it and bring it to attention by marking the node in Degraded state. This is explained further here. So it is important to note that if any valid configuration change to a file is to be made on the nodes, then it should be handled through MCO. However, to troubleshoot and to revert such a faulty change on the node to remove node from Degraded state, using oc debug command will be very useful.

How to Audit and Track If Anyone Used oc Debug on a Node?

We saw earlier that when a node is debugged using oc debug, a debug pod is created with a suffix “-debug” appended to the nodename after removing any dots from the name. So as an example, node ‘node.example.com’ will have a debug pod created with the name ‘nodeexamplecom-debug”.

As per documentation, the below command can be used to check the audit events regarding this debug pod:

$ oc adm node-logs --role master --path=/kube-apiserver/audit.log | grep -i nodeexamplecom-debug

Each request would have a unique “audit ID” associated with it. Through the audit logs, you can also find the user information that initiated the debug session for the node. For more details on auditing, please refer to upstream Kubernetes documentation.

Summary

Understanding the oc debug command in detail helps to know the default behavior of the command to troubleshoot an application or a build at a pod level and a machine issue at node level. The command is also flexible enough to help end users provide their custom configuration in debugging any issue but it has its limitations as well.

Considering the traditional approach in OpenShift version 3 to login to a node individually using SSH and changing a file on the node is no longer a recommended approach in OpenShift version 4. In order to keep configuration across a set of nodes to be consistent and avoid fixing individual nodes, the recommended practice in OpenShift version 4 is to use Machine Config Operator (MCO). If anything goes wrong with regard to the node configuration for a specific node, MCO as well as oc debug command can be used to troubleshoot the issue.

From a security standpoint, it is important to understand the defaults of oc debug command, the differences between different Security Context Constraints(SCC) used across the applications, and to understand the audit events generated to track any changes or breach.

Further Reading

• Blog on 'Bringing OpenShift Dedicated to Life at Red Hat'
• Blog on 'A Guide to OpenShift and UIDs'
• Blog on 'Managing SCCs in Openshift'