Develop your own policy controller to integrate with Red Hat Advanced Cluster Management for Kubernetes

Overview

Red Hat Advanced Cluster Management for Kubernetes governance help enhance enterprises to meet internal standards for software engineering, secure engineering, resiliency, security, and regulatory compliance for workloads hosted on hybrid clouds. Advanced Cluster Management provides the out-of-the-box policy templates that enterprises use to ehance and apply the security check, and remediate any violation in your policies.

Furthermore, enterprises integrate third-party controls with Red Hat Advanced Cluster Management for Kubernetes governance by implementing custom policy controllers using the governance policy framework. Check out our related blog for more details: Applying governance capability of Advanced Cluster Management.

In this article, I am going to show you the policy framework architecture for Red Hat Advanced Cluster Management for Kubernetes governance, how to write a custom controller that does custom security checks and deploy it, and show the result through the governance dashboard.

Prerequisites:

Install Advanced Cluster Management
Review the Kubernetes documentation and custom resources
Install the Red Hat operator-sdk

The design of Red Hat Advanced Cluster Management for Kubernetes governance policy framework follows standard Kubernetes development. It consists of following components:

Dashboard — The Governance and Risk dashboard that allows you to manage policies and view policy results
Policy CRD — A Kubernetes Custom Resource Definition (CRD) that acts as a vehicle to pass embedded policy definition from hub to managed clusters
Policy framework controller — A Kubernetes controller that propagates policies, and synchronizes status and results between hub and managed clusters
Policy controller — A Kubernetes controller that implements the policy against the security or configuration control, monitor the cluster state against the policy definition, report the result and remediate if possible.

Writing a custom policy controller

In order to write a custom policy controller that can be integrated with Red Hat Advanced Cluster Managment for Kubernetes governance, you need to implement the following flow:

govflow

According to the flow diagram, your policy controller needs to have following capabilities:

Watch and retrieve custom policy CR
Based on policy retrieved, extract policy definition and call security control APIs to execute the policy
Retrieve the execution results and update the corresponding policy
Generate an event on an Advanced Cluster Management policy CR with the remediation results

View the open source policy controller sample, which implements the described flow.

The multicloud-operators-policy-controller repository provides all the code necessary to integrate with Advanced Cluster Management governance. It also provides a Configuration policy controller to check if your cluster is compliant with the required number of role binding and cluster role binding rules. For more information, please read the adoption guide in the repository.

Create your own policy and controller

You can customize your policy controller by using the policy controller sample, that is provided in multicloud-operators-policy-controller repository.

Clone the multicloud-operators-policy-controller repository:

git clone git@github.com:open-cluster-management/multicloud-operators-policy-controller.git

Customize the policy and controller.

A policy file defines the desired state. To make your own policy, change the policy schema definition. View the following example to change the schema definition. Let’s create a custom policy named TestPolicy. The schema file is located in the mentioned repository.

Your policy might look like this:

metadata: 
  name: testpolicies.policies.redhat.com
spec:
  group: policy.open-cluster-management.io
  names:
    kind: TestPolicy
    listKind: TestPolicyList
    plural: testpolicies
    singular: testpolicy

Update the policy controller to watch for TestPolicy kind. Run the following command:

for file in $(find . -name "*.go" -type f); do  sed -i "" "s/SamplePolicy/TestPolicy/g" $file; done
for file in $(find . -name "*.go" -type f); do  sed -i "" "s/samplepolicy-controller/testpolicy-controller/g" $file; done

Now that you updated the controller and policy schema with TestPolicy kind, recompile and run the project to see if it really works. To do that, configure kubectl to the Red Hat Advanced Cluster Management for Kubernetes cluster by completing the following steps:

Log in to your cluster.
Select the user icon, then click Configure client.
Copy and paste the configuration information into your command line, and press Enter.

Run the following commands to install TestPolicy CRD and start the controller:

export GO111MODULE=on
kubectl apply -f deploy/crds/policy.open-cluster-management.io_samplepolicies_crd.yaml
export WATCH_NAMESPACE=<cluster_namespace_on_hub>
go run cmd/manager/main.go

The output indicating that a controller called testpolicy-controller is running might resemble the following content:

{“level”:”info”,”ts”:1578503280.511274,”logger”:”controller-runtime.manager”,”msg”:”starting metrics server”,”path”:”/metrics”}
{“level”:”info”,”ts”:1578503281.215883,”logger”:”controller-runtime.controller”,”msg”:”Starting Controller”,”controller”:”testpolicy-controller”}
{“level”:”info”,”ts”:1578503281.3203468,”logger”:”controller-runtime.controller”,”msg”:”Starting workers”,”controller”:”testpolicy-controller”,”worker count”:1}
Waiting for policies to be available for processing…

Let’s create a policy and verify that the controller retrieves it and applies the policy onto the cluster. Run following command in the new terminal to apply the policy:

kubectl apply -f deploy/crds/policy.open-cluster-management.io_samplepolicy_crd.yaml

Once the policy is applied, a message appears indicating the policy has been detected and executed by your controller:

{"level":"info","ts":1578503685.643426,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
{"level":"info","ts":1578503685.855259,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
Available policies in namespaces:
namespace = kube-public; policy = example-samplepolicy 
namespace = default; policy = example-samplepolicy 
namespace = kube-node-lease; policy = example-samplepolicy

Run the following command to view the policy execution result:

kubectl get testpolicy example-samplepolicy -o yaml

Your output might resemble the following status field:

status:
  compliancyDetails:
    example-samplepolicy:
      cluster-wide:
      - 5 violations detected in namespace `cluster-wide`, there are 0 users violations and 5 groups violations
      default:
      - 0 violations detected in namespace `default`, there are 0 users violations and 0 groups violations
      kube-node-lease:
      - 0 violations detected in namespace `kube-node-lease`, there are 0 users violations and 0 groups violations
      kube-public:
      - 1 violations detected in namespace `kube-public`, there are 0 users violations and 1 groups violations
  compliant: NonCompliant

Change the policy rules and execution logic.

Let’s update your created policy by changing the policy rules and execution logic. View the following policy rules and execution logic for the sample policy:
```
maxClusterRoleBindingGroups: 10
maxClusterRoleBindingUsers: 10
maxRoleBindingGroupsPerNamespace: 2
maxRoleBindingUsersPerNamespace: 2
```
Complete the following steps to customize the policy rules and execution logic:
1. Add new fields in CRD-file to introduce new rules. For examples to add a new field see, the redhat repository CRD file.
2. Update the TestPolicySpec structure in samplepolicy_types.go with new fields.
3. Update PeriodicallyExecSamplePolicies function in samplepolicy_controller.go with new logic to execute the policy.
4. Recompile and run it.

You created a policy controller and policy.

Deploy your controller to the cluster

If you follow the above steps to create your own policy controller, you should now have a custom policy and controller ready to use with Red Hat Advanced Cluster Management for Kubernetes governance.

Complete the following steps to build controller image and deploy to your cluster:

Run following command to build the controller image:

make build
docker build . -f build/Dockerfile -t <username>/multicloud-operators-policy-controller:latest

Push the image to a repository of your choice, e.g. quay.io. Run the following commands:
```
podman login
podman push <username>/multicloud-operators-policy-controller
```
Configure kubectl to point to a cluster managed by Red Hat Advanced Cluster Management for Kubernetes.

Replace the operator manifest to use the built-in image name and update the namespace to watch for policies. The namespace must be the cluster namespace, run the following command:

sed -i "" 's|redhat/multicloud-operators-policy-controller|ycao/multicloud-operators-policy-controller|g' deploy/operator.yaml
sed -i "" 's|value: default|value: <namespace>|g' deploy/operator.yaml

Update the RBAC role by running the following commands:

sed -i "" 's|samplepolicies|testpolicies|g' deploy/cluster_role.yaml
sed -i "" 's|namespace: default|namespace: <namespace>|g' deploy/cluster_role_binding.yaml

Deploy the controller to your cluster by running the following commands:

Setup Service Account

kubectl apply -f deploy/service_account.yaml -n <namespace>

Setup RBAC for operator

kubectl apply -f deploy/role.yaml -n <namespace>
kubectl apply -f deploy/role_binding.yaml -n <namespace>

Setup RBAC for SamplePolicyController

kubectl apply -f deploy/cluster_role.yaml
kubectl apply -f deploy/cluster_role_binding.yaml

Setup the CRD

kubectl apply -f deploy/crds/policies.redhat.com_samplepolicies_crd.yaml

Deploy the multicloud-operators-policy-controller

kubectl apply -f deploy/operator.yaml -n <namespace>

Run following command to verify that the the controller is in running state:
```
kubectl get pod -n <namespace>
```

Look at the log and make sure everything is running correctly by running the following commands:

kubectl logs -f -n <namespace> multicloud-operators-policy-controller-854749c8b8-8v9tp
{"level":"info","ts":1578516495.9202724,"logger":"cmd","msg":"Go Version: go1.13.5"}
{"level":"info","ts":1578516495.9204354,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1578516495.9204445,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
{"level":"info","ts":1578516495.9219334,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1578516498.3849623,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1578516498.4195356,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1578516500.7812169,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1578516500.7819993,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1578516500.7826676,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578516500.787235,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578516505.7244387,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"multicloud-operators-policy-controller-metrics","Service.Namespace":"default"}
{"level":"info","ts":1578516508.310978,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"servicemonitors.monitoring.coreos.com \"multicloud-operators-policy-controller-metrics\" already exists"}
{"level":"info","ts":1578516508.3131533,"logger":"cmd","msg":"Starting the Cmd."}
Waiting for policies to be available for processing...
{"level":"info","ts":1578516508.313996,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1578516508.4145792,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testpolicy-controller"}
{"level":"info","ts":1578516508.5149467,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testpolicy-controller","worker count":1}
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...

Create and manage policy with the Red Hat Advanced Cluster Management for Kubernetes dashboard

Now your policy controller has been installed and running on your cluster. Next, create the policy and apply it to the cluster using the Red Hat Advanced Cluster Management for Kubernetes console. Complete the following steps to navigate the console:

Log in to Red Hat Advanced Cluster Management for Kubernetes console and click Govern risk from the menu navigation. The Governance and risk dashboard appears.

Click the Create policy button. Copy and paste your policy into the YAML editor.

Your policy might look like the following content:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: testpolicy
  namespace: default
  annotations:
    policy.open-cluster-management.io/standards: NIST-CSF
    policy.open-cluster-management.io/categories: PR.PT Protective Technology
    policy.open-cluster-management.io/controls: PR.PT-1 Audit Logging
spec:
  complianceType: musthave
  remediationAction: inform
  disabled: false
  namespaces:
    exclude: ["kube-*"]
    include: ["default"]
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: TestPolicy
        metadata:
          name: example-samplepolicy
        spec:
          severity: low
          namespaceSelector:
            include: ["default","kube-*"]
            exclude: ["kube-system"]
          remediationAction: inform
          maxClusterRoleBindingGroups: 0
          maxClusterRoleBindingUsers: 0
          maxRoleBindingGroupsPerNamespace: 0
          maxRoleBindingUsersPerNamespace: 0
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: binding-testpolicy
  namespace: default
placementRef:
  name: placement-testpolicy
  kind: PlacementPolicy
  apiGroup: policy.open-cluster-management.io
subjects:
- name: testpolicy
  kind: Policy
  apiGroup: policy.open-cluster-management.io
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: placement-testpolicy
  namespace: default
spec:
  clusterConditions:
  - status: "True"
    type: ManagedClusterConditionAvailable
  clusterSelector:
    matchExpressions:
      - {key: name, operator: In, values: ["daunts"]}

Select the cluster where your policy controller runs by updating the Cluster binding field. Click the drop-down menu and select.
Click the Create button. Your created policy appears on the Policies tab. Wait for a few seconds and let policy framework propagate the policy to your managed clusters and collect the results. These results for your clusters violations appear shortly.
Click on the policy row and drill down into the details of the policy. Your policy information is displayed from the Details tab.
Click on Violations tab to view the violation details.

Now you have successfully created a custom policy, deployed it to your cluster, and checked the results by using the Governance and risk dashboard.

Conclusion

Red Hat Advanced Cluster Management for Kubernetes governance provides an extensible framework for enterprises to introduce their own security policies. By writing a custom policy controller, enterprises are able to integrate existing enterprise security and configuration controls with Red Hat Advanced Cluster Management for Kubernetes governance. Enterprises are able to take advantage of the Governance and risk dashboard to create and manage both out-of-box policies and custom policies, monitor the policy compliance status, and take further remediation actions.

Learn how to contribute your custom policies to the open-cluster-management community project, and view examples of custom policies and policy controllers in the policy-collection repository.

Learn to integrate the Open Policy Framework. View the related blog, How to integrate Open Policy Agent with Red Hat Advanced Cluster Management for Kubernetes policy framework.

Blog

Develop your own policy controller to integrate with Red Hat Advanced Cluster Management for Kubernetes

Overview

Writing a custom policy controller

Create your own policy and controller

Deploy your controller to the cluster

Create and manage policy with the Red Hat Advanced Cluster Management for Kubernetes dashboard

Conclusion

Keep reading

Ask an OpenShift Admin Office Hour - Installation methods redux

Creating a Two Disk RHEL8 VM with an Immutable OS Disk

Connecting managed clusters with Submariner in Red Hat Advanced Cluster Management for Kubernetes