Overview
Red Hat Advanced Cluster Management for Kubernetes governance help enhance enterprises to meet internal standards for software engineering, secure engineering, resiliency, security, and regulatory compliance for workloads hosted on hybrid clouds. Advanced Cluster Management provides the out-of-the-box policy templates that enterprises use to ehance and apply the security check, and remediate any violation in your policies.
Furthermore, enterprises integrate third-party controls with Red Hat Advanced Cluster Management for Kubernetes governance by implementing custom policy controllers using the governance policy framework. Check out our related blog for more details: Applying governance capability of Advanced Cluster Management.
In this article, I am going to show you the policy framework architecture for Red Hat Advanced Cluster Management for Kubernetes governance, how to write a custom controller that does custom security checks and deploy it, and show the result through the governance dashboard.
Prerequisites:
- Install Advanced Cluster Management
- Review the Kubernetes documentation and custom resources
- Install the Red Hat operator-sdk
The design of Red Hat Advanced Cluster Management for Kubernetes governance policy framework follows standard Kubernetes development. It consists of following components:
-
Dashboard — The Governance and Risk dashboard that allows you to manage policies and view policy results
-
Policy CRD — A Kubernetes Custom Resource Definition (CRD) that acts as a vehicle to pass embedded policy definition from hub to managed clusters
-
Policy framework controller — A Kubernetes controller that propagates policies, and synchronizes status and results between hub and managed clusters
-
Policy controller — A Kubernetes controller that implements the policy against the security or configuration control, monitor the cluster state against the policy definition, report the result and remediate if possible.
Writing a custom policy controller
In order to write a custom policy controller that can be integrated with Red Hat Advanced Cluster Managment for Kubernetes governance, you need to implement the following flow:
According to the flow diagram, your policy controller needs to have following capabilities:
- Watch and retrieve custom policy CR
- Based on policy retrieved, extract policy definition and call security control APIs to execute the policy
- Retrieve the execution results and update the corresponding policy
- Generate an event on an Advanced Cluster Management policy CR with the remediation results
View the open source policy controller sample, which implements the described flow.
The multicloud-operators-policy-controller
repository provides all the code necessary to integrate with Advanced Cluster Management governance. It also provides a Configuration policy controller to check if your cluster is compliant with the required number of role binding and cluster role binding rules. For more information, please read the adoption guide in the repository.
Create your own policy and controller
You can customize your policy controller by using the policy controller sample, that is provided in multicloud-operators-policy-controller
repository.
-
Clone the
multicloud-operators-policy-controller
repository:git clone git@github.com:open-cluster-management/multicloud-operators-policy-controller.git
-
Customize the policy and controller.
A policy file defines the desired state. To make your own policy, change the policy schema definition. View the following example to change the schema definition. Let’s create a custom policy named
TestPolicy
. The schema file is located in the mentioned repository.Your policy might look like this:
metadata:
name: testpolicies.policies.redhat.com
spec:
group: policy.open-cluster-management.io
names:
kind: TestPolicy
listKind: TestPolicyList
plural: testpolicies
singular: testpolicyUpdate the policy controller to watch for
TestPolicy
kind. Run the following command:for file in $(find . -name "*.go" -type f); do sed -i "" "s/SamplePolicy/TestPolicy/g" $file; done
for file in $(find . -name "*.go" -type f); do sed -i "" "s/samplepolicy-controller/testpolicy-controller/g" $file; doneNow that you updated the controller and policy schema with
TestPolicy
kind, recompile and run the project to see if it really works. To do that, configurekubectl
to the Red Hat Advanced Cluster Management for Kubernetes cluster by completing the following steps:-
Log in to your cluster.
-
Select the user icon, then click Configure client.
-
Copy and paste the configuration information into your command line, and press Enter.
-
Run the following commands to install
TestPolicy
CRD and start the controller:export GO111MODULE=on
kubectl apply -f deploy/crds/policy.open-cluster-management.io_samplepolicies_crd.yaml
export WATCH_NAMESPACE=<cluster_namespace_on_hub>
go run cmd/manager/main.goThe output indicating that a controller called
testpolicy-controller
is running might resemble the following content:{“level”:”info”,”ts”:1578503280.511274,”logger”:”controller-runtime.manager”,”msg”:”starting metrics server”,”path”:”/metrics”}
{“level”:”info”,”ts”:1578503281.215883,”logger”:”controller-runtime.controller”,”msg”:”Starting Controller”,”controller”:”testpolicy-controller”}
{“level”:”info”,”ts”:1578503281.3203468,”logger”:”controller-runtime.controller”,”msg”:”Starting workers”,”controller”:”testpolicy-controller”,”worker count”:1}
Waiting for policies to be available for processing…
Let’s create a policy and verify that the controller retrieves it and applies the policy onto the cluster. Run following command in the new terminal to apply the policy:
kubectl apply -f deploy/crds/policy.open-cluster-management.io_samplepolicy_crd.yaml
Once the policy is applied, a message appears indicating the policy has been detected and executed by your controller:
{"level":"info","ts":1578503685.643426,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
{"level":"info","ts":1578503685.855259,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
Available policies in namespaces:
namespace = kube-public; policy = example-samplepolicy
namespace = default; policy = example-samplepolicy
namespace = kube-node-lease; policy = example-samplepolicyRun the following command to view the policy execution result:
kubectl get testpolicy example-samplepolicy -o yaml
Your output might resemble the following status field:
status:
compliancyDetails:
example-samplepolicy:
cluster-wide:
- 5 violations detected in namespace `cluster-wide`, there are 0 users violations and 5 groups violations
default:
- 0 violations detected in namespace `default`, there are 0 users violations and 0 groups violations
kube-node-lease:
- 0 violations detected in namespace `kube-node-lease`, there are 0 users violations and 0 groups violations
kube-public:
- 1 violations detected in namespace `kube-public`, there are 0 users violations and 1 groups violations
compliant: NonCompliant -
-
Change the policy rules and execution logic.
Let’s update your created policy by changing the policy rules and execution logic. View the following policy rules and execution logic for the sample policy:
maxClusterRoleBindingGroups: 10
maxClusterRoleBindingUsers: 10
maxRoleBindingGroupsPerNamespace: 2
maxRoleBindingUsersPerNamespace: 2Complete the following steps to customize the policy rules and execution logic:
- Add new fields in CRD-file to introduce new rules. For examples to add a new field see, the redhat repository CRD file.
- Update the TestPolicySpec structure in samplepolicy_types.go with new fields.
- Update
PeriodicallyExecSamplePolicies
function insamplepolicy_controller.go
with new logic to execute the policy. - Recompile and run it.
You created a policy controller and policy.
Deploy your controller to the cluster
If you follow the above steps to create your own policy controller, you should now have a custom policy and controller ready to use with Red Hat Advanced Cluster Management for Kubernetes governance.
Complete the following steps to build controller image and deploy to your cluster:
-
Run following command to build the controller image:
make build
docker build . -f build/Dockerfile -t <username>/multicloud-operators-policy-controller:latest -
Push the image to a repository of your choice, e.g. quay.io. Run the following commands:
podman login
podman push <username>/multicloud-operators-policy-controller -
Configure
kubectl
to point to a cluster managed by Red Hat Advanced Cluster Management for Kubernetes. -
Replace the operator manifest to use the built-in image name and update the namespace to watch for policies. The namespace must be the cluster namespace, run the following command:
sed -i "" 's|redhat/multicloud-operators-policy-controller|ycao/multicloud-operators-policy-controller|g' deploy/operator.yaml
sed -i "" 's|value: default|value: <namespace>|g' deploy/operator.yaml -
Update the RBAC role by running the following commands:
sed -i "" 's|samplepolicies|testpolicies|g' deploy/cluster_role.yaml
sed -i "" 's|namespace: default|namespace: <namespace>|g' deploy/cluster_role_binding.yaml -
Deploy the controller to your cluster by running the following commands:
Setup Service Account
kubectl apply -f deploy/service_account.yaml -n <namespace>
Setup RBAC for operator
kubectl apply -f deploy/role.yaml -n <namespace>
kubectl apply -f deploy/role_binding.yaml -n <namespace>Setup RBAC for SamplePolicyController
kubectl apply -f deploy/cluster_role.yaml
kubectl apply -f deploy/cluster_role_binding.yamlSetup the CRD
kubectl apply -f deploy/crds/policies.redhat.com_samplepolicies_crd.yaml
Deploy the multicloud-operators-policy-controller
kubectl apply -f deploy/operator.yaml -n <namespace>
-
Run following command to verify that the the controller is in running state:
kubectl get pod -n <namespace>
-
Look at the log and make sure everything is running correctly by running the following commands:
kubectl logs -f -n <namespace> multicloud-operators-policy-controller-854749c8b8-8v9tp
{"level":"info","ts":1578516495.9202724,"logger":"cmd","msg":"Go Version: go1.13.5"}
{"level":"info","ts":1578516495.9204354,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1578516495.9204445,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
{"level":"info","ts":1578516495.9219334,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1578516498.3849623,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1578516498.4195356,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1578516500.7812169,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1578516500.7819993,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1578516500.7826676,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578516500.787235,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578516505.7244387,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"multicloud-operators-policy-controller-metrics","Service.Namespace":"default"}
{"level":"info","ts":1578516508.310978,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"servicemonitors.monitoring.coreos.com \"multicloud-operators-policy-controller-metrics\" already exists"}
{"level":"info","ts":1578516508.3131533,"logger":"cmd","msg":"Starting the Cmd."}
Waiting for policies to be available for processing...
{"level":"info","ts":1578516508.313996,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1578516508.4145792,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testpolicy-controller"}
{"level":"info","ts":1578516508.5149467,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testpolicy-controller","worker count":1}
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Waiting for policies to be available for processing...
Create and manage policy with the Red Hat Advanced Cluster Management for Kubernetes dashboard
Now your policy controller has been installed and running on your cluster. Next, create the policy and apply it to the cluster using the Red Hat Advanced Cluster Management for Kubernetes console. Complete the following steps to navigate the console:
-
Log in to Red Hat Advanced Cluster Management for Kubernetes console and click Govern risk from the menu navigation. The Governance and risk dashboard appears.
-
Click the Create policy button. Copy and paste your policy into the YAML editor.
Your policy might look like the following content:
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
name: testpolicy
namespace: default
annotations:
policy.open-cluster-management.io/standards: NIST-CSF
policy.open-cluster-management.io/categories: PR.PT Protective Technology
policy.open-cluster-management.io/controls: PR.PT-1 Audit Logging
spec:
complianceType: musthave
remediationAction: inform
disabled: false
namespaces:
exclude: ["kube-*"]
include: ["default"]
policy-templates:
- objectDefinition:
apiVersion: policy.open-cluster-management.io/v1
kind: TestPolicy
metadata:
name: example-samplepolicy
spec:
severity: low
namespaceSelector:
include: ["default","kube-*"]
exclude: ["kube-system"]
remediationAction: inform
maxClusterRoleBindingGroups: 0
maxClusterRoleBindingUsers: 0
maxRoleBindingGroupsPerNamespace: 0
maxRoleBindingUsersPerNamespace: 0
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
name: binding-testpolicy
namespace: default
placementRef:
name: placement-testpolicy
kind: PlacementPolicy
apiGroup: policy.open-cluster-management.io
subjects:
- name: testpolicy
kind: Policy
apiGroup: policy.open-cluster-management.io
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
name: placement-testpolicy
namespace: default
spec:
clusterConditions:
- status: "True"
type: ManagedClusterConditionAvailable
clusterSelector:
matchExpressions:
- {key: name, operator: In, values: ["daunts"]} -
Select the cluster where your policy controller runs by updating the Cluster binding field. Click the drop-down menu and select.
-
Click the Create button. Your created policy appears on the Policies tab. Wait for a few seconds and let policy framework propagate the policy to your managed clusters and collect the results. These results for your clusters violations appear shortly.
-
Click on the policy row and drill down into the details of the policy. Your policy information is displayed from the Details tab.
-
Click on Violations tab to view the violation details.
Now you have successfully created a custom policy, deployed it to your cluster, and checked the results by using the Governance and risk dashboard.
Conclusion
Red Hat Advanced Cluster Management for Kubernetes governance provides an extensible framework for enterprises to introduce their own security policies. By writing a custom policy controller, enterprises are able to integrate existing enterprise security and configuration controls with Red Hat Advanced Cluster Management for Kubernetes governance. Enterprises are able to take advantage of the Governance and risk dashboard to create and manage both out-of-box policies and custom policies, monitor the policy compliance status, and take further remediation actions.
Learn how to contribute your custom policies to the open-cluster-management
community project, and view examples of custom policies and policy controllers in the policy-collection
repository.
Learn to integrate the Open Policy Framework. View the related blog, How to integrate Open Policy Agent with Red Hat Advanced Cluster Management for Kubernetes policy framework.
Categories