Overview

Red Hat Advanced Cluster Management for Kubernetes governance help enhance enterprises to meet internal standards for software engineering, secure engineering, resiliency, security, and regulatory compliance for workloads hosted on hybrid clouds. Advanced Cluster Management provides the out-of-the-box policy templates that enterprises use to ehance and apply the security check, and remediate any violation in your policies.

Furthermore, enterprises integrate third-party controls with Red Hat Advanced Cluster Management for Kubernetes governance by implementing custom policy controllers using the governance policy framework. Check out our related blog for more details: Applying governance capability of Advanced Cluster Management.

In this article, I am going to show you the policy framework architecture for Red Hat Advanced Cluster Management for Kubernetes governance, how to write a custom controller that does custom security checks and deploy it, and show the result through the governance dashboard.

Prerequisites:

The design of Red Hat Advanced Cluster Management for Kubernetes governance policy framework follows standard Kubernetes development. It consists of following components:

  • Dashboard — The Governance and Risk dashboard that allows you to manage policies and view policy results

  • Policy CRD — A Kubernetes Custom Resource Definition (CRD) that acts as a vehicle to pass embedded policy definition from hub to managed clusters

  • Policy framework controller — A Kubernetes controller that propagates policies, and synchronizes status and results between hub and managed clusters

  • Policy controller — A Kubernetes controller that implements the policy against the security or configuration control, monitor the cluster state against the policy definition, report the result and remediate if possible.

Writing a custom policy controller

In order to write a custom policy controller that can be integrated with Red Hat Advanced Cluster Managment for Kubernetes governance, you need to implement the following flow:

govflow

According to the flow diagram, your policy controller needs to have following capabilities:

  • Watch and retrieve custom policy CR
  • Based on policy retrieved, extract policy definition and call security control APIs to execute the policy
  • Retrieve the execution results and update the corresponding policy
  • Generate an event on an Advanced Cluster Management policy CR with the remediation results

View the open source policy controller sample, which implements the described flow.

The multicloud-operators-policy-controller repository provides all the code necessary to integrate with Advanced Cluster Management governance. It also provides a Configuration policy controller to check if your cluster is compliant with the required number of role binding and cluster role binding rules. For more information, please read the adoption guide in the repository.

Create your own policy and controller

You can customize your policy controller by using the policy controller sample, that is provided in multicloud-operators-policy-controller repository.

  1. Clone the multicloud-operators-policy-controller repository:

    git clone git@github.com:open-cluster-management/multicloud-operators-policy-controller.git
  2. Customize the policy and controller.

    A policy file defines the desired state. To make your own policy, change the policy schema definition. View the following example to change the schema definition. Let’s create a custom policy named TestPolicy. The schema file is located in the mentioned repository.

    Your policy might look like this:

    metadata: 
    name: testpolicies.policies.redhat.com
    spec:
    group: policy.open-cluster-management.io
    names:
    kind: TestPolicy
    listKind: TestPolicyList
    plural: testpolicies
    singular: testpolicy

    Update the policy controller to watch for TestPolicy kind. Run the following command:

    for file in $(find . -name "*.go" -type f); do  sed -i "" "s/SamplePolicy/TestPolicy/g" $file; done
    for file in $(find . -name "*.go" -type f); do sed -i "" "s/samplepolicy-controller/testpolicy-controller/g" $file; done

    Now that you updated the controller and policy schema with TestPolicy kind, recompile and run the project to see if it really works. To do that, configure kubectl to the Red Hat Advanced Cluster Management for Kubernetes cluster by completing the following steps:

    • Log in to your cluster.

    • Select the user icon, then click Configure client.

    • Copy and paste the configuration information into your command line, and press Enter.

    • Run the following commands to install TestPolicy CRD and start the controller:

      export GO111MODULE=on

      kubectl apply -f deploy/crds/policy.open-cluster-management.io_samplepolicies_crd.yaml
      operator-sdk run --local --verbose

      The output indicating that a controller called testpolicy-controller is running might resemble the following content:

      {“level”:”info”,”ts”:1578503280.511274,”logger”:”controller-runtime.manager”,”msg”:”starting metrics server”,”path”:”/metrics”}
      {“level”:”info”,”ts”:1578503281.215883,”logger”:”controller-runtime.controller”,”msg”:”Starting Controller”,”controller”:”testpolicy-controller”}
      {“level”:”info”,”ts”:1578503281.3203468,”logger”:”controller-runtime.controller”,”msg”:”Starting workers”,”controller”:”testpolicy-controller”,”worker count”:1}
      Waiting for policies to be available for processing…

    Let’s create a policy and verify that the controller retrieves it and applies the policy onto the cluster. Run following command in the new terminal to apply the policy:

    kubectl apply -f deploy/crds/policy.open-cluster-management.io_samplepolicy_crd.yaml

    Once the policy is applied, a message appears indicating the policy has been detected and executed by your controller:

    {"level":"info","ts":1578503685.643426,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
    {"level":"info","ts":1578503685.855259,"logger":"controller_samplepolicy","msg":"Reconciling TestPolicy","Request.Namespace":"default","Request.Name":"example-samplepolicy"}
    Available policies in namespaces:
    namespace = kube-public; policy = example-samplepolicy
    namespace = default; policy = example-samplepolicy
    namespace = kube-node-lease; policy = example-samplepolicy

    Run the following command to view the policy execution result:

    kubectl get testpolicy example-samplepolicy -o yaml

    Your output might resemble the following status field:

    status:
    compliancyDetails:
    example-samplepolicy:
    cluster-wide:
    - 5 violations detected in namespace `cluster-wide`, there are 0 users violations and 5 groups violations
    default:
    - 0 violations detected in namespace `default`, there are 0 users violations and 0 groups violations
    kube-node-lease:
    - 0 violations detected in namespace `kube-node-lease`, there are 0 users violations and 0 groups violations
    kube-public:
    - 1 violations detected in namespace `kube-public`, there are 0 users violations and 1 groups violations
    compliant: NonCompliant
  3. Change the policy rules and execution logic.

    Let’s update your created policy by changing the policy rules and execution logic. View the following policy rules and execution logic for the sample policy:

    maxClusterRoleBindingGroups: 10
    maxClusterRoleBindingUsers: 10
    maxRoleBindingGroupsPerNamespace: 2
    maxRoleBindingUsersPerNamespace: 2

    Complete the following steps to customize the policy rules and execution logic:

    1. Add new fields in CRD-file to introduce new rules. For examples to add a new field see, the redhat repository CRD file.
    2. Update the TestPolicySpec structure in samplepolicy_types.go with new fields.
    3. Update PeriodicallyExecSamplePolicies function in samplepolicy_controller.go with new logic to execute the policy.
    4. Recompile and run it.

You created a policy controller and policy.

Deploy your controller to the cluster

If you follow the above steps to create your own policy controller, you should now have a custom policy and controller ready to use with Red Hat Advanced Cluster Management for Kubernetes governance.

Complete the following steps to build controller image and deploy to your cluster:

  1. Run following command to build the controller image:

    operator-sdk build <username>/multicloud-operators-policy-controller:latest
  2. Push the image to a repository of your choice, e.g. quay.io. Run the following commands:

    podman login

    podman push <username>/multicloud-operators-policy-controller
  3. Configure kubectl to point to a cluster managed by Red Hat Advanced Cluster Management for Kubernetes.

  4. Replace the operator manifest to use the built-in image name and update the namespace to watch for policies. The namespace must be the cluster namespace, run the following command:

    sed -i "" 's|redhat/multicloud-operators-policy-controller|ycao/multicloud-operators-policy-controller|g' deploy/operator.yaml
    sed -i "" 's|value: default|value: <namespace>|g' deploy/operator.yaml
  5. Update the RBAC role by running the following commands:

    sed -i "" 's|samplepolicies|testpolicies|g' deploy/cluster_role.yaml
    sed -i "" 's|namespace: default|namespace: <namespace>|g' deploy/cluster_role_binding.yaml
  6. Deploy the controller to your cluster by running the following commands:

    Setup Service Account

    kubectl apply -f deploy/service_account.yaml -n <namespace>

    Setup RBAC for operator

    kubectl apply -f deploy/role.yaml -n <namespace>
    kubectl apply -f deploy/role_binding.yaml -n <namespace>

    Setup RBAC for SamplePolicyController

    kubectl apply -f deploy/cluster_role.yaml
    kubectl apply -f deploy/cluster_role_binding.yaml

    Setup the CRD

    kubectl apply -f deploy/crds/policies.redhat.com_samplepolicies_crd.yaml

    Deploy the multicloud-operators-policy-controller

    kubectl apply -f deploy/operator.yaml -n <namespace>
  7. Run following command to verify that the the controller is in running state:

    kubectl get pod -n <namespace>
  8. Look at the log and make sure everything is running correctly by running the following commands:

    kubectl logs -f -n <namespace> multicloud-operators-policy-controller-854749c8b8-8v9tp
    {"level":"info","ts":1578516495.9202724,"logger":"cmd","msg":"Go Version: go1.13.5"}
    {"level":"info","ts":1578516495.9204354,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
    {"level":"info","ts":1578516495.9204445,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
    {"level":"info","ts":1578516495.9219334,"logger":"leader","msg":"Trying to become the leader."}
    {"level":"info","ts":1578516498.3849623,"logger":"leader","msg":"No pre-existing lock was found."}
    {"level":"info","ts":1578516498.4195356,"logger":"leader","msg":"Became the leader."}
    {"level":"info","ts":1578516500.7812169,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
    {"level":"info","ts":1578516500.7819993,"logger":"cmd","msg":"Registering Components."}
    {"level":"info","ts":1578516500.7826676,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
    {"level":"info","ts":1578516500.787235,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testpolicy-controller","source":"kind source: /, Kind="}
    {"level":"info","ts":1578516505.7244387,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"multicloud-operators-policy-controller-metrics","Service.Namespace":"default"}
    {"level":"info","ts":1578516508.310978,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"servicemonitors.monitoring.coreos.com \"multicloud-operators-policy-controller-metrics\" already exists"}
    {"level":"info","ts":1578516508.3131533,"logger":"cmd","msg":"Starting the Cmd."}
    Waiting for policies to be available for processing...
    {"level":"info","ts":1578516508.313996,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
    {"level":"info","ts":1578516508.4145792,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testpolicy-controller"}
    {"level":"info","ts":1578516508.5149467,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testpolicy-controller","worker count":1}
    Waiting for policies to be available for processing...
    Waiting for policies to be available for processing...
    Waiting for policies to be available for processing...
    Waiting for policies to be available for processing...
    Waiting for policies to be available for processing...

Create and manage policy with the Red Hat Advanced Cluster Management for Kubernetes dashboard

Now your policy controller has been installed and running on your cluster. Next, create the policy and apply it to the cluster using the Red Hat Advanced Cluster Management for Kubernetes console. Complete the following steps to navigate the console:

  1. Log in to Red Hat Advanced Cluster Management for Kubernetes console and click Govern risk from the menu navigation. The Governance and risk dashboard appears.

  2. Click the Create policy button. Copy and paste your policy into the YAML editor.

    Your policy might look like the following content:

    apiVersion: policy.open-cluster-management.io/v1
    kind: Policy
    metadata:
    name: testpolicy
    namespace: default
    annotations:
    policy.open-cluster-management.io/standards: NIST-CSF
    policy.open-cluster-management.io/categories: PR.PT Protective Technology
    policy.open-cluster-management.io/controls: PR.PT-1 Audit Logging
    spec:
    complianceType: musthave
    remediationAction: inform
    disabled: false
    namespaces:
    exclude: ["kube-*"]
    include: ["default"]
    policy-templates:
    - objectDefinition:
    apiVersion: policy.open-cluster-management.io/v1
    kind: TestPolicy
    metadata:
    name: example-samplepolicy
    spec:
    severity: low
    namespaceSelector:
    include: ["default","kube-*"]
    exclude: ["kube-system"]
    remediationAction: inform
    maxClusterRoleBindingGroups: 0
    maxClusterRoleBindingUsers: 0
    maxRoleBindingGroupsPerNamespace: 0
    maxRoleBindingUsersPerNamespace: 0
    ---
    apiVersion: policy.open-cluster-management.io/v1
    kind: PlacementBinding
    metadata:
    name: binding-testpolicy
    namespace: default
    placementRef:
    name: placement-testpolicy
    kind: PlacementPolicy
    apiGroup: policy.open-cluster-management.io
    subjects:
    - name: testpolicy
    kind: Policy
    apiGroup: policy.open-cluster-management.io
    ---
    apiVersion: apps.open-cluster-management.io/v1
    kind: PlacementRule
    metadata:
    name: placement-testpolicy
    namespace: default
    spec:
    clusterConditions:
    - status: "True"
    type: ManagedClusterConditionAvailable
    clusterSelector:
    matchExpressions:
    - {key: name, operator: In, values: ["daunts"]}
  3. Select the cluster where your policy controller runs by updating the Cluster binding field. Click the drop-down menu and select.

  4. Click the Create button. Your created policy appears on the Policies tab. Wait for a few seconds and let policy framework propagate the policy to your managed clusters and collect the results. These results for your clusters violations appear shortly.

  5. Click on the policy row and drill down into the details of the policy. Your policy information is displayed from the Details tab.

  6. Click on Violations tab to view the violation details.

Now you have successfully created a custom policy, deployed it to your cluster, and checked the results by using the Governance and risk dashboard.

Conclusion

Red Hat Advanced Cluster Management for Kubernetes governance provides an extensible framework for enterprises to introduce their own security policies. By writing a custom policy controller, enterprises are able to integrate existing enterprise security and configuration controls with Red Hat Advanced Cluster Management for Kubernetes governance. Enterprises are able to take advantage of the Governance and risk dashboard to create and manage both out-of-box policies and custom policies, monitor the policy compliance status, and take further remediation actions.

Learn how to contribute your custom policies to the open-cluster-management community project, and view examples of custom policies and policy controllers in the policy-collection repository.

Learn to integrate the Open Policy Framework. View the related blog, How to integrate Open Policy Agent with Red Hat Advanced Cluster Management for Kubernetes policy framework.


Categories

How-tos, Red Hat Advanced Cluster Management, Multi-Cluster

< Back to the blog