In the first blog posts of this series, we introduced the basic concepts for Application Lifecycle on Advanced Cluster Management and deployed an application to multiple clusters.

In the second blog post, we showed how a Blue/Green deployment can be made by using Advanced Cluster Management.

In the third blog post, we migrated our application across different regions by using Advanced Cluster Management.

If you haven't read the previous blog posts yet, go ahead and read them. This post is a continuation and you will need the context from the previous ones.

We are using the same infrastructure we used in the previous posts. See the following diagram:

NOTE: Pay special attention to the different labels, as they will be used during the blog posts examples.

infra-view (1)

Component Version
Red Hat OpenShift Container Platform 4.5.7
Red Hat Advanced Cluster Management 2.0 Fix Pack 2.0.2

Disaster Recovery on Red Hat Advanced Cluster Management

In the previous post we migrated our application across different regions using PlacementRules.

In this blog post we are going to see how PlacementRules can help us with a basic Disaster Recovery scenario.

Configuring required Advanced Cluster Managemen manifests

We will re-use the Red Hat Advanced Cluster Management manifests used in the previous blog post, which means that we will be using the Namespace reverse-words-region and the Subscription reversewords-region-app-subscription.

We will need to create a new PlacementRule. This time we will include new properties to our PlacementRule; let's review them:

apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
name: us-eu-region-clusters
namespace: reverse-words-region
spec:
clusterConditions:
- type: "ManagedClusterConditionAvailable"
status: "True"
clusterSelector:
matchExpressions:
- key: region
operator: In
values:
- EU
- US
matchLabels: {}
clusterReplicas: 1
  1. We are using the matchExpressions property in order to match any cluster that has a label region with a value of either EU or US.
  2. We are using the clusterReplicas property in order to get only one of the clusters that match the previous expression.
  3. Additionally, we are matching only healthy clusters.

This new PlacementRule will make sure that in case one of the clusters moves to a non-healthy state, the cluster returned by the PlacementRule changes to one on the list that has a healthy state.

Configuring Subscription to use the new PlacementRule

  1. Let's create the PlacementRule discussed in the previous section.

    oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-region/05_placement_rule_DR.yaml

    If we look at the clusters reported by the PlacementRule, we will only see one cluster (Production in this case).

    oc --context hub -n reverse-words-region get placementrule us-eu-region-clusters -o yaml
    <OMITTED_OUTPUT>
    status:
    decisions:
    - clusterName: managed-cluster1-dev
    clusterNamespace: managed-cluster1-dev
  2. Now we can go ahead and update the Subscription we used in the previous blog post. We are going to patch it to use the new PlacementRule we just created.

    oc --context hub -n reverse-words-region patch subscription.apps.open-cluster-management.io/reversewords-region-app-subscription -p '{"spec":{"placement":{"placementRef":{"name":"us-eu-region-clusters"}}}}' --type=merge
  3. The application will run in EU cluster (Development). See the following commands and output:

    oc --context dev -n reverse-words-region get deployments,services,pods
    NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.extensions/reverse-words 1/1 1 1 42s

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    service/reverse-words LoadBalancer 172.30.185.94 a520ed21ff982452abeacf63b0b58cc5-31012041.eu-central-1.elb.amazonaws.com 8080:32283/TCP 42s

    NAME READY STATUS RESTARTS AGE
    pod/reverse-words-68795d69ff-crzqp 1/1 Running 0 42s
  4. Now, I'm going to destroy my EU cluster in order to simulate a disaster; let's see what happens.

    NOTE: We actually destroyed the cluster, if you want to try this without destroying your cluster, you can remove the region: EU label from the cluster

    1. As soon as Red Hat Advanced Cluster Management detects my EU cluster is gone, the PlacementRule gets updated and now it points to the US cluster.

      oc --context hub -n reverse-words-region get placementrule us-eu-region-clusters -o yaml

      The PlacementRule now points to US cluster.

      <OMITTED_OUTPUT>
      status:
      decisions:
      - clusterName: managed-cluster2-prod
      clusterNamespace: managed-cluster2-prod
    2. The application has been moved automatically to the US cluster.

      oc --context pro -n reverse-words-region get deployments,services,pods
      NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
      deployment.extensions/reverse-words 1/1 1 1 76s

      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      service/reverse-words LoadBalancer 172.30.187.142 a1c7d218d901c40ac98375f4a9474084-1310645059.us-west-2.elb.amazonaws.com 8080:31095/TCP 78s

      NAME READY STATUS RESTARTS AGE
      pod/reverse-words-68795d69ff-ttzz5 1/1 Running 0 77s
    3. The EU cluster is gone.

      eu-cluster-goneeu-cluster-gone

  5. When the EU cluster is online again, it will get added to the PlacementRule again automatically

Closing Thoughts

During this blog posts series we have discovered the basics around Application Lifecycle management using Red Hat Advanced Cluster Management.

Feel free to share your comments and keep exploring other Red Hat Advanced Cluster Management related blog posts.


Categories

How-tos, Red Hat Advanced Cluster Management, Multi-Cluster

< Back to the blog