In a restricted network environment, updating OpenShift is a different experience because the Cluster Version Operator (CVO) normally uses an internet connection to get directions for upgrades. Without an internet connection, the CVO relies on the user to direct the cluster upgrade by gathering graph data and running some command line operations. This works fine when tracking a few clusters, but it becomes more tedious as more clusters are added and as versions drift apart.

The recent release of the OpenShift Update Service unlocks the ability for admins to take control over the cluster’s update path from the UI. An admin can launch the OpenShift Update Service (OSUS) from the UI, point the OpenShift Update Service at a registry with mirrored OpenShift releases, configure the Cluster Version Operator (CVO) to reach the OSUS route, and voilà. This greatly improves user experience for clusters in restricted networks, however horizontally scaling clusters into the 100s will still be challenging to update, as 100s of UI visits start to add up.

The Advanced Cluster Management for Kubernetes is a multi cluster management solution that unifies cluster management into a single UI. Combining the OpenShift Update Service with Advanced Cluster Management provides administrators the tools necessary to horizontally scale clusters while maintaining a smooth management experience.

Setup the OpenShift Update Service

The Advanced Cluster Management uses a management cluster (Hub Cluster) and managed cluster(s) (Spoke Clusters). There needs to be a place to run the OpenShift Update Service, and the Hub Cluster is a good fit because it will be responsible for overseeing the management operations of the Spoke Clusters. In addition, the Hub and Spoke clusters can connect to one another, so the OpenShift Update Service will advertise an endpoint from the Hub and all the Spoke Cluster CVOs will gather graph data from the endpoint.

 

 

  1. Mirror OpenShift releases to the Hub container image registry
  2. Build the graph data init container and publish it in the Hub container image registry
    FROM registry.access.redhat.com/ubi8/ubi:8.1

    RUN curl -L -o cincinnati-graph-data.tar.gz https://github.com/openshift/cincinnati-graph-data/archive/master.tar.gz

    CMD exec /bin/bash -c "tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati/graph-data/ --strip-components=1"
    $ podman build -f ./dev/Dockerfile -t registry.example.com/graph-data-image:latest

    $ podman push registry.example.com/graph-data-image:latest
  3. Locate the OpenShift Update Service in the UI and start the operator on the Hub Cluster
    1. Set the ‘repository’ parameter to point at the container image registry on the Hub Cluster
    2. Set the ‘graphDataImage’ parameter to the location where you pushed the graph data init container
      apiVersion: cincinnati.openshift.io/v1beta1
      kind: Cincinnati
      metadata:
       name: example
       namespace: example-namespace
      spec:
       replicas: 1
       registry: "hub.registry.example.com"
       repository: "openshift-release-dev/ocp-release"
       graphDataImage: "hub.registry.example.com/graph-data-image:latest"

       

  4. Extract the route to pass to the spoke clusters:
    ROUTE_URI="$(oc -n example-namespace get route example-policy-engine-route -o jsonpath=https://'{.spec.host}'/api/upgrades_info/v1/graph)"
    1. Note: You may also need to provide a custom default certificate on the spoke clusters, so the Cluster Version Operator doesn’t fail on the route’s TLS termination.
  5. Poll until the operand service is ready:
    curl --header 'Accept:application/json' "${ROUTE_URI}"

 

Configure Spoke Clusters

Now that the OpenShift Update Service is installed and configured on the Hub, configure the Spoke clusters to contact the OpenShift Update Service endpoint for graph data. ACM provides the ability to apply changes across multiple clusters with Policies and PlacementRules:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
name: policy-graph-container
namespace: default
annotations:
  policy.open-cluster-management.io/standards: NIST-CSF
  policy.open-cluster-management.io/categories: PR.IP Information Protection Processes and Procedures
  policy.open-cluster-management.io/controls: PR.IP-1 Baseline configuration
spec:
remediationAction: enforce
disabled: false
policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: role-policy-musthave
      spec:
        remediationAction: enforce
        namespaceSelector:
          exclude: ["kube-*"]
          include: ["default"]
        object-templates:
          - complianceType: musthave
            objectDefinition:
              apiVersion: config.openshift.io/v1
              kind: ClusterVersion
              metadata:
                name: version
              spec:
                channel: stable-4.4
                clusterID: exmaple_cluster_id
                upstream: https://hub.updateservice.com/api/upgrades_info/v1/graph
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
name: binding-graph-container
namespace: default
placementRef:
name: placement-policy-graph-container
kind: PlacementRule
apiGroup: apps.open-cluster-management.io
subjects:
- name: policy-graph-container
kind: Policy
apiGroup: policy.open-cluster-management.io
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
name: placement-policy-graph-container
namespace: default
spec:
clusterConditions:
- status: "True"
  type: ManagedClusterConditionAvailable
clusterSelector:
  matchExpressions:
    - {key: name, operator: In, values: ["spoke-cluster-name"]}

Check the Cluster Version Operator’s spec on a Spoke Cluster to verify the change:

$ oc get clusterversion version -o yaml | grep ' upstream:' -A 3 -B 3
spec:
 channel: stable-4.5
 clusterID: 5fc2ade1-c2f3-4e2b-aaf5-eb89da975347
 upstream: https://hub.updateservice.com/api/upgrades_info/v1/graph
status:
 availableUpdates:
 - force: false

Navigate to the Advance Cluster Management UI instance and your spoke clusters will show if there’s an Upgrade available. Use the checkbox to select multiple clusters to upgrade at once or upgrade them all individually:

Improved UX Scales

The improved user experience that the OpenShift Update Service provides when combined with Advanced Cluster Management make it so cluster updates in restricted networks can easily be triggered across the fleet with the click of a button. Administrators no longer need to run command line operations across multiple clusters or potentially lose track of cluster versions; everything can be UI driven, in one location, simplifying the overall upgrade experience.


Categories

Red Hat Advanced Cluster Management, how, OpenShift Update Service

< Back to the blog