You've heard about OpenShift 4 and are ready to take advantage of its new features. You've created an OpenShift 4 cluster. Now you need a way to move all of your stateful and stateless OpenShift apps over to your shiny new cluster, and CAM (Cluster Application Migration Tool) can help.

CAM bridges the gap between OpenShift 3.7+ and OpenShift 4, providing a powerful tool for moving apps and persistent volume (PV) data between clusters. If this is your first time hearing about CAM, the basics of CAM were covered in an earlier post.

CAM has a suite of built-in migration logic to help you avoid pitfalls that you would encounter running a naive "YAML dumpYAML import" app migration. This embedded migration logic includes detection of incompatible API versions between clusters.

In this post you'll learn about...

  • OpenShift 4.4 API version deprecations
  • CAM's pre-migration warning for incompatible API versions
  • Manual API version conversion steps
  • An in-development CAM feature providing automatic API version upgrades

When are API version upgrades required?

If you haven't updated your OpenShift 3 cluster version lately, there's a chance some resource API versions you're using have been phased out in the newest version of OpenShift.

In OpenShift 4.4, several API GVKs have been deprecated in favor of stable APIs following upstream Kubernetes 1.16 deprecations.

List of deprecated APIs GVKS in Kubernetes 1.16 (OpenShift 4.4)

depricated APIS116

Kubernetes API versioning policy summary

kubernetesapivers

When migrating applications with CAM, you'll receive an alert when API version differences are detected between clusters. Let's look at how CAM handles incompatible API versions during migrations today, and how to resolve these issues manually.

CAM pre-migration detection of API version differences

Let's say you've created a Migration Plan with CAM and received a warning on the Plan Summary screen providing details on the GVK incompatibility. Today, this means you must complete the API version upgrade process on your resources manually.

 

Figure - After Migration Plan details are specified, CAM validations catch API version incompatibilities

 

Figure - The "phronetic" namespace contains some batch/v2alpha1 resources which will require manual migration

apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
 name: move-my-cronjobs
 namespace: openshift-migration
spec:
 [...]
 namespaces:
 - phronetic
status:
 [...]
 conditions:
 - category: Warn
   lastTransitionTime: 2020-04-30T17:16:23Z
   message: 'Some namespaces contain GVKs incompatible with destination cluster.
     See: `incompatibleNamespaces` for details'
   status: "True"
   type: GVKsIncompatible
 incompatibleNamespaces:
 - gvks:
   - group: batch
     kind: cronjobs
     version: v2alpha1
   - group: batch
     kind: scheduledjobs
     version: v2alpha1

Figure - Migration Plan warning about incompatible GVKs shows up on the MigPlan CR

If you get a warning like this on your Migration Plan, you can still run the migration, but the resource GVKs listed in the warning won't be migrated. You'll need to extract the resource JSON and perform manual conversion to rectify the issue. Read on to find out how to do this.

Using 'oc convert' to upgrade resources to a newer API version

The 'oc' client tool you normally use for interacting with OpenShift has a built-in 'oc convert' command that takes in a resource or list of resources, and outputs those resources in an API version schema of your choosing.

The 'oc convert' command can run in an online or offline mode. In online mode, 'oc convert' will connect to the API server configured in your KUBECONFIG and ask the API server to perform an API version conversion. In offline mode, conversion logic built into the 'oc' binary is used instead.

For both conversion methods, an efficient "hub and spoke" conversion system is used to reduce the number of conversion routines needed. Conversion routines to and from a Storage/Hub version from all other known API versions defines the "hub and spoke" model.

 

 

Figure - from kubebuilder docs: Hub and Spoke API Version Conversion

Choosing the 'oc' client version(s) to bridge an API version gap

As mentioned in the API version compatibility table above, beta APIs are usually available for at least one minor release cycle. There is no guarantee that beta APIs will be available for extended periods.

This means a single 'oc' client version (or a single APIserver) may not contain all required conversion logic to convert from the oldest known API version up to the newest when the gap between OpenShift versions is large.

fictionalapi1

If we only look at the table above, it looks impossible to get our resources upgraded from v1alpha1 to v1 since neither 'oc' client supports conversion to the others supported version range. But what if we introduce a third 'oc' client version that helps us bridge the gap?

fictionalapi2

With an 'oc' 3.11 bridge client introduced, our resources belonging to this Fictional API can now be migrated up the chain over to an OpenShift 4.3 cluster as API version v1. This requires use of the offline conversion mode that doesn't interact with either APIserver during conversion, since neither the source nor destination cluster API servers have the required set of API conversion routines.

Manual Conversion Steps

If you choose to execute a migration containing incompatible resources, the incompatible resources won't be migrated, but they will be included in the Velero Backup archive uploaded to your object store of choice (S3, Azure, Google).

The process for migrating resources without a shared API version between OpenShift clusters is shown below:

  1. Run a migration with CAM as normal, noting which resources are flagged as being incompatible with the destination cluster.
  2. Download and extract contents of the Velero "Initial Backup" CAM performed. Find the JSON files that weren't migrated (but were backed up) by cross-referencing with the incompatible GVK list on the Migration Plan.
  3. Run conversions with 'oc convert' on all OpenShift resource JSON files that were excluded from restore, create converted resources on destination with 'oc create -f'


Figure - 3 steps to a complete migration when GVK incompatilities are present

For more details on how this process works, check out an illustrative playbook written to perform this "manual conversion" process automatically: gvk-upgrade. This playbook takes a Migration Plan name as input, and outputs a set of converted "incompatible" resources. The playbook attempts conversion with whatever 'oc' version is installed, but recall that different 'oc' versions will be required for different conversion tasks.

Future work: Automating API version conversion in CAM

A feature providing automatic API version upgrades for migrated resources is under active development in CAM right now. When complete, this feature will join a suite of migration logic built into CAM to ease the process of migrating workloads from OpenShift 3 to 4.

Our goal is to embed the steps above in CAM so all conversion takes place without requiring any manual intervention from the user.

CAM is being developed upstream in the Konveyor Project; check it out and let us know if you have feedback!