Since the early days of Kubernetes, other than the API manifests themselves, the primary supporting artifact has been a container image. Container images are stored and accessed by Kubernetes from Container Registries. These registries were originally designed to support the Docker image format, however, in order to promote the use of additional runtimes aside from Docker, the Open Container Initiative (OCI) was created to provide a standardization surrounding container runtimes and image formats. Most container registries support the OCI standardization as it is based on the Docker image manifest V2, Schema 2 format

As the Kubernetes platform has evolved, a variety of artifacts aside from container images have emerged that support not only individual applications, but the platform as a whole. These range from Open Policy Agent (OPA) policies to security and governance to Helm charts and Operators to aid in application deployment. One of the challenges of these supplementary components is how and where they should be stored. While each can make use of their own implementation, it would be desired if a common artifact store and/or format be used. Fortunately, the OCI specification can be used for this purpose, and since container registries can implement the OCI specification, they can be used to provide a solution.

Red Hat Quay is a private container registry that not only stores container images, but an entire ecosystem of supporting tooling to aid in the management of containers. The release of Red Hat Quay 3.4 has introduced a new Technology Preview feature that enables the use of OCI based artifacts and specifically, Helm Charts. The remainder of this discussion will introduce how OCI support for Helm charts can be enabled in Quay and how it can be used.

Support for Helm charts within Quay has actually existed for some time through the concept of an Application Registry. This feature was the first attempt to provide storage for other, mostly text-based artifact types beyond container images, before OCI introduced the concept of media types. It is now deprecated in favor of the long term approach moving forward with the OCI format. 

Deploying and Configuring Quay

Support for Helm charts and OCI artifacts is not yet available in the public hosted quay.io  service and is only available in the self hosted Red Hat Quay. The easiest method to deploy Quay is in OpenShift using the Quay Operator (Quay can also be deployed outside of a Kubernetes based control plane on physical or virtual hosts, but the setup and configuration is outside the scope of this article). 

Ensure that you are logged into an OpenShift environment and create a new project called quay.

$ oc new-project quay

When Quay is deployed, support for Helm and OCI artifacts is not enabled by default. While the Quay Operator helps automate the majority of the setup and configuration of Quay, administrators have the ability to customize the registry configuration. Two properties within the Quay configuration enable the use of OCI artifacts:

FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_HELM_OCI_SUPPORT: true

Customizations to the configuration of Quay can be provided in a secret containing the configuration bundle. Execute the following command which will create a new secret called quay-config-bundle containing the necessary properties to enable OCI support.

oc create -n quay -f - << EOF
apiVersion: v1
stringData:
 config.yaml: |
   FEATURE_GENERAL_OCI_SUPPORT: true
   FEATURE_HELM_OCI_SUPPORT: true
kind: Secret
metadata:
 name: quay-config-bundle
 namespace: quay
type: Opaque
EOF

The next step is to deploy the Quay Operator from OperatorHub. The operator features a “batteries included” configuration that enables a seamless deployment process and supports merging custom Quay configuration directives using the Secret method described above. Review the Product Documentation  including any prerequisite steps that might be required. 

Once the operator has been installed and is running, create the QuayRegistry custom resource and reference the secret containing the config bundle previously created: 

oc create -n quay -f - << EOF
apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
 name: quay
 namespace: quay
spec:
 configBundleSecret: quay-config-bundle
EOF

Once the deployment of Quay and its supporting components are complete the currentVersion property in the status section of the QuayRegistry object will contain the Quay version.

You can now navigate to the web interface. The URL can be found in the registryEndpoint within status field of the QuayRegistry custom resource or by running the following command:

oc get quayregistry -n quay quay -o jsonpath=’{ .status.registryEndpoint }’

Navigate to the URL returned and create an account by clicking the Create Account link. 

Screen Shot 2021-02-09 at 11.52.23 PM

Enter a username, email address and password and once complete, you will be ready to use Quay to store Helm charts in the OCI format.

Helm OCI Support and Quay

Helm, as a graduated project of the Cloud Native Computing Foundation (CNCF), has become the de facto package manager for Kubernetes as it simplifies how applications are packaged and deployed. Helm uses a packaging format called Charts which contain the Kubernetes resources representing an application. Charts can be made available for general distribution and consumption in repositories. A Helm repository is an HTTP server that serves an index.yaml metadata file and optionally a set of packaged charts. Beginning with Helm version 3, support was made available for distributing charts in OCI registries as an alternative to a traditional repository. To demonstrate how Quay can be used as a registry for Helm charts, an existing chart from a Helm repository will be used to showcase the interaction with OCI registries for chart developers and users 

First, we will download an existing chart from a repository locally to mimic the workflow a chart maintainer or end user would follow to publish via Quay. While there are countless charts and repositories in the community, let's use the etherpad chart from the Red Community of Practice (CoP) repository. Etherpad is an online collaboration tool that enables editing of documents in real time.

Ensure you have the Helm CLI (version 3.0.0+) command line tool and add the repository by executing the following command:

$ helm repo add redhat-cop https://redhat-cop.github.io/helm-charts

Once the repository has been added, obtain the latest metadata:

$ helm repo update

Download the chart locally and instruct Helm to untar the chart which will create a directory called etherpad.

$ helm pull redhat-cop/etherpad --version=0.0.4 --untar

Many of the commands for interacting with Helm and OCI registries make use of the helm chart subcommand.  As of the time of this writing, OCI support in Helm was still marked as an “experimental” feature and must be enabled explicitly. This is accomplished by setting the environment variable HELM_EXPERIMENTAL_OCI=1.

$ export HELM_EXPERIMENTAL_OCI=1

With Helm OCI support enabled, the next step is to take the existing chart and tag it using the helm save command. This command takes two parameters, the location of the chart on your local filesystem and the registry reference including the tag. The ultimate goal is to use Quay as the registry to store the chart, but a repository within Quay has yet to be created.

Open the browser again, navigate to Quay and create a new Quay organization for the purpose of storing Helm charts by selecting the + button on the navigation bar and then choosing New Organization

Enter helm as the organization name and then click Create Organization.

Now that the organization has been created in Quay, the helm chart save command can be used to store the chart. Save the etherpad chart to the local OCI registry cache by executing the following command replacing the hostname for Quay with the value obtained in the oc get quayregistry -n quay quay -o jsonpath=’{ .status.registryEndpoint }’ command used previously:

$ helm chart save ./etherpad <QUAY_HOSTNAME>/helm/etherpad:0.0.4

ref:     <QUAY_HOSTNAME>/helm/etherpad:0.0.4
digest:  281b39efd620a350bb964836eea161e6e9b20ed012b86f8970fa3a5a2ed722f0
size:    3.5 KiB
name:    etherpad
version: 0.0.4
0.0.4: saved

For consistency purposes it makes sense to tag the OCI image with the version of the chart that we pulled before: 0.0.4.

Charts that are saved to the local registry cached are located within the default cache directory in a folder called registry. The cache directory varies depending on the operating system. We can view all of the charts in the local registry cache by using the helm chart list subcommand

$ helm chart list

REF           NAME       VERSION DIGEST SIZE    CREATED  
<REF_URL>...   etherpad    0.0.4  62ddb92 3.5 KiB 2 minutes

Interacting with Helm and Quay

Akin to interacting with Quay for traditional container images, we must provide credentials in order to push and pull charts. Communication between the Helm client and Quay is facilitated over HTTPS and as of Helm 3.5, support is only available for registries communicating over HTTPS with trusted certificates. In addition, the operating system must trust the certificates exposed by the registry. Support in future Helm releases will allow for communicating with remote registries insecurely. With that in mind, ensure that your operating system has been configured to trust the certificates used by Quay. 

Next, login to the remote using the helm registry subcommand with your Quay credentials:

$ helm registry login <QUAY_HOSTNAME>

Username: <username>
Password: <password>

Login succeeded

Push the chart to Quay

$ helm chart push <QUAY_HOSTNAME>/helm/etherpad:0.0.4

ref:     <QUAY_HOSTNAME>/helm/etherpad:0.0.4
digest:  281b39efd620a350bb964836eea161e6e9b20ed012b86f8970fa3a5a2ed722f0
size:    3.5 KiB
name:    etherpad
version: 0.0.4
0.0.4: pushed to remote (1 layer, 3.5 KiB total)

You can use the Quay web console to see the newly pushed chart in the registry including listing the available tags on the tags page.

Screen Shot 2021-02-09 at 11.48.44 PM

To confirm that Quay is properly serving the chart contents, delete the chart from the local registry cache and file system

$ helm chart rm <QUAY_HOSTNAME>/helm/etherpad:0.0.4
$ rm -rf etherpad

This is how a maintainer would publish their Helm charts to Quay as an OCI image. Now let’s look at how users would consume this chart.

With the etherpad chart no longer present on the local machine, retrieve it again from Quay using the helm chart pull subcommand.

$ helm chart pull <QUAY_HOSTNAME>/helm/etherpad:0.0.4

0.0.4: Pulling from <QUAY_HOSTNAME>/helm/etherpad
ref:     <QUAY_HOSTNAME>/helm/etherpad:0.0.4
digest:  029558ddda01f1eea8fb2ef5e7054ba3f3619e7540cf962e27662ffcfe82cf65
size:    3.5 KiB
name:    etherpad
version: 0.0.4
Status: Downloaded newer chart for <QUAY_HOSTNAME>/helm/etherpad:0.0.4

Extract the chart directory by using the export command

$ helm chart export <QUAY_HOSTNAME>/helm/etherpad:0.0.4

ref:     <QUAY_HOSTNAME>/helm/etherpad:0.0.4
digest:  3d1564b19c040d5f28439c2479b545e70ee473bdd8b7b8fc6b423d8e40288126
size:    3.5 KiB
name:    etherpad
version: 0.0.4
Exported chart to etherpad/

The contents of the etherpad chart are now present in the etherpad directory and ready for use via helm install. This demonstrates the full lifecycle of how Helm interacts with OCI registries.

As the Kubernetes ecosystem continues to expand, there will be a need to be able to holistically manage the array of supporting components in a centralized manner. The inclusion of OCI based artifacts, and specifically, Helm Charts, is the first step for how the Quay registry can be used to support these workloads along with existing traditional image based content. Future enhancements to Quay is expected to expand the types of OCI based artifacts that can be used to enable future integrations. 


Categories

How-tos, Operators, Red Hat Quay Registry, helm

< Back to the blog