Red Hat Quay 3.4 Is Here

The Quay team at Red Hat is proud to announce that Red Hat Quay 3.4 is generally available as of today. We’ve been working on this release for quite some time now. While we initially planned this as a maintenance release, it includes some important milestones for the future of the open source container registry.

Move to Python 3

While this change isn’t noticeable from the outside, the entire code base has been moved to Python v3 to future-proof Quay and address the end-of-life announcement of Python v2. This was the primary reason why the release cycle was prolonged, providing additional  time for full regression testing.

While version changes in an underlying language might not require a complete rewrite, the shift from Python 2 to 3 was a significant change, and we’ve taken the extra time to make sure this change doesn’t impact the stability or reliability of Quay.

Official Red Hat Images and Aligned Operator Versions

With the 3.4 release, Red Hat Quay and all related components and Operators are available on catalog.redhat.com. An active Quay subscription is all that’s required to download any images. Prior to 3.4, customers were required to pull Quay release images directly from quay.io, using a dedicated access token.

This release also marks a version alignment of the Red Hat Quay product and the accompanying Operators, namely the Quay Operator, Container Security Operator, and the Quay Bridge Operator. The Quay Operator, which is responsible for deploying and managing Quay on top of OpenShift, defines which exact version of Quay is eventually installed. This simplifies versioning across the board since all Quay components now have the same version number. Updating installed components is done by updating the Quay Operator, which works for both new Quay releases as well as patches.

The above diagram shows how updating the Quay Operator moves the installed Quay Registry through various versions. Using the Operator Life Cycle Manager on OpenShift, the Operator will be offered in channels that correspond to a certain Quay release. For example the channel “quay-v3.4” would contain the initial Quay 3.4 release and all subsequent patch releases (3.4.1, 3.4.2, 3.4.x …). These should be automatically applied (by setting the update policy in OLM to “automatic”) or applied as soon as possible after release since they will contain important bug fixes and security updates but no new features with breaking changes. Existing customers using the previous Quay Operator in version 3.3 simply update the Operator by changing the channel from “quay-v3.3” to “quay-v3.4” via the Operator Lifecycle Manager in OpenShift.

Upon a new Quay release, such as 3.5, a new channel will be introduced for the Quay Operator called “quay-v3.5,” which holds the 3.5 release versions. The Operator inside this channel can upgrade existing 3.4 deployments. This is triggered as a manual update by the Quay administrator on OpenShift by switching the channel of the Quay Operator.

New Quay Operator

The Quay Operator 3.4 is a complete rewrite of the previous Quay Setup Operator, which remains available for Quay versions up to 3.3. It follows a different philosophy of providing Quay registries as managed services with a very opinionated way of deploying and managing  scalable Quay installations on top of OpenShift. Previously, the Quay Setup Operator allowed an admin to specify a subset of the options usually found in Quay’s configuration bundle. The operator would use these options to configure certain subsystems of Quay, overriding whatever is in the config bundle. While this worked, it also sometimes created confusion around who is in charge of the Quay application level configuration. With 3.4 there is now a clear demarcation between the Quay Operator and Quay itself. The Operator is responsible for managing all Kubernetes-level resources whereas the Quay Config Editor is governing all Quay-level configuration.

When the user first deploys Quay using the 3.4 Operator, they have the option to choose what components and aspects of Quay to manage. While Quay and Clair are always deployed and managed by the Operator, a user can choose to opt out of managed databases, key/value stores, auto-scaling, mirroring workers, network ingress management, and object storage. In these cases, the Operator does not create any Kubernetes resources for those components, and the user has the ability to provide Quay with third-party services that fulfill these options. Examples include using Red Hat OpenShift Container Storage to provide a scalable object storage back end or using the Crunchy Operator to provide an HA-Postgres Database. In such cases, access credentials are provided via the config editor, which is stood up automatically by the Operator.

If the user accepts the defaults, however, all of the above components are deployed and continuously managed by the Operator. In this case, a PostgreSQL database and a Redis store will be used and provided by the Red Hat Software Collections. Usage of this is covered by the Red Hat Quay subscription. The object storage component is provided by Noobaa, which is a part of the Red Hat OpenShift Container Storage Operator, and is used to make Kubernetes block Storage accessible via the S3 protocol. See a demo walkthrough of how this works!
This setup is supported for production use and enables the use case for Quay to be backed by local storage (single Kubernetes Persistent Volume). For scalable storage back ends, customers have the option to leverage Red Hat OpenShift Container Storage, which is sold separately. The use of NooBaa as part of the OCS Operator for the local storage use case does not require additional subscriptions. For more details, please consult the documentation.

Quay Builders on OpenShift

During the Python v3 transition, the Quay team addressed many end-of-life dependencies. One of these was a key component in the Quay Builder sub-system. As a result, the team took the opportunity to rewrite the Quay Builders to be simpler, easier to troubleshoot, and far more maintainable for the long run. This feature is probably most known via Quay.io, which triggers container builds based on webhook integration with source code management providers like GitHub or GitLab. The same feature is available in Red Hat Quay and previously required you to provide additional RHEL servers for executing the builds.

The new build system in Quay 3.4 is based on Kubernetes jobs, which launch a pod-per-build job that executes the container image build in a containerized qemu-based VM for isolation purposes. This means that the servers that run OpenShift need to support hardware virtualization and thus need to be a bare-metal host. The build chain is now based on podman / buildah. For the future, alternatives are planned to address where the build jobs are executed in containers natively on OpenShift with the help of the OpenShift Pipelines (based on the Tekton project).

In the Quay upstream code base is also a build engine targeting AWS EC2, which mainly exists for Quay.io.

The new Quay Builders are introduced as a Technology Preview feature in Quay with the goal of having OpenShift-based builds using the containerized VM approach supported in Quay 3.5. We encourage customers and users to test-drive the new build system and provide feedback via our public Jira! Existing users who currently have a requirement to run builds in virtual infrastructure should remain on Quay 3.3 for now.

Clair v4 General Availability

Clair has also been completely re-architected and rewritten, and was introduced as Clair v4 in Tech Preview in Quay 3.3. With 3.4, Clair v4 is the standard security scanner solution deployed with Quay and fully supported in production. It features greater content coverage, and thus, more accurate scan results that now extend beyond base-OS level packages up to programming language package managers. Clair v4 introduces support for Python pip packages. This lays the foundation for increasing the scope to other programming language package managers in the future.

Clair v4 now supports air-gapped deployments enabling manual transfer of vulnerability feed data from a network perimeter into a more restricted network, including offline media.

Transitioning to Clair v4 will require existing data to be rescanned, because in the new Clair data model, content is addressed at the manifest level. This can occur with an existing Clair v2 deployment, which continues to be used when scan results are requested for images that have not yet been rescanned by Clair v4.

Looking Ahead

With the Quay 3.4 release and the Python migration finished, we plan to return to quarterly releases with Red Hat Quay while also adding new features to Quay.io. For 2021, we have an exciting list of new planned functionality and completely new aspects of the Quay experience. Quay will be able to manage more content types and provide more control for administrators and better compatibility with environments with strict compliance requirements and even deeper integration into OpenShift. Quay 3.4 brings in many bug fixes and stability improvements and more efficient ways to run Quay at scale on top of OpenShift.