Containers and Bottlenecks

OpenShift users are those enterprise unicorns using Kubernetes in production, and with the announcement last month of the intent to buy CoreOS (with its various projects), it’s safe to say that Red Hat will be in a position to provide even more value for development teams looking to abstract container ops at scale.

I recently attended an OpenShift panel in which one of the speakers, a seasoned engineer working for a leading UK bank, said that OpenShift solves the infrastructure question so effectively for his organization, that his Ops team is now expected to similarly solve all manner of problem in the IT organization. That brought to my mind a bit from the Theory of Constraints, in which removing one bottleneck will expose new ones around it. Where is the “next constraint” for OpenShift users? An interesting and critical area to look at is the pipeline.

Thoughts about pipelines

From our experience of running in containers since 2014, the decision around a container pipeline revolves around these questions:

  1. Who owns the complexity: do you use an open-source stack which has low/no licensing costs, but is expensive to build, automate, scale and maintain—or do you obtain flexibility by outsourcing complexity to a vendor?
  2. Tool leverage vs. specialization: are you looking to leverage existing tools with their familiar, old ways of doing things, or use a tool that is more fit-for-purpose (i.e. in this case, container-native)?

We’ve been running thousands of our customers’ apps in containers for years and think that, for most companies, operational costs can dwarf software cost savings and will end up creating lock-in to an internal way of doing things, slowing them down. It’s tough to scale when each cluster or region runs its own on-prem CI server with automation scripts that vary between teams, or when key people that hold everything together move on.

We also believe that most CI/CD tools ignore the complexity of microservices, cover much less of the process, and add less value compared to what they do for monolithic apps. It’s all about configuration. For example, when your app is made of 12 services and three databases, and can be rebuilt and redeployed within minutes, does build & test hold the same value as it used to?

CDP: a Pipeline that Speaks Configuration

A new class of solution is needed -- a Container Delivery Pipeline, or CDP. Mostly for the better, containers bring developers and operations closer together, and to avoid the complexity of the infrastructure impacting development pace, a CDP solution should facilitate efficient maintenance of container delivery that is consistent with the code.

We talked about what a CDP should provide in our blog post a few months ago. Here’s a graphic representation of that list (functions also available in common CI tools are marked by a green asterisk):

Configuration is Critical for Container Pipelines

As the complexity of software increases, configuration becomes a more challenging aspect of the entire system -- easy environment creation doesn’t mean easy operations! Enterprises deploy software systems across data centers, geographical regions, environments, and business units. This results in a complex mix of fixed configuration values that remain the same, and variable ones that change between each of the above deployments. On top of that, the need to rotate secrets frequently adds even more to the complexity of managing the information that lives outside of the code but is just as important. Container-based infrastructure facilitates creation of new setups and deployments of the same system over and over again, leaving configuration as the only variable between these setups.

While there are a few tools available to developers for managing code complexity and quality, there are very few tools that focus on configuration as a vital part of a software system. Skycap has the configuration challenge presented by cloud-native workloads in mind, and supports multiple variable factors for each configuration item as well as secrets.

Starting down the Pipeline

Let’s look at how our own CDP tool Skycap, which we also use to build our own stack running on Kubernetes, ramps into OpenShift. First, I’ll go into the UI and create a stack, using a simple Git link (I can also direct to a private repo or specific Dockerfile):

Even when using our Simple UI, I received the output of a full, editable .yaml file. I could also upload a .yaml file of my own in our Advanced UI and edit it before the build:

At this point, clicking “Start build” will initiate a service called BuildGrid, as well as going through an open source build workflow called Habitus, ending with artefacts stored in my repository (we provide a private Docker repo, but users can opt for private/other tools such as Artifactory).

I can also view a detailed build history, changes tracking, permissions -- all providing policy and observability for managers and guardrails for developers to work within. Also, any changes to code or config on one service can be built incrementally, while not touching other services.

Security: Image Size and Secrets Management

If we have libraries or packages in the build stage that I don’t need for runtime (e.g. if my app is in Go), it would be useful to minimize the image so that it’s more efficient and less exposed to dependencies risk. Habitus is open source and squashes the image to remove anything that has been flagged as removable. For example, this could be useful when it comes to private information like private keys that need to be on the image during the build (to pull git repos for example) but can't be published as part of the built image.

Which bring us to another tricky area, managing secrets. Normally, we need to copy private keys to our build context, moving it out of the home directory and leaving it exposed to accidental commits to our Git repo. This means that the keys will end up in an image which may go onto an unencrypted repository. Lastly, as we’ve seen, we could squash the image, but the keys might still be in there. Habitus has an internal web server, which only serves to requests coming from inside of the building containers. The secret is delivered dynamically to the build process which can be used and removed in the same layer, leaving no trace in the image.

Repeatable Deploys to Multiple Clusters

A key promise of Kubernetes, especially when using a platform like OpenShift, is that it abstracts the infrastructure, but building automation to on-ramp into different clusters or services can be a daunting task. Formations within Skycap are deployment targets for your application, allowing the user to deploy the same stack to different Kubernetes clusters and namespaces, from one place. This becomes a highly efficient and scalable on-ramp from the pipeline.

Formations allow you to “render” Kubernetes configuration files based on the cluster type; to deploy to OpenShift, you can choose OpenShift as a formation type on Skycap. This will generate the appropriate templates that will work with OpenShift.

A Snapshot of all services generated from git or as pre-built Docker images is taken every time a code change is detected or manually triggered by the user. This means you can go back in time and deploy your application at any point in time to any Kubernetes cluster -- yes, with Skycap we have cracked time-travel and built an infrastructure time machine!

Formations store Kubernetes configuration files in Stencils. Stencils are templates of Kubernetes configuration files. A Stencil gets "rendered" into a working configuration file by merging it with the data from a "snapshot", which is the state of the app/infrastructure at a point in time.

For example, you create a Stencil to add a configuration item to the application (like the access credentials to your database), the actual values are stored in a snapshot, and a Stencil records how those values are fed into Kubernetes. By merging the snapshot with the Stencil, we have a fully working configuration file which we can apply to our cluster via kubectl.

Rendered Stencils will be downloadable via the Skycap UI or CLI:

However disparate or complex the apps, Stencils make it easy to deploy to OpenShift as a destination with a single command in the OC CLI:

Now our containers are in our OpenShift console and our code is being deployed. I did this while using a CDP:

  • That understands that microservices require a pipeline-wide view, not just good-old build & test
  • That automates the process while taking care of security, policies, and observability
  • That speaks configuration from Git to infra
  • That, unlike own-built pipelines, is easily scalable and deployable across teams, clusters and regions