References:

GitOps is the way to do Kubernetes or OpenShift cluster management through declarative management of API manifests (yaml files) rather than configuration drifting.

There are many posts on OpenShift blog talking about Vault Integration:

But there are other options and possibilities out there in the community, so in this post, we will explore a couple of options for secret management the GitOps way:

The code and scripts referred in this blog can be found at:

https://github.com/rahmed-rh/oc4-learn/tree/master/secret-managment

Using bitnami-labs Sealed Secrets

The idea is to Encrypt your Secret into a Sealed Secret, which is safe to store even in public repos as it is encrypted.

So your normal GitOps workflow is not impacted, you will store Sealed Secret yaml files rather than the standard secret yaml files. Sealed Secrets Controller is responsible for doing the conversion for you.

Understand Sealed Secrets

Sealed Secrets is composed of two parts:

  • A cluster-side controller / operator
  • A client-side utility: kubeseal

Installation

I’m using a bash script to download the controller file from bitnami repo. Then using kustomize, I modify some security context attributes to run on ocp. Then it downloads kubeseal utility:

sealed-secrets/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sealed-secrets
commonLabels:
  app: sealed-secrets
resources:
  - sealed-secrets-namespace.yaml
  - controller.yaml
patchesJson6902:
  - target:
      group: apps
      version: v1
      kind: Deployment
      name: sealed-secrets-controller
    path: sc-patch.yaml

To deploy the controller, run the deploy script:

$ ./00_build.sh

What is kubeseal

The kubeseal utility uses asymmetric crypto to encrypt Secrets that only the controller can decrypt. The key certificate (public key portion) is used for sealing Secrets, and needs to be available wherever kubeseal is going to be used.

Kubeseal will fetch the certificate from the controller at runtime, but public key certificate can also be download for offline usage by:

kubeseal --fetch-cert >mycert.pem

Use it offline with:

kubeseal --cert mycert.pem

Create an Encrypted Sealed Secret

Encrypted Sealed Secret cannot allow users to read a Sealed Secret meant for a namespace they wouldn’t have access to and does not just push a copy of it in a namespace where they can read secrets.

Sealed Secrets thus behaves as if each namespace had its own independent encryption key and thus once you seal a Secret for a namespace, it cannot be moved into another namespace and decrypted there. For more information about scopes, please check documentation.

By default, kubeseal will use current context configured in ~/.kube/config.
This will include server connection and active project you can display it by running:

$ oc config current-context

To create the Sealed Secret named “test-secret”:

$ oc create secret generic test-secret --from-literal=dummykey1=supersecret --from-literal=dummykey2=topsecret --dry-run -o yaml >test-secret.yaml

$ cat test-secret.yaml |kubeseal --controller-namespace sealed-secrets -o yaml --scope strict > sealedtest-secret.yaml

Then apply the Sealed Secret:,

$ oc apply -f sealedtest-secret.yaml

Note that the actual Secret is created automatically:

$ oc describe secret/test-secret
$ oc describe sealedsecret/test-secret

Update an Encrypted Sealed Secret

To update the Sealed Secret “test-secret,” reate the Secret data with the new values, then use “--merge-into” to update the existing Sealed Secret:

$ oc create secret generic test-secret --from-literal=dummykey1=supersecret --from-literal=dummykey2=topsecret --from-literal=dummykey3=new-secret --dry-run -o yaml >test-secret.yaml

$ cat test-secret.yaml |kubeseal --controller-namespace sealed-secrets -o yaml --scope strict --merge-into sealedtest-secret.yaml

Then apply the Sealed Secret:

$ oc apply -f sealedtest-secret.yaml
$ oc describe secret/test-secret

Using GoDaddy Kubernetes External Secrets

The idea is to use external Secret management systems, like AWS Secrets Manager or HashiCorp Vault, to securely add Secrets in Kubernetes. The controller is responsible to fetch Secret data from external providers and add it to namespace

Understand External Secrets

An External Secret is composed of a cluster controller, which is responsible to fetch Secret data from external providers and adds it to namespace so the Pods can use it.

You define your external provider configuration through External Secrets custom resource:

 

 

source https://github.com/godaddy/kubernetes-external-secrets/raw/master/architecture.png

Installation

I’m using a bash script to clone the whole repo as the installer using helm charts, and I converted it to template files and used kustomize to modify some resource files to remove helm-related stuff.

You will need helm client to be installed on the machine, follow this link to Install Helm.

In case you changed the bash variable "InstanaceName," you will need to modify the resource name in all patch yaml files (helm generate the name based on instance name):

external-secrets/00_deploy.sh

export InstanaceName=dev (1)

(1) The helm instance name

For example, go to patch file and change the name:

external-secrets/base/clusterrole-patch.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: RELEASE-NAME-kubernetes-external-secrets (1)
  labels:
    helm.sh/chart: kubernetes-external-secrets-4.0.0
    $patch: delet
    app.kubernetes.io/managed-by: Helm
    $patch: delete

(1) Name should match helm instance name:

external-secrets/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubernetes-external-secrets
commonLabels:
  app: kubernetes-external-secrets
  app.kubernetes.io/instance: kubernetes-external-secrets

resources:
- kubernetes-external-secrets-namespace.yaml
- templates/rbac.yaml
- templates/serviceaccount.yaml
- templates/service.yaml
- templates/deployment.yaml

patchesStrategicMerge:
- clusterrole-patch.yaml
- clusterrolebinding-secret-patch.yaml
- clusterrolebinding-secret-auth-patch.yaml
- serviceaccount-patch.yaml
- service-patch.yaml
- deployment-patch.yaml

 

The bash script configures the environment variable needed for each back-end external service, through kustomization overlays.

For example, for Ali Baba, it modifies the needed environment variable through the overlay patch file:

external-secrets/overlays/alibaba/deployment-patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dev-kubernetes-external-secrets
spec:
  template:
    spec:
      containers:
        - name: kubernetes-external-secrets
          env:
          - name: "ALICLOUD_ACCESS_KEY_ID"
            value: "ACCESS_KEY_ID"
          - name: "ALICLOUD_ACCESS_KEY_SECRET"
            value: "ACCESS_KEY_SECRET"
          - name: "ALICLOUD_ENDPOINT"
            value: "kms.eu-central-1.aliyuncs.com"

And in the deploy bash script, it uses this specific overlay:

external-secrets/00_deploy.sh

oc apply -k  overlays/alibaba/ (1)
(1) Will deploy the controller with the needed Deployment environment variables for Ali Baba cloud

To deploy the controller, run the deploy script:

$ ./00_build.sh

What Are External Secrets?

 

An External Secret is the Custom Resource definition that will be created after the installation. It defines the back-end external service that will be used to retrieve the Secret data:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-service
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  dataFrom:
    - hello-service/credentials

 

Create an External Secret

  • Create secret by using the aliyun-cli command:

aliyun kms CreateSecret --SecretName db_cred --SecretData "{\"username\": \"test\", \"password\": \"123456\"}" --VersionId v001

  • Create the CR yaml file:

external-secrets/demo-external-secret.yaml

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: demo
spec:
  backendType: alicloudSecretsManager
  roleArn: acs:ram::228462691213768868:role/demo (1)
  dataFrom:
  - db_cred (2)
  versionStage: ACSCurrent

(1) (optional) Specify role to assume using provided access key ID and access key secret when retrieving the data.

(2) Match the same Secret name you created using aliyun-cli.

(3) version of the secret, ACSCurrent means the latest

  • Apply the CR file:

oc apply -f demo-external-secret.yaml

That's all folks, please let me know your feedback!


Categories

How-tos, Security, GitOps, git

< Back to the blog