Creating a Virtual Machine (VM) that separates the static operating system from the user’s ever changing data has several advantages. It gives the administrator more control over the VM’s configuration and security while allowing the users of the VM to easily customize and use the VM without having to wait on the administrator to perform day to day configuration activities. Separating the two also allows for an easy method to update the operating system (OS) and change the standard configurations without affecting the user’s data.

This blog post walks through the creation of a two disk Red Hat Enterprise 8 (RHEL8) template with an immutable OS. A web server will be configured on the template that allows the user to change the web pages being served. A VM will then be created from the template. Finally, the OS disk referenced by the template and the created VM will be updated to an image containing a newer version of RHEL8.

The procedures discussed in this blog post use OpenShift 4.7 and have the OpenShift Virtualization 2.6 operator installed. OpenShift Virtualization is deployed and storage is configured. See the OpenShift Virtualization documentation on how to install and configure these.

Create the OS and Data PersistentVolumeClaims(PVCs)

PVCs for the immutable OS and persistent user data need to be created. It is possible to create these and then upload them into the OpenShift Virtualization environment, but we will create them within the environment itself.

To create the PVCs, we will create a VM with two disks. One disk will be used for the immutable OS and the other will be used for the changeable data. We will create the VM using an existing rhel8 Boot Source image. This particular image is based on RHEL 8.1.

We will move the PVCs of the immutable OS to the global openshift-virtualization-os-images project namespace. You would need to decide if this is desired in your cluster environment.

Create a RHEL8 VM

In the GUI, navigate to [Workloads Virtualization]

Click the [Create] button and then select [Virtual Machine With Wizard] from the dropdown.

vm-create-01

 

Select the Red Hat Enterprise Linux 8.0+ VM, it should show it has an Available boot source.

 

vm-create-02

Select a project to create the VM in, this is also where the PVCs will be created. We are using a project namespace called images-workspace that had been created in the past. Specify a meaningful name for the VM, use something to make this easily identifiable like rhel8-readonly-work. All the other information should be fine. Select the [Customize virtual machine] button at the bottom of the window.

vm-create-03

 

Since this VM is used only to create the underlying PVCs for the disk images, most of the default settings will work. We just need to add a second disk to the VM. Select the Storage tab and select [Add Disk].

vm-create-04

 

Set the Source for the disk to be Blank (creates PVC). Next, set the Name of the disk to something easy to identify like persistent-data. Specify the Size of the disk 10 GiB. Since this is simply going to be a webserver for demonstration purposes, 10 GiB should be more than we need. Finally, make sure the Storage Class is set. The remaining fields should be populated correctly. Save the configuration by clicking [Add].

vm-create-05

 

The new disk should appear in the Storage section.

vm-create-06

 

Configure the OS to be Immutable

Navigate to the VMs console, [Workloads ⮞ Virtualization] then select the kebab menu to the right of the VM that we created. Select Open Console. The console will open after the disks are Imported.

vm-console-01

 

Login to the console as cloud-user. The password can be viewed by selecting Show password in the blue section above the console login.

vm-console-02

 

The configuration of the OS and any data that will be immutable must be done before making the OS disk read only. This can include installing or updating packages, enabling services, inserting ssh keys, adding user accounts and changing passwords or other configuration.

Note: Once the operating system is immutable, the cloud-user password cannot be changed by cloud-init. This means the password used to login into any VMs created from this image will be the password used just now even if the blue section of the console screen shows something different. You must either take note of what the password currently is or change it before making the OS immutable. There are ways around this, but it is beyond the scope of this blog post.

Switch to the root user by issuing the sudo -i command. This makes configuring the OS a little easier.

# sudo -i

Before installing the needed packages, we first need to configure the systems package repositories. We will use subscription-manager for this, but other methods such as a local repository or Red Hat Satellite Server can be used.

# subscription-manager register

After the system is registered, we can install the needed packages. The httpd package is needed for the webserver and the readonly-root package provides an easy way to make the OS immutable. Install the httpd and readonly-root packages.

# yum install httpd readonly-root

Enable the httpd service and start it to verify it works.

# systemctl enable httpd
# systemctl start httpd

Three files control the configuration of the readonly-root service. These files are the /etc/sysconfig/readonly-root, /etc/statetab (persistent data), and /etc/rwtab (non-persistent data) files. See the RHEL8 Managing File Systems documentation for more information.

There is also the /etc/statetab.d directory that statetab files can be put into and the /etc/rwtab.d directory for rwtab files. We will put files in those directories instead of editing the main files.

The strace command can be used to find out which files are opened by commands. This can be used to determine which entries are needed to be added to the statetab and rwtab file. For example, the following command will show the files that are access by the yum list command.

# strace -e trace=file yum list

Lets start by creating a statetab.d directory. We will put configuration files in this directory that instruct the system to make the data persistent.

# mkdir /etc/statetab.d

Make the users home directories persistent by adding entries in the statetab.

_/etc/statetab.d/users_
```
/root
/home
```

Make sure we can recreate the hosts ssh-keys.

/etc/statetab.d/host_
```
/etc/ssh
```

Add some files and directories so network changes can be configured and we can keep some logs.

_/etc/statetab.d/network_
```
/etc/sysconfig/network-scripts
/etc/motd
/var/log
/etc/hostname
/etc/hosts
``

Make the web server data directory and configuration directory persistent.

_/etc/statetab.d/httpd_
```
/var/www
/etc/httpd
```

Add the cache directory for the dnf package manager to the scratch space. This allows packages to be queried, but not installed.

_/etc/rwtab.d/packages_
```
dirs /var/cache/dnf
```

Enable the read-only filesystem upon reboot. First, edit the /etc/sysconfig/readonly-root file and set the READONLY variable to yes. Also make note of the RW_LABEL and STATE_LABEL values. These are the labels of the filesystems used for scratch and persistent data. I changed the name of the STATE_LABEL from stateless-state to stateful-rw because the name makes more sense to me. But it does not need to be changed as long as they match the labels on the filesystems that will be created later.

/etc/sysconfig/readonly-root

# Set to 'yes' to mount the system filesystems read-only.
# NOTE: It's necessary to append 'ro' to mount options of '/' mount point in
# /etc/fstab as well, otherwise the READONLY option will not work.
READONLY=yes
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
TEMPORARY_STATE=no
# Place to put a tmpfs for temporary scratch writable space
RW_MOUNT=/var/lib/stateless/writable
# Label on local filesystem which can be used for temporary scratch space
RW_LABEL=stateless-rw
# Options to use for temporary mount
RW_OPTIONS=
# Label for partition with persistent data
STATE_LABEL=stateful-rw
# Where to mount to the persistent data
STATE_MOUNT=/var/lib/stateless/state
# Options to use for persistent mount
STATE_OPTIONS=
# NFS server to use for persistent data?
CLIENTSTATE=
# Use slave bind-mounts
SLAVE_MOUNTS=yes

Edit the /etc/fstab file so that the filesystem is mounted read-only upon reboot.

/etc/fstab

UUID=858db48f-a920-4d3f-a50d-629c466d8910 /boot                   xfs     defaults,ro        0 0

Add the ro option to the boot line and make sure the rw option does not exist.

# sed -i \
-e ‘s/^\(GRUB_CMDLINE_LINUX.*\)rw\(.*\)/\1\2/’ \
-e ‘s/^\(GRUB_CMDLINE_LINUX.*\)”/\1 ro”/’ \
/etc/default/grub

Recreate the GRUB2 configuration

# grub2-mkconfig -o /boot/grub2/grub.cfg

Configure the Data Disk

Partition the PVC used for data into two partitions. One for scratch and one for persistent data.

Note: This is just for this blog post, your environment could be different. In fact, you could use two separate disks if desired, one for scratch and one for the persistent data.

First, list all the disks on the system to determine the device for the second disk that will contain the persistent data.

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 366K 0 disk
vdb 253:16 0 11G 0 disk
vdb1 253:17 0 11G 0 part /
vdc 253:32 0 9.5G 0 disk

We can see that the second device is vdc. Use the following command to partition the device with two partitions. The first partition will be 1 GiB in size and the seconde will fill the remaining space.

# cat <<EOF | sfdisk /dev/vdc
label: dos
start=1, size=+1G, type=83
start=+1G, type=83
EOF

No matter how the disk or disks are partitioned, a file system needs to be installed onto them. The filesystem labels need to match the values defined for RW_LABEL and STATE_LABEL in the /etc/sysconfig/readonly-root file. The following command will place an XFS filesystem onto the partitions and label them appropriately. XFS is not the only filesystem type that can be used on the partitions, but the filesystem type must support labels.

# mkfs -t xfs -L stateless-rw /dev/vdc1
# mkfs -t xfs -L stateful-rw /dev/vdc2

Note: Do not mount the filesystems or add entries into the /etc/fstab for them. The readonly-root service controls mounting the partitions and will mount them during boot.

Configure anything else on the filesystem as needed, including user accounts. The next step is to seal the VM to be used as an image to create other VMs from.

Seal the VM

The VM should be sealed before using its disk images to create other VMs. The process of sealing the VM can vary depending on the customizations performed. But there are a few things that should be done on any VM when sealing it.

Unregister the VM from its update source and clean up any subscription information.

# subscription-manager unregister
# rm -fv /var/lib/rhsm/facts/facts.json
# rm -fv /etc/sysconfig/rhn/osad-auth.conf
# rm -fv /etc/sysconfig/rhn/systemid

Clear the hosts ssh keys.

# rm -fv /etc/ssh/ssh_host_*

Note: After a new VM is created using the images, run ssh-keygen -A to create hosts keys for the new VM.

Clean up the network configuration. Remove MAC address and UUID information from the network scripts, remove the udev rules for the MAC addresses, and clear the hostname.

# sed -i ‘/^HWADDR/d’ /etc/sysconfig/network-scripts/ifcfg-eth*
# rm -fv /etc/udev/rules.d/70-*
# > /etc/hostname

Clean up the installation files, /var/logs, and /tmp.

# rm -rfv /var/log/*/*
# rm -fv /root/anaconda-ks.cfg
# rm -rfv /tmp/*

Remove the bash history for the users.

# history -c
# exit

$ history -c
$ exit

The VM should now be sealed and should be immediately powered off. To power off the VM, navigate to [ Workloads ⮞ Virtualization]. Select the kebab menu next to the VM and choose Stop Virtual Machine from the drop down menu.

Make the PVCs Available Cluster Wide

The images can be made available to all project namespaces in the cluster by either cloning the DataVolume or by making the PV available in the openshift-virtualization-os-images namespace. If the images will only be used in the project namespace they were created in, then these steps can be skipped. I will clone the PVCs to the openshift-virtualization-os-images project namespace.

For these steps, make note of the PVC and PV names for each disk connected to the VM. You can get this by Navigating to [Storage ⮞ PersistentVolumeClaims].

 

pvc-list-01

Cloning the DataVolume (DV) between project namespaces is done using the CLI and creating a YAML file. The YAML file contains the information needed to perform the clone. The following YAML file will clone the rootdisk DV from the images-workspace project namespace into a new DV called rhel8-root-ro-web in the openshift-virtualization-os-images project namespace.

dv-clone-ro-rootdisk.yaml

apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: rhel8-root-ro-web
namespace: openshift-virtualization-os-images
spec:
source:
pvc:
namespace: "images-workspace"
name: "rhel8-readonly-work-rootdisk-30de0"
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 12Gi

The name and destination of the newly cloned DV is specified using the metadata.name and metadata.namespace keys. The value of metadata.namespace should be openshift-virtualization-os-images. The DV to clone is specified by the spec.source.pvc.name and spec.source.pvc.namespace keys. Make sure the spec.pvc.resources.requests.storage is a value that is at least the same size as the DV being cloned.

Create a second file for the data disk.

dv-clone-ro-data.yaml

apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: persistent-data
namespace: openshift-virtualization-os-images
spec:
source:
pvc:
namespace: "images-workspace"
name: "rhel8-readonly-work-persistent-data-1g4ru"
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

After the file is created, start the cloning process by applying the configuration to the cluster.

$ oc apply -f dv-clone-ro-rootdisk.yaml
$ oc apply -f dv-clone-ro-data.yaml

The new PVCs should show up under [Storage ⮞ PersistentVolumeClaims] in the openshift-virtualization-os-images project namespace.

 

pvc-list-02

The VM can now be removed.

Create a Template

Although a VM can be created directly from the PVCs, creating a template allows VMs to be created more easily.

To create a template, navigate to [Workloads Virtualization]. Select a Project where the template will be stored. This template will be project namespace specific. We will use the pre-created project namespace called user-space.

After the Project is set, select [Create Template With Wizard].

template-create-01

 

Provide a Name for the template, We will use rhel8-webserver-readonly-os. You must specify a Template provider, this is just to allow people to identify who created or maintains the template. Normally this would be something meaningful, but lets use something fun for this blog post, set it to The Greatest Cluster Admin Ever. Now we will see who people think the best cluster admin on your team is, hopefully it is you.

For the Operating System, select Red Hat Enterprise Linux 8 or Higher. Uncheck Clone available operating system source to this Virtual Machine. Change the Boot Source to Clone existing PVC. Next select the PVC to clone by first setting the Persistent Volume Claim project to the project namespace openshift-virtualization-os-images since this is where we have the cloned DVs stored. After this is set, the Persistent Volume Claim name dropdown will populate with the PVCs in the project namespace. Select rhel8-rootdisk-ro-web. Leave the Flavor and Workload Type set to the defaults. They will be fine for this blog post. template-create-02

 

We are only customizing the Storage section for this blog post. You might need to configure the other sections when doing this in your environment. We need to add the data disk to the template. Select the Storage section then select [Add Disk].

Change the Source to Clone existing PVC. Set the PVC Namespace to openshift-virtualizaton-os-images and select the persistent-data PersistentVolumeClaim.

Change the Name of the disk to data, this is just my preference. This disk can be named something else or the default of disk-0 can be used. The Size should automatically populate with the size of the source PVC. Leave the remaining fields set to the default values and select [Add].

template-disk-add-01

 

The new disk should show up under Disks.

template-create-03

 

Select Review and confirm. You will be taken to the Review section. Scroll to the bottom and select [Create Virtual Machine template] to create the template.

Now virtual machines can be easily created from the template.

Create a VM from the Template

Create a VM using the new rhel8-webserver-readonly-os template. See the previous steps on how to create the VM. We are going to use the defaults for everything including the name. Since a name is not provided, a random name will be generated.

vm-create-ro-01

 

We need to create new host ssh keys for the server.

$ sudo -i
# ssh-keygen -A
ssh-keygen: generating new host keys: DSA

If the ssh key generation did not fail, then we know we can write to the /etc/ssh directory. Lets verify the filesystem is mounted read only.

# touch /test
Touch: cannot touch ‘/test’: Read-only file system

# mount | grep “ on /”
/dev/vdb1 on / type xfs (ro,relatime,seclabel,attr2,inode64,noquota)

Lets verify the OS release installed is RHEL 8.1.

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)

Create a web page and verify it is being served by httpd.

# echo “I am running an immutable RHEL 8.1” > /var/www/html/index.html

# curl http://127.0.0.1
I am running an immutable RHEL 8.1

Updating the Immutable OS

Updating the OS uses the same process we used to create the OS and data PVCs. However, since the data disk has already been created and will be re-used as is, the steps to add and configure the data disk and its PVC can be skipped.

The update process would then be as simple as:

  • Create a VM from an image, either a newly uploaded image or from an existing image.
  • Configure the OS to be immutable. Make sure the OS is configured as it was when the original VM was created. Scripting the configuration or using a toolkit such as Ansible helps keep consistency between updated OS images.
  • Seal the VM. Again, scripting helps here as well.
  • Make the PVC available. Do this if the PVC needs to be cloned or made available cluster wide.
  • Lastly, remove the VM as it is no longer needed.

Update the Template

The template is updated by simply editing the root disk and changing the PVC it clones. Navigate to [Workloads ⮞ Virtualization ⮞ Templates] and click on the template name.

 

template-update-01

Go to the Disks tab and select the kebab menu next to the rootdisk. Select [Edit] from the drop down.

 

template-update-02

Make sure the Source is still set to Clone existing PVC. As before, select the PVC Namespace and PersistentVolumeClaim that should be used as the new rootdisk. The remaining fields should be correct. Click [Save].

 

template-update-03

The template is now updated and can be used to create more VMs.

Update the VMs

Updating the VM is similar to updating the template. However, the disk cannot be edited. The disk must be removed and a new disk added. This currently causes a small issue, the VM no longer knows which disk is the boot device. This is easily remedied by editing the VMs YAML file in the GUI.

To update the VM, we must first stop the VM. Stop the VM under [Workloads ⮞ Virtualization] then navigate to the VMs Disks tab.

Next, remove the current rootdisk by navigating to the Disks tab and selecting the Delete from the kebab menu next to the rootdisk. Normally, the process will also delete the underlying PV. If you wish to keep the PV until it is verified the VM is functioning as expected, uncheck the Delete DISKNAME Data Volume and PVC before selecting Delete.

 

vm-update-01

After the rootdisk is deleted, we need to add the new OS disk. Select [Add Disk]. Make sure the Source is set to Clone existing PVC and then select the PVC Namespace and the new PersistentVolumeClaim. Change the Name to be rootdisk, this will make it easier to identify in the future. The remaining boxes should be correct. Select [Add] and the disk will be added.

 

vm-update-02

Now we need to set the boot disk again. This is done by selecting the YAML tab of the VM and adding a bootOrder entry under spec.template.spec.domain.devices.disks for the disk that is to become the boot disk.

Edit the YAML to reflect the following.

 

vm-update-03

The VM is now updated and should boot with the updated OS.

Verifying the Updated OS

Any new VMs created from the template and any updated VMs should now be created with the RHEL 8.3 version of the OS. Verify everything still functions as desired.

Verify the host ssh keys are still present.

$ sudo -i
# ls -l /etc/ssh

Lets verify the filesystem is still mounted read only.

# touch /test
Touch: cannot touch ‘/test’: Read-only file system

# mount | grep “ on /”
/dev/vdb1 on / type xfs (ro,relatime,seclabel,attr2,inode64,noquota)

Verify the OS release has been updated.

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)

Verify the web page is still being served.

# echo “I am running an immutable RHEL 8.1” > /var/www/html/index.html

# curl http://127.0.0.1
I am running an immutable RHEL 8.1

It should still report the previous version since it is just a static page.

Final Thoughts

So you can see how easy creating and managing a RHEL VM with an immutable OS can be in OpenShift Virtualization. The intent of this blog post was to show that it could be done and how easy it can be. However, in a real world application, it does take a little more planning when configuring the OS to be immutable.

For example, this blog post created the user accounts and passwords prior to making the OS immutable. This is probably not ideal for some VMs. The difficulty lies within the useradd and passwd commands. These commands create several temporary files in the /etc directory when they are run. These files are appended with the process id (PID) of the command. This is very difficult to account for in the statetab file without including the entire /etc directory as a stateful directory. But there are other options available. The users can be added manually or maybe IdM, LDAP, Kerberos or NIS could be used for user and password management.

Please share any work you do as you figure out how to get an application or program to run in an immutable environment. Blog about it or contribute to the projects so others can benefit from your efforts as well.