Image mode for Red Hat Enterprise Linux is designed to simplify the experience of building, deploying, and managing Red Hat Enterprise Linux (RHEL) as a bootc container image. It reduces complexity across the enterprise by enabling development, operations, and solution providers to use the same container-native tools and techniques to manage everything from applications to the underlying OS.
We've written several articles detailing the experience of using image mode. We discussed the various use cases of image mode, creating automated CI/CD pipelines, managing containerized workloads, the full GitOps experience for sysadmins of RHEL, and how image mode facilitates building software appliances. There is plenty to talk about.
In this article, I want to focus on embedding containers into a bootc image. I will describe the number of ways to embed containerized workloads into a bootc image. I will also elaborate on the various use cases for embedding containers and provide examples that may serve as a template.
Quadlet: Containerized workloads in systemd
Embedding container workloads into a bootc image implies declaring the containers in a way. Most of the mechanisms I will present build upon declaring such workloads in the form of a so-called Quadlet.
Running containerized workloads in systemd is a simple yet powerful means for reliable deployments. Podman has an excellent integration with systemd in the form of a Quadlet. A Quadlet is a tool for running Podman containers in systemd in an optimal and declarative way. Workloads can be declared in the form of systemd-unit-like files extended with Podman-specific functionality.
You might be wondering about the benefits of running containerized workloads in systemd. Systemd is the central control instance on modern Linux systems. It also manages system and user services and the dependencies among them. It has many capabilities, such as elaborate restart policies. Hence, Podman’s integration with systemd was an important milestone, integrating traditional Linux sysadmin craftsmanship with modern container technologies.
Podman’s daemonless architecture integrates perfectly with systemd. The sophisticated process management of systemd allows it to monitor a container and restart it, if needed. The combination of systemd and Podman allowed us to tackle new use cases where human intervention is not always possible (i.e., edge computing or Internet of Things (IoT). In addition, Quadlet is a seamless extension for systemd, which makes it very approachable for sysadmins.
Let's take a closer look at Quadlet, starting with the following example of a Quadlet .container file.
[Unit]
Description=A minimal container
[Container]
Image=registry.fedoraproject.org/fedora
# For demo purposes, the container just sleeps
Exec=sleep 60
[Service]
# Restart service when sleep finishes
Restart=always
[Install]
# Start by default on boot
WantedBy=multi-user.target default.targetAs mentioned, Quadlet extends systemd-units with Podman-specific features. Quadlet .container files, for instance, add a [Container] table where we can declare container-specific options such as the image, command, and name of the container, and which volumes and networks it should use.
Quadlet is a systemd-generator executed on boot or when reloading the systemd daemon. If you want to test the previous example, you can create the file in your home directory $HOME/.config/containers/systemd/test.container and run systemctl --user daemon-reload.
Reloading the daemon will fire the Quadlet and generate a systemd service named test.service that you can start with the command: systemctl --user start test.service.
You can think of Quadlet like Docker Compose, but for running containers in systemd. The declarative nature of Quadlet makes it a perfect candidate for installing and embedding workloads on a bootc system. Quadlet also supports running Pods and Kubernetes-compliant YAML definitions. It can manage volumes, networks, images, and support building images. For detailed documentation on Quadlet and additional examples, refer to the upstream documentation.
Embed Quadlets into a bootc image
Quadlets are managed as files, which allows for a smooth and easy integration into the bootc workflow. The center of gravity for working with bootc is the Containerfile. Hence, Quadlets can live next to a Containerfile in the same Git tree and copy onto the image during the container build to make the workloads available at runtime.
Let's use our test.container example and integrate it into a fedora-bootc based bootc image:
FROM quay.io/fedora/fedora-bootc
RUN mkdir -p /etc/containers/systemd
COPY test.container /etc/containers/systemdThe Quadlet service will start on boot, and there is nothing else to do. This allows for a great hands-off experience when managing Linux hosts, as the entire workload is declared at once. Traditional config management and provisioning can shift left into the build process.
Pulling images
Running containers via Quadlets requires container images on the bootc system. You can use the following options for pulling images in combination, depending on the use case requirements and user needs.
On-demand pulls
Container images can be pulled on-demand whenever a Quadlet is started and the referenced image is not yet present in the local container's storage.
The big advantage of this model is the ease of use, since we only need to install and run a Quadlet and let Podman deal with the pulling. Podman containers started by Quadlet and those created by the user or other tools can all share the same store.
A major disadvantage of on-demand pulls is that it delays the starting of workloads until the individual images have been pulled down. This may even impact boot time when Quadlets start at boot. Moreover, disconnected environments cannot make use of on-demand pulls at all due to the lack of network connectivity.
Logically-bound pre-pulled images
Logically-bound images are an improvement of on-demand pulls. They allow images to be pre-pulled at install time and on updates. This way, logically-bound images must not be pulled when a Quadlet starts because those images are already present on the system.
You can specify logically-bound images in the /usr/lib/bootc/bound-images.d directory in the form of symlinks. bootc will automatically pull the images on bootc install, bootc upgrade, and bootc switch. The symlinks in the directory point to Quadlet files on the system. Currently, those Quadlets must either be .container or .image files.
An example of a Containerfile using logically-bound images may look like the following:
FROM quay.io/myorg/myimage:latest
COPY ./my-app.container /usr/share/containers/systemd/another-app.container
RUN ln -s /usr/share/containers/systemd/my-app.container /usr/lib/bootc/bound-images.d/my-app.containerTo access logically-bound images, .container Quadlets need to add the following to the [Container] table:
GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storageThis setting allows Podman to access the so-called additional image store of bootc. Please note that this solution of using GlobalArgs is preferable over a system-wide configuration in storage.conf unless all containers run in Quadlets. For more details on logically-bound images, refer to the upstream documentation of bootc and the Fedora examples.
Physically-bound images
Some use cases require the entire boot image to be fully self-contained. That means that everything needed to execute the workloads is shipped with the bootc image, including container images of the application containers and Quadlets. Such images are also referred to as physically-bound images.
The underlying mechanism is very similar to the one of logically-bound images, in that you can pre-pull physically-bound images during image build time and make them available at runtime. Let’s first dive into how we can achieve such physical embedding. We'll explain later why we recommend doing it this way.
The instruction in a Containerfile to physically embed an image at build time may look like this:
RUN skopeo copy --preserve-digests docker://<IMAGE> dir:/usr/lib/containers-image-cache/<DIRECTORY>At runtime, you can move the image into Podman’s mutable store as follows:
RUN skopeo copy --preserve-digests dir:/usr/lib/containers-image-cache/<DIRECTORY> containers-storage:<IMAGE>Since the embedded images may change with each system update, we cannot use an additional image store for this purpose, as it was not designed to be swapped out. Instead, we make use of the dir transport of the container tools, which allows for storing one or more images in the same directory. The dir transport further allows us to preserve the digest of the image, which is crucial for the original digest to reference the image. However, there is one caveat: we need to keep track of the name of the image.
To improve the experience of physically embedding container images, we propose two scripts that you can find in the upstream Fedora bootc examples.
The following is an example of physically embedding a number of images in a Containerfile:
COPY ./embed_image.sh /usr/bin/
COPY ./copy_embedded_images.sh /usr/bin/
RUN <<PULL
/usr/bin/embed_image.sh registry.fedoraproject.org/fedora:latest
/usr/bin/embed_image.sh docker.io/library/busybox:latest
/usr/bin/embed_image.sh docker.io/library/alpine@sha256:ca1c944a4f8486a153024d9965aafbe24f5723c1d5c02f4964c045a16d19dc54 --all
PULLTo copy the images into Podman's mutable store at runtime, just run /usr/bin/copy_embedded_images.sh. Note that the images must be copied over before any container or service (e.g., Quadlet), depending on if such an image is started. It could be moved into a systemd unit that starts before any Quadlet, for instance.
For more information, see the upstream Fedora bootc examples. In the meantime, we are working on improving the user experience when using physically-embedded images.
Tagging, versioning, and referencing images
An important aspect of embedding images is the way they are referenced. A tag, digest, or a mix of both can reference images. While digests always point to exactly one image, a tag may be updated on a registry. Choosing the right way of referencing an image can impact the quality and robustness of workloads, so we should be intentional about it. If you are interested in this topic, read an earlier article on how to name, version, and reference container images.
Final thoughts
In this article, I have outlined a number of strategies for embedding containers on image mode. They all serve different purposes and use cases, but you can also use them in combination. Our use cases, requirements, and the degree of automation should determine what’s best in a given environment. I hope that this article will help you navigate this space.
