With the growth in the use of containers, the need to bundle your application into a container has never been stronger. Many Red Hat customers will be familiar with Source-to-Image (S2I) as an easy way to build container images from application source code. While S2I is a convenient way to build images on Red Hat OpenShift, Red Hat works to support our customers when using a variety of approaches, and we’ve recently seen an increasing interest in building applications using Cloud Native Computing Foundation (CNCF) Buildpacks.
With the growth in the use of containers, the need to bundle your application into a container has never been stronger. Many Red Hat customers will be familiar with Source-to-Image (S2I) as an easy way to build container images from application source code. While S2I is a convenient way to build images on Red Hat OpenShift, Red Hat works to support our customers when using a variety of approaches, and we’ve recently seen an increasing interest in building applications using Cloud Native Computing Foundation (CNCF) Buildpacks. This blog post covers the journey to enable UBI with the Paketo buildpacks ecosystem
This is the third in a 5-part series of articles on building your applications with CNCF Buildpacks and UBI. The series will include:
- Building applications with Paketo Buildpacks and Red Hat UBI container images
- Running applications with Paketo Buildpacks and Red Hat UBI container images in OpenShift.
- The journey to enable UBI with the Paketo Buildpacks ecosystem (this post).
- Building applications with UBI and Paketo Buildpacks in CI/CD
A Little History..
Buildpacks have fast become an interesting technology to build cloud applications, with the idea originally being credited to Heroku as far back as 2011. However it wasn’t until 2018 that Heroku & Pivotal created the CNCF Project that represents the approach known as “CNCF Buildpacks” today. As envisaged, buildpacks provided a modular mechanism to allow a way for application developers to delegate the act of building their application to a set of buildpacks.
This approach allows for a separation of buildpack authors and application authors, where buildpack authors can focus on the challenges of correctly building and packaging the application to best run in a container, freeing the application developer from caring about Dockerfiles, or similar tooling.
Buildpacks are not alone in offering a solution that aids developers in this manner, there are a variety of other tools that have similar goals, including S2I, Jib and the late Appsody project. Buildpacks however represent a modular approach to the problem, allowing for builder images that can cope with multiple runtimes and application types, prioritizing keeping the business logic small and relevant to each part of the solution.
By 2020, buildpacks were gaining in popularity, and came to the attention of Red Hat. Internally various teams independently started looking into how to provide a Builder image that could work for their scenario, and various small prototypes were created.
This blog post covers some of the history of how those early prototypes led to the contribution of a UBI based stack to Paketo, and how the first Paketo UBI builder image came to be.
Where do your runtimes come from?
Within Red Hat, there are dedicated teams that look after runtimes such as Java and Node.js. They spend countless hours building, testing and patching the runtimes to ensure they work as expected when running in RHEL/UBI. They then package them for distribution as rpms.
When we began looking at adding UBI support to buildpacks we hit an early snag. Buildpacks would not allow rpm distributed runtimes to be installed from a buildpack. While you could create a builder image for each runtime and version, this would quickly lead to an unmaintainable forest of builders. In addition, the forest of builders would shift the responsibility to the user to select the appropriate builder image for each and every project. That in turn would negate one of the main advantages of buildpacks, having a single builder that figures out what to do based on the project being built.
Another alternative would have been to create a single builder/run image pair with every runtime (at every version) pre-installed, and then use configuration to enable the correct one at runtime. Unfortunately, it’s not difficult to see that the resulting image would be too large to be generally acceptable. It would also need to be updated every time any version of any runtime had an update, resulting in an update frequency which would be onerous for users to manage.
Why no rpms?
Why didn’t we just use a buildpack to install rpms? This would have offered a nice solution, except it went against the goals of the CNCF Buildpack project.
Buildpacks in 2020 had a goal of not allowing modifications to the builder, or run images. The intent was that buildpacks would limit all modifications to the images to directories under a single path, and that the system would receive no modifications. This allows for ‘rebasing’, a feature of buildpack built application containers that allows buildpacks to switch out the image underneath an application with an updated one, allowing for easy security updates to already built application images.
If buildpacks were allowed to modify the builder/run images, it could break rebasing. Taking UBI as an example, if the run image was based on the UBI minimal image and a buildpack installed Java, then after the rebase operation Java would be missing, and the application would fail to function.
What then?
The buildpacks community were not unaware of this limitation and in 2020 were looking at a solution that would allow special buildpacks to manipulate the stack, and then those actions would be replayed during a rebase operation. These ‘stackpacks’ would have to be packaged with the application into the run image, but there still could be occasions when the replay would fail. Stackpacks didn’t get much past the early discussion phase, and were quickly superseded by another suggestion.
The suggested alternative would allow ‘extensions’ to direct modification to the builder and run images via generated Dockerfiles. (Indeed for a while, this was known as ‘the dockerfile rfc’ within the buildpacks community). Since the Dockerfiles would be allowed to run instructions with higher privileges, this allowed for the installation of rpms. The rebasing concern was addressed, by allowing the extension to either disable the capability, or set which layers would be retained. Additionally extensions could opt to ‘switch’ the run image to an entirely different one, rather than extending a base run image.
With this capability, an extension could use rpm to install a runtime during the build, and then switch the run image to an appropriate one that had the runtime already installed. Conveniently there are a whole set of these runtime images already published for UBI! By using these existing run images, we are able to take advantage of the significant testing that has already been performed against them.
Why Paketo?
Although the addition of extensions to buildpacks was a game changer for rpm installed runtimes, having an extension that could install Node, or Java, is only a tiny part of building and packaging an application in a container. After the runtime is available, there still remains the tasks of performing the build itself, using whatever build system the project is using, and arranging and selecting the runtime components from the built application. Additionally, there may be other components blended after build to provide cloud functionality, or to update security via custom certificates etc.
Thankfully, Paketo have been doing this a while, and have a mature buildpacks ecosystem consisting of many buildpacks that can harmoniously co-exist within builder images. Even better, by 2021, Paketo had a roadmap goal of adding a UBI based stack.
The choice became clear, to assist Paketo in delivering a UBI based stack, that would use extensions to supply the runtimes.
How Paketo?
A buildpack build is a coordination of one or more buildpacks that have opted to participate in the building of the current application project. The buildpacks can coordinate via a feature of buildpacks called the ‘build plan’, this allows buildpacks to state that they ‘provide’ or ‘require’ things, eg. it allows the maven buildpack to ‘require’ java, and the java buildpack to ‘provide’ it.
That’s a pretty simple example, but it also means that multiple buildpacks could claim to ‘provide’ something, and before actually providing the thing, each buildpack checks the plan to see if the thing is still “in-plan” (if it is still required), once any buildpack has provided a thing, the thing will no longer be in the plan, and subsequent buildpacks can observe this and act appropriately.
Paketo is a whole ecosystem of buildpacks that have a shared understanding and vocabulary around the entries in the buildplan. This allows the buildpacks to act as replaceable modules within a build, where it doesn’t matter which buildpack provides something, as long as any buildpack provides it, under the right build plan entry, the build will succeed.
If it were possible for buildpacks to use rpms, we could just add a UBI-Runtime buildpack to Paketo, and it would install the appropriate version of the appropriate runtime, and claim it ‘provided’ the appropriate thing to the buildplan. This would then allow the rest of Paketo to proceed as if the UBI Runtime were any other Paketo supplied runtime, and complete the build. Because of the image modification restrictions with buildpacks, we can’t do this with a buildpack, however, thanks to the new image extension specification, an extension can!
We created extensions (Node.js, Java) that were able to install Java & NodeJS runtimes via their rpm packages, while appearing to Paketo in the same manner as existing buildpacks that did those tasks. The UBI Builder image includes these new extensions, and builds performed with it will use runtimes from the UBI 8 package repository due to the use of the newly introduced ubi-base-stack.
Much of the work to enable this was not in the creation of the extensions themselves, but in the upgrading of Paketo tooling, and libraries to understand the new extensions specification. This presented significant challenges, not only from a technical perspective of the code itself, but also from a Go ecosystem perspective, related to Go and Paketo version handling.
To create the UBI Builder image required updates to many contributing projects, and much cooperation with the Paketo community. Red Hat continues to work with Paketo, being particularly involved in the Java and Node.js sub teams.
What's next?
Being part of the Paketo community means being aware of the goals, and trying to adopt those for the UBI Builder. Multi-arch support is slowly rolling out, and Java recently updated its default version. We hope to reflect these within the UBI stack as soon as possible along with Red Hat Specific additions like support for UBI 9.
We hope you found this article interesting and that we have motivated you to try out the new Paketo UBI Builder Image. In the follow-up posts we will provide more detail on running applications with Paketo Buildpacks in OpenShift, using buildpacks in CI/CD, and using CNCF Buildpacks with Quarkus.
