Photo by Claudio Schwarz on Unsplash

Currently, we support CentOS Stream 8 for open-source Vespa, and we announced that in the Upcoming changes in OS support for Vespa blog post in 2022. The choice to use CentOS Stream was made around the time that RedHat announced the EOL for CentOS 8 and the new CentOS Stream initiative. Other groups were scrambling to be the successor to CentOS, and the landscape was not settled. This is now about to change.

Introduction

We are committed to providing Vespa releases that we have high confidence in. Internally, at Vespa.ai, we have migrated to AlmaLinux 8 and Red Hat Enterprise Linux 8. CentOS Stream 8 is also approaching its EOL, which is May 31st, 2024. Because of this, we want to change the supported OS for open-source Vespa.

Vespa is released up to 4 times a week depending on internal testing and battle-proven verification in our production systems. Each high-confidence version is published as RPMs and a container image. RPMs are built and published on Copr, and container images are published on Docker Hub. In this blog post, we will look at options going forward and announce the selected OS support for the upcoming Vespa releases.

Options

There is a wide selection of Linux distributions out there with different purposes and target groups. For us, we need to choose an OS that is as close to what we use internally as possible, and that is acceptable for our open-source users. These criteria limit the options significantly to an enterprise Linux-based distribution.

RPMs

Binary releases of Vespa is a set of RPMs. This set has to be built somewhere and must be uploaded to a repository where it can be downloaded by package managers. These RPMs are then installed either on a host machine or in container images. We will still build our RPMs on Copr, but we have a choice there to compile for different environments that are either downstream or upstream of RHEL. In the time between the EOL of CentOS 8 (end 2021) and now, Copr has added support to build for EPEL 8 and 9. This means that we can build for EPEL 8 and install it on RHEL 8 and its downstream clones.

Distribution of RPMs is currently done on Copr as the built RPMs are directly put in the repository there. The repositories have limited capacity, and Copr only guarantees that the latest version is available. It would be nice to have an archive of more than just the most recent version, but this will rely on vendors offering space and network traffic for the RPMs to be hosted.

Container images

Given the choice of building RPMs on Copr for EPEL 8, this opens up a few options when selecting a base image for our container image:

• AlmaLinux
• Rocky Linux
• Oracle Linux

We should be able to select any of the above due to the RHEL ABI compatibility and the distributions’ respective guarantees to be binary compatible with RHEL.

Red Hat has also announced a free version of RHEL for developers and small workloads. However, it is not hassle-free, as it requires registration at Red Hat to be able to use the version. We believe that this will not be well received by the consumers of Vespa releases.

Selection

Container Image

Considering the options, we have chosen AlmaLinux 8 as the supported OS going forward. The main reasons for this decision are:

• AlmaLinux is used in the Vespa Cloud production systems
• The OS is available to anyone free of charge
• We will still be able to leverage the Copr build system for open-source Vespa artifacts

We will use the docker.io/almalinux:8 image as the base for the Vespa container image on Docker Hub.

RPM distribution

Although we will continue to build RPMs on Copr, we are going to switch to a new RPM repository that can keep an archive of a limited set of historic releases. We have been accepted as an open-source project at Cloudsmith and will use the vespa/open-source-rpms repository to distribute our RPMs. Cloudsmith generously allows qualified open-source projects to store 50 GB and have 200 GB of network traffic. The vespa-engine.repo repository definition will be updated shortly, and information about how to install Vespa from RPMs can be found in the documentation. Within our storage limits, we will be able to store approximately 50 Vespa releases.

Compatibility for current Vespa installations

The consumers of Vespa container images should not notice any differences when Vespa changes the base of the container image to AlmaLinux 8. Everything comes preinstalled in the image, and this is tested the same way as it was before. If you use the Vespa container image as a base of custom images, the differences between the latest AlmaLinux 8 and CentOS Stream 8 are minimal. We do not expect any changes to be required.

For consumers that install Vespa RPMs in their container images or install directly on host instances, we will continue to build and publish RPMs for CentOS Stream 8 on Copr until Dec 31st, 2023. RPMs built on EPEL 8 will be forward compatible with CentOS Stream 8 due to the RHEL ABI compatibility. This means that you can make the switch by replacing the repository configuration with the one defined in vespa-engine.repo the next time Vespa is upgraded. If you do not, no new Vespa versions will be available for upgrade once we stop building for CentOS Stream 8.

Future OS support

Predicting the path of future OS support is not trivial in an environment where the landscape is changing. RedHat announced closing off the RHEL sources and strengthening CentOS Stream. Open Enterprise Linux Association has popped up as a response to this, and AlmaLinux commits to binary compatibility. We expect the landscape to change, and hopefully, we will have more clarity when deciding on the next OS to support.

Regardless of the landscape, we are periodically publishing a preview on AlmaLinux 9 that can be used at your own risk here. Please use this for testing purposes only.

Summary

We have selected AlmaLinux 8 as the supported OS for Vespa going forward. The change is expected to have no impact on the consumers of Vespa container images and RPMs. The primary RPM repository has moved to a Cloudsmith-hosted repository where we can have an archive of releases allowing installation of not just the latest Vespa version.