By Arnstein Ressem, Principal Software Systems Engineer, Verizon Media

When Vespa was open sourced in 2017 we looked for a continuous integration platform to build our source code on. We looked at several hosted solutions as well as Screwdriver – an open source CI/CD platform built by Yahoo/Verizon Media – that had just been open sourced in 2016. Another platform seemed the best fit for us at that point in time and we integrated with that.

Photo by Bill Oxford on Unsplash

The Vespa codebase is large with approximately 700 KLOC C++, 700 KLOC Java and more than 10k unit tests. For a given version of Vespa we build the complete codebase and version the artifacts with semantic versioning. We always build from the master branch and have no feature branches.

Compiling and testing this codebase is resource demanding and we soon realized that the default VMs that the provider had were not up to the task and took more than 2 hours to complete. This was a serious issue for the developers waiting for feedback on their pull requests. We ended up subscribing to a premium plan and did more caching of Maven artifacts and compiled C++ objects (ccache) to bring the build time just under one hour.

In the fall of 2020 we became aware of big changes in the selected CI/CD platform and we needed to migrate to something else. As part of this work we took another look at the open sourced version of Screwdriver as we knew that the project had significantly matured over the past years. Screwdriver is an open source build platform designed for Continuous Delivery that can easily be deployed on different IaaS providers and is currently an incubee in the Continuous Delivery Foundation.

Vespa pipeline on Screwdriver

The Vespa team got access to a hosted instance at cd.screwdriver.cd (invite only, but publicly readable with guest access). Working closely with Screwdriver we were able to reduce the build times for the master branch and pull requests from 50 minutes on the previous solution to 18 minutes. This result was obtained by using Screwdriver’s configurable resource management and fast build caches. We also appreciated the small set of requirements on container images allowing us to optimize the build image for our jobs.

Screwdriver integrated with pull request builds on GitHub

To further increase the developer feedback and productivity we decided to do some pull request analysis to check if only C++ or Java source code was touched. In those cases we could only build and test for the respective language. This brought the pull request build times from 18 down to 12 minutes for C++ and 8 minutes for Java. This allowed developers to have more issues discovered in pull requests without having to wait for a long time for the review and merge.

We are very happy with having the time spent on builds and pull requests reduced by 75% on average and this leads to better productivity and happier developers.