Using Apache Yetus with Jenkins and Github: Part 2

Introduction

In the last post, I introduced a Jenkinsfile that used Apache Yetus to do source tree analysis. However, that example ran the entire process in the Apache Yetus’ default container. For some projects that may not work, especially in consideration of any custom requirements. Plus, it didn’t produce an artifact which may also be required. In this post, I’ll cover some other techniques of integrating Apache Yetus into an existing workflow.

Using a Container for an Existing Pipeline

Jenkins can run parts of the pipeline in different containers. Using that functionality, it’s possible to take advantage of Apache Yetus in an existing pipeline without a significant amount of work by just introducing an Apache Yetus to your existing pipeline. Here is an example that leverages the pre-built container image:   This pipeline definition is similar to the one used in Part 1, with a few notable differences. Like the previous one, the pipeline extracts the Github repository to ${WORKSPACE}/src, downloads the Apace Yetus 0.10.0 docker image, and then runs test-patch against the source tree. It generates reports and such in ${WORKSPACE}/out. That directory will be archived later on for further perusal. The HTML report is also published on the Jenkins page as well. If Apache Yetus cannot compile your source tree (e.g., missing resources), uncommenting the nobuild option configures test-patch to not trigger any known source tree building tools, generally only enabling static linters. The most significant difference is that the only part of the pipeline that runs in the Apache Yetus container image is Apache Yetus itself. Other stages execute in either the Jenkins’ worker environment or, if defined with similar docker() statements, in a container. It is important to note that when running in this mode, Jenkins automatically populates some environment variables and volume mounts the entirety of the ${WORKSPACE} directory in the container. Thus this pipeline still uses a modified directory structure to share data to the world outside of its container. Adding additional volumes and adding additional parameters (e.g., --patchdir) to the test-patch command line would allow one to use a different structure if changing the repository clone location is too painful.

Customizing the Build via Local Installation

For complex projects, it may be desirable to run test-patch in the same environment that the build uses typically, either on the build node or in a custom container. This type of setup requires getting a version of Apache Yetus and its dependencies installed in the appropriate environment. First, let us get Apache Yetus installed. Officially, source archives are the official release mechanism of Apache Software Foundation projects. This type of distribution is less than ideal for our purposes. The Apache Yetus project also makes available a pre-built, binary package. Installation of a single version as part of your regular Jenkins slave installation is reasonably trivial: grab the binary package from https://yetus.apache.org/downloads/, verify against the GPG key, and install. Be aware that the URLs of versions change. Apache Software Foundation policy prevents projects from keeping every version installed on a mirror. So as versions get removed, the download link also changes from a local mirror to the central Apache archival server. The Apache Yetus project provides some sample code here to download, verify, and install the latest released version in a directory of your choosing. [Users of this code should be aware that Apache Yetus doesn’t follow semantic versioning so that incompatibilities appear from time-to-time.]   While greatly discouraged, another choice is to run directly from the source tree. Currently, Apache Yetus’ precommit toolset doesn’t require compiling, making this a viable choice if your project only has access to GitHub. To run Apache Yetus directly from a git clone of its source repository:   In this example, we git clone a known release of the Apache Yetus source tree and run from there. This method allows the test-patch to run in the same environment as Jenkins’s worker node without having to worry about changing URLs or installing a script in your source tree.

Custom Docker Environment

The other big benefit of running test-patch from outside the container is that it is possible to run it inside a built-at-runtime container. This method is exactly how Apache Yetus runs against itself. Other examples include Apache Hadoop and Apache HBase. In all of these examples, you can see that the --docker parameter. This option tells test-patch that it should run the test in a container. The --dockerfile option points to the location of a Dockerfile. If provided, test-patch builds that content as the container and then runs inside of it. If the --dockerfile option is not provided, then test-patch will use the same image as apache/yetus-base either by downloading it or by building it from scratch. Another option of interest may be the --docker-cache-from option. This option is the same as using the docker build --cache-from option to speed up builds. Apache Yetus downloads the images passed and then use those to build the run-specific container image. Using this method, building the Dockerfile only when there are changes makes the whole process significantly faster. It is important to note that if a custom container must have all of the executables and support libraries for all of the plug-ins that you want to run with test-patch. A comprehensive list is maintained as part of Apache Yetus’ documentation. Where the necessary pre-requisites are not available, test-patch will warn that the feature is not available and move on.

Conclusion

That’s it for this post. Stayed tuned for a future edition of this series covering freestyle jobs and other secrets of integrating Apache Yetus with Jenkins!