Using Apache Yetus with Jenkins and Github: Part 1

By | 2019-01-24


Apache Yetus is a toolbox for building and releasing software.  One of the tools is a generalized framework for performing patch and full build testing for continuous integration systems. It supports a wide variety of build tools and features. As a result, it can be overwhelming to get started using it. In this blog post, I’ll run through a quick example of configuring Jenkins Multibranch Pipelines using the Github Branch Source Plugin as a starting point for your own usage.  In a follow-up post, I’ll cover a more advanced example, custom Docker images, and a few different use cases.


A file is expected to be committed to the source tree to use Multibranch Pipelines. That file contains the pipeline definition. By default, this file is expected to be named Jenkinsfile, but it may be called anything you want as long as Jenkins is told the name at job definition time. In this example, we will also assume it is called Jenkinsfile.


Credential Pre-work

In order for Jenkins and Apache Yetus to be able to communicate with Github, at least one username and OAuth token combination needs to be generated, as described in the Github documentation.   Due to the suboptimal Github authorization system, repo:status and admin:repo_hook are the required scopes.

This username/token combination should be stored as described by the Jenkins documentation.  The credential should be configured such that it is a username and password combination, where the password is the OAuth token for that account.  In this example, the id of the credential is assumed to be ‘yetus-github-account’ but may be anything you want.

Using the Apache Yetus Convenience Docker Image


Pipelines  on Jenkins can be configured similarly to other CI systems where a Docker image may be used as the environment. Apache Yetus publishes convenience images on Docker Hub that may also be used in this way. With it, you get a reasonably functional image with quite a few development tools and all of the Apache Yetus bits built into it.

A Jenkinsfile that uses the Apache Yetus image should look something like the following:

The ‘agent’ section defines what Docker image to use. In this example, I am using Apache Yetus version 0.9.0. In many setups, it may be desirable to configure the container to run on a particular set of labeled nodes. Uncomment the ‘label’ line and replace the label to match what is required in your particular setup.

Next up is the ‘options’ setting. This section configures a few global settings, including maximum runtime, number of old run logs to keep around, where the source should be checked out, etc. Add/change as desired.

The environment section dictates where things will be located. In this example file, there are two:

  • SOURCEDIR specifies where in the workspace the git repository has been cloned. By default, Jenkins will extract into the workspace. That configuration isn’t desirable in this use case because it also needs to have a directory to store Yetus’ output…
  • YETUS_PATCHDIR is where that output will go. It needs to be relative to the WORKSPACE for artifact archiving to be able to work correctly to grab Yetus’ logs.

In Jenkins Pipelines, the stages stanza tells Jenkins what it is supposed to execute for this pipeline. These stages can be as complex as needed. For this example, it only has one: run Apache Yetus.

The ‘apache-yetus’ stage is the meat of this pipeline. It starts by getting the Github credentials that should be used to use the Github API to both write commentary and get more details about pull requests. Be sure to change the ID to the one that you used previously.  

After that, a relatively simple shell script is executed. It does a few housekeeping things:

  • Forcibly change the directory to WORKSPACE
  • Forcibly set the JENKINS_URL environment variable to work around a Jenkins bug
  • Does some work to ensure the YETUS_PATCHDIR directory is cleaned out

From here on is where the fun stuff happens. The YETUS_ARGS array contains all of the options that need to get passed to the ‘test-patch’ command:

  • --patch-dir sets the output directory. This directory will get archived later.
  • --basedir sets the source directory so that the system knows what directory to process
  • The various report options configure up different output formats that are useful in a variety of contexts.
  • The --github options tell test-patch the username and password to use to talk to Github
  • The --plugins option configure which optional features to use. Here, it turns them all on (including GPL ones if that is an operational concern)
  • The --build-url-artifacts option helps test-patch tailor its output to match how Jenkins will present URLs for the archived output.
  • The --empty-patch option tells test-patch that if no PR was provided, run as though it was in ‘qbt — quality build tool’ mode by treating the entire source tree as a patch.

Then it executes test-patch.

After that stanza is a ‘post’ section. This section tells Jenkins to perform various tasks after the main work is finished. In this example it has:

  • Archive the output directory as job artifacts
  • Publish the HTML report on the job webpage to provide an easier to read a table of output
  • Delete the workspace directory after completion. This function prevents space from being consumed on the slaves.

This file should now be committed as Jenkinsfile to the root of the source tree of the repo in the master branch so that Jenkins will see it after the job is added.

Jenkins Job Configuration

Now that I have Jenkinsfile, it is time to create the actual job. [In this example, I’m going to configure my personal Apache Zookeeper repository.]

Select New Item, give it a name, and then use Multibranch Pipeline:

Next, configure the Branch Source to be Github. The credentials used here should be ones that have access to the repository. Provide the owner of the repo and then select the repo to test. You will also probably want to change the Trust setting as appropriate for your project. The question mark on the right-hand side can provide some guidance:

Save the job. Congratulations, you should now have a working pipeline! Jenkins should be using the user configured to update the Github web hook and processing all of the existing branches and pull requests.  It should also trigger a build for any of those that already have a the Jenkinsfile present.  All new branches pull requests that have the Jenkinsfile should also start processing.  Apache Yetus will process the content and post results on both the Github PR page and on the Jenkins build page.


Web hooks

If for whatever reason you cannot use a token that actually has permission to set the web hook, let the person who does have administrative control set it to be JENKINS_URL/github-webhook/ , where JENKINS_URL is the base URL you use to get to Jenkins.  (For example,  The web hook needs minimally pushes and pull request notifications.  For more help, checkout out the Cloudbees website.


To ensure that our unit tests and the like don’t cause problems, Apache Yetus sets a relatively small process limit. For larger, complex projects, this may not be enough. To increase the upper bound on the number of processes, the --proclimit option may be used. Add the following line to the YETUS_ARGS sequence:

Ignoring Test Plug-in Failures

Running Apache Yetus on existing source trees tends to highlight problems that may or may not have been previously known. For example, excessive whitespace may take a more organized effort to clean. In the meantime, you may not want to have the run fail but still generate a report. Ignoring results can be done by using the --tests-filter option:

This way, tests don’t have to be disabled entirely.

Disabling Plug-ins

In some source trees, it may not be appropriate to run particular Apache Yetus plug-ins. The --plugins option supports a basic logic operator to disable tests selectively while still running everything else:

In this example, the ‘authors’ plugin that checks for the existence of ‘@authors’ lines has been entirely disabled, perhaps because our project wants to use them.

Other Build Tools

While this example used maven, Apache Yetus supports a wide variety of other build tools. In these examples, changing the --build-tool parameter and adding any additional build tool specific parameters should be all that is required. For example, to use Apache Ant:

alternatively, perhaps cmake:

Or, maybe none at all:


This post covers some of the basics required to get a (relatively) simple project working. In a future post, I’ll cover how to import Apache Yetus into an existing Docker image, how to use test-patch to build the image on the fly, and a few more use cases.

Leave a Reply