Integrating Trac, Jenkins and Cobbler—Customizing Linux Operating Systems for Organizational Needs

The daily repositories either are downloaded from the Web on a daily basis or synchronized from the local filesystem. The daily repositories get the "keep updated" flag set, while the test and production repositories do not. For daily repositories that synchronize from a local filesystem, the "breed" should be set to rsync, while daily repositories that synchronize from the Web should set their "breed" to yum. This configuration, through experience, has been chosen because some RPMs do not upgrade well with new kernels nor do they have standard update processes normal to Red Hat or Fedora.

An example of a set of repositories would be as follows:

  • phi-6-x86_64-daily — synchronizes automatically from the local filesystem using rsync once daily.

  • epel-6-x86_64-daily — synchronizes automatically from the Web using reposync once daily.

  • phi-6-x86_64-test — synchronizes manually from phi-6-x86_64-daily using rsync.

  • epel-6-x86_64-test — synchronizes manually from epel-6-x86_64-daily using rsync.

  • phi-6-x86_64-prod — synchronizes manually from phi-6-x86_64-test using rsync.

  • epel-6-x86_64-prod — synchronizes manually from epel-6-x86_64-test using rsync.

To exclude critical packages from the upstream distribution, the "yum options" flags are set on the daily repository to remove them. For example, to exclude the kernel package from from being synchronized, add exclude=kernel*. It's important for administrators to consult both the Cobbler and yum.conf man pages to get the syntax right.

Setting up Cobbler in this way allows administrators to deploy systems using customized critical packages. Cobbler also is used in future phases where the repositories are used to deploy the test and production clusters. The repositories and their relationships are all Cobbler needs to support package building, the test cluster and the production cluster.


Jenkins is a very powerful continuous integration tool used in software development. However, from a system administration view, Jenkins is a mutant cron job on steroids. Jenkins handles periodic source code checkout from source code management (SCM) repositories and downloading of released source code, via HTTP or FTP. It then runs a series of generic jobs that build, test and deploy the resulting software. These generic interfaces work well for building and distributing RPMs to be included by Cobbler.

The use of Jenkins in a software development role is not all that different from building RPMs (see Table 1 for a comparison of the two processes). The first step in the two processes differs in that (hopefully) the software development code required for the build step is in one place. Package developers need to have, at a minimum, two locations to pull code from to continue with the build. The first location is for patches and spec files, normally kept in an SCM. The second is for released source code packages. Source code is released in a single file and usually in some container format (such as tar, rar or zip). These files do not normally belong in an SCM and are more suited to an S3 (, swift ( or blob store-like interface.

Table 1. Packaging vs. Development

Software Development RPM Packaging
Download source code from SCM. Download released source, spec file and patches.
Run the build process. Build the RPMs using Mock.
Run the testing suite. Validate the RPMs using rpmlint.
Publish test results. Save validation output for inspection.
Save source code package to repository. Save built RPMs for later download.
Send notification to pertinent developers. Send notification to pertinent packagers.

Jenkins is built primarily for downloading code from one and only one SCM. However, you can work around this issue by adding another build step. This means that the SCM plugin is used to download the spec file and patches while the first step in the build process downloads the source code package. After these two steps are done, the source code, patches or spec file can be patched with site-specific customization.

The next step is to build RPMs using Mock. This involves several tasks that can be broken up into various build steps (see the Mock Build in Jenkins sidebar). All these steps are done using the Jenkins execute shell build steps. Some of the Jenkins jobs we use are multi-configuration jobs that contain one axis defining the Mock chroot configuration. That chroot configuration should be generated from the daily repositories defined in Cobbler. Following these tasks can get you started on using Mock in Jenkins (Listing 1).

Listing 1.

#!/bin/bash -xe

# keep in mind DIST is defined in multi-configuration axis
MOCK="/usr/bin/mock -r $DIST"
# keep in mind VER could also be a multi-configuration axis
# if you are ripping apart an RPM might have this one too


wget -O $PKG-$VER.tar.gz 
rm -f $OUT/*.src.rpm
if ! $MOCK --resultdir=$OUT --buildsrpm --spec=$PKG.spec 
    more $OUT/*.log | cat
    exit -1

if ! $MOCK --resultdir=$OUT --rebuild $OUT/*.src.rpm
    more $OUT/*.log | cat
    exit -1

rpmlint $OUT/*.rpm > rpmlint.log

Mock Build in Jenkins

  1. Prepare the source and specs.

  2. Run Mock source rpm build.

  3. Run Mock rpm build.

  4. Run rpm validation.


David Brown is a high-performance computing system administrator with a B.S. in Computer Science from Washington State University.