Continuous Integration

The LArSoft Continuous Integration (CI) system is the set of tools, applications and machines that allows users to specify and automate the execution of tests of LArSoft software. CI has five major components:

  1. The central build service
  2. Results database/web server
  3. Server-side driver software (inside the build server)
  4. Client-side driver software (inside each build slave)
  5. Monitoring/reporting software (on the right side in the diagram below)

CI System Overview


The system is deployed on a Jenkins server at Fermilab. When a user runs a script or submits code into the development branch of the repository, a workflow runs and presents information as to the state of the most recent build of the LArSoft code. The Jenkins CI application in the build server runs generic test jobs, also known as builds. To accomplish this, it dispatches jobs to build slaves. There is at least one build slave for each supported operating system. Build slaves can be located anywhere.

At a more detailed level, test jobs are launched in response to a http trigger which starts the LArSoft server-side test driver process. Arguments in the trigger specify run-time configuration options such as the release to use, the branch to build and the test suite to run. The server dispatches LArSoft client-side driver processes to build slaves. Drivers report status, progress, test results, test logs, etc to the database.

Real-time test status, progress and results are viewable via a Web GUI running on a database node called the CI results display page. (More on this after the workflow discussion.)

Default test workflow


A LArSoft script runs in each step of the workflow. The workflow terminates if any step exits with a non-zero return value.

LArSoft CI results display page

The CI results display page displays the status of each step according to the return value. The colors indicate the status of the phase.

The test phases are listed on the top. The boxes are links to summary pages for that phase.


Each point on the graph below is a link to the top-level CI reporting page with the selected test instance listed at the top. (Be patient, it takes time to load.) The total execution time (wall clock) is given for unit test phase versus test instance. And there are links to result summary pages for specific unit tests below the graph. If a test was skipped because a pre-requisite test above failed, the reason is listed.


The left side of the page are links to Jenkins page for the job.


This material is based on Erica Snider’s Introduction to LArSoft CI System. More information (including links to detailed instructions on how to run CI tests) can be found at