The LArSoft Continuous Integration (CI) system is the set of tools, applications and machines that allows users to specify and automate the execution of tests of LArSoft software. CI has five major components:
- The central build service
- Results database/web server
- Server-side driver software (inside the build server)
- Client-side driver software (inside each build slave)
- Monitoring/reporting software (on the right side in the diagram below)
CI System Overview
The system is deployed on a Jenkins server at Fermilab. When a user runs a script or submits code into the development branch of the repository, a workflow runs and presents information as to the state of the most recent build of the LArSoft code. The Jenkins CI application in the build server runs generic test jobs, also known as builds. To accomplish this, it dispatches jobs to build slaves. There is at least one build slave for each supported operating system. Build slaves can be located anywhere.
At a more detailed level, test jobs are launched in response to a http trigger which starts the LArSoft server-side test driver process. Arguments in the trigger specify run-time configuration options such as the release to use, the branch to build and the test suite to run. The server dispatches LArSoft client-side driver processes to build slaves. Drivers report status, progress, test results, test logs, etc to the database.
Real-time test status, progress and results are viewable via a Web GUI running on a database node called the CI results display page. (More on this after the workflow discussion.)
Default test workflow
A LArSoft script runs in each step of the workflow. The workflow terminates if any step exits with a non-zero return value.
LArSoft CI results display page
The CI results display page displays the status of each step according to the return value. The colors indicate the status of the phase.
The test phases are listed on the top. The boxes are links to summary pages for that phase.
Each point on the graph below is a link to the top-level CI reporting page with the selected test instance listed at the top. (Be patient, it takes time to load.) The total execution time (wall clock) is given for unit test phase versus test instance. And there are links to result summary pages for specific unit tests below the graph. If a test was skipped because a pre-requisite test above failed, the reason is listed.
The left side of the page are links to Jenkins page for the job.
Various studies have shown that the later in a project errors are found, the more it costs to fix them.[1][2][3] The Continuous Integration (CI) approach of software development adopted by LArSoft can save time by finding errors earlier in the process. As Lynn Garren noted, “Using CI allows you to test your code in the complete LArSoft environment independent of other people, which can save you time.” Another important benefit is noted by Gianluca Petrillo, “By using the CI system, I gain confidence that my code changes will not break other people’s work since the tests in the CI system will notify me if something went awry.”
The LArSoft Continuous Integration system is the set of tools, applications and machines that allows users to specify and automate the execution of tests of LArSoft and experiment software. Tests are configured via text files in the various repositories and are launched in response to an http trigger initiated manually by a user or automatically on every git push command to the develop branch of the central repositories. Arguments in the trigger specify run-time configuration options such as the base release to use, the branch within each repository to build (the develop branch by default), and the test “suite” to run, where each suite is a workflow of tests specified in the configuration files. Prior to each test, the new code is built against the base release. The CI system reports status, progress, test results, test logs, etc. to a database and notifies users with a summary of results via email. A web application queries the database to display a summary of test results with drill-down links to detailed information.
As Erica Snider said, “People focus on the testing needed for their particular experiment. One benefit of our CI system is that, every time a change is made, it automatically tests the common code against all of the experiment repositories, then runs the test suites defined by each of the experiments. This allows us to catch interoperability problems within minutes after they are committed.”
This material is based on Erica Snider’s Introduction to LArSoft CI System. More information (including links to detailed instructions on how to run CI tests) can be found at https://cdcvs.fnal.gov/redmine/projects/lar_ci/wiki.