Evolution of the LHCb Continuous Integration system
The physics software stack of LHCb is based on Gaudi and is comprised of about 20 interdependent projects, managed across multiple GitLab repositories. At present, the continuous integration (CI) system used for regular building and testing of this software is implemented using Jenkins and runs on a...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2020-01-01
|
| Series: | EPJ Web of Conferences |
| Online Access: | https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_05039.pdf |
| _version_ | 1818885255719813120 |
|---|---|
| author | Currie Robert Mataev Rosen Clemencic Marco |
| author_facet | Currie Robert Mataev Rosen Clemencic Marco |
| author_sort | Currie Robert |
| collection | DOAJ |
| description | The physics software stack of LHCb is based on Gaudi and is comprised of about 20 interdependent projects, managed across multiple GitLab repositories. At present, the continuous integration (CI) system used for regular building and testing of this software is implemented using Jenkins and runs on a cluster of about 300 cores.
LHCb CI pipelines are python-based and relatively modern with some degree of modularity, i.e. the separation of test jobs from build jobs. However, these still suffer from obsoleted design choices that prevent improvements to scalability and reporting. In particular, the resource use and speed have not been thoroughly optimized due to the predominant use of the system for nightly builds, where a feedback time of 8 hours is acceptable. We describe recent work on speeding up pipelines by aggressively splitting and parallelizing checkout, build and test jobs and caching their artifacts. The current state of automatic code quality integration, such as coverage reports, is shown.
This paper presents how feedback time from change (merge request) submission to build and test reports is reduced from “next day” to a few hours by dedicated on-demand pipelines. Custom GitLab integration allows easy triggering of pipelines, including linked changes to multiple projects, and provides immediate feedback as soon as ready. Reporting includes a comparison to tests on a unique stable reference build, dynamically chosen for every set of changes under testing. This work enables isolated testing of changes that integrates well into the development workflow, leaving nightly testing primarily for integration tests. |
| first_indexed | 2024-12-19T16:02:32Z |
| format | Article |
| id | doaj.art-ced0bd22a9c944b189395161c01aa1ca |
| institution | Directory Open Access Journal |
| issn | 2100-014X |
| language | English |
| last_indexed | 2024-12-19T16:02:32Z |
| publishDate | 2020-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | EPJ Web of Conferences |
| spelling | doaj.art-ced0bd22a9c944b189395161c01aa1ca2022-12-21T20:14:54ZengEDP SciencesEPJ Web of Conferences2100-014X2020-01-012450503910.1051/epjconf/202024505039epjconf_chep2020_05039Evolution of the LHCb Continuous Integration systemCurrie Robert0Mataev Rosen1Clemencic Marco2University of EdinburghCERNCERNThe physics software stack of LHCb is based on Gaudi and is comprised of about 20 interdependent projects, managed across multiple GitLab repositories. At present, the continuous integration (CI) system used for regular building and testing of this software is implemented using Jenkins and runs on a cluster of about 300 cores. LHCb CI pipelines are python-based and relatively modern with some degree of modularity, i.e. the separation of test jobs from build jobs. However, these still suffer from obsoleted design choices that prevent improvements to scalability and reporting. In particular, the resource use and speed have not been thoroughly optimized due to the predominant use of the system for nightly builds, where a feedback time of 8 hours is acceptable. We describe recent work on speeding up pipelines by aggressively splitting and parallelizing checkout, build and test jobs and caching their artifacts. The current state of automatic code quality integration, such as coverage reports, is shown. This paper presents how feedback time from change (merge request) submission to build and test reports is reduced from “next day” to a few hours by dedicated on-demand pipelines. Custom GitLab integration allows easy triggering of pipelines, including linked changes to multiple projects, and provides immediate feedback as soon as ready. Reporting includes a comparison to tests on a unique stable reference build, dynamically chosen for every set of changes under testing. This work enables isolated testing of changes that integrates well into the development workflow, leaving nightly testing primarily for integration tests.https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_05039.pdf |
| spellingShingle | Currie Robert Mataev Rosen Clemencic Marco Evolution of the LHCb Continuous Integration system EPJ Web of Conferences |
| title | Evolution of the LHCb Continuous Integration system |
| title_full | Evolution of the LHCb Continuous Integration system |
| title_fullStr | Evolution of the LHCb Continuous Integration system |
| title_full_unstemmed | Evolution of the LHCb Continuous Integration system |
| title_short | Evolution of the LHCb Continuous Integration system |
| title_sort | evolution of the lhcb continuous integration system |
| url | https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_05039.pdf |
| work_keys_str_mv | AT currierobert evolutionofthelhcbcontinuousintegrationsystem AT mataevrosen evolutionofthelhcbcontinuousintegrationsystem AT clemencicmarco evolutionofthelhcbcontinuousintegrationsystem |