Standalone containers with ATLAS offline software
This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can en...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2020-01-01
|
| Series: | EPJ Web of Conferences |
| Online Access: | https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_07010.pdf |
| _version_ | 1818456077133414400 |
|---|---|
| author | Vogel Marcelo Borodin Mikhail Forti Alessandra Heinrich Lukas |
| author_facet | Vogel Marcelo Borodin Mikhail Forti Alessandra Heinrich Lukas |
| author_sort | Vogel Marcelo |
| collection | DOAJ |
| description | This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can encapsulate software packages inside complete file systems. The deployment of offline releases via containers removes the interdependence between the runtime environment needed for job execution and the configuration of the computing nodes at the sites. Docker or Singularity would provide a uniform runtime environment for the grid, HPCs and for a variety of opportunistic resources. Additionally, releases may be supplemented with a detector’s conditions data, thus removing the need for network connectivity at computing nodes, which is normally quite restricted for HPCs. In preparation to achieve this goal, we have built Docker and Singularity images containing single full releases of ATLAS software for running detector simulation and reconstruction jobs in runtime environments without a network connection. Unlike similar efforts to produce containers by packing all possible dependencies of every possible workflow into heavy images (≈ 200GB), our approach is to include only what is needed for specific workflows and to manage dependencies efficiently via software package managers. This approach leads to more stable packaged releases where the dependencies are clear and the resulting images have more portable sizes ( 16GB). In an effort to cover a wider variety of workflows, we are deploying images that can be used in raw data reconstruction. This is particularly challenging due to the high database resource consumption during the access to the experiment’s conditions payload when processing data. We describe here a prototype pipeline in which images are provisioned only with the conditions payload necessary to satisfy the jobs’ requirements. This database-on-demand approach would keep images slim, portable and capable of supporting various workflows in a standalone fashion in environments with no network connectivity. |
| first_indexed | 2024-12-14T22:20:56Z |
| format | Article |
| id | doaj.art-6a8e5ea299824790980e6b665fd668c2 |
| institution | Directory Open Access Journal |
| issn | 2100-014X |
| language | English |
| last_indexed | 2024-12-14T22:20:56Z |
| publishDate | 2020-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | EPJ Web of Conferences |
| spelling | doaj.art-6a8e5ea299824790980e6b665fd668c22022-12-21T22:45:30ZengEDP SciencesEPJ Web of Conferences2100-014X2020-01-012450701010.1051/epjconf/202024507010epjconf_chep2020_07010Standalone containers with ATLAS offline softwareVogel Marcelo0Borodin Mikhail1Forti Alessandra2Heinrich Lukas3Faculty of Mathematics and Natural Sciences, University of WuppertalDepartment of Physics and Astronomy, University of IowaSchool of Physics and Astronomy, University of ManchesterEuropean Organization for Nuclear Research (CERN)This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can encapsulate software packages inside complete file systems. The deployment of offline releases via containers removes the interdependence between the runtime environment needed for job execution and the configuration of the computing nodes at the sites. Docker or Singularity would provide a uniform runtime environment for the grid, HPCs and for a variety of opportunistic resources. Additionally, releases may be supplemented with a detector’s conditions data, thus removing the need for network connectivity at computing nodes, which is normally quite restricted for HPCs. In preparation to achieve this goal, we have built Docker and Singularity images containing single full releases of ATLAS software for running detector simulation and reconstruction jobs in runtime environments without a network connection. Unlike similar efforts to produce containers by packing all possible dependencies of every possible workflow into heavy images (≈ 200GB), our approach is to include only what is needed for specific workflows and to manage dependencies efficiently via software package managers. This approach leads to more stable packaged releases where the dependencies are clear and the resulting images have more portable sizes ( 16GB). In an effort to cover a wider variety of workflows, we are deploying images that can be used in raw data reconstruction. This is particularly challenging due to the high database resource consumption during the access to the experiment’s conditions payload when processing data. We describe here a prototype pipeline in which images are provisioned only with the conditions payload necessary to satisfy the jobs’ requirements. This database-on-demand approach would keep images slim, portable and capable of supporting various workflows in a standalone fashion in environments with no network connectivity.https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_07010.pdf |
| spellingShingle | Vogel Marcelo Borodin Mikhail Forti Alessandra Heinrich Lukas Standalone containers with ATLAS offline software EPJ Web of Conferences |
| title | Standalone containers with ATLAS offline software |
| title_full | Standalone containers with ATLAS offline software |
| title_fullStr | Standalone containers with ATLAS offline software |
| title_full_unstemmed | Standalone containers with ATLAS offline software |
| title_short | Standalone containers with ATLAS offline software |
| title_sort | standalone containers with atlas offline software |
| url | https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_07010.pdf |
| work_keys_str_mv | AT vogelmarcelo standalonecontainerswithatlasofflinesoftware AT borodinmikhail standalonecontainerswithatlasofflinesoftware AT fortialessandra standalonecontainerswithatlasofflinesoftware AT heinrichlukas standalonecontainerswithatlasofflinesoftware |