Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers
Abstract The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occa...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer International Publishing
2021
|
| Online Access: | https://hdl.handle.net/1721.1/132027 |
| _version_ | 1811097445916999680 |
|---|---|
| author | Boccali, T. Cameron, D. Cardo, N. Conciatore, D. Di Girolamo, A. Dissertori, G. Fernandez, P. Filipcic, A. Gila, M. Grab, C. Elmsheuser, J. Jankauskas, V. |
| author2 | Massachusetts Institute of Technology. Department of Physics |
| author_facet | Massachusetts Institute of Technology. Department of Physics Boccali, T. Cameron, D. Cardo, N. Conciatore, D. Di Girolamo, A. Dissertori, G. Fernandez, P. Filipcic, A. Gila, M. Grab, C. Elmsheuser, J. Jankauskas, V. |
| author_sort | Boccali, T. |
| collection | MIT |
| description | Abstract
The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. This is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC. |
| first_indexed | 2024-09-23T16:59:40Z |
| format | Article |
| id | mit-1721.1/132027 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T16:59:40Z |
| publishDate | 2021 |
| publisher | Springer International Publishing |
| record_format | dspace |
| spelling | mit-1721.1/1320272023-11-07T19:56:31Z Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers Boccali, T. Cameron, D. Cardo, N. Conciatore, D. Di Girolamo, A. Dissertori, G. Fernandez, P. Filipcic, A. Gila, M. Grab, C. Elmsheuser, J. Jankauskas, V. Massachusetts Institute of Technology. Department of Physics Abstract The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. This is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC. 2021-09-20T17:41:32Z 2021-09-20T17:41:32Z 2021-02-08 2021-02-11T16:36:38Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/132027 Computing and Software for Big Science. 2021 Feb 08;5(1):7 PUBLISHER_CC en https://doi.org/10.1007/s41781-020-00052-w Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The Author(s) application/pdf Springer International Publishing Springer International Publishing |
| spellingShingle | Boccali, T. Cameron, D. Cardo, N. Conciatore, D. Di Girolamo, A. Dissertori, G. Fernandez, P. Filipcic, A. Gila, M. Grab, C. Elmsheuser, J. Jankauskas, V. Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers |
| title | Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers |
| title_full | Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers |
| title_fullStr | Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers |
| title_full_unstemmed | Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers |
| title_short | Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers |
| title_sort | dynamic distribution of high rate data processing from cern to remote hpc data centers |
| url | https://hdl.handle.net/1721.1/132027 |
| work_keys_str_mv | AT boccalit dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT camerond dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT cardon dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT conciatored dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT digirolamoa dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT dissertorig dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT fernandezp dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT filipcica dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT gilam dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT grabc dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT elmsheuserj dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters AT jankauskasv dynamicdistributionofhighratedataprocessingfromcerntoremotehpcdatacenters |