Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment
The dynamic data federation software Dynafed, developed by CERN IT, provides a federated storage cluster on demand using the HTTP protocol with WebDAV extensions. Traditional storage sites which support an experiment can be added to Dynafed without requiring any changes to the site. Dynafed also sup...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2019-01-01
|
| Series: | EPJ Web of Conferences |
| Online Access: | https://www.epj-conferences.org/articles/epjconf/pdf/2019/19/epjconf_chep2018_04026.pdf |
| _version_ | 1818655977306587136 |
|---|---|
| author | Ebert Marcus Berghaus Frank Casteels Kevin Driemel Colson Leavett-Brown Colin Fernandez Galindo Fernando Paterson Michael Seuster Rolf Sobie Randall Tafirout Reda Taylor Ryan |
| author_facet | Ebert Marcus Berghaus Frank Casteels Kevin Driemel Colson Leavett-Brown Colin Fernandez Galindo Fernando Paterson Michael Seuster Rolf Sobie Randall Tafirout Reda Taylor Ryan |
| author_sort | Ebert Marcus |
| collection | DOAJ |
| description | The dynamic data federation software Dynafed, developed by CERN IT, provides a federated storage cluster on demand using the HTTP protocol with WebDAV extensions. Traditional storage sites which support an experiment can be added to Dynafed without requiring any changes to the site. Dynafed also supports direct access to cloud storage such as S3 and Azure. We report on the usage of Dynafed to support Belle-II production jobs running on a distributed cloud system utilizing clouds across North America. Cloudscheduler, developed by the University of Victoria HEP Research Computing group , federates Openstack, OpenNebula, Amazon, Google, and Microsoft cloud compute resources and provides them as a unified Grid site which on average runs about 3500 Belle-II production jobs in parallel. The input data for those jobs is accessible through a single endpoint, our Dynafed instance. This Dynafed instance unifies storage resources provided by Amazon S3, Ceph, and Minio object stores as endpoints, as well as storage provided by traditional DPM and dCache sites. We report on our long term experience with this setup, the implementation of a grid-mapfile based X509 authentication/authorization for Belle-II access, and we show how a federated cluster can be used by Belle-II through gfalFS. |
| first_indexed | 2024-12-17T03:18:15Z |
| format | Article |
| id | doaj.art-8ac55e0e473c485f8ada2e50ad5c1f13 |
| institution | Directory Open Access Journal |
| issn | 2100-014X |
| language | English |
| last_indexed | 2024-12-17T03:18:15Z |
| publishDate | 2019-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | EPJ Web of Conferences |
| spelling | doaj.art-8ac55e0e473c485f8ada2e50ad5c1f132022-12-21T22:05:36ZengEDP SciencesEPJ Web of Conferences2100-014X2019-01-012140402610.1051/epjconf/201921404026epjconf_chep2018_04026Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environmentEbert MarcusBerghaus FrankCasteels KevinDriemel ColsonLeavett-Brown ColinFernandez Galindo FernandoPaterson MichaelSeuster RolfSobie RandallTafirout RedaTaylor RyanThe dynamic data federation software Dynafed, developed by CERN IT, provides a federated storage cluster on demand using the HTTP protocol with WebDAV extensions. Traditional storage sites which support an experiment can be added to Dynafed without requiring any changes to the site. Dynafed also supports direct access to cloud storage such as S3 and Azure. We report on the usage of Dynafed to support Belle-II production jobs running on a distributed cloud system utilizing clouds across North America. Cloudscheduler, developed by the University of Victoria HEP Research Computing group , federates Openstack, OpenNebula, Amazon, Google, and Microsoft cloud compute resources and provides them as a unified Grid site which on average runs about 3500 Belle-II production jobs in parallel. The input data for those jobs is accessible through a single endpoint, our Dynafed instance. This Dynafed instance unifies storage resources provided by Amazon S3, Ceph, and Minio object stores as endpoints, as well as storage provided by traditional DPM and dCache sites. We report on our long term experience with this setup, the implementation of a grid-mapfile based X509 authentication/authorization for Belle-II access, and we show how a federated cluster can be used by Belle-II through gfalFS.https://www.epj-conferences.org/articles/epjconf/pdf/2019/19/epjconf_chep2018_04026.pdf |
| spellingShingle | Ebert Marcus Berghaus Frank Casteels Kevin Driemel Colson Leavett-Brown Colin Fernandez Galindo Fernando Paterson Michael Seuster Rolf Sobie Randall Tafirout Reda Taylor Ryan Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment EPJ Web of Conferences |
| title | Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment |
| title_full | Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment |
| title_fullStr | Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment |
| title_full_unstemmed | Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment |
| title_short | Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment |
| title_sort | using a dynamic data federation for running belle ii simulation applications in a distributed cloud environment |
| url | https://www.epj-conferences.org/articles/epjconf/pdf/2019/19/epjconf_chep2018_04026.pdf |
| work_keys_str_mv | AT ebertmarcus usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT berghausfrank usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT casteelskevin usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT driemelcolson usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT leavettbrowncolin usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT fernandezgalindofernando usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT patersonmichael usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT seusterrolf usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT sobierandall usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT tafiroutreda usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment AT taylorryan usingadynamicdatafederationforrunningbelleiisimulationapplicationsinadistributedcloudenvironment |