Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing
<p>To promote cloud and HPC computing, GRAPEVINE* project objectives include using these tools along with open data sources to provide a reusable IT service. In this service a predictive model based on Machine learning (ML) techniques is created with the aim of preventing and controlling grape...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2023-03-01
|
| Series: | Advances in Science and Research |
| Online Access: | https://asr.copernicus.org/articles/20/1/2023/asr-20-1-2023.pdf |
| _version_ | 1797867091939819520 |
|---|---|
| author | P. Vourlioti S. Kotsopoulos T. Mamouka A. Agrafiotis F. J. Nieto C. F. Sánchez C. G. Llerena S. García González |
| author_facet | P. Vourlioti S. Kotsopoulos T. Mamouka A. Agrafiotis F. J. Nieto C. F. Sánchez C. G. Llerena S. García González |
| author_sort | P. Vourlioti |
| collection | DOAJ |
| description | <p>To promote cloud and HPC computing, GRAPEVINE* project objectives include using these tools along with open data sources to provide a reusable IT service. In this service a predictive model based on Machine learning (ML) techniques is created with the aim of preventing and
controlling grape vine diseases in the wine cultivation sector. Aside from
the predictive ML, meteorological forecasts are crucial input to train the
ML models and on a second step to be used as input for the operational
prediction of grapevine diseases. To this end, the Weather and Research
Forecasting model (WRF) has been deployed in CESGA's HPC infrastructure to
produce medium-range and sub-seasonal forecasts for the targeted pilot areas (Greece and Spain). The data assimilation component of WRF – WRFDA – has been also introduced for improving the initial conditions of the WRF model by assimilating observations from weather stations and satellite
precipitation products (Integrated Multi-satellitE Retrieval for GPM – IMERG). This methodology for assimilation was developed during STARGATE<span class="inline-formula"><sup>*</sup></span> project, allowing the testing of the methodology in the operational service of GRAPEVINE. The operational production of the forecasts is achieved by the cloudify orchestrator on a Kubernetes cluster. The connections between the Kubernetes cluster and the HPC infrastructure, where the model resides, is achieved with the croupier plugin of cloudify. Blueprints that encapsule the workflows of the meteorological model and its dependencies were created. The instances of the blueprints (deployments) were created automatically to produce operationally weather forecasts and they were made available to the ML models via a THREDDS server. Valuable lessons were learned with regards the automation of the process and the coupling with the HPC in terms of reservations and operational production.</p> |
| first_indexed | 2024-04-09T23:34:46Z |
| format | Article |
| id | doaj.art-11140d99816a45e387de245151e5061e |
| institution | Directory Open Access Journal |
| issn | 1992-0628 1992-0636 |
| language | English |
| last_indexed | 2024-04-09T23:34:46Z |
| publishDate | 2023-03-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | Advances in Science and Research |
| spelling | doaj.art-11140d99816a45e387de245151e5061e2023-03-20T12:43:15ZengCopernicus PublicationsAdvances in Science and Research1992-06281992-06362023-03-01201810.5194/asr-20-1-2023Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computingP. Vourlioti0S. Kotsopoulos1T. Mamouka2A. Agrafiotis3F. J. Nieto4C. F. Sánchez5C. G. Llerena6S. García González7AgroApps, 55133 Thessaloniki, GreeceAgroApps, 55133 Thessaloniki, GreeceAgroApps, 55133 Thessaloniki, GreeceAgroApps, 55133 Thessaloniki, GreeceAtos, Research and Innovation, ATOS Spain SA, 48013 Bilbao, SpainCESGA, 15705 Santiago de Compostela, SpainCESGA, 15705 Santiago de Compostela, SpainAtos IT, Research and Innovation, 39011 Santander, Spain<p>To promote cloud and HPC computing, GRAPEVINE* project objectives include using these tools along with open data sources to provide a reusable IT service. In this service a predictive model based on Machine learning (ML) techniques is created with the aim of preventing and controlling grape vine diseases in the wine cultivation sector. Aside from the predictive ML, meteorological forecasts are crucial input to train the ML models and on a second step to be used as input for the operational prediction of grapevine diseases. To this end, the Weather and Research Forecasting model (WRF) has been deployed in CESGA's HPC infrastructure to produce medium-range and sub-seasonal forecasts for the targeted pilot areas (Greece and Spain). The data assimilation component of WRF – WRFDA – has been also introduced for improving the initial conditions of the WRF model by assimilating observations from weather stations and satellite precipitation products (Integrated Multi-satellitE Retrieval for GPM – IMERG). This methodology for assimilation was developed during STARGATE<span class="inline-formula"><sup>*</sup></span> project, allowing the testing of the methodology in the operational service of GRAPEVINE. The operational production of the forecasts is achieved by the cloudify orchestrator on a Kubernetes cluster. The connections between the Kubernetes cluster and the HPC infrastructure, where the model resides, is achieved with the croupier plugin of cloudify. Blueprints that encapsule the workflows of the meteorological model and its dependencies were created. The instances of the blueprints (deployments) were created automatically to produce operationally weather forecasts and they were made available to the ML models via a THREDDS server. Valuable lessons were learned with regards the automation of the process and the coupling with the HPC in terms of reservations and operational production.</p>https://asr.copernicus.org/articles/20/1/2023/asr-20-1-2023.pdf |
| spellingShingle | P. Vourlioti S. Kotsopoulos T. Mamouka A. Agrafiotis F. J. Nieto C. F. Sánchez C. G. Llerena S. García González Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing Advances in Science and Research |
| title | Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing |
| title_full | Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing |
| title_fullStr | Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing |
| title_full_unstemmed | Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing |
| title_short | Maximizing the potential of numerical weather prediction models: lessons learned from combining high-performance computing and cloud computing |
| title_sort | maximizing the potential of numerical weather prediction models lessons learned from combining high performance computing and cloud computing |
| url | https://asr.copernicus.org/articles/20/1/2023/asr-20-1-2023.pdf |
| work_keys_str_mv | AT pvourlioti maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT skotsopoulos maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT tmamouka maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT aagrafiotis maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT fjnieto maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT cfsanchez maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT cgllerena maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing AT sgarciagonzalez maximizingthepotentialofnumericalweatherpredictionmodelslessonslearnedfromcombininghighperformancecomputingandcloudcomputing |