NDRank: optimised parallel search for weather analogues
ABSTRACTGlobal meteorology data are now widely used in various areas, but one of its applications, weather analogues, still require exhaustive searches on the whole historical data. We present two optimisations for the state-of-the-art weather analogue search algorithms: a parallelization and a heur...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2023-04-01
|
| Series: | Big Earth Data |
| Subjects: | |
| Online Access: | https://www.tandfonline.com/doi/10.1080/20964471.2023.2195468 |
| _version_ | 1827935877807996928 |
|---|---|
| author | David Martins Miguel Ferreira João Nuno Silva |
| author_facet | David Martins Miguel Ferreira João Nuno Silva |
| author_sort | David Martins |
| collection | DOAJ |
| description | ABSTRACTGlobal meteorology data are now widely used in various areas, but one of its applications, weather analogues, still require exhaustive searches on the whole historical data. We present two optimisations for the state-of-the-art weather analogue search algorithms: a parallelization and a heuristic search. The heuristic search (NDRank) limits of the final number of results and does initial searches on a lower resolution dataset to find candidates that, in the second phase, are locally validated. These optimisations were deployed in the Cloud and evaluated with ERA5 data from ECMWF. The proposed parallelization attained speedups close to optimal, and NDRank attains speedups higher than 4. NDRank can be applied to any parallel search, adding similar speedups. A substantial number of executions returned a set of analogues similar to the existing exhaustive search and most of the remaining results presented a numerical value difference lower than 0.1%. The results demonstrate that it is now possible to search for weather analogues in a faster way (even compared with parallel searches) with results with little to no error. Furthermore, NDRank can be applied to existing exhaustive searches, providing faster results with small reduction of the precision of the results. |
| first_indexed | 2024-03-13T08:02:13Z |
| format | Article |
| id | doaj.art-74c5ed15710c45ea92e673654761c228 |
| institution | Directory Open Access Journal |
| issn | 2096-4471 2574-5417 |
| language | English |
| last_indexed | 2024-03-13T08:02:13Z |
| publishDate | 2023-04-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Big Earth Data |
| spelling | doaj.art-74c5ed15710c45ea92e673654761c2282023-06-01T12:20:44ZengTaylor & Francis GroupBig Earth Data2096-44712574-54172023-04-017227629710.1080/20964471.2023.2195468NDRank: optimised parallel search for weather analoguesDavid Martins0Miguel Ferreira1João Nuno Silva2Instituto Superior Técnico, Universidade de Lisboa, Lisboa, PortugalCaravela Energy Partners, Darien, Connecticut, USAInstituto Superior Técnico, Universidade de Lisboa, Lisboa, PortugalABSTRACTGlobal meteorology data are now widely used in various areas, but one of its applications, weather analogues, still require exhaustive searches on the whole historical data. We present two optimisations for the state-of-the-art weather analogue search algorithms: a parallelization and a heuristic search. The heuristic search (NDRank) limits of the final number of results and does initial searches on a lower resolution dataset to find candidates that, in the second phase, are locally validated. These optimisations were deployed in the Cloud and evaluated with ERA5 data from ECMWF. The proposed parallelization attained speedups close to optimal, and NDRank attains speedups higher than 4. NDRank can be applied to any parallel search, adding similar speedups. A substantial number of executions returned a set of analogues similar to the existing exhaustive search and most of the remaining results presented a numerical value difference lower than 0.1%. The results demonstrate that it is now possible to search for weather analogues in a faster way (even compared with parallel searches) with results with little to no error. Furthermore, NDRank can be applied to existing exhaustive searches, providing faster results with small reduction of the precision of the results.https://www.tandfonline.com/doi/10.1080/20964471.2023.2195468Multidimensional arraysweather analogues searchparallel computingalgorithm optimisation |
| spellingShingle | David Martins Miguel Ferreira João Nuno Silva NDRank: optimised parallel search for weather analogues Big Earth Data Multidimensional arrays weather analogues search parallel computing algorithm optimisation |
| title | NDRank: optimised parallel search for weather analogues |
| title_full | NDRank: optimised parallel search for weather analogues |
| title_fullStr | NDRank: optimised parallel search for weather analogues |
| title_full_unstemmed | NDRank: optimised parallel search for weather analogues |
| title_short | NDRank: optimised parallel search for weather analogues |
| title_sort | ndrank optimised parallel search for weather analogues |
| topic | Multidimensional arrays weather analogues search parallel computing algorithm optimisation |
| url | https://www.tandfonline.com/doi/10.1080/20964471.2023.2195468 |
| work_keys_str_mv | AT davidmartins ndrankoptimisedparallelsearchforweatheranalogues AT miguelferreira ndrankoptimisedparallelsearchforweatheranalogues AT joaonunosilva ndrankoptimisedparallelsearchforweatheranalogues |