GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model
<p>We discuss the various performance aspects of parallelizing our transient global-scale groundwater model at 30<span class="inline-formula"><sup>′′</sup></span> resolution (30 arcsec; <span class="inline-formula">∼</span> 1 km at the Eq...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2024-01-01
|
| Series: | Geoscientific Model Development |
| Online Access: | https://gmd.copernicus.org/articles/17/275/2024/gmd-17-275-2024.pdf |
| _version_ | 1827382882358788096 |
|---|---|
| author | J. Verkaik J. Verkaik E. H. Sutanudjaja G. H. P. Oude Essink G. H. P. Oude Essink H. X. Lin H. X. Lin M. F. P. Bierkens M. F. P. Bierkens |
| author_facet | J. Verkaik J. Verkaik E. H. Sutanudjaja G. H. P. Oude Essink G. H. P. Oude Essink H. X. Lin H. X. Lin M. F. P. Bierkens M. F. P. Bierkens |
| author_sort | J. Verkaik |
| collection | DOAJ |
| description | <p>We discuss the various performance aspects of parallelizing our transient global-scale groundwater model at 30<span class="inline-formula"><sup>′′</sup></span> resolution (30 arcsec; <span class="inline-formula">∼</span> 1 km at the Equator) on large distributed memory parallel clusters. This model, referred to as GLOBGM, is the successor of our 5<span class="inline-formula"><sup>′</sup></span> (5 arcmin; <span class="inline-formula">∼</span> 10 km at the Equator) PCR-GLOBWB 2 (PCRaster Global Water Balance model) groundwater model, based on MODFLOW having two model layers. The current version of GLOBGM (v1.0) used in this study also has two model layers, is uncalibrated, and uses available 30<span class="inline-formula"><sup>′′</sup></span> PCR-GLOBWB data. Increasing the model resolution from 5<span class="inline-formula"><sup>′</sup></span> to 30<span class="inline-formula"><sup>′′</sup></span> creates challenges, including increased runtime, memory usage, and data storage that exceed the capacity of a single computer. We show that our parallelization tackles these problems with relatively low parallel hardware requirements to meet the needs of users or modelers who do not have exclusive access to hundreds or thousands of nodes within a supercomputer.</p>
<p>For our simulation, we use unstructured grids and a prototype version of MODFLOW 6 that we have parallelized using the message-passing interface. We construct independent unstructured grids with a total of 278 million active cells to cancel all redundant sea and land cells, while satisfying all necessary boundary conditions, and distribute them over three continental-scale groundwater models (168 million – Afro–Eurasia; 77 million – the Americas; 16 million – Australia) and one remaining model for the smaller islands (17 million). Each of the four groundwater models is partitioned into multiple non-overlapping submodels that are tightly coupled within the MODFLOW linear solver, where each submodel is uniquely assigned to one processor core, and associated submodel data are written in parallel during the pre-processing, using data tiles. For balancing the parallel workload in advance, we apply the widely used METIS graph partitioner in two ways: it is straightforwardly applied to all (lateral) model grid cells, and it is applied in an area-based manner to HydroBASINS catchments that are assigned to submodels for pre-sorting to a future coupling with surface water. We consider an experiment for simulating the years 1958–2015 with daily time steps and monthly input, including a 20-year spin-up, on the Dutch national supercomputer Snellius. Given that the serial simulation would require <span class="inline-formula">∼</span> 4.5 months of runtime, we set a hypothetical target of a maximum of 16 h of simulation runtime. We show that 12 nodes (32 cores per node; 384 cores in total) are sufficient to achieve this target, resulting in a speedup of 138 for the largest Afro–Eurasia model when using 7 nodes (224 cores) in parallel.</p>
<p>A limited evaluation of the model output using the United States Geological Survey (USGS) National Water Information System (NWIS) head observations for the contiguous United States was conducted. This showed that increasing the resolution from 5<span class="inline-formula"><sup>′</sup></span> to 30<span class="inline-formula"><sup>′′</sup></span> results in a significant improvement with GLOBGM for the steady-state simulation when compared to the 5<span class="inline-formula"><sup>′</sup></span> PCR-GLOBWB groundwater model. However, results for the transient simulation are quite similar, and there is much room for improvement. Monthly and multi-year total terrestrial water storage anomalies derived from the GLOBGM and PCR-GLOBWB models, however, compared favorably with observations from the GRACE<span id="page276"/> satellite. For the next versions of GLOBGM, further improvements require a more detailed (hydro)geological schematization and better information on the locations, depths, and pumping rates of abstraction wells.</p> |
| first_indexed | 2024-03-08T14:28:50Z |
| format | Article |
| id | doaj.art-2dd70f928ee34d14b7c4f1e36613f9a5 |
| institution | Directory Open Access Journal |
| issn | 1991-959X 1991-9603 |
| language | English |
| last_indexed | 2024-03-08T14:28:50Z |
| publishDate | 2024-01-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | Geoscientific Model Development |
| spelling | doaj.art-2dd70f928ee34d14b7c4f1e36613f9a52024-01-12T13:26:58ZengCopernicus PublicationsGeoscientific Model Development1991-959X1991-96032024-01-011727530010.5194/gmd-17-275-2024GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater modelJ. Verkaik0J. Verkaik1E. H. Sutanudjaja2G. H. P. Oude Essink3G. H. P. Oude Essink4H. X. Lin5H. X. Lin6M. F. P. Bierkens7M. F. P. Bierkens8Unit Subsurface and Groundwater Systems, Deltares, Utrecht, the NetherlandsDepartment of Physical Geography, Faculty of Geosciences, Utrecht University, Utrecht, the NetherlandsDepartment of Physical Geography, Faculty of Geosciences, Utrecht University, Utrecht, the NetherlandsUnit Subsurface and Groundwater Systems, Deltares, Utrecht, the NetherlandsDepartment of Physical Geography, Faculty of Geosciences, Utrecht University, Utrecht, the NetherlandsDepartment of Applied Mathematical Analysis, Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, the NetherlandsInstitute of Environmental Sciences, Faculty of Science, Leiden University, Leiden, the NetherlandsUnit Subsurface and Groundwater Systems, Deltares, Utrecht, the NetherlandsDepartment of Physical Geography, Faculty of Geosciences, Utrecht University, Utrecht, the Netherlands<p>We discuss the various performance aspects of parallelizing our transient global-scale groundwater model at 30<span class="inline-formula"><sup>′′</sup></span> resolution (30 arcsec; <span class="inline-formula">∼</span> 1 km at the Equator) on large distributed memory parallel clusters. This model, referred to as GLOBGM, is the successor of our 5<span class="inline-formula"><sup>′</sup></span> (5 arcmin; <span class="inline-formula">∼</span> 10 km at the Equator) PCR-GLOBWB 2 (PCRaster Global Water Balance model) groundwater model, based on MODFLOW having two model layers. The current version of GLOBGM (v1.0) used in this study also has two model layers, is uncalibrated, and uses available 30<span class="inline-formula"><sup>′′</sup></span> PCR-GLOBWB data. Increasing the model resolution from 5<span class="inline-formula"><sup>′</sup></span> to 30<span class="inline-formula"><sup>′′</sup></span> creates challenges, including increased runtime, memory usage, and data storage that exceed the capacity of a single computer. We show that our parallelization tackles these problems with relatively low parallel hardware requirements to meet the needs of users or modelers who do not have exclusive access to hundreds or thousands of nodes within a supercomputer.</p> <p>For our simulation, we use unstructured grids and a prototype version of MODFLOW 6 that we have parallelized using the message-passing interface. We construct independent unstructured grids with a total of 278 million active cells to cancel all redundant sea and land cells, while satisfying all necessary boundary conditions, and distribute them over three continental-scale groundwater models (168 million – Afro–Eurasia; 77 million – the Americas; 16 million – Australia) and one remaining model for the smaller islands (17 million). Each of the four groundwater models is partitioned into multiple non-overlapping submodels that are tightly coupled within the MODFLOW linear solver, where each submodel is uniquely assigned to one processor core, and associated submodel data are written in parallel during the pre-processing, using data tiles. For balancing the parallel workload in advance, we apply the widely used METIS graph partitioner in two ways: it is straightforwardly applied to all (lateral) model grid cells, and it is applied in an area-based manner to HydroBASINS catchments that are assigned to submodels for pre-sorting to a future coupling with surface water. We consider an experiment for simulating the years 1958–2015 with daily time steps and monthly input, including a 20-year spin-up, on the Dutch national supercomputer Snellius. Given that the serial simulation would require <span class="inline-formula">∼</span> 4.5 months of runtime, we set a hypothetical target of a maximum of 16 h of simulation runtime. We show that 12 nodes (32 cores per node; 384 cores in total) are sufficient to achieve this target, resulting in a speedup of 138 for the largest Afro–Eurasia model when using 7 nodes (224 cores) in parallel.</p> <p>A limited evaluation of the model output using the United States Geological Survey (USGS) National Water Information System (NWIS) head observations for the contiguous United States was conducted. This showed that increasing the resolution from 5<span class="inline-formula"><sup>′</sup></span> to 30<span class="inline-formula"><sup>′′</sup></span> results in a significant improvement with GLOBGM for the steady-state simulation when compared to the 5<span class="inline-formula"><sup>′</sup></span> PCR-GLOBWB groundwater model. However, results for the transient simulation are quite similar, and there is much room for improvement. Monthly and multi-year total terrestrial water storage anomalies derived from the GLOBGM and PCR-GLOBWB models, however, compared favorably with observations from the GRACE<span id="page276"/> satellite. For the next versions of GLOBGM, further improvements require a more detailed (hydro)geological schematization and better information on the locations, depths, and pumping rates of abstraction wells.</p>https://gmd.copernicus.org/articles/17/275/2024/gmd-17-275-2024.pdf |
| spellingShingle | J. Verkaik J. Verkaik E. H. Sutanudjaja G. H. P. Oude Essink G. H. P. Oude Essink H. X. Lin H. X. Lin M. F. P. Bierkens M. F. P. Bierkens GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model Geoscientific Model Development |
| title | GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model |
| title_full | GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model |
| title_fullStr | GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model |
| title_full_unstemmed | GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model |
| title_short | GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model |
| title_sort | globgm v1 0 a parallel implementation of a 30 thinsp arcsec pcr globwb modflow global scale groundwater model |
| url | https://gmd.copernicus.org/articles/17/275/2024/gmd-17-275-2024.pdf |
| work_keys_str_mv | AT jverkaik globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT jverkaik globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT ehsutanudjaja globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT ghpoudeessink globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT ghpoudeessink globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT hxlin globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT hxlin globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT mfpbierkens globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel AT mfpbierkens globgmv10aparallelimplementationofa30thinsparcsecpcrglobwbmodflowglobalscalegroundwatermodel |