Towards improving the spatial testability of aftershock forecast models
<p>Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2023-07-01
|
| Series: | Natural Hazards and Earth System Sciences |
| Online Access: | https://nhess.copernicus.org/articles/23/2683/2023/nhess-23-2683-2023.pdf |
| _version_ | 1797768724516700160 |
|---|---|
| author | A. M. Khawaja A. M. Khawaja B. Maleki Asayesh B. Maleki Asayesh S. Hainzl S. Hainzl D. Schorlemmer |
| author_facet | A. M. Khawaja A. M. Khawaja B. Maleki Asayesh B. Maleki Asayesh S. Hainzl S. Hainzl D. Schorlemmer |
| author_sort | A. M. Khawaja |
| collection | DOAJ |
| description | <p>Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws in using the ROC for aftershock forecast ranking. We suggest a twofold improvement in the testing strategy. First, we propose to replace ROC with the Matthews correlation coefficient (MCC) and the <span class="inline-formula"><i>F</i><sub>1</sub></span> curve. We also suggest using a multi-resolution test grid adapted to the earthquake density. We conduct a synthetic experiment where we analyse aftershock distributions stemming from a Coulomb failure (<span class="inline-formula">ΔCFS</span>) model, including stress activation and shadow regions. Using these aftershock distributions, we test the true <span class="inline-formula">ΔCFS</span> model as well as a simple distance-based forecast (R), only predicting activation. The standard test cannot clearly distinguish between both forecasts, particularly in the case of some outliers. However, using both MCC-<span class="inline-formula"><i>F</i><sub>1</sub></span> instead of ROC curves and a simple radial multi-resolution grid improves the test capabilities significantly. The novel findings of this study suggest that we should have at least 8 % and 5 % cells with observed earthquakes to differentiate between a near-perfect forecast model and an informationless forecast using ROC and MCC-<span class="inline-formula"><i>F</i><sub>1</sub></span>, respectively. While we cannot change the observed data, we can adjust the spatial grid using a data-driven approach to reduce the disparity between the number of earthquakes and the total number of cells. Using the recently introduced Quadtree approach to generate multi-resolution grids, we test real aftershock forecast models for Chi-Chi and Landers aftershocks following the suggested guideline. Despite the improved tests, we find that the simple R model still outperforms the <span class="inline-formula">ΔCFS</span> model in both cases, indicating that the latter should not be applied without further model adjustments.</p> |
| first_indexed | 2024-03-12T20:58:25Z |
| format | Article |
| id | doaj.art-cd03150725084f0891769f796623e7b9 |
| institution | Directory Open Access Journal |
| issn | 1561-8633 1684-9981 |
| language | English |
| last_indexed | 2024-03-12T20:58:25Z |
| publishDate | 2023-07-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | Natural Hazards and Earth System Sciences |
| spelling | doaj.art-cd03150725084f0891769f796623e7b92023-07-31T13:38:08ZengCopernicus PublicationsNatural Hazards and Earth System Sciences1561-86331684-99812023-07-01232683269610.5194/nhess-23-2683-2023Towards improving the spatial testability of aftershock forecast modelsA. M. Khawaja0A. M. Khawaja1B. Maleki Asayesh2B. Maleki Asayesh3S. Hainzl4S. Hainzl5D. Schorlemmer6GFZ German Research Centre for Geosciences, Telegrafenberg, 14473 Potsdam, GermanyInstitute of Geosciences, University of Potsdam, 14476 Potsdam, GermanyGFZ German Research Centre for Geosciences, Telegrafenberg, 14473 Potsdam, GermanyInstitute of Geosciences, University of Potsdam, 14476 Potsdam, GermanyGFZ German Research Centre for Geosciences, Telegrafenberg, 14473 Potsdam, GermanyInstitute of Geosciences, University of Potsdam, 14476 Potsdam, GermanyGFZ German Research Centre for Geosciences, Telegrafenberg, 14473 Potsdam, Germany<p>Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws in using the ROC for aftershock forecast ranking. We suggest a twofold improvement in the testing strategy. First, we propose to replace ROC with the Matthews correlation coefficient (MCC) and the <span class="inline-formula"><i>F</i><sub>1</sub></span> curve. We also suggest using a multi-resolution test grid adapted to the earthquake density. We conduct a synthetic experiment where we analyse aftershock distributions stemming from a Coulomb failure (<span class="inline-formula">ΔCFS</span>) model, including stress activation and shadow regions. Using these aftershock distributions, we test the true <span class="inline-formula">ΔCFS</span> model as well as a simple distance-based forecast (R), only predicting activation. The standard test cannot clearly distinguish between both forecasts, particularly in the case of some outliers. However, using both MCC-<span class="inline-formula"><i>F</i><sub>1</sub></span> instead of ROC curves and a simple radial multi-resolution grid improves the test capabilities significantly. The novel findings of this study suggest that we should have at least 8 % and 5 % cells with observed earthquakes to differentiate between a near-perfect forecast model and an informationless forecast using ROC and MCC-<span class="inline-formula"><i>F</i><sub>1</sub></span>, respectively. While we cannot change the observed data, we can adjust the spatial grid using a data-driven approach to reduce the disparity between the number of earthquakes and the total number of cells. Using the recently introduced Quadtree approach to generate multi-resolution grids, we test real aftershock forecast models for Chi-Chi and Landers aftershocks following the suggested guideline. Despite the improved tests, we find that the simple R model still outperforms the <span class="inline-formula">ΔCFS</span> model in both cases, indicating that the latter should not be applied without further model adjustments.</p>https://nhess.copernicus.org/articles/23/2683/2023/nhess-23-2683-2023.pdf |
| spellingShingle | A. M. Khawaja A. M. Khawaja B. Maleki Asayesh B. Maleki Asayesh S. Hainzl S. Hainzl D. Schorlemmer Towards improving the spatial testability of aftershock forecast models Natural Hazards and Earth System Sciences |
| title | Towards improving the spatial testability of aftershock forecast models |
| title_full | Towards improving the spatial testability of aftershock forecast models |
| title_fullStr | Towards improving the spatial testability of aftershock forecast models |
| title_full_unstemmed | Towards improving the spatial testability of aftershock forecast models |
| title_short | Towards improving the spatial testability of aftershock forecast models |
| title_sort | towards improving the spatial testability of aftershock forecast models |
| url | https://nhess.copernicus.org/articles/23/2683/2023/nhess-23-2683-2023.pdf |
| work_keys_str_mv | AT amkhawaja towardsimprovingthespatialtestabilityofaftershockforecastmodels AT amkhawaja towardsimprovingthespatialtestabilityofaftershockforecastmodels AT bmalekiasayesh towardsimprovingthespatialtestabilityofaftershockforecastmodels AT bmalekiasayesh towardsimprovingthespatialtestabilityofaftershockforecastmodels AT shainzl towardsimprovingthespatialtestabilityofaftershockforecastmodels AT shainzl towardsimprovingthespatialtestabilityofaftershockforecastmodels AT dschorlemmer towardsimprovingthespatialtestabilityofaftershockforecastmodels |