Towards improving the spatial testability of aftershock forecast models
<p>Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2023-07-01
|
| Series: | Natural Hazards and Earth System Sciences |
| Online Access: | https://nhess.copernicus.org/articles/23/2683/2023/nhess-23-2683-2023.pdf |
| Summary: | <p>Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws in using the ROC for aftershock forecast ranking. We suggest a twofold improvement in the testing strategy. First, we propose to replace ROC with the Matthews correlation coefficient (MCC) and the <span class="inline-formula"><i>F</i><sub>1</sub></span> curve. We also suggest using a multi-resolution test grid adapted to the earthquake density. We conduct a synthetic experiment where we analyse aftershock distributions stemming from a Coulomb failure (<span class="inline-formula">ΔCFS</span>) model, including stress activation and shadow regions. Using these aftershock distributions, we test the true <span class="inline-formula">ΔCFS</span> model as well as a simple distance-based forecast (R), only predicting activation. The standard test cannot clearly distinguish between both forecasts, particularly in the case of some outliers. However, using both MCC-<span class="inline-formula"><i>F</i><sub>1</sub></span> instead of ROC curves and a simple radial multi-resolution grid improves the test capabilities significantly. The novel findings of this study suggest that we should have at least 8 % and 5 % cells with observed earthquakes to differentiate between a near-perfect forecast model and an informationless forecast using ROC and MCC-<span class="inline-formula"><i>F</i><sub>1</sub></span>, respectively. While we cannot change the observed data, we can adjust the spatial grid using a data-driven approach to reduce the disparity between the number of earthquakes and the total number of cells. Using the recently introduced Quadtree approach to generate multi-resolution grids, we test real aftershock forecast models for Chi-Chi and Landers aftershocks following the suggested guideline. Despite the improved tests, we find that the simple R model still outperforms the <span class="inline-formula">ΔCFS</span> model in both cases, indicating that the latter should not be applied without further model adjustments.</p> |
|---|---|
| ISSN: | 1561-8633 1684-9981 |