Evaluation Procedures for Forecasting with Spatiotemporal Data
The increasing use of sensor networks has led to an ever larger number of available spatiotemporal datasets. Forecasting applications using this type of data are frequently motivated by important domains such as environmental monitoring. Being able to properly assess the performance of different for...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-03-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/9/6/691 |
| _version_ | 1797540272254484480 |
|---|---|
| author | Mariana Oliveira Luís Torgo Vítor Santos Costa |
| author_facet | Mariana Oliveira Luís Torgo Vítor Santos Costa |
| author_sort | Mariana Oliveira |
| collection | DOAJ |
| description | The increasing use of sensor networks has led to an ever larger number of available spatiotemporal datasets. Forecasting applications using this type of data are frequently motivated by important domains such as environmental monitoring. Being able to properly assess the performance of different forecasting approaches is fundamental to achieve progress. However, traditional performance estimation procedures, such as cross-validation, face challenges due to the implicit dependence between observations in spatiotemporal datasets. In this paper, we empirically compare several variants of cross-validation (CV) and out-of-sample (OOS) performance estimation procedures, using both artificially generated and real-world spatiotemporal datasets. Our results show both CV and OOS reporting useful estimates, but they suggest that blocking data in space and/or in time may be useful in mitigating CV’s bias to underestimate error. Overall, our study shows the importance of considering data dependencies when estimating the performance of spatiotemporal forecasting models. |
| first_indexed | 2024-03-10T12:58:42Z |
| format | Article |
| id | doaj.art-19a7eeccd77c4c7f971166ca8c2b3dfb |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-10T12:58:42Z |
| publishDate | 2021-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-19a7eeccd77c4c7f971166ca8c2b3dfb2023-11-21T11:41:13ZengMDPI AGMathematics2227-73902021-03-019669110.3390/math9060691Evaluation Procedures for Forecasting with Spatiotemporal DataMariana Oliveira0Luís Torgo1Vítor Santos Costa2Department of Computer Science, Faculty of Sciences, University of Porto, Rua Campo Alegre 1055, 4169-007 Porto, PortugalFaculty of Computer Science, Dalhousie University, 6050 University Av., Halifax, NS B3H 1W5, CanadaDepartment of Computer Science, Faculty of Sciences, University of Porto, Rua Campo Alegre 1055, 4169-007 Porto, PortugalThe increasing use of sensor networks has led to an ever larger number of available spatiotemporal datasets. Forecasting applications using this type of data are frequently motivated by important domains such as environmental monitoring. Being able to properly assess the performance of different forecasting approaches is fundamental to achieve progress. However, traditional performance estimation procedures, such as cross-validation, face challenges due to the implicit dependence between observations in spatiotemporal datasets. In this paper, we empirically compare several variants of cross-validation (CV) and out-of-sample (OOS) performance estimation procedures, using both artificially generated and real-world spatiotemporal datasets. Our results show both CV and OOS reporting useful estimates, but they suggest that blocking data in space and/or in time may be useful in mitigating CV’s bias to underestimate error. Overall, our study shows the importance of considering data dependencies when estimating the performance of spatiotemporal forecasting models.https://www.mdpi.com/2227-7390/9/6/691evaluation methodsperformance estimationcross-validationspatiotemporal datageo-referenced time seriesreproducible research |
| spellingShingle | Mariana Oliveira Luís Torgo Vítor Santos Costa Evaluation Procedures for Forecasting with Spatiotemporal Data Mathematics evaluation methods performance estimation cross-validation spatiotemporal data geo-referenced time series reproducible research |
| title | Evaluation Procedures for Forecasting with Spatiotemporal Data |
| title_full | Evaluation Procedures for Forecasting with Spatiotemporal Data |
| title_fullStr | Evaluation Procedures for Forecasting with Spatiotemporal Data |
| title_full_unstemmed | Evaluation Procedures for Forecasting with Spatiotemporal Data |
| title_short | Evaluation Procedures for Forecasting with Spatiotemporal Data |
| title_sort | evaluation procedures for forecasting with spatiotemporal data |
| topic | evaluation methods performance estimation cross-validation spatiotemporal data geo-referenced time series reproducible research |
| url | https://www.mdpi.com/2227-7390/9/6/691 |
| work_keys_str_mv | AT marianaoliveira evaluationproceduresforforecastingwithspatiotemporaldata AT luistorgo evaluationproceduresforforecastingwithspatiotemporaldata AT vitorsantoscosta evaluationproceduresforforecastingwithspatiotemporaldata |