ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system
Abstract Background Wellington-Dufferin-Guelph Public Health (WDGPH) has conducted an absenteeism-based influenza surveillance program in the WDG region of Ontario, Canada since 2008, using a 10% absenteeism threshold to raise an alert for the implementation of mitigating measures. A recent study in...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-05-01
|
| Series: | BMC Public Health |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12889-023-15747-z |
| _version_ | 1797827300404756480 |
|---|---|
| author | Kayla R. Vanderkruk Lorna E. Deeth Zeny Feng Lise A. Trotz-Williams |
| author_facet | Kayla R. Vanderkruk Lorna E. Deeth Zeny Feng Lise A. Trotz-Williams |
| author_sort | Kayla R. Vanderkruk |
| collection | DOAJ |
| description | Abstract Background Wellington-Dufferin-Guelph Public Health (WDGPH) has conducted an absenteeism-based influenza surveillance program in the WDG region of Ontario, Canada since 2008, using a 10% absenteeism threshold to raise an alert for the implementation of mitigating measures. A recent study indicated that model-based alternatives, such as distributed lag seasonal logistic regression models, provided improved alerts for detecting an upcoming epidemic. However model evaluation and selection was primarily based on alert accuracy, measured by the false alert rate (FAR), and failed to optimize timeliness. Here, a new metric that simultaneously evaluates epidemic alert accuracy and timeliness is proposed. The alert time quality (ATQ) metric is investigated as a model selection criterion on both a simulated and real data set. Methods The ATQ assessed alerts on a gradient, where alerts raised incrementally before or after an optimal day were considered informative, but were penalized for lack of timeliness. Summary statistics of ATQ, average alert time quality (AATQ) and first alert time quality (FATQ), were used for model evaluation and selection. Alerts raised by ATQ and FAR selected models were compared. Daily elementary school absenteeism and laboratory-confirmed influenza case data collected by WDGPH were used for demonstration and evaluation of the proposed metric. A simulation study that mimicked the WDG population and influenza demographics was conducted for further evaluation of the proposed metric. Results The FATQ-selected model raised acceptable first alerts most frequently, while the AATQ-selected model raised first alerts within the ideal range most frequently. Conclusions Models selected by either FATQ or AATQ would more effectively predict community influenza activity with the local community than those selected by FAR. |
| first_indexed | 2024-04-09T12:46:27Z |
| format | Article |
| id | doaj.art-a75369c2493040ae9d624696d04b1ca3 |
| institution | Directory Open Access Journal |
| issn | 1471-2458 |
| language | English |
| last_indexed | 2024-04-09T12:46:27Z |
| publishDate | 2023-05-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Public Health |
| spelling | doaj.art-a75369c2493040ae9d624696d04b1ca32023-05-14T11:30:29ZengBMCBMC Public Health1471-24582023-05-0123111710.1186/s12889-023-15747-zATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance systemKayla R. Vanderkruk0Lorna E. Deeth1Zeny Feng2Lise A. Trotz-Williams3Department of Mathematics and Statistics, University of GuelphDepartment of Mathematics and Statistics, University of GuelphDepartment of Mathematics and Statistics, University of GuelphWellington-Dufferin-Guelph Public HealthAbstract Background Wellington-Dufferin-Guelph Public Health (WDGPH) has conducted an absenteeism-based influenza surveillance program in the WDG region of Ontario, Canada since 2008, using a 10% absenteeism threshold to raise an alert for the implementation of mitigating measures. A recent study indicated that model-based alternatives, such as distributed lag seasonal logistic regression models, provided improved alerts for detecting an upcoming epidemic. However model evaluation and selection was primarily based on alert accuracy, measured by the false alert rate (FAR), and failed to optimize timeliness. Here, a new metric that simultaneously evaluates epidemic alert accuracy and timeliness is proposed. The alert time quality (ATQ) metric is investigated as a model selection criterion on both a simulated and real data set. Methods The ATQ assessed alerts on a gradient, where alerts raised incrementally before or after an optimal day were considered informative, but were penalized for lack of timeliness. Summary statistics of ATQ, average alert time quality (AATQ) and first alert time quality (FATQ), were used for model evaluation and selection. Alerts raised by ATQ and FAR selected models were compared. Daily elementary school absenteeism and laboratory-confirmed influenza case data collected by WDGPH were used for demonstration and evaluation of the proposed metric. A simulation study that mimicked the WDG population and influenza demographics was conducted for further evaluation of the proposed metric. Results The FATQ-selected model raised acceptable first alerts most frequently, while the AATQ-selected model raised first alerts within the ideal range most frequently. Conclusions Models selected by either FATQ or AATQ would more effectively predict community influenza activity with the local community than those selected by FAR.https://doi.org/10.1186/s12889-023-15747-zAbsenteeism surveillance systemInfluenzaEpidemic detectionSimulation studyEvaluation metric |
| spellingShingle | Kayla R. Vanderkruk Lorna E. Deeth Zeny Feng Lise A. Trotz-Williams ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system BMC Public Health Absenteeism surveillance system Influenza Epidemic detection Simulation study Evaluation metric |
| title | ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system |
| title_full | ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system |
| title_fullStr | ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system |
| title_full_unstemmed | ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system |
| title_short | ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system |
| title_sort | atq alert time quality an evaluation metric for assessing timely epidemic detection models within a school absenteeism based surveillance system |
| topic | Absenteeism surveillance system Influenza Epidemic detection Simulation study Evaluation metric |
| url | https://doi.org/10.1186/s12889-023-15747-z |
| work_keys_str_mv | AT kaylarvanderkruk atqalerttimequalityanevaluationmetricforassessingtimelyepidemicdetectionmodelswithinaschoolabsenteeismbasedsurveillancesystem AT lornaedeeth atqalerttimequalityanevaluationmetricforassessingtimelyepidemicdetectionmodelswithinaschoolabsenteeismbasedsurveillancesystem AT zenyfeng atqalerttimequalityanevaluationmetricforassessingtimelyepidemicdetectionmodelswithinaschoolabsenteeismbasedsurveillancesystem AT liseatrotzwilliams atqalerttimequalityanevaluationmetricforassessingtimelyepidemicdetectionmodelswithinaschoolabsenteeismbasedsurveillancesystem |