Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory
Abstract Background Rapidly developing tests for emerging diseases is critical for early disease monitoring. In the early stages of an epidemic, when low prevalences are expected, high specificity tests are desired to avoid numerous false positives. Selecting a cutoff to classify positive and negati...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-02-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-023-02139-5 |
| _version_ | 1797274168933220352 |
|---|---|
| author | Sierra Pugh Bailey K. Fosdick Mary Nehring Emily N. Gallichotte Sue VandeWoude Ander Wilson |
| author_facet | Sierra Pugh Bailey K. Fosdick Mary Nehring Emily N. Gallichotte Sue VandeWoude Ander Wilson |
| author_sort | Sierra Pugh |
| collection | DOAJ |
| description | Abstract Background Rapidly developing tests for emerging diseases is critical for early disease monitoring. In the early stages of an epidemic, when low prevalences are expected, high specificity tests are desired to avoid numerous false positives. Selecting a cutoff to classify positive and negative test results that has the desired operating characteristics, such as specificity, is challenging for new tests because of limited validation data with known disease status. While there is ample statistical literature on estimating quantiles of a distribution, there is limited evidence on estimating extreme quantiles from limited validation data and the resulting test characteristics in the disease testing context. Methods We propose using extreme value theory to select a cutoff with predetermined specificity by fitting a Pareto distribution to the upper tail of the negative controls. We compared this method to five previously proposed cutoff selection methods in a data analysis and simulation study. We analyzed COVID-19 enzyme linked immunosorbent assay antibody test results from long-term care facilities and skilled nursing staff in Colorado between May and December of 2020. Results We found the extreme value approach had minimal bias when targeting a specificity of 0.995. Using the empirical quantile of the negative controls performed well when targeting a specificity of 0.95. The higher target specificity is preferred for overall test accuracy when prevalence is low, whereas the lower target specificity is preferred when prevalence is higher and resulted in less variable prevalence estimation. Discussion While commonly used, the normal based methods showed considerable bias compared to the empirical and extreme value theory-based methods. Conclusions When determining disease testing cutoffs from small training data samples, we recommend using the extreme value based-methods when targeting a high specificity and the empirical quantile when targeting a lower specificity. |
| first_indexed | 2024-03-07T14:55:31Z |
| format | Article |
| id | doaj.art-6db8c817da294352842da54c0793fa73 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-03-07T14:55:31Z |
| publishDate | 2024-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-6db8c817da294352842da54c0793fa732024-03-05T19:28:28ZengBMCBMC Medical Research Methodology1471-22882024-02-0124111410.1186/s12874-023-02139-5Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theorySierra Pugh0Bailey K. Fosdick1Mary Nehring2Emily N. Gallichotte3Sue VandeWoude4Ander Wilson5Department of Statistics, Colorado State UniversityDepartment of Biostatistics and Informatics, Colorado School of Public HealthDepartment of Microbiology, Immunology, and Pathology, Colorado State UniversityDepartment of Microbiology, Immunology, and Pathology, Colorado State UniversityDepartment of Microbiology, Immunology, and Pathology, Colorado State UniversityDepartment of Statistics, Colorado State UniversityAbstract Background Rapidly developing tests for emerging diseases is critical for early disease monitoring. In the early stages of an epidemic, when low prevalences are expected, high specificity tests are desired to avoid numerous false positives. Selecting a cutoff to classify positive and negative test results that has the desired operating characteristics, such as specificity, is challenging for new tests because of limited validation data with known disease status. While there is ample statistical literature on estimating quantiles of a distribution, there is limited evidence on estimating extreme quantiles from limited validation data and the resulting test characteristics in the disease testing context. Methods We propose using extreme value theory to select a cutoff with predetermined specificity by fitting a Pareto distribution to the upper tail of the negative controls. We compared this method to five previously proposed cutoff selection methods in a data analysis and simulation study. We analyzed COVID-19 enzyme linked immunosorbent assay antibody test results from long-term care facilities and skilled nursing staff in Colorado between May and December of 2020. Results We found the extreme value approach had minimal bias when targeting a specificity of 0.995. Using the empirical quantile of the negative controls performed well when targeting a specificity of 0.95. The higher target specificity is preferred for overall test accuracy when prevalence is low, whereas the lower target specificity is preferred when prevalence is higher and resulted in less variable prevalence estimation. Discussion While commonly used, the normal based methods showed considerable bias compared to the empirical and extreme value theory-based methods. Conclusions When determining disease testing cutoffs from small training data samples, we recommend using the extreme value based-methods when targeting a high specificity and the empirical quantile when targeting a lower specificity.https://doi.org/10.1186/s12874-023-02139-5Antibody testCut pointExtreme value theorySerologyThreshold |
| spellingShingle | Sierra Pugh Bailey K. Fosdick Mary Nehring Emily N. Gallichotte Sue VandeWoude Ander Wilson Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory BMC Medical Research Methodology Antibody test Cut point Extreme value theory Serology Threshold |
| title | Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory |
| title_full | Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory |
| title_fullStr | Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory |
| title_full_unstemmed | Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory |
| title_short | Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory |
| title_sort | estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory |
| topic | Antibody test Cut point Extreme value theory Serology Threshold |
| url | https://doi.org/10.1186/s12874-023-02139-5 |
| work_keys_str_mv | AT sierrapugh estimatingcutoffvaluesfordiagnosticteststoachievetargetspecificityusingextremevaluetheory AT baileykfosdick estimatingcutoffvaluesfordiagnosticteststoachievetargetspecificityusingextremevaluetheory AT marynehring estimatingcutoffvaluesfordiagnosticteststoachievetargetspecificityusingextremevaluetheory AT emilyngallichotte estimatingcutoffvaluesfordiagnosticteststoachievetargetspecificityusingextremevaluetheory AT suevandewoude estimatingcutoffvaluesfordiagnosticteststoachievetargetspecificityusingextremevaluetheory AT anderwilson estimatingcutoffvaluesfordiagnosticteststoachievetargetspecificityusingextremevaluetheory |