An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data
Abstract Background Selective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and spe...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-02-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-023-02134-w |
| _version_ | 1797274141739450368 |
|---|---|
| author | Zelalem F. Negeri Brooke Levis John P. A. Ioannidis Brett D. Thombs Andrea Benedetti the DEPRESsion Screening Data (DEPRESSD) EPDS Group |
| author_facet | Zelalem F. Negeri Brooke Levis John P. A. Ioannidis Brett D. Thombs Andrea Benedetti the DEPRESsion Screening Data (DEPRESSD) EPDS Group |
| author_sort | Zelalem F. Negeri |
| collection | DOAJ |
| description | Abstract Background Selective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and specificity at each cut-off via bivariate random-effects models (BREMs) can overcome this problem. However, IPDMA is laborious and depends on the ability to successfully obtain primary datasets, and BREMs ignore the correlation between cut-offs within primary studies. Methods We compared the performance of three recent multiple cut-off models developed by Steinhauser et al., Jones et al., and Hoyer and Kuss, that account for missing cut-offs when meta-analyzing diagnostic accuracy studies with multiple cut-offs, to BREMs fitted at each cut-off. We used data from 22 studies of the accuracy of the Edinburgh Postnatal Depression Scale (EPDS; 4475 participants, 758 major depression cases). We fitted each of the three multiple cut-off models and BREMs to a dataset with results from only published cut-offs from each study (published data) and an IPD dataset with results for all cut-offs (full IPD data). We estimated pooled sensitivity and specificity with 95% confidence intervals (CIs) for each cut-off and the area under the curve. Results Compared to the BREMs fitted to the full IPD data, the Steinhauser et al., Jones et al., and Hoyer and Kuss models fitted to the published data produced similar receiver operating characteristic curves; though, the Hoyer and Kuss model had lower area under the curve, mainly due to estimating slightly lower sensitivity at lower cut-offs. When fitting the three multiple cut-off models to the full IPD data, a similar pattern of results was observed. Importantly, all models had similar 95% CIs for sensitivity and specificity, and the CI width increased with cut-off levels for sensitivity and decreased with an increasing cut-off for specificity, even the BREMs which treat each cut-off separately. Conclusions Multiple cut-off models appear to be the favorable methods when only published data are available. While collecting IPD is expensive and time consuming, IPD can facilitate subgroup analyses that cannot be conducted with published data only. |
| first_indexed | 2024-03-07T14:55:08Z |
| format | Article |
| id | doaj.art-0f3a2dcead574fb4a85d1bcf61474e64 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-03-07T14:55:08Z |
| publishDate | 2024-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-0f3a2dcead574fb4a85d1bcf61474e642024-03-05T19:28:43ZengBMCBMC Medical Research Methodology1471-22882024-02-0124111310.1186/s12874-023-02134-wAn empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant dataZelalem F. Negeri0Brooke Levis1John P. A. Ioannidis2Brett D. Thombs3Andrea Benedetti4the DEPRESsion Screening Data (DEPRESSD) EPDS GroupDepartment of Statistics and Actuarial Science, University of WaterlooLady Davis Institute for Medical Research, Jewish General HospitalDepartment of Medicine, Department of Epidemiology and Population Health, Department of Biomedical Data Science, Department of Statistics, Stanford UniversityLady Davis Institute for Medical Research, Jewish General HospitalDepartment of Epidemiology, Biostatistics, and Occupational Health, McGill UniversityAbstract Background Selective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and specificity at each cut-off via bivariate random-effects models (BREMs) can overcome this problem. However, IPDMA is laborious and depends on the ability to successfully obtain primary datasets, and BREMs ignore the correlation between cut-offs within primary studies. Methods We compared the performance of three recent multiple cut-off models developed by Steinhauser et al., Jones et al., and Hoyer and Kuss, that account for missing cut-offs when meta-analyzing diagnostic accuracy studies with multiple cut-offs, to BREMs fitted at each cut-off. We used data from 22 studies of the accuracy of the Edinburgh Postnatal Depression Scale (EPDS; 4475 participants, 758 major depression cases). We fitted each of the three multiple cut-off models and BREMs to a dataset with results from only published cut-offs from each study (published data) and an IPD dataset with results for all cut-offs (full IPD data). We estimated pooled sensitivity and specificity with 95% confidence intervals (CIs) for each cut-off and the area under the curve. Results Compared to the BREMs fitted to the full IPD data, the Steinhauser et al., Jones et al., and Hoyer and Kuss models fitted to the published data produced similar receiver operating characteristic curves; though, the Hoyer and Kuss model had lower area under the curve, mainly due to estimating slightly lower sensitivity at lower cut-offs. When fitting the three multiple cut-off models to the full IPD data, a similar pattern of results was observed. Importantly, all models had similar 95% CIs for sensitivity and specificity, and the CI width increased with cut-off levels for sensitivity and decreased with an increasing cut-off for specificity, even the BREMs which treat each cut-off separately. Conclusions Multiple cut-off models appear to be the favorable methods when only published data are available. While collecting IPD is expensive and time consuming, IPD can facilitate subgroup analyses that cannot be conducted with published data only.https://doi.org/10.1186/s12874-023-02134-wMultiple cut-offs meta-analysisIndividual participant dataDepression screening accuracySensitivitySpecificitySelective reporting bias |
| spellingShingle | Zelalem F. Negeri Brooke Levis John P. A. Ioannidis Brett D. Thombs Andrea Benedetti the DEPRESsion Screening Data (DEPRESSD) EPDS Group An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data BMC Medical Research Methodology Multiple cut-offs meta-analysis Individual participant data Depression screening accuracy Sensitivity Specificity Selective reporting bias |
| title | An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data |
| title_full | An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data |
| title_fullStr | An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data |
| title_full_unstemmed | An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data |
| title_short | An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data |
| title_sort | empirical comparison of statistical methods for multiple cut off diagnostic test accuracy meta analysis of the edinburgh postnatal depression scale epds depression screening tool using published results vs individual participant data |
| topic | Multiple cut-offs meta-analysis Individual participant data Depression screening accuracy Sensitivity Specificity Selective reporting bias |
| url | https://doi.org/10.1186/s12874-023-02134-w |
| work_keys_str_mv | AT zelalemfnegeri anempiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT brookelevis anempiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT johnpaioannidis anempiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT brettdthombs anempiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT andreabenedetti anempiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT thedepressionscreeningdatadepressdepdsgroup anempiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT zelalemfnegeri empiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT brookelevis empiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT johnpaioannidis empiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT brettdthombs empiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT andreabenedetti empiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata AT thedepressionscreeningdatadepressdepdsgroup empiricalcomparisonofstatisticalmethodsformultiplecutoffdiagnostictestaccuracymetaanalysisoftheedinburghpostnataldepressionscaleepdsdepressionscreeningtoolusingpublishedresultsvsindividualparticipantdata |