The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model
<p>Abstract</p> <p>Background</p> <p>Self-reported height and weight are commonly collected at the population level; however, they can be subject to measurement error. The impact of this error on predicted risk, discrimination, and calibration of a model that uses body...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2012-11-01
|
| Series: | Population Health Metrics |
| Online Access: | http://www.pophealthmetrics.com/content/10/1/20 |
| _version_ | 1829495754424582144 |
|---|---|
| author | Rosella Laura C Corey Paul Stukel Therese A Mustard Cam Hux Jan Manuel Doug G |
| author_facet | Rosella Laura C Corey Paul Stukel Therese A Mustard Cam Hux Jan Manuel Doug G |
| author_sort | Rosella Laura C |
| collection | DOAJ |
| description | <p>Abstract</p> <p>Background</p> <p>Self-reported height and weight are commonly collected at the population level; however, they can be subject to measurement error. The impact of this error on predicted risk, discrimination, and calibration of a model that uses body mass index (BMI) to predict risk of diabetes incidence is not known. The objective of this study is to use simulation to quantify and describe the effect of random and systematic error in self-reported height and weight on the performance of a model for predicting diabetes.</p> <p>Methods</p> <p>Two general categories of error were examined: random (nondirectional) error and systematic (directional) error on an algorithm relating BMI in kg/m<sup>2</sup> to probability of developing diabetes. The cohort used to develop the risk algorithm was derived from 23,403 Ontario residents that responded to the 1996/1997 National Population Health Survey linked to a population-based diabetes registry. The data and algorithm were then simulated to allow for estimation of the impact of these errors on predicted risk using the Hosmer-Lemeshow goodness-of-fit χ<sup>2</sup> and C-statistic. Simulations were done 500 times with sample sizes of 9,177 for males and 10,618 for females.</p> <p>Results</p> <p>Simulation data successfully reproduced discrimination and calibration generated from population data. Increasing levels of random error in height and weight reduced the calibration and discrimination of the model. Random error biased the predicted risk upwards whereas systematic error biased predicted risk in the direction of the bias and reduced calibration; however, it did not affect discrimination.</p> <p>Conclusion</p> <p>This study demonstrates that random and systematic errors in self-reported health data have the potential to influence the performance of risk algorithms. Further research that quantifies the amount and direction of error can improve model performance by allowing for adjustments in exposure measurements.</p> |
| first_indexed | 2024-12-16T07:16:19Z |
| format | Article |
| id | doaj.art-c6ebad8a9a214a81a0353023e61ff184 |
| institution | Directory Open Access Journal |
| issn | 1478-7954 |
| language | English |
| last_indexed | 2024-12-16T07:16:19Z |
| publishDate | 2012-11-01 |
| publisher | BMC |
| record_format | Article |
| series | Population Health Metrics |
| spelling | doaj.art-c6ebad8a9a214a81a0353023e61ff1842022-12-21T22:39:45ZengBMCPopulation Health Metrics1478-79542012-11-011012010.1186/1478-7954-10-20The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction modelRosella Laura CCorey PaulStukel Therese AMustard CamHux JanManuel Doug G<p>Abstract</p> <p>Background</p> <p>Self-reported height and weight are commonly collected at the population level; however, they can be subject to measurement error. The impact of this error on predicted risk, discrimination, and calibration of a model that uses body mass index (BMI) to predict risk of diabetes incidence is not known. The objective of this study is to use simulation to quantify and describe the effect of random and systematic error in self-reported height and weight on the performance of a model for predicting diabetes.</p> <p>Methods</p> <p>Two general categories of error were examined: random (nondirectional) error and systematic (directional) error on an algorithm relating BMI in kg/m<sup>2</sup> to probability of developing diabetes. The cohort used to develop the risk algorithm was derived from 23,403 Ontario residents that responded to the 1996/1997 National Population Health Survey linked to a population-based diabetes registry. The data and algorithm were then simulated to allow for estimation of the impact of these errors on predicted risk using the Hosmer-Lemeshow goodness-of-fit χ<sup>2</sup> and C-statistic. Simulations were done 500 times with sample sizes of 9,177 for males and 10,618 for females.</p> <p>Results</p> <p>Simulation data successfully reproduced discrimination and calibration generated from population data. Increasing levels of random error in height and weight reduced the calibration and discrimination of the model. Random error biased the predicted risk upwards whereas systematic error biased predicted risk in the direction of the bias and reduced calibration; however, it did not affect discrimination.</p> <p>Conclusion</p> <p>This study demonstrates that random and systematic errors in self-reported health data have the potential to influence the performance of risk algorithms. Further research that quantifies the amount and direction of error can improve model performance by allowing for adjustments in exposure measurements.</p>http://www.pophealthmetrics.com/content/10/1/20 |
| spellingShingle | Rosella Laura C Corey Paul Stukel Therese A Mustard Cam Hux Jan Manuel Doug G The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model Population Health Metrics |
| title | The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model |
| title_full | The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model |
| title_fullStr | The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model |
| title_full_unstemmed | The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model |
| title_short | The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model |
| title_sort | influence of measurement error on calibration discrimination and overall estimation of a risk prediction model |
| url | http://www.pophealthmetrics.com/content/10/1/20 |
| work_keys_str_mv | AT rosellalaurac theinfluenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT coreypaul theinfluenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT stukeltheresea theinfluenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT mustardcam theinfluenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT huxjan theinfluenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT manueldougg theinfluenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT rosellalaurac influenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT coreypaul influenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT stukeltheresea influenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT mustardcam influenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT huxjan influenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel AT manueldougg influenceofmeasurementerroroncalibrationdiscriminationandoverallestimationofariskpredictionmodel |