Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques
Abstract Background Epidemiological surveys offer essential data on adolescent substance use. Nevertheless, the precision of these self-report-based surveys often faces mistrust from researchers and the public. We evaluate the efficacy of a direct method to assess data quality by asking adolescents...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-09-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-023-02035-y |
| _version_ | 1797452185526599680 |
|---|---|
| author | Janaka V. Kosgolla Douglas C. Smith Shahana Begum Crystal A. Reinhart |
| author_facet | Janaka V. Kosgolla Douglas C. Smith Shahana Begum Crystal A. Reinhart |
| author_sort | Janaka V. Kosgolla |
| collection | DOAJ |
| description | Abstract Background Epidemiological surveys offer essential data on adolescent substance use. Nevertheless, the precision of these self-report-based surveys often faces mistrust from researchers and the public. We evaluate the efficacy of a direct method to assess data quality by asking adolescents if they were honest. The main goal of our study was to assess the accuracy of a self-report honesty item and designate an optimal threshold for it, allowing us to better account for its impact on point estimates. Methods The participants were from the 2020 Illinois Youth Survey, a self-report school-based survey. We divided the primary dataset into subsets based on responses to an honesty item. Then, for each dataset, we examined two distinct data analysis methodologies: supervised machine learning, using the random forest algorithm, and a conventional inferential statistical method, logistic regression. We evaluated item thresholds from both analyses, investigating probable relationships with reported fake drug use, social desirability biases, and missingness in the datasets. Results The study results corroborate the appropriateness and reliability of the honesty item and its corresponding threshold. These contain the agreeing honesty thresholds determined in both data analyses, the identified association between reported fake drug use and lower honesty scores, increased missingness and lower honesty, and the determined link between the social desirability bias and honesty threshold. Conclusions Confirming the honesty threshold via missing data analysis also strengthens these collective findings, emphasizing our methodology’s and findings’ robustness. Researchers are encouraged to use self-report honesty items in epidemiological research. This will permit the modeling of accurate point estimates by addressing questionable reporting. |
| first_indexed | 2024-03-09T15:05:08Z |
| format | Article |
| id | doaj.art-a3921fef078e4eebb3404898e0b594c9 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-03-09T15:05:08Z |
| publishDate | 2023-09-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-a3921fef078e4eebb3404898e0b594c92023-11-26T13:42:55ZengBMCBMC Medical Research Methodology1471-22882023-09-012311910.1186/s12874-023-02035-yAssessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniquesJanaka V. Kosgolla0Douglas C. Smith1Shahana Begum2Crystal A. Reinhart3School of Social Work, University of Illinois Urbana-ChampaignSchool of Social Work, University of Illinois Urbana-ChampaignSchool of Social Work, University of Illinois Urbana-ChampaignSchool of Social Work, University of Illinois Urbana-ChampaignAbstract Background Epidemiological surveys offer essential data on adolescent substance use. Nevertheless, the precision of these self-report-based surveys often faces mistrust from researchers and the public. We evaluate the efficacy of a direct method to assess data quality by asking adolescents if they were honest. The main goal of our study was to assess the accuracy of a self-report honesty item and designate an optimal threshold for it, allowing us to better account for its impact on point estimates. Methods The participants were from the 2020 Illinois Youth Survey, a self-report school-based survey. We divided the primary dataset into subsets based on responses to an honesty item. Then, for each dataset, we examined two distinct data analysis methodologies: supervised machine learning, using the random forest algorithm, and a conventional inferential statistical method, logistic regression. We evaluated item thresholds from both analyses, investigating probable relationships with reported fake drug use, social desirability biases, and missingness in the datasets. Results The study results corroborate the appropriateness and reliability of the honesty item and its corresponding threshold. These contain the agreeing honesty thresholds determined in both data analyses, the identified association between reported fake drug use and lower honesty scores, increased missingness and lower honesty, and the determined link between the social desirability bias and honesty threshold. Conclusions Confirming the honesty threshold via missing data analysis also strengthens these collective findings, emphasizing our methodology’s and findings’ robustness. Researchers are encouraged to use self-report honesty items in epidemiological research. This will permit the modeling of accurate point estimates by addressing questionable reporting.https://doi.org/10.1186/s12874-023-02035-yAdolescentsSubstance useSelf-reported honestyMachine learningResponse validityEpidemiological surveys |
| spellingShingle | Janaka V. Kosgolla Douglas C. Smith Shahana Begum Crystal A. Reinhart Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques BMC Medical Research Methodology Adolescents Substance use Self-reported honesty Machine learning Response validity Epidemiological surveys |
| title | Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques |
| title_full | Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques |
| title_fullStr | Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques |
| title_full_unstemmed | Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques |
| title_short | Assessing the self-reported honesty threshold in adolescent epidemiological research: comparing supervised machine learning and inferential statistical techniques |
| title_sort | assessing the self reported honesty threshold in adolescent epidemiological research comparing supervised machine learning and inferential statistical techniques |
| topic | Adolescents Substance use Self-reported honesty Machine learning Response validity Epidemiological surveys |
| url | https://doi.org/10.1186/s12874-023-02035-y |
| work_keys_str_mv | AT janakavkosgolla assessingtheselfreportedhonestythresholdinadolescentepidemiologicalresearchcomparingsupervisedmachinelearningandinferentialstatisticaltechniques AT douglascsmith assessingtheselfreportedhonestythresholdinadolescentepidemiologicalresearchcomparingsupervisedmachinelearningandinferentialstatisticaltechniques AT shahanabegum assessingtheselfreportedhonestythresholdinadolescentepidemiologicalresearchcomparingsupervisedmachinelearningandinferentialstatisticaltechniques AT crystalareinhart assessingtheselfreportedhonestythresholdinadolescentepidemiologicalresearchcomparingsupervisedmachinelearningandinferentialstatisticaltechniques |