Predicting Health Material Accessibility: Development of Machine Learning Algorithms
BackgroundCurrent health information understandability research uses medical readability formulas to assess the cognitive difficulty of health education resources. This is based on an implicit assumption that medical domain knowledge represented by uncommon words or jargon fo...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2021-09-01
|
| Series: | JMIR Medical Informatics |
| Online Access: | https://medinform.jmir.org/2021/9/e29175 |
| _version_ | 1797735750165331968 |
|---|---|
| author | Meng Ji Yanmeng Liu Tianyong Hao |
| author_facet | Meng Ji Yanmeng Liu Tianyong Hao |
| author_sort | Meng Ji |
| collection | DOAJ |
| description |
BackgroundCurrent health information understandability research uses medical readability formulas to assess the cognitive difficulty of health education resources. This is based on an implicit assumption that medical domain knowledge represented by uncommon words or jargon form the sole barriers to health information access among the public. Our study challenged this by showing that, for readers from non-English speaking backgrounds with higher education attainment, semantic features of English health texts that underpin the knowledge structure of English health texts, rather than medical jargon, can explain the cognitive accessibility of health materials among readers with better understanding of English health terms yet limited exposure to English-based health education environments and traditions.
ObjectiveOur study explores multidimensional semantic features for developing machine learning algorithms to predict the perceived level of cognitive accessibility of English health materials on health risks and diseases for young adults enrolled in Australian tertiary institutes. We compared algorithms to evaluate the cognitive accessibility of health information for nonnative English speakers with advanced education levels yet limited exposure to English health education environments.
MethodsWe used 113 semantic features to measure the content complexity and accessibility of original English resources. Using 1000 English health texts collected from Australian and international health organization websites rated by overseas tertiary students, we compared machine learning (decision tree, support vector machine [SVM], ensemble tree, and logistic regression) after hyperparameter optimization (grid search for the best hyperparameter combination of minimal classification errors). We applied 5-fold cross-validation on the whole data set for the model training and testing, and calculated the area under the operating characteristic curve (AUC), sensitivity, specificity, and accuracy as the measurement of the model performance.
ResultsWe developed and compared 4 machine learning algorithms using multidimensional semantic features as predictors. The results showed that ensemble classifier (LogitBoost) outperformed in terms of AUC (0.858), sensitivity (0.787), specificity (0.813), and accuracy (0.802). Support vector machine (AUC 0.848, sensitivity 0.783, specificity 0.791, and accuracy 0.786) and decision tree (AUC 0.754, sensitivity 0.7174, specificity 0.7424, and accuracy 0.732) followed. Ensemble classifier (LogitBoost), support vector machine, and decision tree achieved statistically significant improvement over logistic regression in AUC, sensitivity, specificity, and accuracy. Support vector machine reached statistically significant improvement over decision tree in AUC and accuracy. As the best performing algorithm, ensemble classifier (LogitBoost) reached statistically significant improvement over decision tree in AUC, sensitivity, specificity, and accuracy.
ConclusionsOur study shows that cognitive accessibility of English health texts is not limited to word length and sentence length as had been conventionally measured by medical readability formulas. We compared machine learning algorithms based on semantic features to explore the cognitive accessibility of health information for nonnative English speakers. The results showed the new models reached statistically increased AUC, sensitivity, and accuracy to predict health resource accessibility for the target readership. Our study illustrated that semantic features such as cognitive ability–related semantic features, communicative actions and processes, power relationships in health care settings, and lexical familiarity and diversity of health texts are large contributors to the comprehension of health information; for readers such as international students, semantic features of health texts outweigh syntax and domain knowledge. |
| first_indexed | 2024-03-12T13:03:37Z |
| format | Article |
| id | doaj.art-8949af653c3946cd96bdb4e3b1bc4ee7 |
| institution | Directory Open Access Journal |
| issn | 2291-9694 |
| language | English |
| last_indexed | 2024-03-12T13:03:37Z |
| publishDate | 2021-09-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Informatics |
| spelling | doaj.art-8949af653c3946cd96bdb4e3b1bc4ee72023-08-28T18:59:07ZengJMIR PublicationsJMIR Medical Informatics2291-96942021-09-0199e2917510.2196/29175Predicting Health Material Accessibility: Development of Machine Learning AlgorithmsMeng Jihttps://orcid.org/0000-0002-7463-9208Yanmeng Liuhttps://orcid.org/0000-0002-6444-8504Tianyong Haohttps://orcid.org/0000-0002-9792-3949 BackgroundCurrent health information understandability research uses medical readability formulas to assess the cognitive difficulty of health education resources. This is based on an implicit assumption that medical domain knowledge represented by uncommon words or jargon form the sole barriers to health information access among the public. Our study challenged this by showing that, for readers from non-English speaking backgrounds with higher education attainment, semantic features of English health texts that underpin the knowledge structure of English health texts, rather than medical jargon, can explain the cognitive accessibility of health materials among readers with better understanding of English health terms yet limited exposure to English-based health education environments and traditions. ObjectiveOur study explores multidimensional semantic features for developing machine learning algorithms to predict the perceived level of cognitive accessibility of English health materials on health risks and diseases for young adults enrolled in Australian tertiary institutes. We compared algorithms to evaluate the cognitive accessibility of health information for nonnative English speakers with advanced education levels yet limited exposure to English health education environments. MethodsWe used 113 semantic features to measure the content complexity and accessibility of original English resources. Using 1000 English health texts collected from Australian and international health organization websites rated by overseas tertiary students, we compared machine learning (decision tree, support vector machine [SVM], ensemble tree, and logistic regression) after hyperparameter optimization (grid search for the best hyperparameter combination of minimal classification errors). We applied 5-fold cross-validation on the whole data set for the model training and testing, and calculated the area under the operating characteristic curve (AUC), sensitivity, specificity, and accuracy as the measurement of the model performance. ResultsWe developed and compared 4 machine learning algorithms using multidimensional semantic features as predictors. The results showed that ensemble classifier (LogitBoost) outperformed in terms of AUC (0.858), sensitivity (0.787), specificity (0.813), and accuracy (0.802). Support vector machine (AUC 0.848, sensitivity 0.783, specificity 0.791, and accuracy 0.786) and decision tree (AUC 0.754, sensitivity 0.7174, specificity 0.7424, and accuracy 0.732) followed. Ensemble classifier (LogitBoost), support vector machine, and decision tree achieved statistically significant improvement over logistic regression in AUC, sensitivity, specificity, and accuracy. Support vector machine reached statistically significant improvement over decision tree in AUC and accuracy. As the best performing algorithm, ensemble classifier (LogitBoost) reached statistically significant improvement over decision tree in AUC, sensitivity, specificity, and accuracy. ConclusionsOur study shows that cognitive accessibility of English health texts is not limited to word length and sentence length as had been conventionally measured by medical readability formulas. We compared machine learning algorithms based on semantic features to explore the cognitive accessibility of health information for nonnative English speakers. The results showed the new models reached statistically increased AUC, sensitivity, and accuracy to predict health resource accessibility for the target readership. Our study illustrated that semantic features such as cognitive ability–related semantic features, communicative actions and processes, power relationships in health care settings, and lexical familiarity and diversity of health texts are large contributors to the comprehension of health information; for readers such as international students, semantic features of health texts outweigh syntax and domain knowledge.https://medinform.jmir.org/2021/9/e29175 |
| spellingShingle | Meng Ji Yanmeng Liu Tianyong Hao Predicting Health Material Accessibility: Development of Machine Learning Algorithms JMIR Medical Informatics |
| title | Predicting Health Material Accessibility: Development of Machine Learning Algorithms |
| title_full | Predicting Health Material Accessibility: Development of Machine Learning Algorithms |
| title_fullStr | Predicting Health Material Accessibility: Development of Machine Learning Algorithms |
| title_full_unstemmed | Predicting Health Material Accessibility: Development of Machine Learning Algorithms |
| title_short | Predicting Health Material Accessibility: Development of Machine Learning Algorithms |
| title_sort | predicting health material accessibility development of machine learning algorithms |
| url | https://medinform.jmir.org/2021/9/e29175 |
| work_keys_str_mv | AT mengji predictinghealthmaterialaccessibilitydevelopmentofmachinelearningalgorithms AT yanmengliu predictinghealthmaterialaccessibilitydevelopmentofmachinelearningalgorithms AT tianyonghao predictinghealthmaterialaccessibilitydevelopmentofmachinelearningalgorithms |