An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study
BackgroundPredicting the likelihood of success of weight loss interventions using machine learning (ML) models may enhance intervention effectiveness by enabling timely and dynamic modification of intervention components for nonresponders to treatment. However, a lack of unde...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-09-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2023/1/e42047 |
| _version_ | 1797691136941228032 |
|---|---|
| author | Glenn J Fernandes Arthur Choi Jacob Michael Schauer Angela F Pfammatter Bonnie J Spring Adnan Darwiche Nabil I Alshurafa |
| author_facet | Glenn J Fernandes Arthur Choi Jacob Michael Schauer Angela F Pfammatter Bonnie J Spring Adnan Darwiche Nabil I Alshurafa |
| author_sort | Glenn J Fernandes |
| collection | DOAJ |
| description |
BackgroundPredicting the likelihood of success of weight loss interventions using machine learning (ML) models may enhance intervention effectiveness by enabling timely and dynamic modification of intervention components for nonresponders to treatment. However, a lack of understanding and trust in these ML models impacts adoption among weight management experts. Recent advances in the field of explainable artificial intelligence enable the interpretation of ML models, yet it is unknown whether they enhance model understanding, trust, and adoption among weight management experts.
ObjectiveThis study aimed to build and evaluate an ML model that can predict 6-month weight loss success (ie, ≥7% weight loss) from 5 engagement and diet-related features collected over the initial 2 weeks of an intervention, to assess whether providing ML-based explanations increases weight management experts’ agreement with ML model predictions, and to inform factors that influence the understanding and trust of ML models to advance explainability in early prediction of weight loss among weight management experts.
MethodsWe trained an ML model using the random forest (RF) algorithm and data from a 6-month weight loss intervention (N=419). We leveraged findings from existing explainability metrics to develop Prime Implicant Maintenance of Outcome (PRIMO), an interactive tool to understand predictions made by the RF model. We asked 14 weight management experts to predict hypothetical participants’ weight loss success before and after using PRIMO. We compared PRIMO with 2 other explainability methods, one based on feature ranking and the other based on conditional probability. We used generalized linear mixed-effects models to evaluate participants’ agreement with ML predictions and conducted likelihood ratio tests to examine the relationship between explainability methods and outcomes for nested models. We conducted guided interviews and thematic analysis to study the impact of our tool on experts’ understanding and trust in the model.
ResultsOur RF model had 81% accuracy in the early prediction of weight loss success. Weight management experts were significantly more likely to agree with the model when using PRIMO (χ2=7.9; P=.02) compared with the other 2 methods with odds ratios of 2.52 (95% CI 0.91-7.69) and 3.95 (95% CI 1.50-11.76). From our study, we inferred that our software not only influenced experts’ understanding and trust but also impacted decision-making. Several themes were identified through interviews: preference for multiple explanation types, need to visualize uncertainty in explanations provided by PRIMO, and need for model performance metrics on similar participant test instances.
ConclusionsOur results show the potential for weight management experts to agree with the ML-based early prediction of success in weight loss treatment programs, enabling timely and dynamic modification of intervention components to enhance intervention effectiveness. Our findings provide methods for advancing the understandability and trust of ML models among weight management experts. |
| first_indexed | 2024-03-12T02:09:03Z |
| format | Article |
| id | doaj.art-478238e9e6774c62bd15b6110ebe795b |
| institution | Directory Open Access Journal |
| issn | 1438-8871 |
| language | English |
| last_indexed | 2024-03-12T02:09:03Z |
| publishDate | 2023-09-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj.art-478238e9e6774c62bd15b6110ebe795b2023-09-06T14:30:46ZengJMIR PublicationsJournal of Medical Internet Research1438-88712023-09-0125e4204710.2196/42047An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods StudyGlenn J Fernandeshttps://orcid.org/0000-0001-9070-4594Arthur Choihttps://orcid.org/0000-0002-1821-4221Jacob Michael Schauerhttps://orcid.org/0000-0002-9041-7082Angela F Pfammatterhttps://orcid.org/0000-0003-0081-4090Bonnie J Springhttps://orcid.org/0000-0003-0692-9868Adnan Darwichehttps://orcid.org/0000-0003-3976-6735Nabil I Alshurafahttps://orcid.org/0000-0001-6681-7564 BackgroundPredicting the likelihood of success of weight loss interventions using machine learning (ML) models may enhance intervention effectiveness by enabling timely and dynamic modification of intervention components for nonresponders to treatment. However, a lack of understanding and trust in these ML models impacts adoption among weight management experts. Recent advances in the field of explainable artificial intelligence enable the interpretation of ML models, yet it is unknown whether they enhance model understanding, trust, and adoption among weight management experts. ObjectiveThis study aimed to build and evaluate an ML model that can predict 6-month weight loss success (ie, ≥7% weight loss) from 5 engagement and diet-related features collected over the initial 2 weeks of an intervention, to assess whether providing ML-based explanations increases weight management experts’ agreement with ML model predictions, and to inform factors that influence the understanding and trust of ML models to advance explainability in early prediction of weight loss among weight management experts. MethodsWe trained an ML model using the random forest (RF) algorithm and data from a 6-month weight loss intervention (N=419). We leveraged findings from existing explainability metrics to develop Prime Implicant Maintenance of Outcome (PRIMO), an interactive tool to understand predictions made by the RF model. We asked 14 weight management experts to predict hypothetical participants’ weight loss success before and after using PRIMO. We compared PRIMO with 2 other explainability methods, one based on feature ranking and the other based on conditional probability. We used generalized linear mixed-effects models to evaluate participants’ agreement with ML predictions and conducted likelihood ratio tests to examine the relationship between explainability methods and outcomes for nested models. We conducted guided interviews and thematic analysis to study the impact of our tool on experts’ understanding and trust in the model. ResultsOur RF model had 81% accuracy in the early prediction of weight loss success. Weight management experts were significantly more likely to agree with the model when using PRIMO (χ2=7.9; P=.02) compared with the other 2 methods with odds ratios of 2.52 (95% CI 0.91-7.69) and 3.95 (95% CI 1.50-11.76). From our study, we inferred that our software not only influenced experts’ understanding and trust but also impacted decision-making. Several themes were identified through interviews: preference for multiple explanation types, need to visualize uncertainty in explanations provided by PRIMO, and need for model performance metrics on similar participant test instances. ConclusionsOur results show the potential for weight management experts to agree with the ML-based early prediction of success in weight loss treatment programs, enabling timely and dynamic modification of intervention components to enhance intervention effectiveness. Our findings provide methods for advancing the understandability and trust of ML models among weight management experts.https://www.jmir.org/2023/1/e42047 |
| spellingShingle | Glenn J Fernandes Arthur Choi Jacob Michael Schauer Angela F Pfammatter Bonnie J Spring Adnan Darwiche Nabil I Alshurafa An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study Journal of Medical Internet Research |
| title | An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study |
| title_full | An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study |
| title_fullStr | An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study |
| title_full_unstemmed | An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study |
| title_short | An Explainable Artificial Intelligence Software Tool for Weight Management Experts (PRIMO): Mixed Methods Study |
| title_sort | explainable artificial intelligence software tool for weight management experts primo mixed methods study |
| url | https://www.jmir.org/2023/1/e42047 |
| work_keys_str_mv | AT glennjfernandes anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT arthurchoi anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT jacobmichaelschauer anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT angelafpfammatter anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT bonniejspring anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT adnandarwiche anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT nabilialshurafa anexplainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT glennjfernandes explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT arthurchoi explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT jacobmichaelschauer explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT angelafpfammatter explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT bonniejspring explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT adnandarwiche explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy AT nabilialshurafa explainableartificialintelligencesoftwaretoolforweightmanagementexpertsprimomixedmethodsstudy |