Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review

BackgroundThe field of eHealth is growing rapidly and chaotically. Health care professionals need guidance on reviewing and assessing health-related smartphone apps to propose appropriate ones to their patients. However, to date, no framework or evaluation tool fulfills this...

Full description

Bibliographic Details
Main Authors: Janette Ribaut, Annette DeVito Dabbs, Fabienne Dobbels, Alexandra Teynor, Elisabeth Veronica Mess, Theresa Hoffmann, Sabina De Geest
Format: Article
Language:English
Published: JMIR Publications 2024-01-01
Series:JMIR mHealth and uHealth
Online Access:https://mhealth.jmir.org/2024/1/e48625
_version_ 1797354866247467008
author Janette Ribaut
Annette DeVito Dabbs
Fabienne Dobbels
Alexandra Teynor
Elisabeth Veronica Mess
Theresa Hoffmann
Sabina De Geest
author_facet Janette Ribaut
Annette DeVito Dabbs
Fabienne Dobbels
Alexandra Teynor
Elisabeth Veronica Mess
Theresa Hoffmann
Sabina De Geest
author_sort Janette Ribaut
collection DOAJ
description BackgroundThe field of eHealth is growing rapidly and chaotically. Health care professionals need guidance on reviewing and assessing health-related smartphone apps to propose appropriate ones to their patients. However, to date, no framework or evaluation tool fulfills this purpose. ObjectiveBefore developing a tool to help health care professionals assess and recommend apps to their patients, we aimed to create an overview of published criteria to describe and evaluate health apps. MethodsWe conducted a systematic review to identify existing criteria for eHealth smartphone app evaluation. Relevant databases and trial registers were queried for articles. Articles were included that (1) described tools, guidelines, dimensions, or criteria to evaluate apps, (2) were available in full text, and (3) were written in English, French, German, Italian, Portuguese, or Spanish. We proposed a conceptual framework for app evaluation based on the dimensions reported in the selected articles. This was revised iteratively in discussion rounds with international stakeholders. The conceptual framework was used to synthesize the reported evaluation criteria. The list of criteria was discussed and refined by the research team. ResultsScreening of 1258 articles yielded 128 (10.17%) that met the inclusion criteria. Of these 128 articles, 30 (23.4%) reported the use of self-developed criteria and described their development processes incompletely. Although 43 evaluation instruments were used only once, 6 were used in multiple studies. Most articles (83/128, 64.8%) did not report following theoretical guidelines; those that did noted 37 theoretical frameworks. On the basis of the selected articles, we proposed a conceptual framework to explore 6 app evaluation dimensions: context, stakeholder involvement, features and requirements, development processes, implementation, and evaluation. After standardizing the definitions, we identified 205 distinct criteria. Through consensus, the research team relabeled 12 of these and added 11 more—mainly related to ethical, legal, and social aspects—resulting in 216 evaluation criteria. No criteria had to be moved between dimensions. ConclusionsThis study provides a comprehensive overview of criteria currently used in clinical practice to describe and evaluate apps. This is necessary as no reviewed criteria sets were inclusive, and none included consistent definitions and terminology. Although the resulting overview is impractical for use in clinical practice in its current form, it confirms the need to craft it into a purpose-built, theory-driven tool. Therefore, in a subsequent step, based on our current criteria set, we plan to construct an app evaluation tool with 2 parts: a short section (including 1-3 questions/dimension) to quickly disqualify clearly unsuitable apps and a longer one to investigate more likely candidates in closer detail. We will use a Delphi consensus-building process and develop a user manual to prepare for this undertaking. Trial RegistrationPROSPERO International Prospective Register of Systematic Reviews CRD42021227064; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021227064
first_indexed 2024-03-08T13:55:02Z
format Article
id doaj.art-410d9441dbe34173bc705c90ee761ddd
institution Directory Open Access Journal
issn 2291-5222
language English
last_indexed 2024-03-08T13:55:02Z
publishDate 2024-01-01
publisher JMIR Publications
record_format Article
series JMIR mHealth and uHealth
spelling doaj.art-410d9441dbe34173bc705c90ee761ddd2024-01-15T15:30:34ZengJMIR PublicationsJMIR mHealth and uHealth2291-52222024-01-0112e4862510.2196/48625Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic ReviewJanette Ribauthttps://orcid.org/0000-0003-0654-4052Annette DeVito Dabbshttps://orcid.org/0000-0003-3325-435XFabienne Dobbelshttps://orcid.org/0000-0002-9502-7374Alexandra Teynorhttps://orcid.org/0000-0002-8141-0428Elisabeth Veronica Messhttps://orcid.org/0000-0003-1205-9141Theresa Hoffmannhttps://orcid.org/0009-0002-2742-6362Sabina De Geesthttps://orcid.org/0000-0001-6596-7237 BackgroundThe field of eHealth is growing rapidly and chaotically. Health care professionals need guidance on reviewing and assessing health-related smartphone apps to propose appropriate ones to their patients. However, to date, no framework or evaluation tool fulfills this purpose. ObjectiveBefore developing a tool to help health care professionals assess and recommend apps to their patients, we aimed to create an overview of published criteria to describe and evaluate health apps. MethodsWe conducted a systematic review to identify existing criteria for eHealth smartphone app evaluation. Relevant databases and trial registers were queried for articles. Articles were included that (1) described tools, guidelines, dimensions, or criteria to evaluate apps, (2) were available in full text, and (3) were written in English, French, German, Italian, Portuguese, or Spanish. We proposed a conceptual framework for app evaluation based on the dimensions reported in the selected articles. This was revised iteratively in discussion rounds with international stakeholders. The conceptual framework was used to synthesize the reported evaluation criteria. The list of criteria was discussed and refined by the research team. ResultsScreening of 1258 articles yielded 128 (10.17%) that met the inclusion criteria. Of these 128 articles, 30 (23.4%) reported the use of self-developed criteria and described their development processes incompletely. Although 43 evaluation instruments were used only once, 6 were used in multiple studies. Most articles (83/128, 64.8%) did not report following theoretical guidelines; those that did noted 37 theoretical frameworks. On the basis of the selected articles, we proposed a conceptual framework to explore 6 app evaluation dimensions: context, stakeholder involvement, features and requirements, development processes, implementation, and evaluation. After standardizing the definitions, we identified 205 distinct criteria. Through consensus, the research team relabeled 12 of these and added 11 more—mainly related to ethical, legal, and social aspects—resulting in 216 evaluation criteria. No criteria had to be moved between dimensions. ConclusionsThis study provides a comprehensive overview of criteria currently used in clinical practice to describe and evaluate apps. This is necessary as no reviewed criteria sets were inclusive, and none included consistent definitions and terminology. Although the resulting overview is impractical for use in clinical practice in its current form, it confirms the need to craft it into a purpose-built, theory-driven tool. Therefore, in a subsequent step, based on our current criteria set, we plan to construct an app evaluation tool with 2 parts: a short section (including 1-3 questions/dimension) to quickly disqualify clearly unsuitable apps and a longer one to investigate more likely candidates in closer detail. We will use a Delphi consensus-building process and develop a user manual to prepare for this undertaking. Trial RegistrationPROSPERO International Prospective Register of Systematic Reviews CRD42021227064; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021227064https://mhealth.jmir.org/2024/1/e48625
spellingShingle Janette Ribaut
Annette DeVito Dabbs
Fabienne Dobbels
Alexandra Teynor
Elisabeth Veronica Mess
Theresa Hoffmann
Sabina De Geest
Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review
JMIR mHealth and uHealth
title Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review
title_full Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review
title_fullStr Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review
title_full_unstemmed Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review
title_short Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review
title_sort developing a comprehensive list of criteria to evaluate the characteristics and quality of ehealth smartphone apps systematic review
url https://mhealth.jmir.org/2024/1/e48625
work_keys_str_mv AT janetteribaut developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview
AT annettedevitodabbs developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview
AT fabiennedobbels developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview
AT alexandrateynor developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview
AT elisabethveronicamess developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview
AT theresahoffmann developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview
AT sabinadegeest developingacomprehensivelistofcriteriatoevaluatethecharacteristicsandqualityofehealthsmartphoneappssystematicreview