Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study
BackgroundReproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproducti...
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-12-01
|
| Series: | JMIR mHealth and uHealth |
| Online Access: | https://mhealth.jmir.org/2023/1/e46718 |
| _version_ | 1827595215024685056 |
|---|---|
| author | Kimberly Peven Aidan P Wickham Octavia Wilks Yusuf C Kaplan Andrei Marhol Saddif Ahmed Ryan Bamford Adam C Cunningham Carley Prentice András Meczner Matthew Fenech Stephen Gilbert Anna Klepchukova Sonia Ponzo Liudmila Zhaunova |
| author_facet | Kimberly Peven Aidan P Wickham Octavia Wilks Yusuf C Kaplan Andrei Marhol Saddif Ahmed Ryan Bamford Adam C Cunningham Carley Prentice András Meczner Matthew Fenech Stephen Gilbert Anna Klepchukova Sonia Ponzo Liudmila Zhaunova |
| author_sort | Kimberly Peven |
| collection | DOAJ |
| description |
BackgroundReproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions.
ObjectiveThis study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools.
MethodsIndependent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value.
ResultsA total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS.
ConclusionsThe single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways. |
| first_indexed | 2024-03-09T02:48:27Z |
| format | Article |
| id | doaj.art-a27f2a74d01e4778ab0db20174d8af04 |
| institution | Directory Open Access Journal |
| issn | 2291-5222 |
| language | English |
| last_indexed | 2024-03-09T02:48:27Z |
| publishDate | 2023-12-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR mHealth and uHealth |
| spelling | doaj.art-a27f2a74d01e4778ab0db20174d8af042023-12-05T15:00:46ZengJMIR PublicationsJMIR mHealth and uHealth2291-52222023-12-0111e4671810.2196/46718Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes StudyKimberly Pevenhttps://orcid.org/0000-0002-0047-4084Aidan P Wickhamhttps://orcid.org/0000-0001-6569-059XOctavia Wilkshttps://orcid.org/0000-0001-6299-4627Yusuf C Kaplanhttps://orcid.org/0000-0003-0369-7934Andrei Marholhttps://orcid.org/0000-0002-5402-2593Saddif Ahmedhttps://orcid.org/0000-0002-9087-0899Ryan Bamfordhttps://orcid.org/0000-0002-1229-3782Adam C Cunninghamhttps://orcid.org/0000-0002-9791-7813Carley Prenticehttps://orcid.org/0000-0003-1710-1388András Mecznerhttps://orcid.org/0000-0001-8136-7768Matthew Fenechhttps://orcid.org/0000-0001-9970-0215Stephen Gilberthttps://orcid.org/0000-0002-1997-1689Anna Klepchukovahttps://orcid.org/0000-0002-3035-3267Sonia Ponzohttps://orcid.org/0000-0002-6754-5078Liudmila Zhaunovahttps://orcid.org/0000-0001-6000-1898 BackgroundReproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions. ObjectiveThis study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools. MethodsIndependent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value. ResultsA total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS. ConclusionsThe single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways.https://mhealth.jmir.org/2023/1/e46718 |
| spellingShingle | Kimberly Peven Aidan P Wickham Octavia Wilks Yusuf C Kaplan Andrei Marhol Saddif Ahmed Ryan Bamford Adam C Cunningham Carley Prentice András Meczner Matthew Fenech Stephen Gilbert Anna Klepchukova Sonia Ponzo Liudmila Zhaunova Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study JMIR mHealth and uHealth |
| title | Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study |
| title_full | Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study |
| title_fullStr | Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study |
| title_full_unstemmed | Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study |
| title_short | Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study |
| title_sort | assessment of a digital symptom checker tool s accuracy in suggesting reproductive health conditions clinical vignettes study |
| url | https://mhealth.jmir.org/2023/1/e46718 |
| work_keys_str_mv | AT kimberlypeven assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT aidanpwickham assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT octaviawilks assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT yusufckaplan assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT andreimarhol assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT saddifahmed assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT ryanbamford assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT adamccunningham assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT carleyprentice assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT andrasmeczner assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT matthewfenech assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT stephengilbert assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT annaklepchukova assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT soniaponzo assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy AT liudmilazhaunova assessmentofadigitalsymptomcheckertoolsaccuracyinsuggestingreproductivehealthconditionsclinicalvignettesstudy |