Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study

BackgroundStory recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for a...

Full description

Bibliographic Details
Main Authors: Caroline Skirrow, Marton Meszaros, Udeepa Meepegama, Raphael Lenain, Kathryn V Papp, Jack Weston, Emil Fristed
Format: Article
Language:English
Published: JMIR Publications 2022-09-01
Series:JMIR Aging
Online Access:https://aging.jmir.org/2022/3/e37090
_version_ 1797734698400612352
author Caroline Skirrow
Marton Meszaros
Udeepa Meepegama
Raphael Lenain
Kathryn V Papp
Jack Weston
Emil Fristed
author_facet Caroline Skirrow
Marton Meszaros
Udeepa Meepegama
Raphael Lenain
Kathryn V Papp
Jack Weston
Emil Fristed
author_sort Caroline Skirrow
collection DOAJ
description BackgroundStory recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for automated administration and scoring. Multiple parallel test stimuli are required for higher-frequency disease monitoring. ObjectiveThis study aims to develop and validate a remote and fully automated story recall task, suitable for longitudinal assessment, in a population of older adults with and without mild cognitive impairment (MCI) or mild AD. MethodsThe “Amyloid Prediction in Early Stage Alzheimer’s disease” (AMYPRED) studies recruited participants in the United Kingdom (AMYPRED-UK: NCT04828122) and the United States (AMYPRED-US: NCT04928976). Participants were asked to complete optional daily self-administered assessments remotely on their smart devices over 7 to 8 days. Assessments included immediate and delayed recall of 3 stories from the Automatic Story Recall Task (ASRT), a test with multiple parallel stimuli (18 short stories and 18 long stories) balanced for key linguistic and discourse metrics. Verbal responses were recorded and securely transferred from participants’ personal devices and automatically transcribed and scored using text similarity metrics between the source text and retelling to derive a generalized match score. Group differences in adherence and task performance were examined using logistic and linear mixed models, respectively. Correlational analysis examined parallel-forms reliability of ASRTs and convergent validity with cognitive tests (Logical Memory Test and Preclinical Alzheimer’s Cognitive Composite with semantic processing). Acceptability and usability data were obtained using a remotely administered questionnaire. ResultsOf the 200 participants recruited in the AMYPRED studies, 151 (75.5%)—78 cognitively unimpaired (CU) and 73 MCI or mild AD—engaged in optional remote assessments. Adherence to daily assessment was moderate and did not decline over time but was higher in CU participants (ASRTs were completed each day by 73/106, 68.9% participants with MCI or mild AD and 78/94, 83% CU participants). Participants reported favorable task usability: infrequent technical problems, easy use of the app, and a broad interest in the tasks. Task performance improved modestly across the week and was better for immediate recall. The generalized match scores were lower in participants with MCI or mild AD (Cohen d=1.54). Parallel-forms reliability of ASRT stories was moderate to strong for immediate recall (mean rho 0.73, range 0.56-0.88) and delayed recall (mean rho=0.73, range=0.54-0.86). The ASRTs showed moderate convergent validity with established cognitive tests. ConclusionsThe unsupervised, self-administered ASRT task is sensitive to cognitive impairments in MCI and mild AD. The task showed good usability, high parallel-forms reliability, and high convergent validity with established cognitive tests. Remote, low-cost, low-burden, and automatically scored speech assessments could support diagnostic screening, health care, and treatment monitoring.
first_indexed 2024-03-12T12:47:09Z
format Article
id doaj.art-2907720129894251bf9d21602e99ec2f
institution Directory Open Access Journal
issn 2561-7605
language English
last_indexed 2024-03-12T12:47:09Z
publishDate 2022-09-01
publisher JMIR Publications
record_format Article
series JMIR Aging
spelling doaj.art-2907720129894251bf9d21602e99ec2f2023-08-28T23:09:28ZengJMIR PublicationsJMIR Aging2561-76052022-09-0153e3709010.2196/37090Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational StudyCaroline Skirrowhttps://orcid.org/0000-0001-8692-7787Marton Meszaroshttps://orcid.org/0000-0003-4937-3062Udeepa Meepegamahttps://orcid.org/0000-0001-8718-7530Raphael Lenainhttps://orcid.org/0000-0001-5345-0628Kathryn V Papphttps://orcid.org/0000-0003-1687-4370Jack Westonhttps://orcid.org/0000-0001-5344-7840Emil Fristedhttps://orcid.org/0000-0002-9590-7275 BackgroundStory recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for automated administration and scoring. Multiple parallel test stimuli are required for higher-frequency disease monitoring. ObjectiveThis study aims to develop and validate a remote and fully automated story recall task, suitable for longitudinal assessment, in a population of older adults with and without mild cognitive impairment (MCI) or mild AD. MethodsThe “Amyloid Prediction in Early Stage Alzheimer’s disease” (AMYPRED) studies recruited participants in the United Kingdom (AMYPRED-UK: NCT04828122) and the United States (AMYPRED-US: NCT04928976). Participants were asked to complete optional daily self-administered assessments remotely on their smart devices over 7 to 8 days. Assessments included immediate and delayed recall of 3 stories from the Automatic Story Recall Task (ASRT), a test with multiple parallel stimuli (18 short stories and 18 long stories) balanced for key linguistic and discourse metrics. Verbal responses were recorded and securely transferred from participants’ personal devices and automatically transcribed and scored using text similarity metrics between the source text and retelling to derive a generalized match score. Group differences in adherence and task performance were examined using logistic and linear mixed models, respectively. Correlational analysis examined parallel-forms reliability of ASRTs and convergent validity with cognitive tests (Logical Memory Test and Preclinical Alzheimer’s Cognitive Composite with semantic processing). Acceptability and usability data were obtained using a remotely administered questionnaire. ResultsOf the 200 participants recruited in the AMYPRED studies, 151 (75.5%)—78 cognitively unimpaired (CU) and 73 MCI or mild AD—engaged in optional remote assessments. Adherence to daily assessment was moderate and did not decline over time but was higher in CU participants (ASRTs were completed each day by 73/106, 68.9% participants with MCI or mild AD and 78/94, 83% CU participants). Participants reported favorable task usability: infrequent technical problems, easy use of the app, and a broad interest in the tasks. Task performance improved modestly across the week and was better for immediate recall. The generalized match scores were lower in participants with MCI or mild AD (Cohen d=1.54). Parallel-forms reliability of ASRT stories was moderate to strong for immediate recall (mean rho 0.73, range 0.56-0.88) and delayed recall (mean rho=0.73, range=0.54-0.86). The ASRTs showed moderate convergent validity with established cognitive tests. ConclusionsThe unsupervised, self-administered ASRT task is sensitive to cognitive impairments in MCI and mild AD. The task showed good usability, high parallel-forms reliability, and high convergent validity with established cognitive tests. Remote, low-cost, low-burden, and automatically scored speech assessments could support diagnostic screening, health care, and treatment monitoring.https://aging.jmir.org/2022/3/e37090
spellingShingle Caroline Skirrow
Marton Meszaros
Udeepa Meepegama
Raphael Lenain
Kathryn V Papp
Jack Weston
Emil Fristed
Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
JMIR Aging
title Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
title_full Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
title_fullStr Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
title_full_unstemmed Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
title_short Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
title_sort validation of a remote and fully automated story recall task to assess for early cognitive impairment in older adults longitudinal case control observational study
url https://aging.jmir.org/2022/3/e37090
work_keys_str_mv AT carolineskirrow validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy
AT martonmeszaros validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy
AT udeepameepegama validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy
AT raphaellenain validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy
AT kathrynvpapp validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy
AT jackweston validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy
AT emilfristed validationofaremoteandfullyautomatedstoryrecalltasktoassessforearlycognitiveimpairmentinolderadultslongitudinalcasecontrolobservationalstudy