Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach
BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the mai...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2024-03-01
|
| Series: | JMIR Formative Research |
| Online Access: | https://formative.jmir.org/2024/1/e52482 |
| _version_ | 1797246087603421184 |
|---|---|
| author | Duy-Anh Nguyen Minyi Li Gavin Lambert Ryszard Kowalczyk Rachael McDonald Quoc Bao Vo |
| author_facet | Duy-Anh Nguyen Minyi Li Gavin Lambert Ryszard Kowalczyk Rachael McDonald Quoc Bao Vo |
| author_sort | Duy-Anh Nguyen |
| collection | DOAJ |
| description |
BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context.
ObjectiveThis study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately.
MethodsExisting approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model’s performance.
ResultsWe demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F1-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases.
ConclusionsThe proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts. |
| first_indexed | 2024-04-24T19:37:13Z |
| format | Article |
| id | doaj.art-22cba6d6940146d0a4772eca4f5abf85 |
| institution | Directory Open Access Journal |
| issn | 2561-326X |
| language | English |
| last_indexed | 2024-04-24T19:37:13Z |
| publishDate | 2024-03-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Formative Research |
| spelling | doaj.art-22cba6d6940146d0a4772eca4f5abf852024-03-25T12:45:32ZengJMIR PublicationsJMIR Formative Research2561-326X2024-03-018e5248210.2196/52482Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction ApproachDuy-Anh Nguyenhttps://orcid.org/0009-0004-4812-8692Minyi Lihttps://orcid.org/0000-0002-9314-5799Gavin Lamberthttps://orcid.org/0000-0003-0315-645XRyszard Kowalczykhttps://orcid.org/0000-0003-0937-4028Rachael McDonaldhttps://orcid.org/0000-0002-9991-1255Quoc Bao Vohttps://orcid.org/0000-0002-7404-110X BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context. ObjectiveThis study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately. MethodsExisting approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model’s performance. ResultsWe demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F1-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases. ConclusionsThe proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts.https://formative.jmir.org/2024/1/e52482 |
| spellingShingle | Duy-Anh Nguyen Minyi Li Gavin Lambert Ryszard Kowalczyk Rachael McDonald Quoc Bao Vo Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach JMIR Formative Research |
| title | Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach |
| title_full | Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach |
| title_fullStr | Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach |
| title_full_unstemmed | Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach |
| title_short | Efficient Machine Reading Comprehension for Health Care Applications: Algorithm Development and Validation of a Context Extraction Approach |
| title_sort | efficient machine reading comprehension for health care applications algorithm development and validation of a context extraction approach |
| url | https://formative.jmir.org/2024/1/e52482 |
| work_keys_str_mv | AT duyanhnguyen efficientmachinereadingcomprehensionforhealthcareapplicationsalgorithmdevelopmentandvalidationofacontextextractionapproach AT minyili efficientmachinereadingcomprehensionforhealthcareapplicationsalgorithmdevelopmentandvalidationofacontextextractionapproach AT gavinlambert efficientmachinereadingcomprehensionforhealthcareapplicationsalgorithmdevelopmentandvalidationofacontextextractionapproach AT ryszardkowalczyk efficientmachinereadingcomprehensionforhealthcareapplicationsalgorithmdevelopmentandvalidationofacontextextractionapproach AT rachaelmcdonald efficientmachinereadingcomprehensionforhealthcareapplicationsalgorithmdevelopmentandvalidationofacontextextractionapproach AT quocbaovo efficientmachinereadingcomprehensionforhealthcareapplicationsalgorithmdevelopmentandvalidationofacontextextractionapproach |