A medical multimodal large language model for future pandemics
Deep neural networks have been integrated into the whole clinical decision procedure which can improve the efficiency of diagnosis and alleviate the heavy workload of physicians. Since most neural networks are supervised, their performance heavily depends on the volume and quality of available label...
| Päätekijät: | , , , , , , , , , , , , |
|---|---|
| Aineistotyyppi: | Journal article |
| Kieli: | English |
| Julkaistu: |
Springer Nature
2023
|
| _version_ | 1826311779341303808 |
|---|---|
| author | Liu, F Zhu, T Wu, X Yang, B You, C Wang, C Lu, L Liu, Z Zheng, Y Sun, X Yang, Y Clifton, L Clifton, DA |
| author_facet | Liu, F Zhu, T Wu, X Yang, B You, C Wang, C Lu, L Liu, Z Zheng, Y Sun, X Yang, Y Clifton, L Clifton, DA |
| author_sort | Liu, F |
| collection | OXFORD |
| description | Deep neural networks have been integrated into the whole clinical decision procedure which can improve the efficiency of diagnosis and alleviate the heavy workload of physicians. Since most neural networks are supervised, their performance heavily depends on the volume and quality of available labels. However, few such labels exist for rare diseases (e.g., new pandemics). Here we report a medical multimodal large language model (Med-MLLM) for radiograph representation learning, which can learn broad medical knowledge (e.g., image understanding, text semantics, and clinical phenotypes) from unlabelled data. As a result, when encountering a rare disease, our Med-MLLM can be rapidly deployed and easily adapted to them with limited labels. Furthermore, our model supports medical data across visual modality (e.g., chest X-ray and CT) and textual modality (e.g., medical report and free-text clinical note); therefore, it can be used for clinical tasks that involve both visual and textual data. We demonstrate the effectiveness of our Med-MLLM by showing how it would perform using the COVID-19 pandemic “in replay”. In the retrospective setting, we test the model on the early COVID-19 datasets; and in the prospective setting, we test the model on the new variant COVID-19-Omicron. The experiments are conducted on 1) three kinds of input data; 2) three kinds of downstream tasks, including disease reporting, diagnosis, and prognosis; 3) five COVID-19 datasets; and 4) three different languages, including English, Chinese, and Spanish. All experiments show that our model can make accurate and robust COVID-19 decision-support with little labelled data. |
| first_indexed | 2024-03-07T08:14:47Z |
| format | Journal article |
| id | oxford-uuid:dc358e68-1be1-4c7b-a478-7e0023fda5c5 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T08:14:47Z |
| publishDate | 2023 |
| publisher | Springer Nature |
| record_format | dspace |
| spelling | oxford-uuid:dc358e68-1be1-4c7b-a478-7e0023fda5c52023-12-19T11:31:52ZA medical multimodal large language model for future pandemicsJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:dc358e68-1be1-4c7b-a478-7e0023fda5c5EnglishSymplectic ElementsSpringer Nature2023Liu, FZhu, TWu, XYang, BYou, CWang, CLu, LLiu, ZZheng, YSun, XYang, YClifton, LClifton, DADeep neural networks have been integrated into the whole clinical decision procedure which can improve the efficiency of diagnosis and alleviate the heavy workload of physicians. Since most neural networks are supervised, their performance heavily depends on the volume and quality of available labels. However, few such labels exist for rare diseases (e.g., new pandemics). Here we report a medical multimodal large language model (Med-MLLM) for radiograph representation learning, which can learn broad medical knowledge (e.g., image understanding, text semantics, and clinical phenotypes) from unlabelled data. As a result, when encountering a rare disease, our Med-MLLM can be rapidly deployed and easily adapted to them with limited labels. Furthermore, our model supports medical data across visual modality (e.g., chest X-ray and CT) and textual modality (e.g., medical report and free-text clinical note); therefore, it can be used for clinical tasks that involve both visual and textual data. We demonstrate the effectiveness of our Med-MLLM by showing how it would perform using the COVID-19 pandemic “in replay”. In the retrospective setting, we test the model on the early COVID-19 datasets; and in the prospective setting, we test the model on the new variant COVID-19-Omicron. The experiments are conducted on 1) three kinds of input data; 2) three kinds of downstream tasks, including disease reporting, diagnosis, and prognosis; 3) five COVID-19 datasets; and 4) three different languages, including English, Chinese, and Spanish. All experiments show that our model can make accurate and robust COVID-19 decision-support with little labelled data. |
| spellingShingle | Liu, F Zhu, T Wu, X Yang, B You, C Wang, C Lu, L Liu, Z Zheng, Y Sun, X Yang, Y Clifton, L Clifton, DA A medical multimodal large language model for future pandemics |
| title | A medical multimodal large language model for future pandemics |
| title_full | A medical multimodal large language model for future pandemics |
| title_fullStr | A medical multimodal large language model for future pandemics |
| title_full_unstemmed | A medical multimodal large language model for future pandemics |
| title_short | A medical multimodal large language model for future pandemics |
| title_sort | medical multimodal large language model for future pandemics |
| work_keys_str_mv | AT liuf amedicalmultimodallargelanguagemodelforfuturepandemics AT zhut amedicalmultimodallargelanguagemodelforfuturepandemics AT wux amedicalmultimodallargelanguagemodelforfuturepandemics AT yangb amedicalmultimodallargelanguagemodelforfuturepandemics AT youc amedicalmultimodallargelanguagemodelforfuturepandemics AT wangc amedicalmultimodallargelanguagemodelforfuturepandemics AT lul amedicalmultimodallargelanguagemodelforfuturepandemics AT liuz amedicalmultimodallargelanguagemodelforfuturepandemics AT zhengy amedicalmultimodallargelanguagemodelforfuturepandemics AT sunx amedicalmultimodallargelanguagemodelforfuturepandemics AT yangy amedicalmultimodallargelanguagemodelforfuturepandemics AT cliftonl amedicalmultimodallargelanguagemodelforfuturepandemics AT cliftonda amedicalmultimodallargelanguagemodelforfuturepandemics AT liuf medicalmultimodallargelanguagemodelforfuturepandemics AT zhut medicalmultimodallargelanguagemodelforfuturepandemics AT wux medicalmultimodallargelanguagemodelforfuturepandemics AT yangb medicalmultimodallargelanguagemodelforfuturepandemics AT youc medicalmultimodallargelanguagemodelforfuturepandemics AT wangc medicalmultimodallargelanguagemodelforfuturepandemics AT lul medicalmultimodallargelanguagemodelforfuturepandemics AT liuz medicalmultimodallargelanguagemodelforfuturepandemics AT zhengy medicalmultimodallargelanguagemodelforfuturepandemics AT sunx medicalmultimodallargelanguagemodelforfuturepandemics AT yangy medicalmultimodallargelanguagemodelforfuturepandemics AT cliftonl medicalmultimodallargelanguagemodelforfuturepandemics AT cliftonda medicalmultimodallargelanguagemodelforfuturepandemics |