A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence
Abstract Media convergence is a media change led by technological innovation. Applying media convergence technology to the study of clustering in Chinese medicine can significantly exploit the advantages of media fusion. Obtaining consistent and complementary information among multiple modalities th...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2023-06-01
|
| Series: | CAAI Transactions on Intelligence Technology |
| Subjects: | |
| Online Access: | https://doi.org/10.1049/cit2.12230 |
| _version_ | 1797641480004698112 |
|---|---|
| author | Jingna Si Ziwei Tian Dongmei Li Lei Zhang Lei Yao Wenjuan Jiang Jia Liu Runshun Zhang Xiaoping Zhang |
| author_facet | Jingna Si Ziwei Tian Dongmei Li Lei Zhang Lei Yao Wenjuan Jiang Jia Liu Runshun Zhang Xiaoping Zhang |
| author_sort | Jingna Si |
| collection | DOAJ |
| description | Abstract Media convergence is a media change led by technological innovation. Applying media convergence technology to the study of clustering in Chinese medicine can significantly exploit the advantages of media fusion. Obtaining consistent and complementary information among multiple modalities through media convergence can provide technical support for clustering. This article presents an approach based on Media Convergence and Graph convolution Encoder Clustering (MCGEC) for traditional Chinese medicine (TCM) clinical data. It feeds modal information and graph structure from media information into a multi‐modal graph convolution encoder to obtain the media feature representation learnt from multiple modalities. MCGEC captures latent information from various modalities by fusion and optimises the feature representations and network architecture with learnt clustering labels. The experiment is conducted on real‐world multi‐modal TCM clinical data, including information like images and text. MCGEC has improved clustering results compared to the generic single‐modal clustering methods and the current more advanced multi‐modal clustering methods. MCGEC applied to TCM clinical datasets can achieve better results. Integrating multimedia features into clustering algorithms offers significant benefits compared to single‐modal clustering approaches that simply concatenate features from different modalities. It provides practical technical support for multi‐modal clustering in the TCM field incorporating multimedia features. |
| first_indexed | 2024-03-11T13:46:11Z |
| format | Article |
| id | doaj.art-4fd34d5008cf41d4ad904868f5ccc147 |
| institution | Directory Open Access Journal |
| issn | 2468-2322 |
| language | English |
| last_indexed | 2024-03-11T13:46:11Z |
| publishDate | 2023-06-01 |
| publisher | Wiley |
| record_format | Article |
| series | CAAI Transactions on Intelligence Technology |
| spelling | doaj.art-4fd34d5008cf41d4ad904868f5ccc1472023-11-02T10:21:18ZengWileyCAAI Transactions on Intelligence Technology2468-23222023-06-018239040010.1049/cit2.12230A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergenceJingna Si0Ziwei Tian1Dongmei Li2Lei Zhang3Lei Yao4Wenjuan Jiang5Jia Liu6Runshun Zhang7Xiaoping Zhang8State Key Laboratory of Tree Genetics and Breeding College of Biological Sciences and Technology Beijing Forestry University Beijing ChinaSchool of Information Science and Technology Beijing Forestry University Beijing ChinaSchool of Information Science and Technology Beijing Forestry University Beijing ChinaNational Data Center of Traditional Chinese Medicine China Academy of Chinese Medical Sciences Beijing ChinaUniversity of Wisconsin‐Milwaukee Milwaukee Wisconsin USASchool of Information Science and Technology Beijing Forestry University Beijing ChinaExperimental Research Center China Academy of Chinese Medical Sciences Beijing ChinaGuang'anmen Hospital China Academy of Chinese Medical Sciences Beijing ChinaNational Data Center of Traditional Chinese Medicine China Academy of Chinese Medical Sciences Beijing ChinaAbstract Media convergence is a media change led by technological innovation. Applying media convergence technology to the study of clustering in Chinese medicine can significantly exploit the advantages of media fusion. Obtaining consistent and complementary information among multiple modalities through media convergence can provide technical support for clustering. This article presents an approach based on Media Convergence and Graph convolution Encoder Clustering (MCGEC) for traditional Chinese medicine (TCM) clinical data. It feeds modal information and graph structure from media information into a multi‐modal graph convolution encoder to obtain the media feature representation learnt from multiple modalities. MCGEC captures latent information from various modalities by fusion and optimises the feature representations and network architecture with learnt clustering labels. The experiment is conducted on real‐world multi‐modal TCM clinical data, including information like images and text. MCGEC has improved clustering results compared to the generic single‐modal clustering methods and the current more advanced multi‐modal clustering methods. MCGEC applied to TCM clinical datasets can achieve better results. Integrating multimedia features into clustering algorithms offers significant benefits compared to single‐modal clustering approaches that simply concatenate features from different modalities. It provides practical technical support for multi‐modal clustering in the TCM field incorporating multimedia features.https://doi.org/10.1049/cit2.12230graph convolutional encodermedia convergencemulti‐modal clusteringtraditional Chinese medicine |
| spellingShingle | Jingna Si Ziwei Tian Dongmei Li Lei Zhang Lei Yao Wenjuan Jiang Jia Liu Runshun Zhang Xiaoping Zhang A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence CAAI Transactions on Intelligence Technology graph convolutional encoder media convergence multi‐modal clustering traditional Chinese medicine |
| title | A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence |
| title_full | A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence |
| title_fullStr | A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence |
| title_full_unstemmed | A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence |
| title_short | A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence |
| title_sort | multi modal clustering method for traditional chinese medicine clinical data via media convergence |
| topic | graph convolutional encoder media convergence multi‐modal clustering traditional Chinese medicine |
| url | https://doi.org/10.1049/cit2.12230 |
| work_keys_str_mv | AT jingnasi amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT ziweitian amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT dongmeili amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT leizhang amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT leiyao amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT wenjuanjiang amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT jialiu amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT runshunzhang amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT xiaopingzhang amultimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT jingnasi multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT ziweitian multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT dongmeili multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT leizhang multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT leiyao multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT wenjuanjiang multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT jialiu multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT runshunzhang multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence AT xiaopingzhang multimodalclusteringmethodfortraditionalchinesemedicineclinicaldataviamediaconvergence |