Probing the Impacts of Visual Context in Multimodal Entity Alignment
Abstract We study the problem of multimodal embedding-based entity alignment (EA) between different knowledge graphs. Recent works have attempted to incorporate images (visual context) to address EA in a multimodal view. While the benefits of multimodal information have been observed, its negative i...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-04-01
|
| Series: | Data Science and Engineering |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s41019-023-00208-9 |
| _version_ | 1797822732111446016 |
|---|---|
| author | Meng Wang Yinghui Shi Han Yang Ziheng Zhang Zhenxi Lin Yefeng Zheng |
| author_facet | Meng Wang Yinghui Shi Han Yang Ziheng Zhang Zhenxi Lin Yefeng Zheng |
| author_sort | Meng Wang |
| collection | DOAJ |
| description | Abstract We study the problem of multimodal embedding-based entity alignment (EA) between different knowledge graphs. Recent works have attempted to incorporate images (visual context) to address EA in a multimodal view. While the benefits of multimodal information have been observed, its negative impacts are non-negligible as injecting images without constraints brings much noise. It also remains unknown under what circumstances or to what extent visual context is truly helpful to the task. In this work, we propose to learn entity representations from graph structures and visual context, and combine feature similarities to find alignments at the output level. On top of this, we explore a mechanism which utilizes classification techniques and entity types to remove potentially un-helpful images (visual noises) during alignment learning and inference. We conduct extensive experiments to examine this mechanism and provide thorough analysis about impacts of the visual modality on EA. |
| first_indexed | 2024-03-13T10:13:30Z |
| format | Article |
| id | doaj.art-510093c75cc349888d059ab84bcbe647 |
| institution | Directory Open Access Journal |
| issn | 2364-1185 2364-1541 |
| language | English |
| last_indexed | 2024-03-13T10:13:30Z |
| publishDate | 2023-04-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Data Science and Engineering |
| spelling | doaj.art-510093c75cc349888d059ab84bcbe6472023-05-21T11:22:27ZengSpringerOpenData Science and Engineering2364-11852364-15412023-04-018212413410.1007/s41019-023-00208-9Probing the Impacts of Visual Context in Multimodal Entity AlignmentMeng Wang0Yinghui Shi1Han Yang2Ziheng Zhang3Zhenxi Lin4Yefeng Zheng5College of Design and Innovation, Tongji UniversitySchool of Cyber Science and Engineering, Southeast UniversityZEEKR Intelligent Technology Holding Ltd.Tencent Jarvis LabTencent Jarvis LabTencent Jarvis LabAbstract We study the problem of multimodal embedding-based entity alignment (EA) between different knowledge graphs. Recent works have attempted to incorporate images (visual context) to address EA in a multimodal view. While the benefits of multimodal information have been observed, its negative impacts are non-negligible as injecting images without constraints brings much noise. It also remains unknown under what circumstances or to what extent visual context is truly helpful to the task. In this work, we propose to learn entity representations from graph structures and visual context, and combine feature similarities to find alignments at the output level. On top of this, we explore a mechanism which utilizes classification techniques and entity types to remove potentially un-helpful images (visual noises) during alignment learning and inference. We conduct extensive experiments to examine this mechanism and provide thorough analysis about impacts of the visual modality on EA.https://doi.org/10.1007/s41019-023-00208-9Entity alignmentMultimodalityVisual contextKnowledge graph |
| spellingShingle | Meng Wang Yinghui Shi Han Yang Ziheng Zhang Zhenxi Lin Yefeng Zheng Probing the Impacts of Visual Context in Multimodal Entity Alignment Data Science and Engineering Entity alignment Multimodality Visual context Knowledge graph |
| title | Probing the Impacts of Visual Context in Multimodal Entity Alignment |
| title_full | Probing the Impacts of Visual Context in Multimodal Entity Alignment |
| title_fullStr | Probing the Impacts of Visual Context in Multimodal Entity Alignment |
| title_full_unstemmed | Probing the Impacts of Visual Context in Multimodal Entity Alignment |
| title_short | Probing the Impacts of Visual Context in Multimodal Entity Alignment |
| title_sort | probing the impacts of visual context in multimodal entity alignment |
| topic | Entity alignment Multimodality Visual context Knowledge graph |
| url | https://doi.org/10.1007/s41019-023-00208-9 |
| work_keys_str_mv | AT mengwang probingtheimpactsofvisualcontextinmultimodalentityalignment AT yinghuishi probingtheimpactsofvisualcontextinmultimodalentityalignment AT hanyang probingtheimpactsofvisualcontextinmultimodalentityalignment AT zihengzhang probingtheimpactsofvisualcontextinmultimodalentityalignment AT zhenxilin probingtheimpactsofvisualcontextinmultimodalentityalignment AT yefengzheng probingtheimpactsofvisualcontextinmultimodalentityalignment |