DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification
Skin cancer is a dangerous form of cancer that develops slowly in skin cells. Delays in diagnosing and treating these malignant skin conditions may have serious repercussions. Likewise, early skin cancer detection has been shown to improve treatment outcomes. This paper proposes DeepMetaForge, a dee...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10366268/ |
| _version_ | 1827396116185874432 |
|---|---|
| author | Sirawich Vachmanus Thanapon Noraset Waritsara Piyanonpong Teerapong Rattananukrom Suppawong Tuarob |
| author_facet | Sirawich Vachmanus Thanapon Noraset Waritsara Piyanonpong Teerapong Rattananukrom Suppawong Tuarob |
| author_sort | Sirawich Vachmanus |
| collection | DOAJ |
| description | Skin cancer is a dangerous form of cancer that develops slowly in skin cells. Delays in diagnosing and treating these malignant skin conditions may have serious repercussions. Likewise, early skin cancer detection has been shown to improve treatment outcomes. This paper proposes DeepMetaForge, a deep-learning framework for skin cancer detection from metadata-accompanied images. The proposed framework utilizes BEiT, a vision transformer pre-trained as a masked image modeling task, as the image-encoding backbone. We further propose merging the encoded metadata with the derived visual characteristics while compressing the aggregate information simultaneously, simulating how photos with metadata are interpreted. The experiment results on four public datasets of dermoscopic and smartphone skin lesion images reveal that the best configuration of our proposed framework yields 87.1% macro-average F1 on average. The empirical scalability analysis further shows that the proposed framework can be implemented in a variety of machine-learning paradigms, including applications on low-resource devices and as services. The findings shed light on not only the possibility of implementing telemedicine solutions for skin cancer on a nationwide scale that could benefit those in need of quality healthcare but also open doors to many intelligent applications in medicine where images and metadata are collected together, such as disease detection from CT-scan images and patients’ expression recognition from facial images. |
| first_indexed | 2024-03-08T18:46:07Z |
| format | Article |
| id | doaj.art-735aef0d6fd346b994569d35d850588f |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-08T18:46:07Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-735aef0d6fd346b994569d35d850588f2023-12-29T00:03:49ZengIEEEIEEE Access2169-35362023-01-011114546714548410.1109/ACCESS.2023.334522510366268DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion ClassificationSirawich Vachmanus0https://orcid.org/0000-0002-6472-8426Thanapon Noraset1Waritsara Piyanonpong2Teerapong Rattananukrom3Suppawong Tuarob4https://orcid.org/0000-0002-5201-5699Faculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, ThailandFaculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, ThailandDivision of Dermatology, Faculty of Medicine, Ramathibodi Hospital, Mahidol University, Bangkok, ThailandDivision of Dermatology, Faculty of Medicine, Ramathibodi Hospital, Mahidol University, Bangkok, ThailandFaculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, ThailandSkin cancer is a dangerous form of cancer that develops slowly in skin cells. Delays in diagnosing and treating these malignant skin conditions may have serious repercussions. Likewise, early skin cancer detection has been shown to improve treatment outcomes. This paper proposes DeepMetaForge, a deep-learning framework for skin cancer detection from metadata-accompanied images. The proposed framework utilizes BEiT, a vision transformer pre-trained as a masked image modeling task, as the image-encoding backbone. We further propose merging the encoded metadata with the derived visual characteristics while compressing the aggregate information simultaneously, simulating how photos with metadata are interpreted. The experiment results on four public datasets of dermoscopic and smartphone skin lesion images reveal that the best configuration of our proposed framework yields 87.1% macro-average F1 on average. The empirical scalability analysis further shows that the proposed framework can be implemented in a variety of machine-learning paradigms, including applications on low-resource devices and as services. The findings shed light on not only the possibility of implementing telemedicine solutions for skin cancer on a nationwide scale that could benefit those in need of quality healthcare but also open doors to many intelligent applications in medicine where images and metadata are collected together, such as disease detection from CT-scan images and patients’ expression recognition from facial images.https://ieeexplore.ieee.org/document/10366268/Image-metadata fusiondeep learningskin lesion classification |
| spellingShingle | Sirawich Vachmanus Thanapon Noraset Waritsara Piyanonpong Teerapong Rattananukrom Suppawong Tuarob DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification IEEE Access Image-metadata fusion deep learning skin lesion classification |
| title | DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification |
| title_full | DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification |
| title_fullStr | DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification |
| title_full_unstemmed | DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification |
| title_short | DeepMetaForge: A Deep Vision-Transformer Metadata-Fusion Network for Automatic Skin Lesion Classification |
| title_sort | deepmetaforge a deep vision transformer metadata fusion network for automatic skin lesion classification |
| topic | Image-metadata fusion deep learning skin lesion classification |
| url | https://ieeexplore.ieee.org/document/10366268/ |
| work_keys_str_mv | AT sirawichvachmanus deepmetaforgeadeepvisiontransformermetadatafusionnetworkforautomaticskinlesionclassification AT thanaponnoraset deepmetaforgeadeepvisiontransformermetadatafusionnetworkforautomaticskinlesionclassification AT waritsarapiyanonpong deepmetaforgeadeepvisiontransformermetadatafusionnetworkforautomaticskinlesionclassification AT teerapongrattananukrom deepmetaforgeadeepvisiontransformermetadatafusionnetworkforautomaticskinlesionclassification AT suppawongtuarob deepmetaforgeadeepvisiontransformermetadatafusionnetworkforautomaticskinlesionclassification |