Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification
Traditional methods of multi-label text classification, particularly deep learning, have achieved remarkable results. However, most of these methods use word2vec technology to represent sequential text information, while ignoring the logic and internal hierarchy of the text itself. Although these ap...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8988213/ |
| _version_ | 1819182035895320576 |
|---|---|
| author | Jibing Gong Zhiyong Teng Qi Teng Hekai Zhang Linfeng Du Shuai Chen Md Zakirul Alam Bhuiyan Jianhua Li Mingsheng Liu Hongyuan Ma |
| author_facet | Jibing Gong Zhiyong Teng Qi Teng Hekai Zhang Linfeng Du Shuai Chen Md Zakirul Alam Bhuiyan Jianhua Li Mingsheng Liu Hongyuan Ma |
| author_sort | Jibing Gong |
| collection | DOAJ |
| description | Traditional methods of multi-label text classification, particularly deep learning, have achieved remarkable results. However, most of these methods use word2vec technology to represent sequential text information, while ignoring the logic and internal hierarchy of the text itself. Although these approaches can learn the hypothetical hierarchy and logic of the text, it is unexplained. In addition, the traditional approach treats labels as independent individuals and ignores the relationships between them, which not only does not reflect reality but also causes significant loss of semantic information. In this paper, we propose a novel Hierarchical Graph Transformer based deep learning model for large-scale multi-label text classification. We first model the text into a graph structure that can embody the different semantics of the text and the connections between them. We then use a multi-layer transformer structure with a multi-head attention mechanism at the word, sentence, and graph levels to fully capture the features of the text and observe the importance of the separate parts. Finally, we use the hierarchical relationship of the labels to generate the representation of the labels, and design a weighted loss function based on the semantic distances of the labels. Extensive experiments conducted on three benchmark datasets demonstrated that the proposed model can realistically capture the hierarchy and logic of text and improve performance compared with the state-of-the-art methods. |
| first_indexed | 2024-12-22T22:39:44Z |
| format | Article |
| id | doaj.art-d172022294864d1b8ccde55ba9c09846 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T22:39:44Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-d172022294864d1b8ccde55ba9c098462022-12-21T18:10:14ZengIEEEIEEE Access2169-35362020-01-018308853089610.1109/ACCESS.2020.29727518988213Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text ClassificationJibing Gong0https://orcid.org/0000-0003-4449-5845Zhiyong Teng1https://orcid.org/0000-0002-4761-7724Qi Teng2Hekai Zhang3Linfeng Du4https://orcid.org/0000-0002-4389-5575Shuai Chen5https://orcid.org/0000-0003-1753-0115Md Zakirul Alam Bhuiyan6Jianhua Li7https://orcid.org/0000-0003-1025-7910Mingsheng Liu8Hongyuan Ma9School of Information Science and Engineering, Yanshan University, Qinhuangdao, ChinaSchool of Information Science and Engineering, Yanshan University, Qinhuangdao, ChinaSchool of Computer Science and Engineering, Beihang University, Beijing, ChinaSchool of Information Science and Engineering, Yanshan University, Qinhuangdao, ChinaSchool of Computer Science and Engineering, Beihang University, Beijing, ChinaSchool of Information Science and Engineering, Yanshan University, Qinhuangdao, ChinaDepartment of Computer and Information Sciences, Fordham University, New York, NY, USASchool of Information Science and Technology, Shijiazhuang Tiedao University, Shijiazhuang, ChinaCollege of Electrical Engineering, Hebei University of Technology, Tianjin, ChinaNational Computer Network Emergency Response Technical Team/Coordination Center of China, Beijing, ChinaTraditional methods of multi-label text classification, particularly deep learning, have achieved remarkable results. However, most of these methods use word2vec technology to represent sequential text information, while ignoring the logic and internal hierarchy of the text itself. Although these approaches can learn the hypothetical hierarchy and logic of the text, it is unexplained. In addition, the traditional approach treats labels as independent individuals and ignores the relationships between them, which not only does not reflect reality but also causes significant loss of semantic information. In this paper, we propose a novel Hierarchical Graph Transformer based deep learning model for large-scale multi-label text classification. We first model the text into a graph structure that can embody the different semantics of the text and the connections between them. We then use a multi-layer transformer structure with a multi-head attention mechanism at the word, sentence, and graph levels to fully capture the features of the text and observe the importance of the separate parts. Finally, we use the hierarchical relationship of the labels to generate the representation of the labels, and design a weighted loss function based on the semantic distances of the labels. Extensive experiments conducted on three benchmark datasets demonstrated that the proposed model can realistically capture the hierarchy and logic of text and improve performance compared with the state-of-the-art methods.https://ieeexplore.ieee.org/document/8988213/Multi-label text classificationgraph modelinggraph transformerdeep learning |
| spellingShingle | Jibing Gong Zhiyong Teng Qi Teng Hekai Zhang Linfeng Du Shuai Chen Md Zakirul Alam Bhuiyan Jianhua Li Mingsheng Liu Hongyuan Ma Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification IEEE Access Multi-label text classification graph modeling graph transformer deep learning |
| title | Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification |
| title_full | Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification |
| title_fullStr | Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification |
| title_full_unstemmed | Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification |
| title_short | Hierarchical Graph Transformer-Based Deep Learning Model for Large-Scale Multi-Label Text Classification |
| title_sort | hierarchical graph transformer based deep learning model for large scale multi label text classification |
| topic | Multi-label text classification graph modeling graph transformer deep learning |
| url | https://ieeexplore.ieee.org/document/8988213/ |
| work_keys_str_mv | AT jibinggong hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT zhiyongteng hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT qiteng hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT hekaizhang hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT linfengdu hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT shuaichen hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT mdzakirulalambhuiyan hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT jianhuali hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT mingshengliu hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification AT hongyuanma hierarchicalgraphtransformerbaseddeeplearningmodelforlargescalemultilabeltextclassification |