A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification
The multi-label text classification task aims to tag a document with a series of labels. Previous studies usually treated labels as symbols without semantics and ignored the relation among labels, which caused information loss. In this paper, we show that explicitly modeling label semantics can impr...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9169885/ |
| _version_ | 1818407785950347264 |
|---|---|
| author | Linkun Cai Yu Song Tao Liu Kunli Zhang |
| author_facet | Linkun Cai Yu Song Tao Liu Kunli Zhang |
| author_sort | Linkun Cai |
| collection | DOAJ |
| description | The multi-label text classification task aims to tag a document with a series of labels. Previous studies usually treated labels as symbols without semantics and ignored the relation among labels, which caused information loss. In this paper, we show that explicitly modeling label semantics can improve multi-label text classification. We propose a hybrid neural network model to simultaneously take advantage of both label semantics and fine-grained text information. Specifically, we utilize the pre-trained BERT model to compute context-aware representation of documents. Furthermore, we incorporate the label semantics in two stages. First, a novel label graph construction approach is proposed to capture the label structures and correlations. Second, we propose a neoteric attention mechanism-adjustive attention to establish the semantic connections between labels and words and to obtain the label-specific word representation. The hybrid representation that combines context-aware feature and label-special word feature is fed into a document encoder to classify. Experimental results on two publicly available datasets show that our model is superior to other state-of-the-art classification methods. |
| first_indexed | 2024-12-14T09:33:22Z |
| format | Article |
| id | doaj.art-5c048166d8834bc388cf9048f92edc22 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-14T09:33:22Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-5c048166d8834bc388cf9048f92edc222022-12-21T23:08:00ZengIEEEIEEE Access2169-35362020-01-01815218315219210.1109/ACCESS.2020.30173829169885A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text ClassificationLinkun Cai0https://orcid.org/0000-0002-9728-8164Yu Song1Tao Liu2https://orcid.org/0000-0003-1873-5462Kunli Zhang3School of Information and Engineering, Zhengzhou University, Zhengzhou, ChinaSchool of Information and Engineering, Zhengzhou University, Zhengzhou, ChinaSchool of Information and Engineering, Zhengzhou University, Zhengzhou, ChinaSchool of Information and Engineering, Zhengzhou University, Zhengzhou, ChinaThe multi-label text classification task aims to tag a document with a series of labels. Previous studies usually treated labels as symbols without semantics and ignored the relation among labels, which caused information loss. In this paper, we show that explicitly modeling label semantics can improve multi-label text classification. We propose a hybrid neural network model to simultaneously take advantage of both label semantics and fine-grained text information. Specifically, we utilize the pre-trained BERT model to compute context-aware representation of documents. Furthermore, we incorporate the label semantics in two stages. First, a novel label graph construction approach is proposed to capture the label structures and correlations. Second, we propose a neoteric attention mechanism-adjustive attention to establish the semantic connections between labels and words and to obtain the label-specific word representation. The hybrid representation that combines context-aware feature and label-special word feature is fed into a document encoder to classify. Experimental results on two publicly available datasets show that our model is superior to other state-of-the-art classification methods.https://ieeexplore.ieee.org/document/9169885/Multi-label text classificationlabel embeddingBERTattention mechanism |
| spellingShingle | Linkun Cai Yu Song Tao Liu Kunli Zhang A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification IEEE Access Multi-label text classification label embedding BERT attention mechanism |
| title | A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification |
| title_full | A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification |
| title_fullStr | A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification |
| title_full_unstemmed | A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification |
| title_short | A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification |
| title_sort | hybrid bert model that incorporates label semantics via adjustive attention for multi label text classification |
| topic | Multi-label text classification label embedding BERT attention mechanism |
| url | https://ieeexplore.ieee.org/document/9169885/ |
| work_keys_str_mv | AT linkuncai ahybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT yusong ahybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT taoliu ahybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT kunlizhang ahybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT linkuncai hybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT yusong hybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT taoliu hybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification AT kunlizhang hybridbertmodelthatincorporateslabelsemanticsviaadjustiveattentionformultilabeltextclassification |