Sentence Compression Using BERT and Graph Convolutional Networks
Sentence compression is a natural language-processing task that produces a short paraphrase of an input sentence by deleting words from the input sentence while ensuring grammatical correctness and preserving meaningful core information. This study introduces a graph convolutional network (GCN) into...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-10-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/11/21/9910 |
| _version_ | 1797512893105700864 |
|---|---|
| author | Yo-Han Park Gyong-Ho Lee Yong-Seok Choi Kong-Joo Lee |
| author_facet | Yo-Han Park Gyong-Ho Lee Yong-Seok Choi Kong-Joo Lee |
| author_sort | Yo-Han Park |
| collection | DOAJ |
| description | Sentence compression is a natural language-processing task that produces a short paraphrase of an input sentence by deleting words from the input sentence while ensuring grammatical correctness and preserving meaningful core information. This study introduces a graph convolutional network (GCN) into a sentence compression task to encode syntactic information, such as dependency trees. As we upgrade the GCN to activate a directed edge, the compression model with the GCN layers can distinguish between parent and child nodes in a dependency tree when aggregating adjacent nodes. Furthermore, by increasing the number of GCN layers, the model can gradually collect high-order information of a dependency tree when propagating node information through the layers. We implement a sentence compression model for Korean and English, respectively. This model consists of three components: pre-trained BERT model, GCN layers, and a scoring layer. The scoring layer can determine whether a word should remain in a compressed sentence by relying on the word vector containing contextual and syntactic information encoded by BERT and GCN layers. To train and evaluate the proposed model, we used the Google sentence compression dataset for English and a Korean sentence compression corpus containing about 140,000 sentence pairs for Korean. The experimental results demonstrate that the proposed model achieves state-of-the-art performance for English. To the best of our knowledge, this sentence compression model based on the deep learning model trained with a large-scale corpus is the first attempt for Korean. |
| first_indexed | 2024-03-10T06:07:33Z |
| format | Article |
| id | doaj.art-f662c617032640ec80e68ff4dde1c624 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T06:07:33Z |
| publishDate | 2021-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-f662c617032640ec80e68ff4dde1c6242023-11-22T20:24:58ZengMDPI AGApplied Sciences2076-34172021-10-011121991010.3390/app11219910Sentence Compression Using BERT and Graph Convolutional NetworksYo-Han Park0Gyong-Ho Lee1Yong-Seok Choi2Kong-Joo Lee3Department of Radio and Information Communications Engineering, ChungNam National University, 99 Daehak-ro, Yuseong-gu, Daejeon 34134, KoreaAI Laboratory, Drama & Company, Seoul 06158, KoreaDepartment of Radio and Information Communications Engineering, ChungNam National University, 99 Daehak-ro, Yuseong-gu, Daejeon 34134, KoreaDepartment of Radio and Information Communications Engineering, ChungNam National University, 99 Daehak-ro, Yuseong-gu, Daejeon 34134, KoreaSentence compression is a natural language-processing task that produces a short paraphrase of an input sentence by deleting words from the input sentence while ensuring grammatical correctness and preserving meaningful core information. This study introduces a graph convolutional network (GCN) into a sentence compression task to encode syntactic information, such as dependency trees. As we upgrade the GCN to activate a directed edge, the compression model with the GCN layers can distinguish between parent and child nodes in a dependency tree when aggregating adjacent nodes. Furthermore, by increasing the number of GCN layers, the model can gradually collect high-order information of a dependency tree when propagating node information through the layers. We implement a sentence compression model for Korean and English, respectively. This model consists of three components: pre-trained BERT model, GCN layers, and a scoring layer. The scoring layer can determine whether a word should remain in a compressed sentence by relying on the word vector containing contextual and syntactic information encoded by BERT and GCN layers. To train and evaluate the proposed model, we used the Google sentence compression dataset for English and a Korean sentence compression corpus containing about 140,000 sentence pairs for Korean. The experimental results demonstrate that the proposed model achieves state-of-the-art performance for English. To the best of our knowledge, this sentence compression model based on the deep learning model trained with a large-scale corpus is the first attempt for Korean.https://www.mdpi.com/2076-3417/11/21/9910dependency treegraph convolutional networkgraph neural networkspre-trained modelsentence compression |
| spellingShingle | Yo-Han Park Gyong-Ho Lee Yong-Seok Choi Kong-Joo Lee Sentence Compression Using BERT and Graph Convolutional Networks Applied Sciences dependency tree graph convolutional network graph neural networks pre-trained model sentence compression |
| title | Sentence Compression Using BERT and Graph Convolutional Networks |
| title_full | Sentence Compression Using BERT and Graph Convolutional Networks |
| title_fullStr | Sentence Compression Using BERT and Graph Convolutional Networks |
| title_full_unstemmed | Sentence Compression Using BERT and Graph Convolutional Networks |
| title_short | Sentence Compression Using BERT and Graph Convolutional Networks |
| title_sort | sentence compression using bert and graph convolutional networks |
| topic | dependency tree graph convolutional network graph neural networks pre-trained model sentence compression |
| url | https://www.mdpi.com/2076-3417/11/21/9910 |
| work_keys_str_mv | AT yohanpark sentencecompressionusingbertandgraphconvolutionalnetworks AT gyongholee sentencecompressionusingbertandgraphconvolutionalnetworks AT yongseokchoi sentencecompressionusingbertandgraphconvolutionalnetworks AT kongjoolee sentencecompressionusingbertandgraphconvolutionalnetworks |