Metamorphic Testing of Relation Extraction Models
Relation extraction (RE) is a fundamental NLP task that aims to identify relations between some entities regarding a given text. RE forms the basis for many advanced NLP tasks, such as question answering and text summarization, and thus its quality is critical to the relevant downstream applications...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Algorithms |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-4893/16/2/102 |
| _version_ | 1797622857018114048 |
|---|---|
| author | Yuhe Sun Zuohua Ding Hongyun Huang Senhao Zou Mingyue Jiang |
| author_facet | Yuhe Sun Zuohua Ding Hongyun Huang Senhao Zou Mingyue Jiang |
| author_sort | Yuhe Sun |
| collection | DOAJ |
| description | Relation extraction (RE) is a fundamental NLP task that aims to identify relations between some entities regarding a given text. RE forms the basis for many advanced NLP tasks, such as question answering and text summarization, and thus its quality is critical to the relevant downstream applications. However, evaluating the quality of RE models is non-trivial. On the one hand, obtaining ground truth labels for individual test inputs is tedious and even difficult. On the other hand, there is an increasing need to understand the characteristics of RE models in terms of various aspects. To mitigate these issues, this study proposes evaluating RE models by applying metamorphic testing (MT). A total of eight metamorphic relations (MRs) are identified based on three categories of transformation operations, namely replacement, swap, and combination. These MRs encode some expected properties of different aspects of RE. We further apply MT to three popular RE models. Our experiments reveal a large number of prediction failures in the subject RE models, confirming that MT is effective for evaluating RE models. Further analysis of the experimental results reveals the advantages and disadvantages of our subject models and also uncovers some typical issues of RE models. |
| first_indexed | 2024-03-11T09:16:11Z |
| format | Article |
| id | doaj.art-9aad79944dff444897ae2dcbb82f0516 |
| institution | Directory Open Access Journal |
| issn | 1999-4893 |
| language | English |
| last_indexed | 2024-03-11T09:16:11Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Algorithms |
| spelling | doaj.art-9aad79944dff444897ae2dcbb82f05162023-11-16T18:37:47ZengMDPI AGAlgorithms1999-48932023-02-0116210210.3390/a16020102Metamorphic Testing of Relation Extraction ModelsYuhe Sun0Zuohua Ding1Hongyun Huang2Senhao Zou3Mingyue Jiang4School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, ChinaSchool of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, ChinaLibrary Multimedia Big Data Center, Zhejiang Sci-Tech University, Hangzhou 310018, ChinaSchool of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, ChinaSchool of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, ChinaRelation extraction (RE) is a fundamental NLP task that aims to identify relations between some entities regarding a given text. RE forms the basis for many advanced NLP tasks, such as question answering and text summarization, and thus its quality is critical to the relevant downstream applications. However, evaluating the quality of RE models is non-trivial. On the one hand, obtaining ground truth labels for individual test inputs is tedious and even difficult. On the other hand, there is an increasing need to understand the characteristics of RE models in terms of various aspects. To mitigate these issues, this study proposes evaluating RE models by applying metamorphic testing (MT). A total of eight metamorphic relations (MRs) are identified based on three categories of transformation operations, namely replacement, swap, and combination. These MRs encode some expected properties of different aspects of RE. We further apply MT to three popular RE models. Our experiments reveal a large number of prediction failures in the subject RE models, confirming that MT is effective for evaluating RE models. Further analysis of the experimental results reveals the advantages and disadvantages of our subject models and also uncovers some typical issues of RE models.https://www.mdpi.com/1999-4893/16/2/102relation extractionmetamorphic testingmetamorphic relationquality evaluationtesting and validation |
| spellingShingle | Yuhe Sun Zuohua Ding Hongyun Huang Senhao Zou Mingyue Jiang Metamorphic Testing of Relation Extraction Models Algorithms relation extraction metamorphic testing metamorphic relation quality evaluation testing and validation |
| title | Metamorphic Testing of Relation Extraction Models |
| title_full | Metamorphic Testing of Relation Extraction Models |
| title_fullStr | Metamorphic Testing of Relation Extraction Models |
| title_full_unstemmed | Metamorphic Testing of Relation Extraction Models |
| title_short | Metamorphic Testing of Relation Extraction Models |
| title_sort | metamorphic testing of relation extraction models |
| topic | relation extraction metamorphic testing metamorphic relation quality evaluation testing and validation |
| url | https://www.mdpi.com/1999-4893/16/2/102 |
| work_keys_str_mv | AT yuhesun metamorphictestingofrelationextractionmodels AT zuohuading metamorphictestingofrelationextractionmodels AT hongyunhuang metamorphictestingofrelationextractionmodels AT senhaozou metamorphictestingofrelationextractionmodels AT mingyuejiang metamorphictestingofrelationextractionmodels |