Label-aware debiased causal reasoning for Natural Language Inference
Recently, researchers have argued that the impressive performance of Natural Language Inference (NLI) models is highly due to the spurious correlations existing in training data, which makes models vulnerable and poorly generalized. Some work has made preliminary debiased attempts by developing data...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KeAi Communications Co. Ltd.
2024-01-01
|
| Series: | AI Open |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666651024000081 |
| _version_ | 1797259677297278976 |
|---|---|
| author | Kun Zhang Dacao Zhang Le Wu Richang Hong Ye Zhao Meng Wang |
| author_facet | Kun Zhang Dacao Zhang Le Wu Richang Hong Ye Zhao Meng Wang |
| author_sort | Kun Zhang |
| collection | DOAJ |
| description | Recently, researchers have argued that the impressive performance of Natural Language Inference (NLI) models is highly due to the spurious correlations existing in training data, which makes models vulnerable and poorly generalized. Some work has made preliminary debiased attempts by developing data-driven interventions or model-level debiased learning. Despite the progress, existing debiased methods either suffered from the high cost of data annotation processing, or required elaborate design to identify biased factors. By conducting detailed investigations and data analysis, we argue that label information can provide meaningful guidance to identify these spurious correlations in training data, which has not been paid enough attention. Thus, we design a novel Label-aware Debiased Causal Reasoning Network (LDCRN). Specifically, according to the data analysis, we first build a causal graph to describe causal relations and spurious correlations in NLI. Then, we employ an NLI model (e.g., RoBERTa) to calculate total causal effect of input sentences to labels. Meanwhile, we design a novel label-aware biased module to model spurious correlations and calculate their causal effect in a fine-grained manner. The debiasing process is realized by subtracting this causal effect from total causal effect. Finally, extensive experiments over two well-known NLI datasets and multiple human-annotated challenging test sets are conducted to prove the superiority of LDCRN. Moreover, we have developed novel challenging test sets based on MultiNLI to facilitate the community. |
| first_indexed | 2024-04-24T23:13:14Z |
| format | Article |
| id | doaj.art-f9bedbe8aa6244498a31981c76d79b16 |
| institution | Directory Open Access Journal |
| issn | 2666-6510 |
| language | English |
| last_indexed | 2024-04-24T23:13:14Z |
| publishDate | 2024-01-01 |
| publisher | KeAi Communications Co. Ltd. |
| record_format | Article |
| series | AI Open |
| spelling | doaj.art-f9bedbe8aa6244498a31981c76d79b162024-03-17T07:58:51ZengKeAi Communications Co. Ltd.AI Open2666-65102024-01-0157078Label-aware debiased causal reasoning for Natural Language InferenceKun Zhang0Dacao Zhang1Le Wu2Richang Hong3Ye Zhao4Meng Wang5School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China; Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, 230601, Anhui, China; Corresponding author at: School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China.School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China; Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, 230601, Anhui, ChinaSchool of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China; Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, 230601, Anhui, China; Institute of Dataspace, Hefei Comprehensive National Science Center, Hefei, 230009, Anhui, ChinaSchool of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China; Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, 230601, Anhui, China; Institute of Dataspace, Hefei Comprehensive National Science Center, Hefei, 230009, Anhui, ChinaSchool of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China; Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, 230601, Anhui, ChinaSchool of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, Anhui, China; Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, 230601, Anhui, ChinaRecently, researchers have argued that the impressive performance of Natural Language Inference (NLI) models is highly due to the spurious correlations existing in training data, which makes models vulnerable and poorly generalized. Some work has made preliminary debiased attempts by developing data-driven interventions or model-level debiased learning. Despite the progress, existing debiased methods either suffered from the high cost of data annotation processing, or required elaborate design to identify biased factors. By conducting detailed investigations and data analysis, we argue that label information can provide meaningful guidance to identify these spurious correlations in training data, which has not been paid enough attention. Thus, we design a novel Label-aware Debiased Causal Reasoning Network (LDCRN). Specifically, according to the data analysis, we first build a causal graph to describe causal relations and spurious correlations in NLI. Then, we employ an NLI model (e.g., RoBERTa) to calculate total causal effect of input sentences to labels. Meanwhile, we design a novel label-aware biased module to model spurious correlations and calculate their causal effect in a fine-grained manner. The debiasing process is realized by subtracting this causal effect from total causal effect. Finally, extensive experiments over two well-known NLI datasets and multiple human-annotated challenging test sets are conducted to prove the superiority of LDCRN. Moreover, we have developed novel challenging test sets based on MultiNLI to facilitate the community.http://www.sciencedirect.com/science/article/pii/S2666651024000081Natural language inferenceSpurious correlationsDebiased reasoningCausal effect |
| spellingShingle | Kun Zhang Dacao Zhang Le Wu Richang Hong Ye Zhao Meng Wang Label-aware debiased causal reasoning for Natural Language Inference AI Open Natural language inference Spurious correlations Debiased reasoning Causal effect |
| title | Label-aware debiased causal reasoning for Natural Language Inference |
| title_full | Label-aware debiased causal reasoning for Natural Language Inference |
| title_fullStr | Label-aware debiased causal reasoning for Natural Language Inference |
| title_full_unstemmed | Label-aware debiased causal reasoning for Natural Language Inference |
| title_short | Label-aware debiased causal reasoning for Natural Language Inference |
| title_sort | label aware debiased causal reasoning for natural language inference |
| topic | Natural language inference Spurious correlations Debiased reasoning Causal effect |
| url | http://www.sciencedirect.com/science/article/pii/S2666651024000081 |
| work_keys_str_mv | AT kunzhang labelawaredebiasedcausalreasoningfornaturallanguageinference AT dacaozhang labelawaredebiasedcausalreasoningfornaturallanguageinference AT lewu labelawaredebiasedcausalreasoningfornaturallanguageinference AT richanghong labelawaredebiasedcausalreasoningfornaturallanguageinference AT yezhao labelawaredebiasedcausalreasoningfornaturallanguageinference AT mengwang labelawaredebiasedcausalreasoningfornaturallanguageinference |