Fairness testing of machine translation systems
Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach t...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/180446 |
| _version_ | 1824456270269120512 |
|---|---|
| author | Sun,Zeyu Chen, Zhenpeng Zhang, Jie Hao, Dan |
| author2 | College of Computing and Data Science |
| author_facet | College of Computing and Data Science Sun,Zeyu Chen, Zhenpeng Zhang, Jie Hao, Dan |
| author_sort | Sun,Zeyu |
| collection | NTU |
| description | Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach tailored for machine translation systems. FairMT operates on the assumption that translations of semantically similar sentences, containing protected attributes from distinct demographic groups, should maintain comparable meanings. It comprises three key steps: (1) test input generation, producing inputs covering various demographic groups; (2) test oracle generation, identifying potential unfair translations based on semantic similarity measurements; and (3) regression, discerning genuine fairness issues from those caused by low-quality translation. Leveraging FairMT, we conduct an empirical study on three leading machine translation systems-Google Translate, T5, and Transformer. Our investigation uncovers up to 832, 1,984, and 2,627 unfair translations across the three systems, respectively. Intriguingly, we observe that fair translations tend to exhibit superior translation performance, challenging the conventional wisdom of a fairness-performance tradeoff prevalent in the fairness literature. |
| first_indexed | 2025-02-19T03:51:26Z |
| format | Journal Article |
| id | ntu-10356/180446 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:51:26Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1804462024-10-07T08:38:08Z Fairness testing of machine translation systems Sun,Zeyu Chen, Zhenpeng Zhang, Jie Hao, Dan College of Computing and Data Science Computer and Information Science Fairness testing Metamorphic testing Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach tailored for machine translation systems. FairMT operates on the assumption that translations of semantically similar sentences, containing protected attributes from distinct demographic groups, should maintain comparable meanings. It comprises three key steps: (1) test input generation, producing inputs covering various demographic groups; (2) test oracle generation, identifying potential unfair translations based on semantic similarity measurements; and (3) regression, discerning genuine fairness issues from those caused by low-quality translation. Leveraging FairMT, we conduct an empirical study on three leading machine translation systems-Google Translate, T5, and Transformer. Our investigation uncovers up to 832, 1,984, and 2,627 unfair translations across the three systems, respectively. Intriguingly, we observe that fair translations tend to exhibit superior translation performance, challenging the conventional wisdom of a fairness-performance tradeoff prevalent in the fairness literature. Published version This work was supported by National Natural Science Foundation of China under Grant No. 62372005. 2024-10-07T08:38:08Z 2024-10-07T08:38:08Z 2024 Journal Article Sun, Z., Chen, Z., Zhang, J. & Hao, D. (2024). Fairness testing of machine translation systems. ACM Transactions On Software Engineering and Methodology, 33(6), 156-. https://dx.doi.org/10.1145/3664608 1049-331X https://hdl.handle.net/10356/180446 10.1145/3664608 2-s2.0-85198860702 6 33 156 en ACM Transactions on Software Engineering and Methodology © 2024 Copyright held by the owner/author(s). This work is licensed under a Creative Commons Attribution International 4.0 License. application/pdf |
| spellingShingle | Computer and Information Science Fairness testing Metamorphic testing Sun,Zeyu Chen, Zhenpeng Zhang, Jie Hao, Dan Fairness testing of machine translation systems |
| title | Fairness testing of machine translation systems |
| title_full | Fairness testing of machine translation systems |
| title_fullStr | Fairness testing of machine translation systems |
| title_full_unstemmed | Fairness testing of machine translation systems |
| title_short | Fairness testing of machine translation systems |
| title_sort | fairness testing of machine translation systems |
| topic | Computer and Information Science Fairness testing Metamorphic testing |
| url | https://hdl.handle.net/10356/180446 |
| work_keys_str_mv | AT sunzeyu fairnesstestingofmachinetranslationsystems AT chenzhenpeng fairnesstestingofmachinetranslationsystems AT zhangjie fairnesstestingofmachinetranslationsystems AT haodan fairnesstestingofmachinetranslationsystems |