ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence
The purpose of this study is to evaluate ChatGPT-4’s language proficiency in Italian dialects. At the outset, it is clarified what is meant by ‘language ability’ within the context of Large Language Models. This involves identifying the tasks that ChatGPT might face in real-world scenarios, from whi...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
University of Bologna
2023-12-01
|
| Series: | Umanistica Digitale |
| Subjects: | |
| Online Access: | https://umanisticadigitale.unibo.it/article/view/18221 |
| _version_ | 1797389950851743744 |
|---|---|
| author | Silvia Lilli |
| author_facet | Silvia Lilli |
| author_sort | Silvia Lilli |
| collection | DOAJ |
| description | The purpose of this study is to evaluate ChatGPT-4’s language proficiency in Italian dialects. At the outset, it is clarified what is meant by ‘language ability’ within the context of Large Language Models. This involves identifying the tasks that ChatGPT might face in real-world scenarios, from which we can derive inferential assumptions regarding its linguistic ability. The skills identified, which served as foundation for test design, include comprehension and translation, dialect recognition, analysis of the distinctive features, error detection, text production, interaction, theoretical background, and self-assessment. The tests were crafted to mimic situations requiring these competencies, trying to emulate authentic ChatGPT-User interactions. The results highlight ChatGPT’s excellent prowess in understanding and recognizing Italian dialects and their subvarieties, and a robust background and awareness of its own knowledge. However, the model exhibits significant gaps in analytical skills and struggles with text production and interactive tasks, suggesting superior passive linguistic capabilities compared to active ones. |
| first_indexed | 2024-03-08T23:04:33Z |
| format | Article |
| id | doaj.art-8757185abdb0416fb3a37e03fdc44049 |
| institution | Directory Open Access Journal |
| issn | 2532-8816 |
| language | English |
| last_indexed | 2024-03-08T23:04:33Z |
| publishDate | 2023-12-01 |
| publisher | University of Bologna |
| record_format | Article |
| series | Umanistica Digitale |
| spelling | doaj.art-8757185abdb0416fb3a37e03fdc440492023-12-15T14:17:41ZengUniversity of BolognaUmanistica Digitale2532-88162023-12-011623526310.6092/issn.2532-8816/1822116580ChatGPT-4 and Italian Dialects: Assessing Linguistic CompetenceSilvia Lilli0University of Rome "Tor Vergata"The purpose of this study is to evaluate ChatGPT-4’s language proficiency in Italian dialects. At the outset, it is clarified what is meant by ‘language ability’ within the context of Large Language Models. This involves identifying the tasks that ChatGPT might face in real-world scenarios, from which we can derive inferential assumptions regarding its linguistic ability. The skills identified, which served as foundation for test design, include comprehension and translation, dialect recognition, analysis of the distinctive features, error detection, text production, interaction, theoretical background, and self-assessment. The tests were crafted to mimic situations requiring these competencies, trying to emulate authentic ChatGPT-User interactions. The results highlight ChatGPT’s excellent prowess in understanding and recognizing Italian dialects and their subvarieties, and a robust background and awareness of its own knowledge. However, the model exhibits significant gaps in analytical skills and struggles with text production and interactive tasks, suggesting superior passive linguistic capabilities compared to active ones.https://umanisticadigitale.unibo.it/article/view/18221chatgptllmsitalian dialectslanguage testinglanguage abilitiesmachine translation |
| spellingShingle | Silvia Lilli ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence Umanistica Digitale chatgpt llms italian dialects language testing language abilities machine translation |
| title | ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence |
| title_full | ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence |
| title_fullStr | ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence |
| title_full_unstemmed | ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence |
| title_short | ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence |
| title_sort | chatgpt 4 and italian dialects assessing linguistic competence |
| topic | chatgpt llms italian dialects language testing language abilities machine translation |
| url | https://umanisticadigitale.unibo.it/article/view/18221 |
| work_keys_str_mv | AT silvialilli chatgpt4anditaliandialectsassessinglinguisticcompetence |