AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System
In this work, we present the use of one artificial intelligence (AI) application (ChatGPT) to train another AI-based application. As the latter one, we show a dialogue system named Terabot, which was used in the therapy of psychiatric patients. Our study was motivated by the fact that for such a dom...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/22/4694 |
| _version_ | 1797459497574203392 |
|---|---|
| author | Karolina Gabor-Siatkowska Marcin Sowański Rafał Rzatkiewicz Izabela Stefaniak Marek Kozłowski Artur Janicki |
| author_facet | Karolina Gabor-Siatkowska Marcin Sowański Rafał Rzatkiewicz Izabela Stefaniak Marek Kozłowski Artur Janicki |
| author_sort | Karolina Gabor-Siatkowska |
| collection | DOAJ |
| description | In this work, we present the use of one artificial intelligence (AI) application (ChatGPT) to train another AI-based application. As the latter one, we show a dialogue system named Terabot, which was used in the therapy of psychiatric patients. Our study was motivated by the fact that for such a domain-specific system, it was difficult to acquire large real-life data samples to increase the training database: this would require recruiting more patients, which is both time-consuming and costly. To address this gap, we have employed a neural large language model: ChatGPT version 3.5, to generate data solely for training our dialogue system. During initial experiments, we identified intents that were most often misrecognized. Next, we fed ChatGPT with a series of prompts, which triggered the language model to generate numerous additional training entries, e.g., alternatives to the phrases that had been collected during initial experiments with healthy users. This way, we have enlarged the training dataset by 112%. In our case study, for testing, we used 2802 speech recordings originating from 32 psychiatric patients. As an evaluation metric, we used the accuracy of intent recognition. The speech samples were converted into text using automatic speech recognition (ASR). The analysis showed that the patients’ speech challenged the ASR module significantly, resulting in deteriorated speech recognition and, consequently, low accuracy of intent recognition. However, thanks to the augmentation of the training data with ChatGPT-generated data, the intent recognition accuracy increased by 13% relatively, reaching 86% in total. We also emulated the case of an error-free ASR and showed the impact of ASR misrecognitions on the intent recognition accuracy. Our study showcased the potential of using generative language models to develop other AI-based tools, such as dialogue systems. |
| first_indexed | 2024-03-09T16:52:11Z |
| format | Article |
| id | doaj.art-d950f1b607404f49a440eaf0225ed996 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-09T16:52:11Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-d950f1b607404f49a440eaf0225ed9962023-11-24T14:39:43ZengMDPI AGElectronics2079-92922023-11-011222469410.3390/electronics12224694AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue SystemKarolina Gabor-Siatkowska0Marcin Sowański1Rafał Rzatkiewicz2Izabela Stefaniak3Marek Kozłowski4Artur Janicki5Faculty of Electronics and Information Technology, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, PolandFaculty of Electronics and Information Technology, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, PolandFaculty of Electronics and Information Technology, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, PolandFaculty of Medicine, Lazarski University, ul. Świeradowska 43, 02-662 Warsaw, PolandFaculty of Electronics and Information Technology, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, PolandFaculty of Electronics and Information Technology, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, PolandIn this work, we present the use of one artificial intelligence (AI) application (ChatGPT) to train another AI-based application. As the latter one, we show a dialogue system named Terabot, which was used in the therapy of psychiatric patients. Our study was motivated by the fact that for such a domain-specific system, it was difficult to acquire large real-life data samples to increase the training database: this would require recruiting more patients, which is both time-consuming and costly. To address this gap, we have employed a neural large language model: ChatGPT version 3.5, to generate data solely for training our dialogue system. During initial experiments, we identified intents that were most often misrecognized. Next, we fed ChatGPT with a series of prompts, which triggered the language model to generate numerous additional training entries, e.g., alternatives to the phrases that had been collected during initial experiments with healthy users. This way, we have enlarged the training dataset by 112%. In our case study, for testing, we used 2802 speech recordings originating from 32 psychiatric patients. As an evaluation metric, we used the accuracy of intent recognition. The speech samples were converted into text using automatic speech recognition (ASR). The analysis showed that the patients’ speech challenged the ASR module significantly, resulting in deteriorated speech recognition and, consequently, low accuracy of intent recognition. However, thanks to the augmentation of the training data with ChatGPT-generated data, the intent recognition accuracy increased by 13% relatively, reaching 86% in total. We also emulated the case of an error-free ASR and showed the impact of ASR misrecognitions on the intent recognition accuracy. Our study showcased the potential of using generative language models to develop other AI-based tools, such as dialogue systems.https://www.mdpi.com/2079-9292/12/22/4694spoken dialogue systemspeech recognitionChatGPTdata augmentationcomputer-aided therapycognitive-behavioral therapy |
| spellingShingle | Karolina Gabor-Siatkowska Marcin Sowański Rafał Rzatkiewicz Izabela Stefaniak Marek Kozłowski Artur Janicki AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System Electronics spoken dialogue system speech recognition ChatGPT data augmentation computer-aided therapy cognitive-behavioral therapy |
| title | AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System |
| title_full | AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System |
| title_fullStr | AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System |
| title_full_unstemmed | AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System |
| title_short | AI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System |
| title_sort | ai to train ai using chatgpt to improve the accuracy of a therapeutic dialogue system |
| topic | spoken dialogue system speech recognition ChatGPT data augmentation computer-aided therapy cognitive-behavioral therapy |
| url | https://www.mdpi.com/2079-9292/12/22/4694 |
| work_keys_str_mv | AT karolinagaborsiatkowska aitotrainaiusingchatgpttoimprovetheaccuracyofatherapeuticdialoguesystem AT marcinsowanski aitotrainaiusingchatgpttoimprovetheaccuracyofatherapeuticdialoguesystem AT rafałrzatkiewicz aitotrainaiusingchatgpttoimprovetheaccuracyofatherapeuticdialoguesystem AT izabelastefaniak aitotrainaiusingchatgpttoimprovetheaccuracyofatherapeuticdialoguesystem AT marekkozłowski aitotrainaiusingchatgpttoimprovetheaccuracyofatherapeuticdialoguesystem AT arturjanicki aitotrainaiusingchatgpttoimprovetheaccuracyofatherapeuticdialoguesystem |