ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems
Large Language Models (LLMs), like ChatGPT, have sparked considerable interest among researchers across diverse disciplines owing to their remarkable text processing and generation capabilities. While ChatGPT is typically employed for tasks involving general knowledge, researchers increasingly explo...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-03-01
|
| Series: | Intelligent Systems with Applications |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2667305323001333 |
| _version_ | 1797287602304319488 |
|---|---|
| author | Yunjian Qiu Yan Jin |
| author_facet | Yunjian Qiu Yan Jin |
| author_sort | Yunjian Qiu |
| collection | DOAJ |
| description | Large Language Models (LLMs), like ChatGPT, have sparked considerable interest among researchers across diverse disciplines owing to their remarkable text processing and generation capabilities. While ChatGPT is typically employed for tasks involving general knowledge, researchers increasingly explore the potential of this LLM-based tool in specific domains to enhance productivity. This study aims to compare the performance of a finetuned BERT model with that of ChatGPT on a domain-specific dataset in the context of developing an intelligent design support system. Through experiments conducted on classification and generation tasks, the knowledge transfer and elicitation abilities of ChatGPT are examined and contrasted with those of the finetuned BERT model. The findings indicate that ChatGPT exhibits comparable performance to the finetuned BERT model in sentence-level classification tasks but struggles with short sequences. However, ChatGPT's classification performance significantly improves when a few-shot setting is applied. Moreover, it can filter out unrelated data and enhance dataset quality by assimilating the underlying domain knowledge. Regarding content generation, ChatGPT with a zero-shot setting produces informative and readable output for domain-specific questions, albeit with an excessive amount of unrelated information, which can burden readers. In conclusion, ChatGPT demonstrates a promising potential for application in facilitating data labeling, knowledge transfer, and knowledge elicitation tasks. With minimal guidance, ChatGPT can substantially enhance the efficiency of domain experts in accomplishing their objectives. The findings suggest a nuanced integration of artificial intelligence (AI) with human expertise, bridging the gap from mere classification models to sophisticated human-analogous text generation systems. This signals a future in AI-augmented engineering design where the robust capabilities of AI technologies integrate with human creativity and innovation, creating a dynamic interactions to redefine how we tackle design challenges. |
| first_indexed | 2024-03-07T18:35:43Z |
| format | Article |
| id | doaj.art-e6bc28223b814f46a0e2411b2fb24cf9 |
| institution | Directory Open Access Journal |
| issn | 2667-3053 |
| language | English |
| last_indexed | 2024-03-07T18:35:43Z |
| publishDate | 2024-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Intelligent Systems with Applications |
| spelling | doaj.art-e6bc28223b814f46a0e2411b2fb24cf92024-03-02T04:55:16ZengElsevierIntelligent Systems with Applications2667-30532024-03-0121200308ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systemsYunjian Qiu0Yan Jin1Department of Aerospace & Mechanical Engineering, University of Southern California, Los Angeles, CA 90089, United StatesCorresponding author.; Department of Aerospace & Mechanical Engineering, University of Southern California, Los Angeles, CA 90089, United StatesLarge Language Models (LLMs), like ChatGPT, have sparked considerable interest among researchers across diverse disciplines owing to their remarkable text processing and generation capabilities. While ChatGPT is typically employed for tasks involving general knowledge, researchers increasingly explore the potential of this LLM-based tool in specific domains to enhance productivity. This study aims to compare the performance of a finetuned BERT model with that of ChatGPT on a domain-specific dataset in the context of developing an intelligent design support system. Through experiments conducted on classification and generation tasks, the knowledge transfer and elicitation abilities of ChatGPT are examined and contrasted with those of the finetuned BERT model. The findings indicate that ChatGPT exhibits comparable performance to the finetuned BERT model in sentence-level classification tasks but struggles with short sequences. However, ChatGPT's classification performance significantly improves when a few-shot setting is applied. Moreover, it can filter out unrelated data and enhance dataset quality by assimilating the underlying domain knowledge. Regarding content generation, ChatGPT with a zero-shot setting produces informative and readable output for domain-specific questions, albeit with an excessive amount of unrelated information, which can burden readers. In conclusion, ChatGPT demonstrates a promising potential for application in facilitating data labeling, knowledge transfer, and knowledge elicitation tasks. With minimal guidance, ChatGPT can substantially enhance the efficiency of domain experts in accomplishing their objectives. The findings suggest a nuanced integration of artificial intelligence (AI) with human expertise, bridging the gap from mere classification models to sophisticated human-analogous text generation systems. This signals a future in AI-augmented engineering design where the robust capabilities of AI technologies integrate with human creativity and innovation, creating a dynamic interactions to redefine how we tackle design challenges.http://www.sciencedirect.com/science/article/pii/S2667305323001333Language modelKnowledge transferringKnowledge elicitationText classificationText generation |
| spellingShingle | Yunjian Qiu Yan Jin ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems Intelligent Systems with Applications Language model Knowledge transferring Knowledge elicitation Text classification Text generation |
| title | ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems |
| title_full | ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems |
| title_fullStr | ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems |
| title_full_unstemmed | ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems |
| title_short | ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems |
| title_sort | chatgpt and finetuned bert a comparative study for developing intelligent design support systems |
| topic | Language model Knowledge transferring Knowledge elicitation Text classification Text generation |
| url | http://www.sciencedirect.com/science/article/pii/S2667305323001333 |
| work_keys_str_mv | AT yunjianqiu chatgptandfinetunedbertacomparativestudyfordevelopingintelligentdesignsupportsystems AT yanjin chatgptandfinetunedbertacomparativestudyfordevelopingintelligentdesignsupportsystems |