Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks
The purpose of this paper is to study the application of ChatGPT and BERT models in the field of mechanical engineering. In the context of machine learning, the ChatGPT and BERT models can be applied to various natural language processing tasks such as analyzing technical documentation and building...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2024-01-01
|
| Series: | E3S Web of Conferences |
| Online Access: | https://www.e3s-conferences.org/articles/e3sconf/pdf/2024/45/e3sconf_tt21c-2024_03016.pdf |
| _version_ | 1827278841103515648 |
|---|---|
| author | Kim Radmir Kotsenko Anton Andreev Aleksandr Bazanova Anastasiia Aladin Dmitry Todua David Marushchenko Aleksei Varlamov Oleg |
| author_facet | Kim Radmir Kotsenko Anton Andreev Aleksandr Bazanova Anastasiia Aladin Dmitry Todua David Marushchenko Aleksei Varlamov Oleg |
| author_sort | Kim Radmir |
| collection | DOAJ |
| description | The purpose of this paper is to study the application of ChatGPT and BERT models in the field of mechanical engineering. In the context of machine learning, the ChatGPT and BERT models can be applied to various natural language processing tasks such as analyzing technical documentation and building instructions according to a particular version of the documentation, diagnosing malfunctions or customer service. The paper discusses the fundamental features of BERT and ChatGPT models, their origin, and also investigates the main architectural features and identifies the main advantages and disadvantages of the models. The paper analyzes and selects various natural language processing tasks to test the models’ ability to understand natural language in the context of machine learning. The selected criterion tasks are divided into semantic groups to identify the capabilities of ChatGPT and BERT models in each of three areas: logical inference tasks, paraphrasing tasks, and text similarity tasks. The paper also discusses the concept of operational design, which involves developing inputs that guide the models to produce desired outputs. The paper quantitatively analyzes and compares the performance of BERT and ChatGPT based models. The reasons for the bottlenecks of ChatGPT model in natural language understanding tasks are discovered and investigated. Possible improvements of ChatGPT model performance using the mivar approach are considered. |
| first_indexed | 2024-04-24T08:07:06Z |
| format | Article |
| id | doaj.art-6cc64802f43147bbb85dc83638f1feaf |
| institution | Directory Open Access Journal |
| issn | 2267-1242 |
| language | English |
| last_indexed | 2024-04-24T08:07:06Z |
| publishDate | 2024-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | E3S Web of Conferences |
| spelling | doaj.art-6cc64802f43147bbb85dc83638f1feaf2024-04-17T09:12:18ZengEDP SciencesE3S Web of Conferences2267-12422024-01-015150301610.1051/e3sconf/202451503016e3sconf_tt21c-2024_03016Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasksKim Radmir0Kotsenko Anton1Andreev Aleksandr2Bazanova Anastasiia3Aladin Dmitry4Todua David5Marushchenko Aleksei6Varlamov Oleg7Bauman Moscow State Technical UniversityBauman Moscow State Technical UniversityBauman Moscow State Technical UniversityBauman Moscow State Technical UniversityBauman Moscow State Technical UniversityBauman Moscow State Technical UniversityBauman Moscow State Technical UniversityBauman Moscow State Technical UniversityThe purpose of this paper is to study the application of ChatGPT and BERT models in the field of mechanical engineering. In the context of machine learning, the ChatGPT and BERT models can be applied to various natural language processing tasks such as analyzing technical documentation and building instructions according to a particular version of the documentation, diagnosing malfunctions or customer service. The paper discusses the fundamental features of BERT and ChatGPT models, their origin, and also investigates the main architectural features and identifies the main advantages and disadvantages of the models. The paper analyzes and selects various natural language processing tasks to test the models’ ability to understand natural language in the context of machine learning. The selected criterion tasks are divided into semantic groups to identify the capabilities of ChatGPT and BERT models in each of three areas: logical inference tasks, paraphrasing tasks, and text similarity tasks. The paper also discusses the concept of operational design, which involves developing inputs that guide the models to produce desired outputs. The paper quantitatively analyzes and compares the performance of BERT and ChatGPT based models. The reasons for the bottlenecks of ChatGPT model in natural language understanding tasks are discovered and investigated. Possible improvements of ChatGPT model performance using the mivar approach are considered.https://www.e3s-conferences.org/articles/e3sconf/pdf/2024/45/e3sconf_tt21c-2024_03016.pdf |
| spellingShingle | Kim Radmir Kotsenko Anton Andreev Aleksandr Bazanova Anastasiia Aladin Dmitry Todua David Marushchenko Aleksei Varlamov Oleg Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks E3S Web of Conferences |
| title | Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks |
| title_full | Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks |
| title_fullStr | Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks |
| title_full_unstemmed | Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks |
| title_short | Evaluation of BERT and ChatGPT models in inference, paraphrase and similarity tasks |
| title_sort | evaluation of bert and chatgpt models in inference paraphrase and similarity tasks |
| url | https://www.e3s-conferences.org/articles/e3sconf/pdf/2024/45/e3sconf_tt21c-2024_03016.pdf |
| work_keys_str_mv | AT kimradmir evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT kotsenkoanton evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT andreevaleksandr evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT bazanovaanastasiia evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT aladindmitry evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT toduadavid evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT marushchenkoaleksei evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks AT varlamovoleg evaluationofbertandchatgptmodelsininferenceparaphraseandsimilaritytasks |