Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review
As artificial intelligence advances, source code completion assistants are becoming more advanced and powerful. Existing traditional assistants are no longer up to all the developers’ challenges. Traditional assistants usually present proposals in alphabetically sorted lists, which does not make a d...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/24/13061 |
| _version_ | 1797382144875560960 |
|---|---|
| author | Tilen Hliš Luka Četina Tina Beranič Luka Pavlič |
| author_facet | Tilen Hliš Luka Četina Tina Beranič Luka Pavlič |
| author_sort | Tilen Hliš |
| collection | DOAJ |
| description | As artificial intelligence advances, source code completion assistants are becoming more advanced and powerful. Existing traditional assistants are no longer up to all the developers’ challenges. Traditional assistants usually present proposals in alphabetically sorted lists, which does not make a developer’s tasks any easier (i.e., they still have to search and filter an appropriate proposal manually). As a possible solution to the presented issue, intelligent assistants that can classify suggestions according to relevance in particular contexts have emerged. Artificial intelligence methods have proven to be successful in solving such problems. Advanced intelligent assistants not only take into account the context of a particular source code but also, more importantly, examine other available projects in detail to extract possible patterns related to particular source code intentions. This is how intelligent assistants try to provide developers with relevant suggestions. By conducting a systematic literature review, we examined the current intelligent assistant landscape. Based on our review, we tested four intelligent assistants and compared them according to their functionality. GitHub Copilot, which stood out, allows suggestions in the form of complete source code sections. One would expect that intelligent assistants, with their outstanding functionalities, would be one of the most popular helpers in a developer’s toolbox. However, through a survey we conducted among practitioners, the results, surprisingly, contradicted this idea. Although intelligent assistants promise high usability, our questionnaires indicate that usability improvements are still needed. However, our research data show that experienced developers value intelligent assistants highly, highlighting their significant utility for the experienced developers group when compared to less experienced individuals. The unexpectedly low net promoter score (NPS) for intelligent code assistants in our study was quite surprising, highlighting a stark contrast between the anticipated impact of these advanced tools and their actual reception among developers. |
| first_indexed | 2024-03-08T21:01:17Z |
| format | Article |
| id | doaj.art-16c269fdac9e4f4581ded9b1ca693deb |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-08T21:01:17Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-16c269fdac9e4f4581ded9b1ca693deb2023-12-22T13:50:50ZengMDPI AGApplied Sciences2076-34172023-12-0113241306110.3390/app132413061Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive ReviewTilen Hliš0Luka Četina1Tina Beranič2Luka Pavlič3Faculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, SloveniaFaculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, SloveniaFaculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, SloveniaFaculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, SloveniaAs artificial intelligence advances, source code completion assistants are becoming more advanced and powerful. Existing traditional assistants are no longer up to all the developers’ challenges. Traditional assistants usually present proposals in alphabetically sorted lists, which does not make a developer’s tasks any easier (i.e., they still have to search and filter an appropriate proposal manually). As a possible solution to the presented issue, intelligent assistants that can classify suggestions according to relevance in particular contexts have emerged. Artificial intelligence methods have proven to be successful in solving such problems. Advanced intelligent assistants not only take into account the context of a particular source code but also, more importantly, examine other available projects in detail to extract possible patterns related to particular source code intentions. This is how intelligent assistants try to provide developers with relevant suggestions. By conducting a systematic literature review, we examined the current intelligent assistant landscape. Based on our review, we tested four intelligent assistants and compared them according to their functionality. GitHub Copilot, which stood out, allows suggestions in the form of complete source code sections. One would expect that intelligent assistants, with their outstanding functionalities, would be one of the most popular helpers in a developer’s toolbox. However, through a survey we conducted among practitioners, the results, surprisingly, contradicted this idea. Although intelligent assistants promise high usability, our questionnaires indicate that usability improvements are still needed. However, our research data show that experienced developers value intelligent assistants highly, highlighting their significant utility for the experienced developers group when compared to less experienced individuals. The unexpectedly low net promoter score (NPS) for intelligent code assistants in our study was quite surprising, highlighting a stark contrast between the anticipated impact of these advanced tools and their actual reception among developers.https://www.mdpi.com/2076-3417/13/24/13061intelligent assistantssource code completionsource code |
| spellingShingle | Tilen Hliš Luka Četina Tina Beranič Luka Pavlič Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review Applied Sciences intelligent assistants source code completion source code |
| title | Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review |
| title_full | Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review |
| title_fullStr | Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review |
| title_full_unstemmed | Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review |
| title_short | Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review |
| title_sort | evaluating the usability and functionality of intelligent source code completion assistants a comprehensive review |
| topic | intelligent assistants source code completion source code |
| url | https://www.mdpi.com/2076-3417/13/24/13061 |
| work_keys_str_mv | AT tilenhlis evaluatingtheusabilityandfunctionalityofintelligentsourcecodecompletionassistantsacomprehensivereview AT lukacetina evaluatingtheusabilityandfunctionalityofintelligentsourcecodecompletionassistantsacomprehensivereview AT tinaberanic evaluatingtheusabilityandfunctionalityofintelligentsourcecodecompletionassistantsacomprehensivereview AT lukapavlic evaluatingtheusabilityandfunctionalityofintelligentsourcecodecompletionassistantsacomprehensivereview |