Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings
After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verifica...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9982595/ |
| _version_ | 1818296162211332096 |
|---|---|
| author | Ranci Ren Sara Perez-Soler John W. Castro Oscar Dieste Silvia T. Acuna |
| author_facet | Ranci Ren Sara Perez-Soler John W. Castro Oscar Dieste Silvia T. Acuna |
| author_sort | Ranci Ren |
| collection | DOAJ |
| description | After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation. |
| first_indexed | 2024-12-13T03:59:09Z |
| format | Article |
| id | doaj.art-ca5c096c65de452aa6046a30079f46a0 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-13T03:59:09Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-ca5c096c65de452aa6046a30079f46a02022-12-22T00:00:31ZengIEEEIEEE Access2169-35362022-01-011013054213056210.1109/ACCESS.2022.32287729982595Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic SettingsRanci Ren0Sara Perez-Soler1https://orcid.org/0000-0002-4558-7111John W. Castro2https://orcid.org/0000-0002-7938-7564Oscar Dieste3https://orcid.org/0000-0002-3060-7853Silvia T. Acuna4https://orcid.org/0000-0003-2135-6113Departamento de Ingeniería Informática, Escuela Politécnica Superior, Universidad Autónoma de Madrid, Madrid, SpainDepartamento de Ingeniería Informática, Escuela Politécnica Superior, Universidad Autónoma de Madrid, Madrid, SpainDepartamento de Ingeniería Informática y Ciencias de la Computación, Universidad de Atacama, Copiapó, ChileEscuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Boadilla del Monte, SpainDepartamento de Ingeniería Informática, Escuela Politécnica Superior, Universidad Autónoma de Madrid, Madrid, SpainAfter improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation.https://ieeexplore.ieee.org/document/9982595/Chatbotusabilityfamily of experiments |
| spellingShingle | Ranci Ren Sara Perez-Soler John W. Castro Oscar Dieste Silvia T. Acuna Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings IEEE Access Chatbot usability family of experiments |
| title | Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings |
| title_full | Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings |
| title_fullStr | Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings |
| title_full_unstemmed | Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings |
| title_short | Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings |
| title_sort | using the socio chatbot for uml modeling a second family of experiments on usability in academic settings |
| topic | Chatbot usability family of experiments |
| url | https://ieeexplore.ieee.org/document/9982595/ |
| work_keys_str_mv | AT ranciren usingthesociochatbotforumlmodelingasecondfamilyofexperimentsonusabilityinacademicsettings AT saraperezsoler usingthesociochatbotforumlmodelingasecondfamilyofexperimentsonusabilityinacademicsettings AT johnwcastro usingthesociochatbotforumlmodelingasecondfamilyofexperimentsonusabilityinacademicsettings AT oscardieste usingthesociochatbotforumlmodelingasecondfamilyofexperimentsonusabilityinacademicsettings AT silviatacuna usingthesociochatbotforumlmodelingasecondfamilyofexperimentsonusabilityinacademicsettings |