Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues
Research on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert anno...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Future Internet |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-5903/15/3/110 |
| _version_ | 1797611604108378112 |
|---|---|
| author | Zixiu Wu Simone Balloccu Vivek Kumar Rim Helaoui Diego Reforgiato Recupero Daniele Riboni |
| author_facet | Zixiu Wu Simone Balloccu Vivek Kumar Rim Helaoui Diego Reforgiato Recupero Daniele Riboni |
| author_sort | Zixiu Wu |
| collection | DOAJ |
| description | Research on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert annotations, but it has been alleviated thanks to the recent release of AnnoMI, the first publicly and freely available conversation dataset of 133 faithfully transcribed and expert-annotated demonstrations of high- and low-quality motivational interviewing (MI)—an effective therapy strategy that evokes client motivation for positive change. In this work, we introduce new expert-annotated utterance attributes to AnnoMI and describe the entire data collection process in more detail, including dialogue source selection, transcription, annotation, and post-processing. Based on the expert annotations on key MI aspects, we carry out thorough analyses of AnnoMI with respect to counselling-related properties on the utterance, conversation, and corpus levels. Furthermore, we introduce utterance-level prediction tasks with potential real-world impacts and build baseline models. Finally, we examine the performance of the models on dialogues of different topics and probe the generalisability of the models to unseen topics. |
| first_indexed | 2024-03-11T06:31:02Z |
| format | Article |
| id | doaj.art-6de5b2ef7d0644399df3aa82a5572420 |
| institution | Directory Open Access Journal |
| issn | 1999-5903 |
| language | English |
| last_indexed | 2024-03-11T06:31:02Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Future Internet |
| spelling | doaj.art-6de5b2ef7d0644399df3aa82a55724202023-11-17T11:13:15ZengMDPI AGFuture Internet1999-59032023-03-0115311010.3390/fi15030110Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling DialoguesZixiu Wu0Simone Balloccu1Vivek Kumar2Rim Helaoui3Diego Reforgiato Recupero4Daniele Riboni5Philips Research, High Tech Campus, 5656 AE Eindhoven, The NetherlandsDepartment of Computing Science, University of Aberdeen, Aberdeen AB24 3FX, UKDepartment of Mathematics and Computer Science, University of Cagliari, 09124 Cagliari, ItalyPhilips Research, High Tech Campus, 5656 AE Eindhoven, The NetherlandsDepartment of Mathematics and Computer Science, University of Cagliari, 09124 Cagliari, ItalyDepartment of Mathematics and Computer Science, University of Cagliari, 09124 Cagliari, ItalyResearch on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert annotations, but it has been alleviated thanks to the recent release of AnnoMI, the first publicly and freely available conversation dataset of 133 faithfully transcribed and expert-annotated demonstrations of high- and low-quality motivational interviewing (MI)—an effective therapy strategy that evokes client motivation for positive change. In this work, we introduce new expert-annotated utterance attributes to AnnoMI and describe the entire data collection process in more detail, including dialogue source selection, transcription, annotation, and post-processing. Based on the expert annotations on key MI aspects, we carry out thorough analyses of AnnoMI with respect to counselling-related properties on the utterance, conversation, and corpus levels. Furthermore, we introduce utterance-level prediction tasks with potential real-world impacts and build baseline models. Finally, we examine the performance of the models on dialogues of different topics and probe the generalisability of the models to unseen topics.https://www.mdpi.com/1999-5903/15/3/110dialoguecounsellingmotivational interviewingnatural language processingdataset |
| spellingShingle | Zixiu Wu Simone Balloccu Vivek Kumar Rim Helaoui Diego Reforgiato Recupero Daniele Riboni Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues Future Internet dialogue counselling motivational interviewing natural language processing dataset |
| title | Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues |
| title_full | Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues |
| title_fullStr | Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues |
| title_full_unstemmed | Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues |
| title_short | Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues |
| title_sort | creation analysis and evaluation of annomi a dataset of expert annotated counselling dialogues |
| topic | dialogue counselling motivational interviewing natural language processing dataset |
| url | https://www.mdpi.com/1999-5903/15/3/110 |
| work_keys_str_mv | AT zixiuwu creationanalysisandevaluationofannomiadatasetofexpertannotatedcounsellingdialogues AT simoneballoccu creationanalysisandevaluationofannomiadatasetofexpertannotatedcounsellingdialogues AT vivekkumar creationanalysisandevaluationofannomiadatasetofexpertannotatedcounsellingdialogues AT rimhelaoui creationanalysisandevaluationofannomiadatasetofexpertannotatedcounsellingdialogues AT diegoreforgiatorecupero creationanalysisandevaluationofannomiadatasetofexpertannotatedcounsellingdialogues AT danieleriboni creationanalysisandevaluationofannomiadatasetofexpertannotatedcounsellingdialogues |