Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application
Intent recognition models, which match a written or spoken input's class in order to guide an interaction, are an essential part of modern voice user interfaces, chatbots, and social robots. However, getting enough data to train these models can be very expensive and challenging, especially whe...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
ACM
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137130 |
| _version_ | 1826197514069475328 |
|---|---|
| author | Huggins, M Alghowinem, S Jeong, S Colon-Hernandez, P Breazeal, C Park, HW |
| author2 | Massachusetts Institute of Technology. Media Laboratory |
| author_facet | Massachusetts Institute of Technology. Media Laboratory Huggins, M Alghowinem, S Jeong, S Colon-Hernandez, P Breazeal, C Park, HW |
| author_sort | Huggins, M |
| collection | MIT |
| description | Intent recognition models, which match a written or spoken input's class in order to guide an interaction, are an essential part of modern voice user interfaces, chatbots, and social robots. However, getting enough data to train these models can be very expensive and challenging, especially when designing novel applications such as real-world human-robot interactions. In this work, wefi rst investigate how much training data is needed for high performance in an intent classification task. We train and evaluate BiLSTM and BERT models on various subsets of the ATIS and Snips datasets. Wefi nd that only 25 training examples per intent are required for our BERT model to achieve 94% intent accuracy compared to 98% with the entire datasets, challenging the belief that large amounts of labeled data are required for high performance in intent recognition. We apply this knowledge to train models for a real-world HRI application, character strength recognition during a positive psychology interaction with a social robot, and evaluate against the Character Strength dataset collected in our previous HRI study. Our real-world HRI application results also confirm that our model can produce 76% intent accuracy with 25 examples per intent compared to 80% with 100 examples. In a real-world scenario, the difference is only one additional error per 25 classifications. Finally, we investigate the limitations of our minimal data models and offer suggestions on developing high quality datasets. We conclude with practical guidelines for training BERT intent recognition models with minimal training data and make our code and evaluation framework available for others to replicate our results and easily develop models for their own applications. |
| first_indexed | 2024-09-23T10:48:50Z |
| format | Article |
| id | mit-1721.1/137130 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:48:50Z |
| publishDate | 2021 |
| publisher | ACM |
| record_format | dspace |
| spelling | mit-1721.1/1371302023-04-11T19:55:46Z Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application Huggins, M Alghowinem, S Jeong, S Colon-Hernandez, P Breazeal, C Park, HW Massachusetts Institute of Technology. Media Laboratory Intent recognition models, which match a written or spoken input's class in order to guide an interaction, are an essential part of modern voice user interfaces, chatbots, and social robots. However, getting enough data to train these models can be very expensive and challenging, especially when designing novel applications such as real-world human-robot interactions. In this work, wefi rst investigate how much training data is needed for high performance in an intent classification task. We train and evaluate BiLSTM and BERT models on various subsets of the ATIS and Snips datasets. Wefi nd that only 25 training examples per intent are required for our BERT model to achieve 94% intent accuracy compared to 98% with the entire datasets, challenging the belief that large amounts of labeled data are required for high performance in intent recognition. We apply this knowledge to train models for a real-world HRI application, character strength recognition during a positive psychology interaction with a social robot, and evaluate against the Character Strength dataset collected in our previous HRI study. Our real-world HRI application results also confirm that our model can produce 76% intent accuracy with 25 examples per intent compared to 80% with 100 examples. In a real-world scenario, the difference is only one additional error per 25 classifications. Finally, we investigate the limitations of our minimal data models and offer suggestions on developing high quality datasets. We conclude with practical guidelines for training BERT intent recognition models with minimal training data and make our code and evaluation framework available for others to replicate our results and easily develop models for their own applications. 2021-11-02T17:26:47Z 2021-11-02T17:26:47Z 2021 2021-06-24T16:30:12Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137130 Huggins, M, Alghowinem, S, Jeong, S, Colon-Hernandez, P, Breazeal, C et al. 2021. "Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application." ACM/IEEE International Conference on Human-Robot Interaction. en 10.1145/3434073.3444671 ACM/IEEE International Conference on Human-Robot Interaction Creative Commons Attribution NonCommercial License 4.0 https://creativecommons.org/licenses/by-nc/4.0/ application/pdf ACM ACM |
| spellingShingle | Huggins, M Alghowinem, S Jeong, S Colon-Hernandez, P Breazeal, C Park, HW Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application |
| title | Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application |
| title_full | Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application |
| title_fullStr | Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application |
| title_full_unstemmed | Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application |
| title_short | Practical guidelines for intent recognition: BERT with minimal training data evaluated in real-world HRI application |
| title_sort | practical guidelines for intent recognition bert with minimal training data evaluated in real world hri application |
| url | https://hdl.handle.net/1721.1/137130 |
| work_keys_str_mv | AT hugginsm practicalguidelinesforintentrecognitionbertwithminimaltrainingdataevaluatedinrealworldhriapplication AT alghowinems practicalguidelinesforintentrecognitionbertwithminimaltrainingdataevaluatedinrealworldhriapplication AT jeongs practicalguidelinesforintentrecognitionbertwithminimaltrainingdataevaluatedinrealworldhriapplication AT colonhernandezp practicalguidelinesforintentrecognitionbertwithminimaltrainingdataevaluatedinrealworldhriapplication AT breazealc practicalguidelinesforintentrecognitionbertwithminimaltrainingdataevaluatedinrealworldhriapplication AT parkhw practicalguidelinesforintentrecognitionbertwithminimaltrainingdataevaluatedinrealworldhriapplication |