MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model
Transit riders' feedback provided in ridership surveys, customer relationship management (CRM) channels, and in more recent times, through social media is key for transit agencies to better gauge the efficacy of their services and initiatives. Getting a holistic understanding of riders' ex...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publications
2024
|
| Online Access: | https://hdl.handle.net/1721.1/156437 |
| _version_ | 1824457900540559360 |
|---|---|
| author | Leong, Michael Abdelhalim, Awad Ha, Jude Patterson, Diane Pincus, Gabriel L Harris, Anthony B Eichler, Michael Zhao, Jinhua |
| author2 | Massachusetts Institute of Technology. Department of Urban Studies and Planning |
| author_facet | Massachusetts Institute of Technology. Department of Urban Studies and Planning Leong, Michael Abdelhalim, Awad Ha, Jude Patterson, Diane Pincus, Gabriel L Harris, Anthony B Eichler, Michael Zhao, Jinhua |
| author_sort | Leong, Michael |
| collection | MIT |
| description | Transit riders' feedback provided in ridership surveys, customer relationship management (CRM) channels, and in more recent times, through social media is key for transit agencies to better gauge the efficacy of their services and initiatives. Getting a holistic understanding of riders' experience through the feedback shared in those instruments is often challenging, mostly due to the open-ended, unstructured nature of text feedback. In this paper, we propose leveraging traditional transit CRM feedback to develop and deploy a transit-topic-aware large language model (LLM) capable of classifying open-ended text feedback to relevant transit-specific topics. First, we utilize semi-supervised learning to engineer a training dataset of 11 broad transit topics detected in a corpus of 6 years of customer feedback provided to the Washington Metropolitan Area Transit Authority (WMATA). We then use this dataset to train and thoroughly evaluate a language model based on the RoBERTa architecture. We compare our LLM, MetRoBERTa, to classical machine learning approaches utilizing keyword-based and lexicon representations. Our model outperforms those methods across all evaluation metrics, providing an average topic classification accuracy of 90%. Finally, we provide a value proposition of this work demonstrating how the language model, alongside additional text processing tools, can be applied to add structure to open-ended text sources of feedback like Twitter. The framework and results we present provide a pathway for an automated, generalizable approach for ingesting, visualizing, and reporting transit riders' feedback at scale, enabling agencies to better understand and improve customer experience. |
| first_indexed | 2024-09-23T08:43:20Z |
| format | Article |
| id | mit-1721.1/156437 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:17:21Z |
| publishDate | 2024 |
| publisher | SAGE Publications |
| record_format | dspace |
| spelling | mit-1721.1/1564372024-12-23T06:09:37Z MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model Leong, Michael Abdelhalim, Awad Ha, Jude Patterson, Diane Pincus, Gabriel L Harris, Anthony B Eichler, Michael Zhao, Jinhua Massachusetts Institute of Technology. Department of Urban Studies and Planning Transit riders' feedback provided in ridership surveys, customer relationship management (CRM) channels, and in more recent times, through social media is key for transit agencies to better gauge the efficacy of their services and initiatives. Getting a holistic understanding of riders' experience through the feedback shared in those instruments is often challenging, mostly due to the open-ended, unstructured nature of text feedback. In this paper, we propose leveraging traditional transit CRM feedback to develop and deploy a transit-topic-aware large language model (LLM) capable of classifying open-ended text feedback to relevant transit-specific topics. First, we utilize semi-supervised learning to engineer a training dataset of 11 broad transit topics detected in a corpus of 6 years of customer feedback provided to the Washington Metropolitan Area Transit Authority (WMATA). We then use this dataset to train and thoroughly evaluate a language model based on the RoBERTa architecture. We compare our LLM, MetRoBERTa, to classical machine learning approaches utilizing keyword-based and lexicon representations. Our model outperforms those methods across all evaluation metrics, providing an average topic classification accuracy of 90%. Finally, we provide a value proposition of this work demonstrating how the language model, alongside additional text processing tools, can be applied to add structure to open-ended text sources of feedback like Twitter. The framework and results we present provide a pathway for an automated, generalizable approach for ingesting, visualizing, and reporting transit riders' feedback at scale, enabling agencies to better understand and improve customer experience. 2024-08-28T18:45:34Z 2024-08-28T18:45:34Z 2024 2024-08-28T18:38:56Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/156437 Leong, M., Abdelhalim, A., Ha, J., Patterson, D., Pincus, G. L., Harris, A. B., Eichler, M., & Zhao, J. (2024). MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model. Transportation Research Record. en 10.1177/03611981231225655 Transportation Research Record: Journal of the Transportation Research Board Creative Commons Attribution-Noncommercial-ShareAlike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf SAGE Publications arxiv |
| spellingShingle | Leong, Michael Abdelhalim, Awad Ha, Jude Patterson, Diane Pincus, Gabriel L Harris, Anthony B Eichler, Michael Zhao, Jinhua MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model |
| title | MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model |
| title_full | MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model |
| title_fullStr | MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model |
| title_full_unstemmed | MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model |
| title_short | MetRoBERTa: Leveraging Traditional Customer Relationship Management Data to Develop a Transit-Topic-Aware Language Model |
| title_sort | metroberta leveraging traditional customer relationship management data to develop a transit topic aware language model |
| url | https://hdl.handle.net/1721.1/156437 |
| work_keys_str_mv | AT leongmichael metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT abdelhalimawad metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT hajude metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT pattersondiane metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT pincusgabriell metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT harrisanthonyb metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT eichlermichael metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel AT zhaojinhua metrobertaleveragingtraditionalcustomerrelationshipmanagementdatatodevelopatransittopicawarelanguagemodel |