The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis
In this study, we demonstrate how supervised learning can extract interpretable survey motivation measurements from a large number of responses to an open-ended question. We manually coded a subsample of 5,000 responses to an open-ended question on survey motivation from the GESIS Panel (25,000 resp...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Big Data |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fdata.2022.880554/full |
| _version_ | 1798037872720216064 |
|---|---|
| author | Anna-Carolina Haensch Bernd Weiß Patricia Steins Priscilla Chyrva Priscilla Chyrva Katja Bitz |
| author_facet | Anna-Carolina Haensch Bernd Weiß Patricia Steins Priscilla Chyrva Priscilla Chyrva Katja Bitz |
| author_sort | Anna-Carolina Haensch |
| collection | DOAJ |
| description | In this study, we demonstrate how supervised learning can extract interpretable survey motivation measurements from a large number of responses to an open-ended question. We manually coded a subsample of 5,000 responses to an open-ended question on survey motivation from the GESIS Panel (25,000 responses in total); we utilized supervised machine learning to classify the remaining responses. We can demonstrate that the responses on survey motivation in the GESIS Panel are particularly well suited for automated classification, since they are mostly one-dimensional. The evaluation of the test set also indicates very good overall performance. We present the pre-processing steps and methods we used for our data, and by discussing other popular options that might be more suitable in other cases, we also generalize beyond our use case. We also discuss various minor problems, such as a necessary spelling correction. Finally, we can showcase the analytic potential of the resulting categorization of panelists' motivation through an event history analysis of panel dropout. The analytical results allow a close look at respondents' motivations: they span a wide range, from the urge to help to interest in questions or the incentive and the wish to influence those in power through their participation. We conclude our paper by discussing the re-usability of the hand-coded responses for other surveys, including similar open questions to the GESIS Panel question. |
| first_indexed | 2024-04-11T21:32:29Z |
| format | Article |
| id | doaj.art-bff58afa6461462bbc6ffd96fe72dd74 |
| institution | Directory Open Access Journal |
| issn | 2624-909X |
| language | English |
| last_indexed | 2024-04-11T21:32:29Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Big Data |
| spelling | doaj.art-bff58afa6461462bbc6ffd96fe72dd742022-12-22T04:01:55ZengFrontiers Media S.A.Frontiers in Big Data2624-909X2022-08-01510.3389/fdata.2022.880554880554The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysisAnna-Carolina Haensch0Bernd Weiß1Patricia Steins2Priscilla Chyrva3Priscilla Chyrva4Katja Bitz5Department of Statistics, Ludwig-Maximilians-University Munich, Munich, GermanyGESIS-Leibniz-Institute for the Social Sciences, Mannheim, GermanySchool of Social Sciences, University of Mannheim, Mannheim, GermanyGESIS-Leibniz-Institute for the Social Sciences, Mannheim, GermanySchool of Social Sciences, University of Mannheim, Mannheim, GermanyFaculty of Economics and Social Sciences, Eberhard Karl University of Tübingen, Tübingen, GermanyIn this study, we demonstrate how supervised learning can extract interpretable survey motivation measurements from a large number of responses to an open-ended question. We manually coded a subsample of 5,000 responses to an open-ended question on survey motivation from the GESIS Panel (25,000 responses in total); we utilized supervised machine learning to classify the remaining responses. We can demonstrate that the responses on survey motivation in the GESIS Panel are particularly well suited for automated classification, since they are mostly one-dimensional. The evaluation of the test set also indicates very good overall performance. We present the pre-processing steps and methods we used for our data, and by discussing other popular options that might be more suitable in other cases, we also generalize beyond our use case. We also discuss various minor problems, such as a necessary spelling correction. Finally, we can showcase the analytic potential of the resulting categorization of panelists' motivation through an event history analysis of panel dropout. The analytical results allow a close look at respondents' motivations: they span a wide range, from the urge to help to interest in questions or the incentive and the wish to influence those in power through their participation. We conclude our paper by discussing the re-usability of the hand-coded responses for other surveys, including similar open questions to the GESIS Panel question.https://www.frontiersin.org/articles/10.3389/fdata.2022.880554/fulltext analysissupport vector machine (SVM)survey methodologysemi-automated analysismachine learningsurvey research |
| spellingShingle | Anna-Carolina Haensch Bernd Weiß Patricia Steins Priscilla Chyrva Priscilla Chyrva Katja Bitz The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis Frontiers in Big Data text analysis support vector machine (SVM) survey methodology semi-automated analysis machine learning survey research |
| title | The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis |
| title_full | The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis |
| title_fullStr | The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis |
| title_full_unstemmed | The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis |
| title_short | The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis |
| title_sort | semi automatic classification of an open ended question on panel survey motivation and its application in attrition analysis |
| topic | text analysis support vector machine (SVM) survey methodology semi-automated analysis machine learning survey research |
| url | https://www.frontiersin.org/articles/10.3389/fdata.2022.880554/full |
| work_keys_str_mv | AT annacarolinahaensch thesemiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT berndweiß thesemiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT patriciasteins thesemiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT priscillachyrva thesemiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT priscillachyrva thesemiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT katjabitz thesemiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT annacarolinahaensch semiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT berndweiß semiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT patriciasteins semiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT priscillachyrva semiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT priscillachyrva semiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis AT katjabitz semiautomaticclassificationofanopenendedquestiononpanelsurveymotivationanditsapplicationinattritionanalysis |