Enhancing response time thresholds with response behaviors for detecting disengaged examinees
Abstract The item responses of examinees who rapid-guess, who do not spend enough time reading and engaging with an item, will not reflect their true ability on that item. Rapid-disengagement refers to rapidly selecting a response to multiple-choice items (i.e., rapid-guess), omitting items, or prov...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2020-03-01
|
| Series: | Large-scale Assessments in Education |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s40536-020-00082-1 |
| _version_ | 1818852153756745728 |
|---|---|
| author | Füsun Sahin Kimberly F. Colvin |
| author_facet | Füsun Sahin Kimberly F. Colvin |
| author_sort | Füsun Sahin |
| collection | DOAJ |
| description | Abstract The item responses of examinees who rapid-guess, who do not spend enough time reading and engaging with an item, will not reflect their true ability on that item. Rapid-disengagement refers to rapidly selecting a response to multiple-choice items (i.e., rapid-guess), omitting items, or providing short-unrelated answers to open-ended items in an unreasonably short time. Such rapid responses need to be identified to enhance the validity of the arguments from the test data. Detection of rapid-guessing behaviors is typically based on identifying a threshold to represent the minimum response time required for the student to have thoughtfully considered a given item. This study investigates whether using response behaviors can improve the detection of rapid-disengagement by investigating two approaches: (a) using response behaviors to decide on the size of the threshold, and (b) using response behaviors as a condition for detecting disengaged examinees in addition to response times, referred as enhanced methods. Process data and item responses from the PISA 2012 computer-based mathematics assessment were used to examine both approaches under threshold values varying from very small (5 s) to very large (60 s). Results suggested that response behaviors can provide meaningful input on establishing the size of the threshold and that while enhanced methods performed better than using only response times in recognizing rapid-disengagement in some cases, no clear pattern was observed as to when such improvement occurs. This study makes a unique contribution by inspecting the response behaviors of disengaged examinees and providing guidelines on using response behaviors to decide on the size of the threshold. This study suggests response behavior categories that can be applicable to many item and response types, which make them suitable for use in digitally-based large-scale assessments. |
| first_indexed | 2024-12-19T07:16:24Z |
| format | Article |
| id | doaj.art-d26b5f2405a949e6a70df381b9b8379e |
| institution | Directory Open Access Journal |
| issn | 2196-0739 |
| language | English |
| last_indexed | 2024-12-19T07:16:24Z |
| publishDate | 2020-03-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Large-scale Assessments in Education |
| spelling | doaj.art-d26b5f2405a949e6a70df381b9b8379e2022-12-21T20:31:04ZengSpringerOpenLarge-scale Assessments in Education2196-07392020-03-018112410.1186/s40536-020-00082-1Enhancing response time thresholds with response behaviors for detecting disengaged examineesFüsun Sahin0Kimberly F. Colvin1American Institutes for ResearchUniversity at Albany, State University of New YorkAbstract The item responses of examinees who rapid-guess, who do not spend enough time reading and engaging with an item, will not reflect their true ability on that item. Rapid-disengagement refers to rapidly selecting a response to multiple-choice items (i.e., rapid-guess), omitting items, or providing short-unrelated answers to open-ended items in an unreasonably short time. Such rapid responses need to be identified to enhance the validity of the arguments from the test data. Detection of rapid-guessing behaviors is typically based on identifying a threshold to represent the minimum response time required for the student to have thoughtfully considered a given item. This study investigates whether using response behaviors can improve the detection of rapid-disengagement by investigating two approaches: (a) using response behaviors to decide on the size of the threshold, and (b) using response behaviors as a condition for detecting disengaged examinees in addition to response times, referred as enhanced methods. Process data and item responses from the PISA 2012 computer-based mathematics assessment were used to examine both approaches under threshold values varying from very small (5 s) to very large (60 s). Results suggested that response behaviors can provide meaningful input on establishing the size of the threshold and that while enhanced methods performed better than using only response times in recognizing rapid-disengagement in some cases, no clear pattern was observed as to when such improvement occurs. This study makes a unique contribution by inspecting the response behaviors of disengaged examinees and providing guidelines on using response behaviors to decide on the size of the threshold. This study suggests response behavior categories that can be applicable to many item and response types, which make them suitable for use in digitally-based large-scale assessments.http://link.springer.com/article/10.1186/s40536-020-00082-1Rapid guessingComputer-based assessmentsResponse timeResponse behaviorsProcess dataLog data |
| spellingShingle | Füsun Sahin Kimberly F. Colvin Enhancing response time thresholds with response behaviors for detecting disengaged examinees Large-scale Assessments in Education Rapid guessing Computer-based assessments Response time Response behaviors Process data Log data |
| title | Enhancing response time thresholds with response behaviors for detecting disengaged examinees |
| title_full | Enhancing response time thresholds with response behaviors for detecting disengaged examinees |
| title_fullStr | Enhancing response time thresholds with response behaviors for detecting disengaged examinees |
| title_full_unstemmed | Enhancing response time thresholds with response behaviors for detecting disengaged examinees |
| title_short | Enhancing response time thresholds with response behaviors for detecting disengaged examinees |
| title_sort | enhancing response time thresholds with response behaviors for detecting disengaged examinees |
| topic | Rapid guessing Computer-based assessments Response time Response behaviors Process data Log data |
| url | http://link.springer.com/article/10.1186/s40536-020-00082-1 |
| work_keys_str_mv | AT fusunsahin enhancingresponsetimethresholdswithresponsebehaviorsfordetectingdisengagedexaminees AT kimberlyfcolvin enhancingresponsetimethresholdswithresponsebehaviorsfordetectingdisengagedexaminees |