Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis
Abstract Background In testing contexts that are predominately concerned with power, rapid guessing (RG) has the potential to undermine the validity of inferences made from educational assessments, as such responses are unreflective of the knowledge, skills, and abilities assessed. Given this concer...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2021-08-01
|
| Series: | Large-scale Assessments in Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40536-021-00110-8 |
| _version_ | 1831769986495938560 |
|---|---|
| author | Joseph A. Rios Jiayi Deng |
| author_facet | Joseph A. Rios Jiayi Deng |
| author_sort | Joseph A. Rios |
| collection | DOAJ |
| description | Abstract Background In testing contexts that are predominately concerned with power, rapid guessing (RG) has the potential to undermine the validity of inferences made from educational assessments, as such responses are unreflective of the knowledge, skills, and abilities assessed. Given this concern, practitioners/researchers have utilized a multitude of response time threshold procedures that classify RG responses in these contexts based on either the use of no empirical data (e.g., an arbitrary time limit), response time distributions, and the combination of response time and accuracy information. As there is little understanding of how these procedures compare to each other, this meta-analysis sought to investigate whether threshold typology is related to differences in descriptive, measurement property, and performance outcomes in these contexts. Methods Studies were sampled that: (a) employed two or more response time (RT) threshold procedures to identify and exclude RG responses on the same computer-administered low-stakes power test; and (b) evaluated differences between procedures on the proportion of RG responses and responders, measurement properties, and test performance. Results Based on as many as 86 effect sizes, our findings indicated non-negligible differences between RT threshold procedures in the proportion of RG responses and responders. The largest differences for these outcomes were observed between procedures using no empirical data and those relying on response time and accuracy information. However, these differences were not related to variability in aggregate-level measurement properties and test performance. Conclusions When filtering RG responses to improve inferences concerning item properties and group score outcomes, the actual threshold procedure chosen may be of less importance than the act of identifying such deleterious responses. However, given the conservative nature of RT thresholds that use no empirical data, practitioners may look to avoid the use of these procedures when making inferences at the individual-level, given their potential for underclassifying RG. |
| first_indexed | 2024-12-22T07:32:05Z |
| format | Article |
| id | doaj.art-2c5c209c69f74df8a277a0dfa709b0a2 |
| institution | Directory Open Access Journal |
| issn | 2196-0739 |
| language | English |
| last_indexed | 2024-12-22T07:32:05Z |
| publishDate | 2021-08-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Large-scale Assessments in Education |
| spelling | doaj.art-2c5c209c69f74df8a277a0dfa709b0a22022-12-21T18:33:59ZengSpringerOpenLarge-scale Assessments in Education2196-07392021-08-019112510.1186/s40536-021-00110-8Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysisJoseph A. Rios0Jiayi Deng1Department of Educational Psychology, University of Minnesota, Twin CitiesDepartment of Educational Psychology, University of Minnesota, Twin CitiesAbstract Background In testing contexts that are predominately concerned with power, rapid guessing (RG) has the potential to undermine the validity of inferences made from educational assessments, as such responses are unreflective of the knowledge, skills, and abilities assessed. Given this concern, practitioners/researchers have utilized a multitude of response time threshold procedures that classify RG responses in these contexts based on either the use of no empirical data (e.g., an arbitrary time limit), response time distributions, and the combination of response time and accuracy information. As there is little understanding of how these procedures compare to each other, this meta-analysis sought to investigate whether threshold typology is related to differences in descriptive, measurement property, and performance outcomes in these contexts. Methods Studies were sampled that: (a) employed two or more response time (RT) threshold procedures to identify and exclude RG responses on the same computer-administered low-stakes power test; and (b) evaluated differences between procedures on the proportion of RG responses and responders, measurement properties, and test performance. Results Based on as many as 86 effect sizes, our findings indicated non-negligible differences between RT threshold procedures in the proportion of RG responses and responders. The largest differences for these outcomes were observed between procedures using no empirical data and those relying on response time and accuracy information. However, these differences were not related to variability in aggregate-level measurement properties and test performance. Conclusions When filtering RG responses to improve inferences concerning item properties and group score outcomes, the actual threshold procedure chosen may be of less importance than the act of identifying such deleterious responses. However, given the conservative nature of RT thresholds that use no empirical data, practitioners may look to avoid the use of these procedures when making inferences at the individual-level, given their potential for underclassifying RG.https://doi.org/10.1186/s40536-021-00110-8Rapid guessingTest-taking effortResponse timesMeta-analysis |
| spellingShingle | Joseph A. Rios Jiayi Deng Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis Large-scale Assessments in Education Rapid guessing Test-taking effort Response times Meta-analysis |
| title | Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis |
| title_full | Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis |
| title_fullStr | Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis |
| title_full_unstemmed | Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis |
| title_short | Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis |
| title_sort | does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses a meta analysis |
| topic | Rapid guessing Test-taking effort Response times Meta-analysis |
| url | https://doi.org/10.1186/s40536-021-00110-8 |
| work_keys_str_mv | AT josepharios doesthechoiceofresponsetimethresholdproceduresubstantiallyaffectinferencesconcerningtheidentificationandexclusionofrapidguessingresponsesametaanalysis AT jiayideng doesthechoiceofresponsetimethresholdproceduresubstantiallyaffectinferencesconcerningtheidentificationandexclusionofrapidguessingresponsesametaanalysis |