Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries
Abstract Unfavorable test-taking behaviors, such as speededness and disengagement, have long been a validity concern for large-scale low-stakes assessments. Understanding the presence and extent of such behaviors is important for ensuring the validity of inferences based on test scores. This study e...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-09-01
|
| Series: | Large-scale Assessments in Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40536-023-00179-3 |
| _version_ | 1797557640416460800 |
|---|---|
| author | Xiaying Zheng Fusun Sahin Ebru Erberber Frank Fonseca |
| author_facet | Xiaying Zheng Fusun Sahin Ebru Erberber Frank Fonseca |
| author_sort | Xiaying Zheng |
| collection | DOAJ |
| description | Abstract Unfavorable test-taking behaviors, such as speededness and disengagement, have long been a validity concern for large-scale low-stakes assessments. Understanding the presence and extent of such behaviors is important for ensuring the validity of inferences based on test scores. This study examined test-taking behaviors using item response time (RT), a process data-derived variable from the TIMSS 2019 database. Analyses compared the United States to three other countries (England, Singapore, and the United Arab Emirates) that administered the digital version of TIMSS (eTIMSS) 2019 in English at grade 8. Test-taking behaviors were identified within each country and compared within and across countries. Specifically, to identify distinct types of test-taking behaviors, mixture modeling was employed on RT and item scores from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment. The results indicated that each country had several latent classes of students with different pacing trajectories and performance. The test-taking behaviors of these latent classes were labeled as Steady; Disengaged or Very disengaged; Speeded or Very speeded; and Efficient and high-performing. Most of the students in each country had a Steady pace (medium to high sum score; steady RT throughout the test): 71% in England, 74% in both Singapore and the United Arab Emirates, and 84% in the United States. Disengaged or Very disengaged students (low sum score; short RT) were identified in each country but were more prevalent in England and the United Arab Emirates (above 20% in both) than in the United States and Singapore (both below 10%). The study also revealed small percentages of Speeded or Very speeded students (low to medium sum score; long RT at first but very short RT toward the end) in England, the United Arab Emirates, and the United States (1%, 5%, and 6%, respectively) but not in Singapore. A unique class of Efficient and high-performing students (high sum score; short RT) was identified only in Singapore (24%). This study demonstrated that mixture modeling is a useful technique for identifying distinct test-taking behaviors and highlighted the presence and extent of unfavorable test-taking behaviors within each selected country using data from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment. |
| first_indexed | 2024-03-10T17:19:28Z |
| format | Article |
| id | doaj.art-a29c1468886440c094f806e33eaf71c9 |
| institution | Directory Open Access Journal |
| issn | 2196-0739 |
| language | English |
| last_indexed | 2024-03-10T17:19:28Z |
| publishDate | 2023-09-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Large-scale Assessments in Education |
| spelling | doaj.art-a29c1468886440c094f806e33eaf71c92023-11-20T10:22:46ZengSpringerOpenLarge-scale Assessments in Education2196-07392023-09-0111111410.1186/s40536-023-00179-3Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countriesXiaying Zheng0Fusun Sahin1Ebru Erberber2Frank Fonseca3American Institutes for ResearchCurriculum AssociatesAmerican Institutes for ResearchAmerican Institutes for ResearchAbstract Unfavorable test-taking behaviors, such as speededness and disengagement, have long been a validity concern for large-scale low-stakes assessments. Understanding the presence and extent of such behaviors is important for ensuring the validity of inferences based on test scores. This study examined test-taking behaviors using item response time (RT), a process data-derived variable from the TIMSS 2019 database. Analyses compared the United States to three other countries (England, Singapore, and the United Arab Emirates) that administered the digital version of TIMSS (eTIMSS) 2019 in English at grade 8. Test-taking behaviors were identified within each country and compared within and across countries. Specifically, to identify distinct types of test-taking behaviors, mixture modeling was employed on RT and item scores from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment. The results indicated that each country had several latent classes of students with different pacing trajectories and performance. The test-taking behaviors of these latent classes were labeled as Steady; Disengaged or Very disengaged; Speeded or Very speeded; and Efficient and high-performing. Most of the students in each country had a Steady pace (medium to high sum score; steady RT throughout the test): 71% in England, 74% in both Singapore and the United Arab Emirates, and 84% in the United States. Disengaged or Very disengaged students (low sum score; short RT) were identified in each country but were more prevalent in England and the United Arab Emirates (above 20% in both) than in the United States and Singapore (both below 10%). The study also revealed small percentages of Speeded or Very speeded students (low to medium sum score; long RT at first but very short RT toward the end) in England, the United Arab Emirates, and the United States (1%, 5%, and 6%, respectively) but not in Singapore. A unique class of Efficient and high-performing students (high sum score; short RT) was identified only in Singapore (24%). This study demonstrated that mixture modeling is a useful technique for identifying distinct test-taking behaviors and highlighted the presence and extent of unfavorable test-taking behaviors within each selected country using data from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment.https://doi.org/10.1186/s40536-023-00179-3Process dataLog dataFinite mixture modelingDisengagementRapid-guessingSpeededness |
| spellingShingle | Xiaying Zheng Fusun Sahin Ebru Erberber Frank Fonseca Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries Large-scale Assessments in Education Process data Log data Finite mixture modeling Disengagement Rapid-guessing Speededness |
| title | Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries |
| title_full | Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries |
| title_fullStr | Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries |
| title_full_unstemmed | Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries |
| title_short | Identification and cross-country comparison of students’ test-taking behaviors in selected eTIMSS 2019 countries |
| title_sort | identification and cross country comparison of students test taking behaviors in selected etimss 2019 countries |
| topic | Process data Log data Finite mixture modeling Disengagement Rapid-guessing Speededness |
| url | https://doi.org/10.1186/s40536-023-00179-3 |
| work_keys_str_mv | AT xiayingzheng identificationandcrosscountrycomparisonofstudentstesttakingbehaviorsinselectedetimss2019countries AT fusunsahin identificationandcrosscountrycomparisonofstudentstesttakingbehaviorsinselectedetimss2019countries AT ebruerberber identificationandcrosscountrycomparisonofstudentstesttakingbehaviorsinselectedetimss2019countries AT frankfonseca identificationandcrosscountrycomparisonofstudentstesttakingbehaviorsinselectedetimss2019countries |