Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests
Abstract In low-stakes assessment settings, students’ performance is not only influenced by students’ ability level but also their test-taking engagement. In computerized adaptive tests (CATs), disengaged responses (e.g., rapid guesses) that fail to reflect students’ true ability levels may lead to...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-07-01
|
| Series: | Large-scale Assessments in Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40536-023-00177-5 |
| _version_ | 1797774026555260928 |
|---|---|
| author | Guher Gorgun Okan Bulut |
| author_facet | Guher Gorgun Okan Bulut |
| author_sort | Guher Gorgun |
| collection | DOAJ |
| description | Abstract In low-stakes assessment settings, students’ performance is not only influenced by students’ ability level but also their test-taking engagement. In computerized adaptive tests (CATs), disengaged responses (e.g., rapid guesses) that fail to reflect students’ true ability levels may lead to the selection of less informative items and thereby contaminate item selection and ability estimation procedures. To date, researchers have developed various approaches to detect and remove disengaged responses after test administration is completed to alleviate the negative impact of low test-taking engagement on test scores. This study proposes an alternative item selection method based on Maximum Fisher Information (MFI) that considers test-taking engagement as a secondary latent trait to select the most optimal items based on both ability and engagement. The results of post-hoc simulation studies indicated that the proposed method could optimize item selection and improve the accuracy of final ability estimates, especially for low-ability students. Overall, the proposed method showed great promise for tailoring CATs based on test-taking engagement. Practitioners are encouraged to consider incorporating engagement into the item selection algorithm to enhance the validity of inferences made from low-stakes CATs. |
| first_indexed | 2024-03-12T22:14:58Z |
| format | Article |
| id | doaj.art-74793c77e0934ff9938f37cb2547fa04 |
| institution | Directory Open Access Journal |
| issn | 2196-0739 |
| language | English |
| last_indexed | 2024-03-12T22:14:58Z |
| publishDate | 2023-07-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Large-scale Assessments in Education |
| spelling | doaj.art-74793c77e0934ff9938f37cb2547fa042023-07-23T11:21:01ZengSpringerOpenLarge-scale Assessments in Education2196-07392023-07-0111112110.1186/s40536-023-00177-5Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive testsGuher Gorgun0Okan Bulut1Measurement, Evaluation, and Data Science, Faculty of Education, University of AlbertaCentre for Research in Applied Measurement and Evaluation, Faculty of Education, University of AlbertaAbstract In low-stakes assessment settings, students’ performance is not only influenced by students’ ability level but also their test-taking engagement. In computerized adaptive tests (CATs), disengaged responses (e.g., rapid guesses) that fail to reflect students’ true ability levels may lead to the selection of less informative items and thereby contaminate item selection and ability estimation procedures. To date, researchers have developed various approaches to detect and remove disengaged responses after test administration is completed to alleviate the negative impact of low test-taking engagement on test scores. This study proposes an alternative item selection method based on Maximum Fisher Information (MFI) that considers test-taking engagement as a secondary latent trait to select the most optimal items based on both ability and engagement. The results of post-hoc simulation studies indicated that the proposed method could optimize item selection and improve the accuracy of final ability estimates, especially for low-ability students. Overall, the proposed method showed great promise for tailoring CATs based on test-taking engagement. Practitioners are encouraged to consider incorporating engagement into the item selection algorithm to enhance the validity of inferences made from low-stakes CATs.https://doi.org/10.1186/s40536-023-00177-5CATLow-stakesTest-taking engagementResponse timeMaximum fisher information |
| spellingShingle | Guher Gorgun Okan Bulut Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests Large-scale Assessments in Education CAT Low-stakes Test-taking engagement Response time Maximum fisher information |
| title | Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests |
| title_full | Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests |
| title_fullStr | Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests |
| title_full_unstemmed | Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests |
| title_short | Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests |
| title_sort | incorporating test taking engagement into the item selection algorithm in low stakes computerized adaptive tests |
| topic | CAT Low-stakes Test-taking engagement Response time Maximum fisher information |
| url | https://doi.org/10.1186/s40536-023-00177-5 |
| work_keys_str_mv | AT guhergorgun incorporatingtesttakingengagementintotheitemselectionalgorithminlowstakescomputerizedadaptivetests AT okanbulut incorporatingtesttakingengagementintotheitemselectionalgorithminlowstakescomputerizedadaptivetests |