Effects of DIF in MST routing in ILSAs
Abstract The advance and access to technology have allowed for the implementation of adaptive testing in international large-scale assessments (ILSAs). Multistage testing (MST) in ILSAs offers opportunities and advantages compared to linear testing. However, when dozens of highly heterogeneous syste...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-06-01
|
| Series: | Large-scale Assessments in Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40536-023-00165-9 |
| _version_ | 1797801294910455808 |
|---|---|
| author | Montserrat Valdivia Medinaceli Leslie Rutkowski Dubravka Svetina Valdivia David Rutkowski |
| author_facet | Montserrat Valdivia Medinaceli Leslie Rutkowski Dubravka Svetina Valdivia David Rutkowski |
| author_sort | Montserrat Valdivia Medinaceli |
| collection | DOAJ |
| description | Abstract The advance and access to technology have allowed for the implementation of adaptive testing in international large-scale assessments (ILSAs). Multistage testing (MST) in ILSAs offers opportunities and advantages compared to linear testing. However, when dozens of highly heterogeneous systems participate in ILSAs, the sources of heterogeneity pose challenges to any cross-cultural measurement endeavor. With the recent implementation of adaptive designs in ILSAs, little is known about differential item functioning (DIF) effects, mainly when an MST design is used in an ILSA context. Through a simulation study grounded on the empirical basis of the Trends in International Mathematics and Science Study (TIMSS) data, this paper examines the impact of DIF in MST routing in ILSA under different routing strategies. Results showed that Merit routing is highly accurate even when the amount and magnitude of DIF is high, whereas suboptimal routing showed poor accuracy across DIF conditions. As expected, Merit routing has better proficiency recovery parameters than a suboptimal routing mechanism. Implications and recommendations for test developers are included in the discussion section. |
| first_indexed | 2024-03-13T04:48:17Z |
| format | Article |
| id | doaj.art-38dd0141c84b4d529ca45a486cfccf7d |
| institution | Directory Open Access Journal |
| issn | 2196-0739 |
| language | English |
| last_indexed | 2024-03-13T04:48:17Z |
| publishDate | 2023-06-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Large-scale Assessments in Education |
| spelling | doaj.art-38dd0141c84b4d529ca45a486cfccf7d2023-06-18T11:20:41ZengSpringerOpenLarge-scale Assessments in Education2196-07392023-06-0111112210.1186/s40536-023-00165-9Effects of DIF in MST routing in ILSAsMontserrat Valdivia Medinaceli0Leslie Rutkowski1Dubravka Svetina Valdivia2David Rutkowski3Counseling and Educational Psychology, Indiana University BloomingtonCounseling and Educational Psychology, Indiana University BloomingtonCounseling and Educational Psychology, Indiana University BloomingtonCounseling and Educational Psychology, Indiana University BloomingtonAbstract The advance and access to technology have allowed for the implementation of adaptive testing in international large-scale assessments (ILSAs). Multistage testing (MST) in ILSAs offers opportunities and advantages compared to linear testing. However, when dozens of highly heterogeneous systems participate in ILSAs, the sources of heterogeneity pose challenges to any cross-cultural measurement endeavor. With the recent implementation of adaptive designs in ILSAs, little is known about differential item functioning (DIF) effects, mainly when an MST design is used in an ILSA context. Through a simulation study grounded on the empirical basis of the Trends in International Mathematics and Science Study (TIMSS) data, this paper examines the impact of DIF in MST routing in ILSA under different routing strategies. Results showed that Merit routing is highly accurate even when the amount and magnitude of DIF is high, whereas suboptimal routing showed poor accuracy across DIF conditions. As expected, Merit routing has better proficiency recovery parameters than a suboptimal routing mechanism. Implications and recommendations for test developers are included in the discussion section.https://doi.org/10.1186/s40536-023-00165-9Differential item functioningMultistage testingInternational large-scale assessment |
| spellingShingle | Montserrat Valdivia Medinaceli Leslie Rutkowski Dubravka Svetina Valdivia David Rutkowski Effects of DIF in MST routing in ILSAs Large-scale Assessments in Education Differential item functioning Multistage testing International large-scale assessment |
| title | Effects of DIF in MST routing in ILSAs |
| title_full | Effects of DIF in MST routing in ILSAs |
| title_fullStr | Effects of DIF in MST routing in ILSAs |
| title_full_unstemmed | Effects of DIF in MST routing in ILSAs |
| title_short | Effects of DIF in MST routing in ILSAs |
| title_sort | effects of dif in mst routing in ilsas |
| topic | Differential item functioning Multistage testing International large-scale assessment |
| url | https://doi.org/10.1186/s40536-023-00165-9 |
| work_keys_str_mv | AT montserratvaldiviamedinaceli effectsofdifinmstroutinginilsas AT leslierutkowski effectsofdifinmstroutinginilsas AT dubravkasvetinavaldivia effectsofdifinmstroutinginilsas AT davidrutkowski effectsofdifinmstroutinginilsas |