Impact of differential item functioning on group score reporting in the context of large-scale assessments
Abstract We investigated the potential impact of differential item functioning (DIF) on group-level mean and standard deviation estimates using empirical and simulated data in the context of large-scale assessment. For the empirical investigation, PISA 2018 cognitive domains (Reading, Mathematics, a...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2022-11-01
|
| Series: | Large-scale Assessments in Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40536-022-00135-7 |
| _version_ | 1811309229356613632 |
|---|---|
| author | Sean Joo Usama Ali Frederic Robin Hyo Jeong Shin |
| author_facet | Sean Joo Usama Ali Frederic Robin Hyo Jeong Shin |
| author_sort | Sean Joo |
| collection | DOAJ |
| description | Abstract We investigated the potential impact of differential item functioning (DIF) on group-level mean and standard deviation estimates using empirical and simulated data in the context of large-scale assessment. For the empirical investigation, PISA 2018 cognitive domains (Reading, Mathematics, and Science) data were analyzed using Jackknife sampling to explore the impact of DIF on the country scores and their standard errors. We found that the countries that have a large number of DIF items tend to increase the difference of the country scores computed with and without the DIF adjustment. In addition, standard errors of the country score differences also increased with the number of DIF items. For the simulation study, we evaluated bias and root mean squared error (RMSE) of the group mean and standard deviation estimates using the multigroup item response theory (IRT) model to explore the extent to which DIF items create a bias of the group mean scores and how effectively the DIF adjustment corrects the bias under various conditions. We found that the DIF adjustment reduced the bias by 50% on average. The implications and limitations of the study are further discussed. |
| first_indexed | 2024-04-13T09:38:33Z |
| format | Article |
| id | doaj.art-1b45a72472214bd1840762792a218c8a |
| institution | Directory Open Access Journal |
| issn | 2196-0739 |
| language | English |
| last_indexed | 2024-04-13T09:38:33Z |
| publishDate | 2022-11-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Large-scale Assessments in Education |
| spelling | doaj.art-1b45a72472214bd1840762792a218c8a2022-12-22T02:52:01ZengSpringerOpenLarge-scale Assessments in Education2196-07392022-11-0110112110.1186/s40536-022-00135-7Impact of differential item functioning on group score reporting in the context of large-scale assessmentsSean Joo0Usama Ali1Frederic Robin2Hyo Jeong Shin3University of KansasEducational Testing ServiceEducational Testing ServiceSogang UniversityAbstract We investigated the potential impact of differential item functioning (DIF) on group-level mean and standard deviation estimates using empirical and simulated data in the context of large-scale assessment. For the empirical investigation, PISA 2018 cognitive domains (Reading, Mathematics, and Science) data were analyzed using Jackknife sampling to explore the impact of DIF on the country scores and their standard errors. We found that the countries that have a large number of DIF items tend to increase the difference of the country scores computed with and without the DIF adjustment. In addition, standard errors of the country score differences also increased with the number of DIF items. For the simulation study, we evaluated bias and root mean squared error (RMSE) of the group mean and standard deviation estimates using the multigroup item response theory (IRT) model to explore the extent to which DIF items create a bias of the group mean scores and how effectively the DIF adjustment corrects the bias under various conditions. We found that the DIF adjustment reduced the bias by 50% on average. The implications and limitations of the study are further discussed.https://doi.org/10.1186/s40536-022-00135-7Large-scale assessmentProgramme for International Student AssessmentDifferential item functioningGroup score reportingJackknife sampling |
| spellingShingle | Sean Joo Usama Ali Frederic Robin Hyo Jeong Shin Impact of differential item functioning on group score reporting in the context of large-scale assessments Large-scale Assessments in Education Large-scale assessment Programme for International Student Assessment Differential item functioning Group score reporting Jackknife sampling |
| title | Impact of differential item functioning on group score reporting in the context of large-scale assessments |
| title_full | Impact of differential item functioning on group score reporting in the context of large-scale assessments |
| title_fullStr | Impact of differential item functioning on group score reporting in the context of large-scale assessments |
| title_full_unstemmed | Impact of differential item functioning on group score reporting in the context of large-scale assessments |
| title_short | Impact of differential item functioning on group score reporting in the context of large-scale assessments |
| title_sort | impact of differential item functioning on group score reporting in the context of large scale assessments |
| topic | Large-scale assessment Programme for International Student Assessment Differential item functioning Group score reporting Jackknife sampling |
| url | https://doi.org/10.1186/s40536-022-00135-7 |
| work_keys_str_mv | AT seanjoo impactofdifferentialitemfunctioningongroupscorereportinginthecontextoflargescaleassessments AT usamaali impactofdifferentialitemfunctioningongroupscorereportinginthecontextoflargescaleassessments AT fredericrobin impactofdifferentialitemfunctioningongroupscorereportinginthecontextoflargescaleassessments AT hyojeongshin impactofdifferentialitemfunctioningongroupscorereportinginthecontextoflargescaleassessments |