The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination
Purpose Diagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rare...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Korea Health Personnel Licensing Examination Institute
2021-07-01
|
| Series: | Journal of Educational Evaluation for Health Professions |
| Subjects: | |
| Online Access: | http://www.jeehp.org/upload/jeehp-18-15.pdf |
| _version_ | 1797763569389928448 |
|---|---|
| author | Dong Gi Seo Jae Kum Kim |
| author_facet | Dong Gi Seo Jae Kum Kim |
| author_sort | Dong Gi Seo |
| collection | DOAJ |
| description | Purpose Diagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rarely been reported compared with classical test theory (CTT) and item response theory models. This paper compared the accuracy of high-level attribute mastery between deterministic inputs, noisy “and” gate (DINA) and Rasch models, along with sub-scores based on CTT. Methods First, a simulation study explored the effects of attribute length (number of items per attribute) and the correlations among attributes with respect to the accuracy of mastery. Second, a real-data study examined model and item fit and investigated the consistency of mastery for each attribute among the 3 models using the 2017 Korean Medical Licensing Examination with 360 items. Results Accuracy of mastery increased with a higher number of items measuring each attribute across all conditions. The DINA model was more accurate than the CTT and Rasch models for attributes with high correlations (>0.5) and few items. In the real-data analysis, the DINA and Rasch models generally showed better item fits and appropriate model fit. The consistency of mastery between the Rasch and DINA models ranged from 0.541 to 0.633 and the correlations of person attribute scores between the Rasch and DINA models ranged from 0.579 to 0.786. Conclusion Although all 3 models provide a mastery decision for each examinee, the individual mastery profile using the DINA model provides more accurate decisions for attributes with high correlations than the CTT and Rasch models. The DINA model can also be directly applied to tests with complex structures, unlike the CTT and Rasch models, and it provides different diagnostic information from the CTT and Rasch models. |
| first_indexed | 2024-03-12T19:43:20Z |
| format | Article |
| id | doaj.art-7cfd4ce730ae48b19894af726db5a1bf |
| institution | Directory Open Access Journal |
| issn | 1975-5937 |
| language | English |
| last_indexed | 2024-03-12T19:43:20Z |
| publishDate | 2021-07-01 |
| publisher | Korea Health Personnel Licensing Examination Institute |
| record_format | Article |
| series | Journal of Educational Evaluation for Health Professions |
| spelling | doaj.art-7cfd4ce730ae48b19894af726db5a1bf2023-08-02T03:41:27ZengKorea Health Personnel Licensing Examination InstituteJournal of Educational Evaluation for Health Professions1975-59372021-07-011810.3352/jeehp.2021.18.15407The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing ExaminationDong Gi Seo0Jae Kum Kim1Department of Psychology, College of Social Science, Hallym University, Chuncheon, KoreaKorea International University in Ferghana, Ferghana, UzbekistanPurpose Diagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rarely been reported compared with classical test theory (CTT) and item response theory models. This paper compared the accuracy of high-level attribute mastery between deterministic inputs, noisy “and” gate (DINA) and Rasch models, along with sub-scores based on CTT. Methods First, a simulation study explored the effects of attribute length (number of items per attribute) and the correlations among attributes with respect to the accuracy of mastery. Second, a real-data study examined model and item fit and investigated the consistency of mastery for each attribute among the 3 models using the 2017 Korean Medical Licensing Examination with 360 items. Results Accuracy of mastery increased with a higher number of items measuring each attribute across all conditions. The DINA model was more accurate than the CTT and Rasch models for attributes with high correlations (>0.5) and few items. In the real-data analysis, the DINA and Rasch models generally showed better item fits and appropriate model fit. The consistency of mastery between the Rasch and DINA models ranged from 0.541 to 0.633 and the correlations of person attribute scores between the Rasch and DINA models ranged from 0.579 to 0.786. Conclusion Although all 3 models provide a mastery decision for each examinee, the individual mastery profile using the DINA model provides more accurate decisions for attributes with high correlations than the CTT and Rasch models. The DINA model can also be directly applied to tests with complex structures, unlike the CTT and Rasch models, and it provides different diagnostic information from the CTT and Rasch models.http://www.jeehp.org/upload/jeehp-18-15.pdfdata collectiondata analysispsychometricsrepublic of koreastatistical models |
| spellingShingle | Dong Gi Seo Jae Kum Kim The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination Journal of Educational Evaluation for Health Professions data collection data analysis psychometrics republic of korea statistical models |
| title | The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination |
| title_full | The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination |
| title_fullStr | The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination |
| title_full_unstemmed | The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination |
| title_short | The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination |
| title_sort | accuracy and consistency of mastery for each content domain using the rasch and deterministic inputs noisy and gate diagnostic classification models a simulation study and a real world analysis using data from the korean medical licensing examination |
| topic | data collection data analysis psychometrics republic of korea statistical models |
| url | http://www.jeehp.org/upload/jeehp-18-15.pdf |
| work_keys_str_mv | AT donggiseo theaccuracyandconsistencyofmasteryforeachcontentdomainusingtheraschanddeterministicinputsnoisyandgatediagnosticclassificationmodelsasimulationstudyandarealworldanalysisusingdatafromthekoreanmedicallicensingexamination AT jaekumkim theaccuracyandconsistencyofmasteryforeachcontentdomainusingtheraschanddeterministicinputsnoisyandgatediagnosticclassificationmodelsasimulationstudyandarealworldanalysisusingdatafromthekoreanmedicallicensingexamination AT donggiseo accuracyandconsistencyofmasteryforeachcontentdomainusingtheraschanddeterministicinputsnoisyandgatediagnosticclassificationmodelsasimulationstudyandarealworldanalysisusingdatafromthekoreanmedicallicensingexamination AT jaekumkim accuracyandconsistencyofmasteryforeachcontentdomainusingtheraschanddeterministicinputsnoisyandgatediagnosticclassificationmodelsasimulationstudyandarealworldanalysisusingdatafromthekoreanmedicallicensingexamination |