Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods
BackgroundScoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-03-01
|
| Series: | JMIR Medical Education |
| Online Access: | https://mededu.jmir.org/2023/1/e43792 |
| _version_ | 1797734246833455104 |
|---|---|
| author | Philipp Kanzow Dennis Schmidt Manfred Herrmann Torsten Wassmann Annette Wiegand Tobias Raupach |
| author_facet | Philipp Kanzow Dennis Schmidt Manfred Herrmann Torsten Wassmann Annette Wiegand Tobias Raupach |
| author_sort | Philipp Kanzow |
| collection | DOAJ |
| description |
BackgroundScoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data.
ObjectiveThis study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6).
MethodsItems and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05).
ResultsWe analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977).
ConclusionsEducators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees’ scores and overall examination results. |
| first_indexed | 2024-03-12T12:41:32Z |
| format | Article |
| id | doaj.art-8834317bb9744ed9b2917bdc25b4f293 |
| institution | Directory Open Access Journal |
| issn | 2369-3762 |
| language | English |
| last_indexed | 2024-03-12T12:41:32Z |
| publishDate | 2023-03-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Education |
| spelling | doaj.art-8834317bb9744ed9b2917bdc25b4f2932023-08-28T23:48:53ZengJMIR PublicationsJMIR Medical Education2369-37622023-03-019e4379210.2196/43792Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring MethodsPhilipp Kanzowhttps://orcid.org/0000-0002-2169-561XDennis Schmidthttps://orcid.org/0000-0002-9828-5299Manfred Herrmannhttps://orcid.org/0000-0003-2637-2198Torsten Wassmannhttps://orcid.org/0000-0002-7041-8774Annette Wiegandhttps://orcid.org/0000-0001-5640-8775Tobias Raupachhttps://orcid.org/0000-0003-2555-8097 BackgroundScoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data. ObjectiveThis study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6). MethodsItems and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05). ResultsWe analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977). ConclusionsEducators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees’ scores and overall examination results.https://mededu.jmir.org/2023/1/e43792 |
| spellingShingle | Philipp Kanzow Dennis Schmidt Manfred Herrmann Torsten Wassmann Annette Wiegand Tobias Raupach Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods JMIR Medical Education |
| title | Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods |
| title_full | Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods |
| title_fullStr | Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods |
| title_full_unstemmed | Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods |
| title_short | Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods |
| title_sort | use of multiple select multiple choice items in a dental undergraduate curriculum retrospective study involving the application of different scoring methods |
| url | https://mededu.jmir.org/2023/1/e43792 |
| work_keys_str_mv | AT philippkanzow useofmultipleselectmultiplechoiceitemsinadentalundergraduatecurriculumretrospectivestudyinvolvingtheapplicationofdifferentscoringmethods AT dennisschmidt useofmultipleselectmultiplechoiceitemsinadentalundergraduatecurriculumretrospectivestudyinvolvingtheapplicationofdifferentscoringmethods AT manfredherrmann useofmultipleselectmultiplechoiceitemsinadentalundergraduatecurriculumretrospectivestudyinvolvingtheapplicationofdifferentscoringmethods AT torstenwassmann useofmultipleselectmultiplechoiceitemsinadentalundergraduatecurriculumretrospectivestudyinvolvingtheapplicationofdifferentscoringmethods AT annettewiegand useofmultipleselectmultiplechoiceitemsinadentalundergraduatecurriculumretrospectivestudyinvolvingtheapplicationofdifferentscoringmethods AT tobiasraupach useofmultipleselectmultiplechoiceitemsinadentalundergraduatecurriculumretrospectivestudyinvolvingtheapplicationofdifferentscoringmethods |