Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions
Purpose This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: in...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Korea Health Personnel Licensing Examination Institute
2020-04-01
|
| Series: | Journal of Educational Evaluation for Health Professions |
| Subjects: | |
| Online Access: | http://www.jeehp.org/upload/pdf/jeehp-17-12.pdf |
| _version_ | 1827246783711936512 |
|---|---|
| author | Jimmy Bourque Haley Skinner Jonathan Dupré Maria Bacchus Martha Ainslie Irene W. Y. Ma Gary Cole |
| author_facet | Jimmy Bourque Haley Skinner Jonathan Dupré Maria Bacchus Martha Ainslie Irene W. Y. Ma Gary Cole |
| author_sort | Jimmy Bourque |
| collection | DOAJ |
| description | Purpose This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: inter-rater agreement, the correlations between Ebel scores and item facility indices, the impact of raters’ knowledge of correct answers on the Ebel score, and the effects of raters’ specialty on inter-rater agreement and Ebel scores. Methods Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. The Ebel method was applied to 203 multiple-choice questions by 49 raters. Facility indices came from 194 candidates. We computed the Fleiss kappa and the Pearson correlations between Ebel scores and item facility indices. We investigated differences in the Ebel score according to whether correct answers were provided or not and differences between internists and other specialists using the t-test. Results The Fleiss kappa was below 0.15 for both facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same whether the correct answers were provided or not. Inter-rater agreement and Ebel scores were not significantly different between internists and other specialists. Conclusion Inter-rater agreement and correlations between item Ebel scores and facility indices were consistently low; furthermore, raters’ knowledge of the correct answers and raters’ specialty had no effect on Ebel scores in the present setting. |
| first_indexed | 2024-03-12T19:51:47Z |
| format | Article |
| id | doaj.art-2c3b9eaaf40e4c56867fef34e24f2a1e |
| institution | Directory Open Access Journal |
| issn | 1975-5937 |
| language | English |
| last_indexed | 2025-03-21T23:05:27Z |
| publishDate | 2020-04-01 |
| publisher | Korea Health Personnel Licensing Examination Institute |
| record_format | Article |
| series | Journal of Educational Evaluation for Health Professions |
| spelling | doaj.art-2c3b9eaaf40e4c56867fef34e24f2a1e2024-05-22T06:57:37ZengKorea Health Personnel Licensing Examination InstituteJournal of Educational Evaluation for Health Professions1975-59372020-04-011710.3352/jeehp.2020.17.12360Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questionsJimmy Bourque0Haley Skinner1Jonathan Dupré2Maria Bacchus3Martha Ainslie4Irene W. Y. Ma5Gary Cole6Exam Quality and Analytics Unit, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CanadaExam Quality and Analytics Unit, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CanadaExam Quality and Analytics Unit, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CanadaDepartment of Medicine, University of Calgary, Calgary, AB, CanadaDepartment of Medicine, University of Manitoba, Winnipeg, MB, CanadaDepartment of Medicine, University of Calgary, Calgary, AB, CanadaExam Quality and Analytics Unit, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CanadaPurpose This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: inter-rater agreement, the correlations between Ebel scores and item facility indices, the impact of raters’ knowledge of correct answers on the Ebel score, and the effects of raters’ specialty on inter-rater agreement and Ebel scores. Methods Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. The Ebel method was applied to 203 multiple-choice questions by 49 raters. Facility indices came from 194 candidates. We computed the Fleiss kappa and the Pearson correlations between Ebel scores and item facility indices. We investigated differences in the Ebel score according to whether correct answers were provided or not and differences between internists and other specialists using the t-test. Results The Fleiss kappa was below 0.15 for both facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same whether the correct answers were provided or not. Inter-rater agreement and Ebel scores were not significantly different between internists and other specialists. Conclusion Inter-rater agreement and correlations between item Ebel scores and facility indices were consistently low; furthermore, raters’ knowledge of the correct answers and raters’ specialty had no effect on Ebel scores in the present setting.http://www.jeehp.org/upload/pdf/jeehp-17-12.pdfcanadacertificationmedicinespecializationstandard-setting |
| spellingShingle | Jimmy Bourque Haley Skinner Jonathan Dupré Maria Bacchus Martha Ainslie Irene W. Y. Ma Gary Cole Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions Journal of Educational Evaluation for Health Professions canada certification medicine specialization standard-setting |
| title | Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions |
| title_full | Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions |
| title_fullStr | Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions |
| title_full_unstemmed | Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions |
| title_short | Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions |
| title_sort | performance of the ebel standard setting method for the spring 2019 royal college of physicians and surgeons of canada internal medicine certification examination consisting of multiple choice questions |
| topic | canada certification medicine specialization standard-setting |
| url | http://www.jeehp.org/upload/pdf/jeehp-17-12.pdf |
| work_keys_str_mv | AT jimmybourque performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions AT haleyskinner performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions AT jonathandupre performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions AT mariabacchus performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions AT marthaainslie performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions AT irenewyma performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions AT garycole performanceoftheebelstandardsettingmethodforthespring2019royalcollegeofphysiciansandsurgeonsofcanadainternalmedicinecertificationexaminationconsistingofmultiplechoicequestions |