Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School
Background: OSCE are widely used for assessing clinical skills training in medical schools. Use of traditional pass fail cut off yields wide variations in the results of different cohorts of students. This has led to a growing emphasis on the application of standard setting procedures in OSCEs. Purp...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publishing
2020-12-01
|
| Series: | Journal of Medical Education and Curricular Development |
| Online Access: | https://doi.org/10.1177/2382120520981992 |
| _version_ | 1797375362179530752 |
|---|---|
| author | Neelam Rekha Dwivedi Narasimha Prasad Vijayashankar Manisha Hansda Arun Kumar Dubey Fidelis Nwachukwu Vernon Curran Joseph Jillwin |
| author_facet | Neelam Rekha Dwivedi Narasimha Prasad Vijayashankar Manisha Hansda Arun Kumar Dubey Fidelis Nwachukwu Vernon Curran Joseph Jillwin |
| author_sort | Neelam Rekha Dwivedi |
| collection | DOAJ |
| description | Background: OSCE are widely used for assessing clinical skills training in medical schools. Use of traditional pass fail cut off yields wide variations in the results of different cohorts of students. This has led to a growing emphasis on the application of standard setting procedures in OSCEs. Purpose/aim: The purpose of the study was comparing the utility, feasibility and appropriateness of 4 different standard setting methods with OSCEs at XUSOM. Methods: A 15-station OSCE was administered to 173 students over 6 months. Five stations were conducted for each organ system (Respiratory, Gastrointestinal and Cardiovascular). Students were assessed for their clinical skills in 15 stations. Four different standard setting methods were applied and compared with a control (Traditional method) to establish cut off scores for pass/fail decisions. Results: OSCE checklist scores revealed a Cronbach’s alpha of 0.711, demonstrating acceptable level of internal consistency. About 13 of 15 OSCE stations performed well with “Alpha if deleted values” lower that 0.711 emphasizing the reliability of OSCE stations. The traditional standard setting method (cut off score of 70) resulted in highest failure rate. The Modified Angoff Method and Relative methods yielded the lowest failure rates, which were typically less than 10% for each system. Failure rates for the Borderline methods ranged from 28% to 57% across systems. Conclusions: In our study, Modified Angoff method and Borderline regression method have shown to be consistently reliable and practically suitable to provide acceptable cut-off score across different organ system. Therefore, an average of Modified Angoff Method and Borderline Regression Method appeared to provide an acceptable cutoff score in OSCE. Further studies, in high-stake clinical examinations, utilizing larger number of judges and OSCE stations are recommended to reinforce the validity of combining multiple methods for standard setting. |
| first_indexed | 2024-03-08T19:23:13Z |
| format | Article |
| id | doaj.art-238e0f88aeea40e1851175920de74b61 |
| institution | Directory Open Access Journal |
| issn | 2382-1205 |
| language | English |
| last_indexed | 2024-03-08T19:23:13Z |
| publishDate | 2020-12-01 |
| publisher | SAGE Publishing |
| record_format | Article |
| series | Journal of Medical Education and Curricular Development |
| spelling | doaj.art-238e0f88aeea40e1851175920de74b612023-12-26T13:03:20ZengSAGE PublishingJournal of Medical Education and Curricular Development2382-12052020-12-01710.1177/2382120520981992Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical SchoolNeelam Rekha Dwivedi0Narasimha Prasad Vijayashankar1Manisha Hansda2Arun Kumar Dubey3Fidelis Nwachukwu4Vernon Curran5Joseph Jillwin6Xavier University School of Medicine, Oranjestad, ArubaXavier University School of Medicine, Oranjestad, ArubaXavier University School of Medicine, Oranjestad, ArubaXavier University School of Medicine, Oranjestad, ArubaXavier University School of Medicine, Oranjestad, ArubaFaculty of Medicine, Memorial University of Newfoundland, St. John’s, NL, CanadaXavier University School of Medicine, Oranjestad, ArubaBackground: OSCE are widely used for assessing clinical skills training in medical schools. Use of traditional pass fail cut off yields wide variations in the results of different cohorts of students. This has led to a growing emphasis on the application of standard setting procedures in OSCEs. Purpose/aim: The purpose of the study was comparing the utility, feasibility and appropriateness of 4 different standard setting methods with OSCEs at XUSOM. Methods: A 15-station OSCE was administered to 173 students over 6 months. Five stations were conducted for each organ system (Respiratory, Gastrointestinal and Cardiovascular). Students were assessed for their clinical skills in 15 stations. Four different standard setting methods were applied and compared with a control (Traditional method) to establish cut off scores for pass/fail decisions. Results: OSCE checklist scores revealed a Cronbach’s alpha of 0.711, demonstrating acceptable level of internal consistency. About 13 of 15 OSCE stations performed well with “Alpha if deleted values” lower that 0.711 emphasizing the reliability of OSCE stations. The traditional standard setting method (cut off score of 70) resulted in highest failure rate. The Modified Angoff Method and Relative methods yielded the lowest failure rates, which were typically less than 10% for each system. Failure rates for the Borderline methods ranged from 28% to 57% across systems. Conclusions: In our study, Modified Angoff method and Borderline regression method have shown to be consistently reliable and practically suitable to provide acceptable cut-off score across different organ system. Therefore, an average of Modified Angoff Method and Borderline Regression Method appeared to provide an acceptable cutoff score in OSCE. Further studies, in high-stake clinical examinations, utilizing larger number of judges and OSCE stations are recommended to reinforce the validity of combining multiple methods for standard setting.https://doi.org/10.1177/2382120520981992 |
| spellingShingle | Neelam Rekha Dwivedi Narasimha Prasad Vijayashankar Manisha Hansda Arun Kumar Dubey Fidelis Nwachukwu Vernon Curran Joseph Jillwin Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School Journal of Medical Education and Curricular Development |
| title | Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School |
| title_full | Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School |
| title_fullStr | Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School |
| title_full_unstemmed | Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School |
| title_short | Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School |
| title_sort | comparing standard setting methods for objective structured clinical examinations in a caribbean medical school |
| url | https://doi.org/10.1177/2382120520981992 |
| work_keys_str_mv | AT neelamrekhadwivedi comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool AT narasimhaprasadvijayashankar comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool AT manishahansda comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool AT arunkumardubey comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool AT fidelisnwachukwu comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool AT vernoncurran comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool AT josephjillwin comparingstandardsettingmethodsforobjectivestructuredclinicalexaminationsinacaribbeanmedicalschool |