Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism
Abstract Background Assessment of emergent, rare or complex medical conditions in Endocrinology and Metabolism (E&M) is an integral component of training. However, data is lacking on how this could be best achieved. The purpose of this study was to develop and administer an Objective Structured...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2021-05-01
|
| Series: | BMC Medical Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12909-021-02653-4 |
| _version_ | 1818597457316020224 |
|---|---|
| author | Stephanie Dizon Janine C Malcolm Jan-Joost Rethans Debra Pugh |
| author_facet | Stephanie Dizon Janine C Malcolm Jan-Joost Rethans Debra Pugh |
| author_sort | Stephanie Dizon |
| collection | DOAJ |
| description | Abstract Background Assessment of emergent, rare or complex medical conditions in Endocrinology and Metabolism (E&M) is an integral component of training. However, data is lacking on how this could be best achieved. The purpose of this study was to develop and administer an Objective Structured Clinical Examination (OSCE) for E&M residents, and to gather validity evidence for its use. Methods A needs assessment survey was distributed to all Canadian E&M Program Directors and recent graduates to determine which topics to include in the OSCE. The top 5 topics were selected using a modified Delphi technique. OSCE cases based on these topics were subsequently developed. Five E&M residents (PGY4-5) and five junior Internal Medicine (IM) residents participated in the OSCE. Performance of E&M and IM residents was compared and results were analyzed using a Generalizability study. Examiners and candidates completed a survey following the OSCE to evaluate their experiences. Results The mean score of IM and E&M residents was 41.7 and 69.3 % (p < 0.001), respectively, with a large effect size (partial η2 = 0.75). Overall reliability of the OSCE was 0.74. Standard setting using a borderline regression method resulted in a pass rate of 100 % of E&M residents and 0 % of IM residents. All residents felt the OSCE had high value for learning as a formative exam. Conclusions The E&M OSCE is a feasible method for assessing emergent, rare and complex medical conditions and this study provides validity evidence to support its use in a competency-based curriculum. |
| first_indexed | 2024-12-16T11:48:06Z |
| format | Article |
| id | doaj.art-9e843c5c11e54e9e9b14703ec7f0dc0d |
| institution | Directory Open Access Journal |
| issn | 1472-6920 |
| language | English |
| last_indexed | 2024-12-16T11:48:06Z |
| publishDate | 2021-05-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Education |
| spelling | doaj.art-9e843c5c11e54e9e9b14703ec7f0dc0d2022-12-21T22:32:47ZengBMCBMC Medical Education1472-69202021-05-012111910.1186/s12909-021-02653-4Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolismStephanie Dizon0Janine C Malcolm1Jan-Joost Rethans2Debra Pugh3Department of Medicine, The Ottawa HospitalDepartment of Medicine, The Ottawa HospitalFaculty of Health, Medicine and Life Sciences, Maastricht UniversityDepartment of Medicine, The Ottawa HospitalAbstract Background Assessment of emergent, rare or complex medical conditions in Endocrinology and Metabolism (E&M) is an integral component of training. However, data is lacking on how this could be best achieved. The purpose of this study was to develop and administer an Objective Structured Clinical Examination (OSCE) for E&M residents, and to gather validity evidence for its use. Methods A needs assessment survey was distributed to all Canadian E&M Program Directors and recent graduates to determine which topics to include in the OSCE. The top 5 topics were selected using a modified Delphi technique. OSCE cases based on these topics were subsequently developed. Five E&M residents (PGY4-5) and five junior Internal Medicine (IM) residents participated in the OSCE. Performance of E&M and IM residents was compared and results were analyzed using a Generalizability study. Examiners and candidates completed a survey following the OSCE to evaluate their experiences. Results The mean score of IM and E&M residents was 41.7 and 69.3 % (p < 0.001), respectively, with a large effect size (partial η2 = 0.75). Overall reliability of the OSCE was 0.74. Standard setting using a borderline regression method resulted in a pass rate of 100 % of E&M residents and 0 % of IM residents. All residents felt the OSCE had high value for learning as a formative exam. Conclusions The E&M OSCE is a feasible method for assessing emergent, rare and complex medical conditions and this study provides validity evidence to support its use in a competency-based curriculum.https://doi.org/10.1186/s12909-021-02653-4OSCEKane’s validity frameworkEndocrinologyCompetency-Based Medical Education |
| spellingShingle | Stephanie Dizon Janine C Malcolm Jan-Joost Rethans Debra Pugh Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism BMC Medical Education OSCE Kane’s validity framework Endocrinology Competency-Based Medical Education |
| title | Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism |
| title_full | Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism |
| title_fullStr | Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism |
| title_full_unstemmed | Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism |
| title_short | Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism |
| title_sort | assessing the validity of an osce developed to assess rare emergent or complex clinical conditions in endocrinology metabolism |
| topic | OSCE Kane’s validity framework Endocrinology Competency-Based Medical Education |
| url | https://doi.org/10.1186/s12909-021-02653-4 |
| work_keys_str_mv | AT stephaniedizon assessingthevalidityofanoscedevelopedtoassessrareemergentorcomplexclinicalconditionsinendocrinologymetabolism AT janinecmalcolm assessingthevalidityofanoscedevelopedtoassessrareemergentorcomplexclinicalconditionsinendocrinologymetabolism AT janjoostrethans assessingthevalidityofanoscedevelopedtoassessrareemergentorcomplexclinicalconditionsinendocrinologymetabolism AT debrapugh assessingthevalidityofanoscedevelopedtoassessrareemergentorcomplexclinicalconditionsinendocrinologymetabolism |