Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA)
Abstract Background Ensuring equivalence of examiners’ judgements across different groups of examiners is a priority for large scale performance assessments in clinical education, both to enhance fairness and reassure the public. This study extends insight into an innovation called Video-based Exami...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-01-01
|
| Series: | BMC Medical Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12909-022-03115-1 |
| _version_ | 1818953789559799808 |
|---|---|
| author | Peter Yeates Gareth McCray Alice Moult Natalie Cope Richard Fuller Robert McKinley |
| author_facet | Peter Yeates Gareth McCray Alice Moult Natalie Cope Richard Fuller Robert McKinley |
| author_sort | Peter Yeates |
| collection | DOAJ |
| description | Abstract Background Ensuring equivalence of examiners’ judgements across different groups of examiners is a priority for large scale performance assessments in clinical education, both to enhance fairness and reassure the public. This study extends insight into an innovation called Video-based Examiner Score Comparison and Adjustment (VESCA) which uses video scoring to link otherwise unlinked groups of examiners. This linkage enables comparison of the influence of different examiner-groups within a common frame of reference and provision of adjusted “fair” scores to students. Whilst this innovation promises substantial benefit to quality assurance of distributed Objective Structured Clinical Exams (OSCEs), questions remain about how the resulting score adjustments might be influenced by the specific parameters used to operationalise VESCA. Research questions, How similar are estimates of students’ score adjustments when the model is run with either: fewer comparison videos per participating examiner?; reduced numbers of participating examiners? Methods Using secondary analysis of recent research which used VESCA to compare scoring tendencies of different examiner groups, we made numerous copies of the original data then selectively deleted video scores to reduce the number of 1/ linking videos per examiner (4 versus several permutations of 3,2,or 1 videos) or 2/examiner participation rates (all participating examiners (76%) versus several permutations of 70%, 60% or 50% participation). After analysing all resulting datasets with Many Facet Rasch Modelling (MFRM) we calculated students’ score adjustments for each dataset and compared these with score adjustments in the original data using Spearman’s correlations. Results Students’ score adjustments derived form 3 videos per examiner correlated highly with score adjustments derived from 4 linking videos (median Rho = 0.93,IQR0.90–0.95,p < 0.001), with 2 (median Rho 0.85,IQR0.81–0.87,p < 0.001) and 1 linking videos (median Rho = 0.52(IQR0.46–0.64,p < 0.001) producing progressively smaller correlations. Score adjustments were similar for 76% participating examiners and 70% (median Rho = 0.97,IQR0.95–0.98,p < 0.001), and 60% (median Rho = 0.95,IQR0.94–0.98,p < 0.001) participation, but were lower and more variable for 50% examiner participation (median Rho = 0.78,IQR0.65–0.83, some ns). Conclusions Whilst VESCA showed some sensitivity to the examined parameters, modest reductions in examiner participation rates or video numbers produced highly similar results. Employing VESCA in distributed or national exams could enhance quality assurance or exam fairness. |
| first_indexed | 2024-12-20T10:11:51Z |
| format | Article |
| id | doaj.art-19061dbb84ca46d7bbee447cec118b50 |
| institution | Directory Open Access Journal |
| issn | 1472-6920 |
| language | English |
| last_indexed | 2024-12-20T10:11:51Z |
| publishDate | 2022-01-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Education |
| spelling | doaj.art-19061dbb84ca46d7bbee447cec118b502022-12-21T19:44:08ZengBMCBMC Medical Education1472-69202022-01-012211910.1186/s12909-022-03115-1Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA)Peter Yeates0Gareth McCray1Alice Moult2Natalie Cope3Richard Fuller4Robert McKinley5School of Medicine, David Weatherall Building, Keele UniversitySchool of Medicine, David Weatherall Building, Keele UniversitySchool of Medicine, David Weatherall Building, Keele UniversitySchool of Medicine, David Weatherall Building, Keele UniversityChristie Education, Christie Hospitals NHS Foundation TrustSchool of Medicine, David Weatherall Building, Keele UniversityAbstract Background Ensuring equivalence of examiners’ judgements across different groups of examiners is a priority for large scale performance assessments in clinical education, both to enhance fairness and reassure the public. This study extends insight into an innovation called Video-based Examiner Score Comparison and Adjustment (VESCA) which uses video scoring to link otherwise unlinked groups of examiners. This linkage enables comparison of the influence of different examiner-groups within a common frame of reference and provision of adjusted “fair” scores to students. Whilst this innovation promises substantial benefit to quality assurance of distributed Objective Structured Clinical Exams (OSCEs), questions remain about how the resulting score adjustments might be influenced by the specific parameters used to operationalise VESCA. Research questions, How similar are estimates of students’ score adjustments when the model is run with either: fewer comparison videos per participating examiner?; reduced numbers of participating examiners? Methods Using secondary analysis of recent research which used VESCA to compare scoring tendencies of different examiner groups, we made numerous copies of the original data then selectively deleted video scores to reduce the number of 1/ linking videos per examiner (4 versus several permutations of 3,2,or 1 videos) or 2/examiner participation rates (all participating examiners (76%) versus several permutations of 70%, 60% or 50% participation). After analysing all resulting datasets with Many Facet Rasch Modelling (MFRM) we calculated students’ score adjustments for each dataset and compared these with score adjustments in the original data using Spearman’s correlations. Results Students’ score adjustments derived form 3 videos per examiner correlated highly with score adjustments derived from 4 linking videos (median Rho = 0.93,IQR0.90–0.95,p < 0.001), with 2 (median Rho 0.85,IQR0.81–0.87,p < 0.001) and 1 linking videos (median Rho = 0.52(IQR0.46–0.64,p < 0.001) producing progressively smaller correlations. Score adjustments were similar for 76% participating examiners and 70% (median Rho = 0.97,IQR0.95–0.98,p < 0.001), and 60% (median Rho = 0.95,IQR0.94–0.98,p < 0.001) participation, but were lower and more variable for 50% examiner participation (median Rho = 0.78,IQR0.65–0.83, some ns). Conclusions Whilst VESCA showed some sensitivity to the examined parameters, modest reductions in examiner participation rates or video numbers produced highly similar results. Employing VESCA in distributed or national exams could enhance quality assurance or exam fairness.https://doi.org/10.1186/s12909-022-03115-1AssessmentObjective Structured Clinical ExamsAssessor variabilityDistributed assessmentsTest EquatingMany Facet Rasch Modelling |
| spellingShingle | Peter Yeates Gareth McCray Alice Moult Natalie Cope Richard Fuller Robert McKinley Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA) BMC Medical Education Assessment Objective Structured Clinical Exams Assessor variability Distributed assessments Test Equating Many Facet Rasch Modelling |
| title | Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA) |
| title_full | Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA) |
| title_fullStr | Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA) |
| title_full_unstemmed | Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA) |
| title_short | Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA) |
| title_sort | determining the influence of different linking patterns on the stability of students score adjustments produced using video based examiner score comparison and adjustment vesca |
| topic | Assessment Objective Structured Clinical Exams Assessor variability Distributed assessments Test Equating Many Facet Rasch Modelling |
| url | https://doi.org/10.1186/s12909-022-03115-1 |
| work_keys_str_mv | AT peteryeates determiningtheinfluenceofdifferentlinkingpatternsonthestabilityofstudentsscoreadjustmentsproducedusingvideobasedexaminerscorecomparisonandadjustmentvesca AT garethmccray determiningtheinfluenceofdifferentlinkingpatternsonthestabilityofstudentsscoreadjustmentsproducedusingvideobasedexaminerscorecomparisonandadjustmentvesca AT alicemoult determiningtheinfluenceofdifferentlinkingpatternsonthestabilityofstudentsscoreadjustmentsproducedusingvideobasedexaminerscorecomparisonandadjustmentvesca AT nataliecope determiningtheinfluenceofdifferentlinkingpatternsonthestabilityofstudentsscoreadjustmentsproducedusingvideobasedexaminerscorecomparisonandadjustmentvesca AT richardfuller determiningtheinfluenceofdifferentlinkingpatternsonthestabilityofstudentsscoreadjustmentsproducedusingvideobasedexaminerscorecomparisonandadjustmentvesca AT robertmckinley determiningtheinfluenceofdifferentlinkingpatternsonthestabilityofstudentsscoreadjustmentsproducedusingvideobasedexaminerscorecomparisonandadjustmentvesca |