Effects of a rater training on rating accuracy in a physical examination skills assessment
[english] Background: The accuracy and reproducibility of medical skills assessment is generally low. Rater training has little or no effect. Our knowledge in this field, however, relies on studies involving video ratings of overall clinical performances. We hypothesised that a rater training focus...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | deu |
| Published: |
German Medical Science GMS Publishing House
2014-11-01
|
| Series: | GMS Zeitschrift für Medizinische Ausbildung |
| Subjects: | |
| Online Access: | http://www.egms.de/static/en/journals/zma/2014-31/zma000933.shtml |
| _version_ | 1818662637943128064 |
|---|---|
| author | Weitz, Gunther Vinzentius, Christian Twesten, Christoph Lehnert, Hendrik Bonnemeier, Hendrik König, Inke R. |
| author_facet | Weitz, Gunther Vinzentius, Christian Twesten, Christoph Lehnert, Hendrik Bonnemeier, Hendrik König, Inke R. |
| author_sort | Weitz, Gunther |
| collection | DOAJ |
| description | [english] Background: The accuracy and reproducibility of medical skills assessment is generally low. Rater training has little or no effect. Our knowledge in this field, however, relies on studies involving video ratings of overall clinical performances. We hypothesised that a rater training focussing on the frame of reference could improve accuracy in grading the curricular assessment of a highly standardised physical head-to-toe examination.Methods: Twenty-one raters assessed the performance of 242 third-year medical students. Eleven raters had been randomly assigned to undergo a brief frame-of-reference training a few days before the assessment. 218 encounters were successfully recorded on video and re-assessed independently by three additional observers. Accuracy was defined as the concordance between the raters' grade and the median of the observers' grade. After the assessment, both students and raters filled in a questionnaire about their views on the assessment.Results: Rater training did not have a measurable influence on accuracy. However, trained raters rated significantly more stringently than untrained raters, and their overall stringency was closer to the stringency of the observers. The questionnaire indicated a higher awareness of the halo effect in the trained raters group. Although the self-assessment of the students mirrored the assessment of the raters in both groups, the students assessed by trained raters felt more discontent with their grade.Conclusions: While training had some marginal effects, it failed to have an impact on the individual accuracy. These results in real-life encounters are consistent with previous studies on rater training using video assessments of clinical performances. The high degree of standardisation in this study was not suitable to harmonize the trained raters’ grading. The data support the notion that the process of appraising medical performance is highly individual. A frame-of-reference training as applied does not effectively adjust the physicians' judgement on medical students in real-live assessments. |
| first_indexed | 2024-12-17T05:04:07Z |
| format | Article |
| id | doaj.art-b67585edde3143bca4da89bf8db4da7b |
| institution | Directory Open Access Journal |
| issn | 1860-3572 |
| language | deu |
| last_indexed | 2024-12-17T05:04:07Z |
| publishDate | 2014-11-01 |
| publisher | German Medical Science GMS Publishing House |
| record_format | Article |
| series | GMS Zeitschrift für Medizinische Ausbildung |
| spelling | doaj.art-b67585edde3143bca4da89bf8db4da7b2022-12-21T22:02:28ZdeuGerman Medical Science GMS Publishing HouseGMS Zeitschrift für Medizinische Ausbildung1860-35722014-11-01314Doc4110.3205/zma000933Effects of a rater training on rating accuracy in a physical examination skills assessmentWeitz, Gunther0Vinzentius, Christian1Twesten, Christoph2Lehnert, Hendrik3Bonnemeier, Hendrik4König, Inke R.5Universitätsklinikum Schleswig-Holstein, Campus Lübeck, Medizinische Klinik I, Lübeck, DeutschlandInstitut für Qualitätsentwicklung an Schulen Schleswig-Holstein, Kronshagen, DeutschlandUniversitätsklinikum Schleswig-Holstein, Campus Lübeck, Medizinische Klinik I, Lübeck, DeutschlandUniversitätsklinikum Schleswig-Holstein, Campus Lübeck, Medizinische Klinik I, Lübeck, DeutschlandUniversitätsklinikum Schlesweig-Holstein, Campus Kiel, Medizinische Klinik III, Kiel, DeutschlandUniversität zu Lübeck, Institut für Medizinische Biometrie und Statistik, Lübeck, Deutschland[english] Background: The accuracy and reproducibility of medical skills assessment is generally low. Rater training has little or no effect. Our knowledge in this field, however, relies on studies involving video ratings of overall clinical performances. We hypothesised that a rater training focussing on the frame of reference could improve accuracy in grading the curricular assessment of a highly standardised physical head-to-toe examination.Methods: Twenty-one raters assessed the performance of 242 third-year medical students. Eleven raters had been randomly assigned to undergo a brief frame-of-reference training a few days before the assessment. 218 encounters were successfully recorded on video and re-assessed independently by three additional observers. Accuracy was defined as the concordance between the raters' grade and the median of the observers' grade. After the assessment, both students and raters filled in a questionnaire about their views on the assessment.Results: Rater training did not have a measurable influence on accuracy. However, trained raters rated significantly more stringently than untrained raters, and their overall stringency was closer to the stringency of the observers. The questionnaire indicated a higher awareness of the halo effect in the trained raters group. Although the self-assessment of the students mirrored the assessment of the raters in both groups, the students assessed by trained raters felt more discontent with their grade.Conclusions: While training had some marginal effects, it failed to have an impact on the individual accuracy. These results in real-life encounters are consistent with previous studies on rater training using video assessments of clinical performances. The high degree of standardisation in this study was not suitable to harmonize the trained raters’ grading. The data support the notion that the process of appraising medical performance is highly individual. A frame-of-reference training as applied does not effectively adjust the physicians' judgement on medical students in real-live assessments.http://www.egms.de/static/en/journals/zma/2014-31/zma000933.shtmlrater trainingrating accuracyskills assessmentphysical examination skillsrandomised controlled trial |
| spellingShingle | Weitz, Gunther Vinzentius, Christian Twesten, Christoph Lehnert, Hendrik Bonnemeier, Hendrik König, Inke R. Effects of a rater training on rating accuracy in a physical examination skills assessment GMS Zeitschrift für Medizinische Ausbildung rater training rating accuracy skills assessment physical examination skills randomised controlled trial |
| title | Effects of a rater training on rating accuracy in a physical examination skills assessment |
| title_full | Effects of a rater training on rating accuracy in a physical examination skills assessment |
| title_fullStr | Effects of a rater training on rating accuracy in a physical examination skills assessment |
| title_full_unstemmed | Effects of a rater training on rating accuracy in a physical examination skills assessment |
| title_short | Effects of a rater training on rating accuracy in a physical examination skills assessment |
| title_sort | effects of a rater training on rating accuracy in a physical examination skills assessment |
| topic | rater training rating accuracy skills assessment physical examination skills randomised controlled trial |
| url | http://www.egms.de/static/en/journals/zma/2014-31/zma000933.shtml |
| work_keys_str_mv | AT weitzgunther effectsofaratertrainingonratingaccuracyinaphysicalexaminationskillsassessment AT vinzentiuschristian effectsofaratertrainingonratingaccuracyinaphysicalexaminationskillsassessment AT twestenchristoph effectsofaratertrainingonratingaccuracyinaphysicalexaminationskillsassessment AT lehnerthendrik effectsofaratertrainingonratingaccuracyinaphysicalexaminationskillsassessment AT bonnemeierhendrik effectsofaratertrainingonratingaccuracyinaphysicalexaminationskillsassessment AT koniginker effectsofaratertrainingonratingaccuracyinaphysicalexaminationskillsassessment |