Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study
Objective: We investigated the feasibility of applying a standard-setting procedure for the rapid film reporting examination of the Malaysian National Conjoint Board of Radiology. Methods: We selected the modified Angoff standard-setting process. Judges were nominated and trained, performance catego...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Published: |
Hong Kong Academy of Medicine Press
2021
|
| _version_ | 1825937507936632832 |
|---|---|
| author | Abdul Rahim, A. F. Roslan, N. S. Shuaib, I. L. Abdullah, M. S. Sayuti, K. A. Abu Hassan, H. |
| author_facet | Abdul Rahim, A. F. Roslan, N. S. Shuaib, I. L. Abdullah, M. S. Sayuti, K. A. Abu Hassan, H. |
| author_sort | Abdul Rahim, A. F. |
| collection | UPM |
| description | Objective: We investigated the feasibility of applying a standard-setting procedure for the rapid film reporting examination of the Malaysian National Conjoint Board of Radiology. Methods: We selected the modified Angoff standard-setting process. Judges were nominated and trained, performance categories were discussed, and judges’ ratings on films were collected after an iterative procedure. The process was then evaluated for evidence of validity. Results: A cut-off score of 92% resulted, compared with the 80% usually used. Judges were satisfied with the training and understood the procedure and their roles. In all, 27.3% felt that time given for the task was not sufficient. A total of 54.5% of judges thought the final passing cut-off score of 92% was too high, and 27.3% were not confident regarding its appropriateness. The inter-rater reliability was 0.928. External comparison with a ‘gold standard’ of supervisor ratings revealed a sensitivity of 0.25 and specificity of 1.00 compared with the traditional cut-off score having a sensitivity of 0.92 and specificity of 0.33. In this kind of situation, high specificity is considered to be more important than high sensitivity. Conclusion: Standard setting for the rapid film reporting examination using the modified Angoff method was feasible with robust procedural, internal, and external validity evidence. Areas for improvement were identified to address the perceived high cut-off score obtained and improve the overall process. |
| first_indexed | 2024-03-06T11:00:33Z |
| format | Article |
| id | upm.eprints-95054 |
| institution | Universiti Putra Malaysia |
| last_indexed | 2024-03-06T11:00:33Z |
| publishDate | 2021 |
| publisher | Hong Kong Academy of Medicine Press |
| record_format | dspace |
| spelling | upm.eprints-950542023-04-12T04:15:00Z http://psasir.upm.edu.my/id/eprint/95054/ Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study Abdul Rahim, A. F. Roslan, N. S. Shuaib, I. L. Abdullah, M. S. Sayuti, K. A. Abu Hassan, H. Objective: We investigated the feasibility of applying a standard-setting procedure for the rapid film reporting examination of the Malaysian National Conjoint Board of Radiology. Methods: We selected the modified Angoff standard-setting process. Judges were nominated and trained, performance categories were discussed, and judges’ ratings on films were collected after an iterative procedure. The process was then evaluated for evidence of validity. Results: A cut-off score of 92% resulted, compared with the 80% usually used. Judges were satisfied with the training and understood the procedure and their roles. In all, 27.3% felt that time given for the task was not sufficient. A total of 54.5% of judges thought the final passing cut-off score of 92% was too high, and 27.3% were not confident regarding its appropriateness. The inter-rater reliability was 0.928. External comparison with a ‘gold standard’ of supervisor ratings revealed a sensitivity of 0.25 and specificity of 1.00 compared with the traditional cut-off score having a sensitivity of 0.92 and specificity of 0.33. In this kind of situation, high specificity is considered to be more important than high sensitivity. Conclusion: Standard setting for the rapid film reporting examination using the modified Angoff method was feasible with robust procedural, internal, and external validity evidence. Areas for improvement were identified to address the perceived high cut-off score obtained and improve the overall process. Hong Kong Academy of Medicine Press 2021 Article PeerReviewed Abdul Rahim, A. F. and Roslan, N. S. and Shuaib, I. L. and Abdullah, M. S. and Sayuti, K. A. and Abu Hassan, H. (2021) Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study. Hong Kong Journal of Radiology, 24 (1). 23 - 30. ISSN 2223-6619; ESSN: 2307-4620 https://www.hkjr.org/article/v24n1/23 10.12809/hkjr2117199 |
| spellingShingle | Abdul Rahim, A. F. Roslan, N. S. Shuaib, I. L. Abdullah, M. S. Sayuti, K. A. Abu Hassan, H. Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study |
| title | Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study |
| title_full | Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study |
| title_fullStr | Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study |
| title_full_unstemmed | Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study |
| title_short | Setting standards for judging the rapid film reporting section in postgraduate radiology assessment: a feasibility study |
| title_sort | setting standards for judging the rapid film reporting section in postgraduate radiology assessment a feasibility study |
| work_keys_str_mv | AT abdulrahimaf settingstandardsforjudgingtherapidfilmreportingsectioninpostgraduateradiologyassessmentafeasibilitystudy AT roslanns settingstandardsforjudgingtherapidfilmreportingsectioninpostgraduateradiologyassessmentafeasibilitystudy AT shuaibil settingstandardsforjudgingtherapidfilmreportingsectioninpostgraduateradiologyassessmentafeasibilitystudy AT abdullahms settingstandardsforjudgingtherapidfilmreportingsectioninpostgraduateradiologyassessmentafeasibilitystudy AT sayutika settingstandardsforjudgingtherapidfilmreportingsectioninpostgraduateradiologyassessmentafeasibilitystudy AT abuhassanh settingstandardsforjudgingtherapidfilmreportingsectioninpostgraduateradiologyassessmentafeasibilitystudy |