Tutor assessment of PBL process: does tutor variability affect objectivity and reliability?
Abstract Background Ensuring objectivity and maintaining reliability are necessary in order to consider any form of assessment valid. Evaluation of students in Problem-Based Learning (PBL) tutorials by the tutors has drawn the attention of critiques citing many challenges and limitations. The aim of...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2019-03-01
|
| Series: | BMC Medical Education |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s12909-019-1508-z |
| _version_ | 1819081037825703936 |
|---|---|
| author | Bidyadhar Sa Chidum Ezenwaka Keerti Singh Sehlule Vuma Md. Anwarul Azim Majumder |
| author_facet | Bidyadhar Sa Chidum Ezenwaka Keerti Singh Sehlule Vuma Md. Anwarul Azim Majumder |
| author_sort | Bidyadhar Sa |
| collection | DOAJ |
| description | Abstract Background Ensuring objectivity and maintaining reliability are necessary in order to consider any form of assessment valid. Evaluation of students in Problem-Based Learning (PBL) tutorials by the tutors has drawn the attention of critiques citing many challenges and limitations. The aim of this study was to determine the extent of tutor variability in assessing the PBL process in the Faculty of Medical Sciences, The University of the West Indies, St Augustine Campus, Trinidad and Tobago. Method All 181 students of year 3 MBBS were assigned randomly to 14 PBL groups. Out of 18 tutors, 12 had an opportunity to assess three groups: one assessed 2 groups and 4 tutors assessed one group each; at the end each group had been assessed three times by different tutors. The tutors used a PBL assessment rating scale of 12 different criteria on a six-point scale to assess each PBL Group. To test the stated hypotheses, independent t-test, one-way ANOVA followed by post-hoc Bonferroni test, Intra Class Correlation, and Pearson product moment correlations were performed. Result The analysis revealed significant differences between the highest- and lowest-rated groups (t-ratio = 12.64; p < 0.05) and between the most lenient and most stringent raters (t-ratio = 27.96; p < 0.05). ANOVA and post-hoc analysis for highest and lowest rated groups revealed that lenient- and stringent-raters significantly contribute (p < 0.01) in diluting the score in their respective category. The intra class correlations (ICC) among rating of different tutors for different groups showed low agreement among various ratings except three groups (Groups 6, 8 and 13) (r = 0.40). The correlation between tutors’ PBL experiences and their mean ratings was found to be moderately significant (r = 0.52; p > 0.05). Conclusion Leniency and stringency factors amongst raters affect objectivity and reliability to a great extent as is evident from the present study. Thus, more rigorous training in the areas of principles of assessment for the tutors are recommended. Moreover, putting that knowledge into practice to overcome the leniency and stringency factors is essential. |
| first_indexed | 2024-12-21T19:54:25Z |
| format | Article |
| id | doaj.art-032b0558350249fa8b2762bf5c43151d |
| institution | Directory Open Access Journal |
| issn | 1472-6920 |
| language | English |
| last_indexed | 2024-12-21T19:54:25Z |
| publishDate | 2019-03-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Education |
| spelling | doaj.art-032b0558350249fa8b2762bf5c43151d2022-12-21T18:52:07ZengBMCBMC Medical Education1472-69202019-03-011911810.1186/s12909-019-1508-zTutor assessment of PBL process: does tutor variability affect objectivity and reliability?Bidyadhar Sa0Chidum Ezenwaka1Keerti Singh2Sehlule Vuma3Md. Anwarul Azim Majumder4Centre for Medical Sciences Education, The Faculty of Medical Sciences, The University of the West IndiesDepartment of Para-clinical Sciences, Faculty of Medical Sciences, The University of the West IndiesFaculty of Medical Sciences, The University of the West IndiesDepartment of Para-clinical Sciences, Faculty of Medical Sciences, The University of the West IndiesFaculty of Medical Sciences, The University of the West IndiesAbstract Background Ensuring objectivity and maintaining reliability are necessary in order to consider any form of assessment valid. Evaluation of students in Problem-Based Learning (PBL) tutorials by the tutors has drawn the attention of critiques citing many challenges and limitations. The aim of this study was to determine the extent of tutor variability in assessing the PBL process in the Faculty of Medical Sciences, The University of the West Indies, St Augustine Campus, Trinidad and Tobago. Method All 181 students of year 3 MBBS were assigned randomly to 14 PBL groups. Out of 18 tutors, 12 had an opportunity to assess three groups: one assessed 2 groups and 4 tutors assessed one group each; at the end each group had been assessed three times by different tutors. The tutors used a PBL assessment rating scale of 12 different criteria on a six-point scale to assess each PBL Group. To test the stated hypotheses, independent t-test, one-way ANOVA followed by post-hoc Bonferroni test, Intra Class Correlation, and Pearson product moment correlations were performed. Result The analysis revealed significant differences between the highest- and lowest-rated groups (t-ratio = 12.64; p < 0.05) and between the most lenient and most stringent raters (t-ratio = 27.96; p < 0.05). ANOVA and post-hoc analysis for highest and lowest rated groups revealed that lenient- and stringent-raters significantly contribute (p < 0.01) in diluting the score in their respective category. The intra class correlations (ICC) among rating of different tutors for different groups showed low agreement among various ratings except three groups (Groups 6, 8 and 13) (r = 0.40). The correlation between tutors’ PBL experiences and their mean ratings was found to be moderately significant (r = 0.52; p > 0.05). Conclusion Leniency and stringency factors amongst raters affect objectivity and reliability to a great extent as is evident from the present study. Thus, more rigorous training in the areas of principles of assessment for the tutors are recommended. Moreover, putting that knowledge into practice to overcome the leniency and stringency factors is essential.http://link.springer.com/article/10.1186/s12909-019-1508-zProblem based learningProcess AssessmentTutor variabilityObjectivityReliability |
| spellingShingle | Bidyadhar Sa Chidum Ezenwaka Keerti Singh Sehlule Vuma Md. Anwarul Azim Majumder Tutor assessment of PBL process: does tutor variability affect objectivity and reliability? BMC Medical Education Problem based learning Process Assessment Tutor variability Objectivity Reliability |
| title | Tutor assessment of PBL process: does tutor variability affect objectivity and reliability? |
| title_full | Tutor assessment of PBL process: does tutor variability affect objectivity and reliability? |
| title_fullStr | Tutor assessment of PBL process: does tutor variability affect objectivity and reliability? |
| title_full_unstemmed | Tutor assessment of PBL process: does tutor variability affect objectivity and reliability? |
| title_short | Tutor assessment of PBL process: does tutor variability affect objectivity and reliability? |
| title_sort | tutor assessment of pbl process does tutor variability affect objectivity and reliability |
| topic | Problem based learning Process Assessment Tutor variability Objectivity Reliability |
| url | http://link.springer.com/article/10.1186/s12909-019-1508-z |
| work_keys_str_mv | AT bidyadharsa tutorassessmentofpblprocessdoestutorvariabilityaffectobjectivityandreliability AT chidumezenwaka tutorassessmentofpblprocessdoestutorvariabilityaffectobjectivityandreliability AT keertisingh tutorassessmentofpblprocessdoestutorvariabilityaffectobjectivityandreliability AT sehlulevuma tutorassessmentofpblprocessdoestutorvariabilityaffectobjectivityandreliability AT mdanwarulazimmajumder tutorassessmentofpblprocessdoestutorvariabilityaffectobjectivityandreliability |