Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students
Abstract Background Students´ assessment should be carried out in an effective and objective manner, which reduces the possibility of different evaluators giving different scores, thus influencing the qualification obtained and the consistency of education. The aim of the present study was to determ...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-03-01
|
| Series: | BMC Medical Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12909-023-04187-3 |
| _version_ | 1827974479779725312 |
|---|---|
| author | Nuria Escribano Virginia Belliard Bruno Baracco Dayana Da Silva Laura Ceballos M. Victoria Fuentes |
| author_facet | Nuria Escribano Virginia Belliard Bruno Baracco Dayana Da Silva Laura Ceballos M. Victoria Fuentes |
| author_sort | Nuria Escribano |
| collection | DOAJ |
| description | Abstract Background Students´ assessment should be carried out in an effective and objective manner, which reduces the possibility of different evaluators giving different scores, thus influencing the qualification obtained and the consistency of education. The aim of the present study was to determine the agreement among four evaluators and compare the overall scores awarded when assessing portfolios of endodontic preclinical treatments performed by dental students by using an analytic rubric and a numeric rating scale. Methods A random sample of 42 portfolios performed by fourth-year dental students at preclinical endodontic practices were blindly assessed by four evaluators using two different evaluation methods: an analytic rubric specifically designed and a numeric rating scale. Six categories were analyzed: radiographic assessment, access preparation, shaping procedure, obturation, content of the portfolio, and presentation of the portfolio. The maximum global score was 10 points. The overall scores obtained with both methods from each evaluator were compared by Student’s t, while agreement among evaluators was measured by Intraclass correlation coefficients (ICC). The influence of the difficulty of the endodontic treatment on the evaluators´ scores was analyzed by one-way ANOVA. Statistical tests were performed at a pre-set alpha of 0.05 using Stata 16. Results Difficulty of canal treatment did not influence the scores of evaluators, irrespective of the evaluation method used. When the analytic rubric was used, inter-evaluator agreement was substantial for radiographic assessment, access preparation, shaping procedure, obturation, and overall scores. Inter-evaluator agreement ranged from moderate to fair with the numeric rating scale. Mean higher overall scores were achieved when numeric rating scale was used. Presentation and content of the portfolio showed slight and fair agreement, respectively, among evaluators, regardless the evaluation method applied. Conclusions Assessment guided by an analytic rubric allowed evaluators to reach higher levels of agreement than those obtained when using a numeric rating scale. However, the rubric negatively affected overall scores. |
| first_indexed | 2024-04-09T19:55:02Z |
| format | Article |
| id | doaj.art-3749a81263ca40eeb0ce5cb510739bc1 |
| institution | Directory Open Access Journal |
| issn | 1472-6920 |
| language | English |
| last_indexed | 2024-04-09T19:55:02Z |
| publishDate | 2023-03-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Education |
| spelling | doaj.art-3749a81263ca40eeb0ce5cb510739bc12023-04-03T05:31:01ZengBMCBMC Medical Education1472-69202023-03-012311910.1186/s12909-023-04187-3Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental studentsNuria Escribano0Virginia Belliard1Bruno Baracco2Dayana Da Silva3Laura Ceballos4M. Victoria Fuentes5Faculty of Health Sciences, IDIBO Research Group, Rey Juan Carlos UniversityFaculty of Health Sciences, IDIBO Research Group, Rey Juan Carlos UniversityFaculty of Health Sciences, IDIBO Research Group, Rey Juan Carlos UniversityFaculty of Health Sciences, IDIBO Research Group, Rey Juan Carlos UniversityFaculty of Health Sciences, IDIBO Research Group, Rey Juan Carlos UniversityFaculty of Health Sciences, IDIBO Research Group, Rey Juan Carlos UniversityAbstract Background Students´ assessment should be carried out in an effective and objective manner, which reduces the possibility of different evaluators giving different scores, thus influencing the qualification obtained and the consistency of education. The aim of the present study was to determine the agreement among four evaluators and compare the overall scores awarded when assessing portfolios of endodontic preclinical treatments performed by dental students by using an analytic rubric and a numeric rating scale. Methods A random sample of 42 portfolios performed by fourth-year dental students at preclinical endodontic practices were blindly assessed by four evaluators using two different evaluation methods: an analytic rubric specifically designed and a numeric rating scale. Six categories were analyzed: radiographic assessment, access preparation, shaping procedure, obturation, content of the portfolio, and presentation of the portfolio. The maximum global score was 10 points. The overall scores obtained with both methods from each evaluator were compared by Student’s t, while agreement among evaluators was measured by Intraclass correlation coefficients (ICC). The influence of the difficulty of the endodontic treatment on the evaluators´ scores was analyzed by one-way ANOVA. Statistical tests were performed at a pre-set alpha of 0.05 using Stata 16. Results Difficulty of canal treatment did not influence the scores of evaluators, irrespective of the evaluation method used. When the analytic rubric was used, inter-evaluator agreement was substantial for radiographic assessment, access preparation, shaping procedure, obturation, and overall scores. Inter-evaluator agreement ranged from moderate to fair with the numeric rating scale. Mean higher overall scores were achieved when numeric rating scale was used. Presentation and content of the portfolio showed slight and fair agreement, respectively, among evaluators, regardless the evaluation method applied. Conclusions Assessment guided by an analytic rubric allowed evaluators to reach higher levels of agreement than those obtained when using a numeric rating scale. However, the rubric negatively affected overall scores.https://doi.org/10.1186/s12909-023-04187-3Dental educationRubricEndodontics trainingPortfolio evaluationEvaluators´ agreement |
| spellingShingle | Nuria Escribano Virginia Belliard Bruno Baracco Dayana Da Silva Laura Ceballos M. Victoria Fuentes Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students BMC Medical Education Dental education Rubric Endodontics training Portfolio evaluation Evaluators´ agreement |
| title | Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students |
| title_full | Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students |
| title_fullStr | Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students |
| title_full_unstemmed | Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students |
| title_short | Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students |
| title_sort | rubric vs numeric rating scale agreement among evaluators on endodontic treatments performed by dental students |
| topic | Dental education Rubric Endodontics training Portfolio evaluation Evaluators´ agreement |
| url | https://doi.org/10.1186/s12909-023-04187-3 |
| work_keys_str_mv | AT nuriaescribano rubricvsnumericratingscaleagreementamongevaluatorsonendodontictreatmentsperformedbydentalstudents AT virginiabelliard rubricvsnumericratingscaleagreementamongevaluatorsonendodontictreatmentsperformedbydentalstudents AT brunobaracco rubricvsnumericratingscaleagreementamongevaluatorsonendodontictreatmentsperformedbydentalstudents AT dayanadasilva rubricvsnumericratingscaleagreementamongevaluatorsonendodontictreatmentsperformedbydentalstudents AT lauraceballos rubricvsnumericratingscaleagreementamongevaluatorsonendodontictreatmentsperformedbydentalstudents AT mvictoriafuentes rubricvsnumericratingscaleagreementamongevaluatorsonendodontictreatmentsperformedbydentalstudents |