The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model
Writing assessment relies closely on scoring the excellence of a subject’s thoughts. This creates a faceted measurement structure regarding rubrics, tasks, and raters. Nevertheless, most studies did not consider the differences among raters systematically. This study examines the raters’ differences...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-12-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.988272/full |
| _version_ | 1797980678907756544 |
|---|---|
| author | Harun Baharudin Zunita Mohamad Maskor Mohd Effendi Ewan Mohd Matore |
| author_facet | Harun Baharudin Zunita Mohamad Maskor Mohd Effendi Ewan Mohd Matore |
| author_sort | Harun Baharudin |
| collection | DOAJ |
| description | Writing assessment relies closely on scoring the excellence of a subject’s thoughts. This creates a faceted measurement structure regarding rubrics, tasks, and raters. Nevertheless, most studies did not consider the differences among raters systematically. This study examines the raters’ differences in association with the reliability and validity of writing rubrics using the Many-Facet Rasch measurement model (MFRM) to model these differences. A set of standards for evaluating the quality of rating based on writing assessment was examined. Rating quality was tested within four writing domains from an analytic rubric using a scale of one to three. The writing domains explored were vocabulary, grammar, language, use, and organization; whereas the data were obtained from 15 Arabic essays gathered from religious secondary school students under the supervision of the Malaysia Ministry of Education. Five raters in the field of practice were selected to evaluate all the essays. As a result, (a) raters range considerably on the lenient-severity dimension, so rater variations ought to be modeled; (b) the combination of findings between raters avoids the doubt of scores, thereby reducing the measurement error which could lower the criterion validity with the external variable; and (c) MFRM adjustments effectively increased the correlations of the scores obtained from partial and full data. Predominant findings revealed that rating quality varies across analytic rubric domains. This also depicts that MFRM is an effective way to model rater differences and evaluate the validity and reliability of writing rubrics. |
| first_indexed | 2024-04-11T05:58:39Z |
| format | Article |
| id | doaj.art-76e7ea9282794ffaab4294a5a5d7a3c8 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-04-11T05:58:39Z |
| publishDate | 2022-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-76e7ea9282794ffaab4294a5a5d7a3c82022-12-22T04:41:49ZengFrontiers Media S.A.Frontiers in Psychology1664-10782022-12-011310.3389/fpsyg.2022.988272988272The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement modelHarun Baharudin0Zunita Mohamad Maskor1Mohd Effendi Ewan Mohd Matore2Faculty of Education, Center of Diversity and Education, Universiti Kebangsaan Malaysia (UKM), Selangor, MalaysiaSMK Khir Johari, Tanjong Malim, Perak, MalaysiaFaculty of Education, Research Centre of Education Leadership and Policy, Universiti Kebangsaan Malaysia (UKM), Selangor, MalaysiaWriting assessment relies closely on scoring the excellence of a subject’s thoughts. This creates a faceted measurement structure regarding rubrics, tasks, and raters. Nevertheless, most studies did not consider the differences among raters systematically. This study examines the raters’ differences in association with the reliability and validity of writing rubrics using the Many-Facet Rasch measurement model (MFRM) to model these differences. A set of standards for evaluating the quality of rating based on writing assessment was examined. Rating quality was tested within four writing domains from an analytic rubric using a scale of one to three. The writing domains explored were vocabulary, grammar, language, use, and organization; whereas the data were obtained from 15 Arabic essays gathered from religious secondary school students under the supervision of the Malaysia Ministry of Education. Five raters in the field of practice were selected to evaluate all the essays. As a result, (a) raters range considerably on the lenient-severity dimension, so rater variations ought to be modeled; (b) the combination of findings between raters avoids the doubt of scores, thereby reducing the measurement error which could lower the criterion validity with the external variable; and (c) MFRM adjustments effectively increased the correlations of the scores obtained from partial and full data. Predominant findings revealed that rating quality varies across analytic rubric domains. This also depicts that MFRM is an effective way to model rater differences and evaluate the validity and reliability of writing rubrics.https://www.frontiersin.org/articles/10.3389/fpsyg.2022.988272/fullratersanalytic rubricwriting assessmentMany-Facet Rasch measurement model (MFRM)writing domainsArabic essays validation |
| spellingShingle | Harun Baharudin Zunita Mohamad Maskor Mohd Effendi Ewan Mohd Matore The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model Frontiers in Psychology raters analytic rubric writing assessment Many-Facet Rasch measurement model (MFRM) writing domains Arabic essays validation |
| title | The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model |
| title_full | The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model |
| title_fullStr | The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model |
| title_full_unstemmed | The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model |
| title_short | The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model |
| title_sort | raters differences in arabic writing rubrics through the many facet rasch measurement model |
| topic | raters analytic rubric writing assessment Many-Facet Rasch measurement model (MFRM) writing domains Arabic essays validation |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.988272/full |
| work_keys_str_mv | AT harunbaharudin theratersdifferencesinarabicwritingrubricsthroughthemanyfacetraschmeasurementmodel AT zunitamohamadmaskor theratersdifferencesinarabicwritingrubricsthroughthemanyfacetraschmeasurementmodel AT mohdeffendiewanmohdmatore theratersdifferencesinarabicwritingrubricsthroughthemanyfacetraschmeasurementmodel AT harunbaharudin ratersdifferencesinarabicwritingrubricsthroughthemanyfacetraschmeasurementmodel AT zunitamohamadmaskor ratersdifferencesinarabicwritingrubricsthroughthemanyfacetraschmeasurementmodel AT mohdeffendiewanmohdmatore ratersdifferencesinarabicwritingrubricsthroughthemanyfacetraschmeasurementmodel |