| Summary: | Performance assessment, unlike the traditional fixed-response assessment, has features peculiar to its assessment setting (the task choice, the task processing conditions, the raters,the rating scale, and the rating procedures) that make it much more vulnerable to construct
irrelevant variance (McNamara, 1997; Upshur & Turner, 1999). Of these potential sources of variability, those associated with raters are considered to be extensive and pose serious threats to the validity of ratings (Linacre, 1989; McNamara, 1996).For performance assessment to yield valid and reliable results, it is essential that these sources of variability are eliminated or minimized. This paper illustrates how sources of rater-related variability or rater effects can be identified and controlled for using the Many-facet Rasch Model.It also illustrates how the idiosyncratic rating behaviour of individual raters can be explicated through the use of this measurement model. In this illustration, the ratings of thirty-five English language instructors on 12 paragraphs written by new intake students at the Centre for Foundation Studies, IIUM were analysed using FACETS (Linacre, 2003), a computer
application which implements the Many-facet Rasch Model. The results of the analysis indicate substantial differences in rater severity, and the presence of other rater effects.
|
|---|