Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates
The Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen’s weighted kappa, the ove...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-10-01
|
| Series: | Education Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7102/11/10/648 |
| _version_ | 1797514773419524096 |
|---|---|
| author | Scott A. Lyness Kent Peterson Kenneth Yates |
| author_facet | Scott A. Lyness Kent Peterson Kenneth Yates |
| author_sort | Scott A. Lyness |
| collection | DOAJ |
| description | The Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen’s weighted kappa, the overall IRR estimate was 0.17 (poor strength of agreement). IRR estimates ranged from −0.29 (worse than expected by chance) to 0.54 (moderate strength of agreement); all were below the standard of 0.70 for consensus agreement. Follow-up interviews of 10 evaluators revealed possible reasons we observed low IRR, such as departures from established PACT scoring protocol, and lack of, or inconsistent, use of a scoring aid document. Evaluators reported difficulties scoring the materials that candidates submitted, particularly the use of Academic Language. Cognitive Task Analysis (CTA) is suggested as a method to improve IRR in the PACT and other teacher performance assessments such as the edTPA. |
| first_indexed | 2024-03-10T06:36:19Z |
| format | Article |
| id | doaj.art-24b8fc1cc78d42fc94ed094eb2241795 |
| institution | Directory Open Access Journal |
| issn | 2227-7102 |
| language | English |
| last_indexed | 2024-03-10T06:36:19Z |
| publishDate | 2021-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Education Sciences |
| spelling | doaj.art-24b8fc1cc78d42fc94ed094eb22417952023-11-22T18:01:15ZengMDPI AGEducation Sciences2227-71022021-10-01111064810.3390/educsci11100648Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher CandidatesScott A. Lyness0Kent Peterson1Kenneth Yates2Rossier School of Education, University of Southern California, Los Angeles, CA 90089, USARossier School of Education, University of Southern California, Los Angeles, CA 90089, USARossier School of Education, University of Southern California, Los Angeles, CA 90089, USAThe Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen’s weighted kappa, the overall IRR estimate was 0.17 (poor strength of agreement). IRR estimates ranged from −0.29 (worse than expected by chance) to 0.54 (moderate strength of agreement); all were below the standard of 0.70 for consensus agreement. Follow-up interviews of 10 evaluators revealed possible reasons we observed low IRR, such as departures from established PACT scoring protocol, and lack of, or inconsistent, use of a scoring aid document. Evaluators reported difficulties scoring the materials that candidates submitted, particularly the use of Academic Language. Cognitive Task Analysis (CTA) is suggested as a method to improve IRR in the PACT and other teacher performance assessments such as the edTPA.https://www.mdpi.com/2227-7102/11/10/648inter-rater reliabilitypreservice teacher performance assessmentPACTedTPAweighted kappacognitive task analysis |
| spellingShingle | Scott A. Lyness Kent Peterson Kenneth Yates Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates Education Sciences inter-rater reliability preservice teacher performance assessment PACT edTPA weighted kappa cognitive task analysis |
| title | Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates |
| title_full | Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates |
| title_fullStr | Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates |
| title_full_unstemmed | Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates |
| title_short | Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates |
| title_sort | low inter rater reliability of a high stakes performance assessment of teacher candidates |
| topic | inter-rater reliability preservice teacher performance assessment PACT edTPA weighted kappa cognitive task analysis |
| url | https://www.mdpi.com/2227-7102/11/10/648 |
| work_keys_str_mv | AT scottalyness lowinterraterreliabilityofahighstakesperformanceassessmentofteachercandidates AT kentpeterson lowinterraterreliabilityofahighstakesperformanceassessmentofteachercandidates AT kennethyates lowinterraterreliabilityofahighstakesperformanceassessmentofteachercandidates |