Interrater reliability estimators tested against true interrater reliabilities
Abstract Background Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observati...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-08-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-022-01707-5 |
| _version_ | 1798037350900563968 |
|---|---|
| author | Xinshu Zhao Guangchao Charles Feng Song Harris Ao Piper Liping Liu |
| author_facet | Xinshu Zhao Guangchao Charles Feng Song Harris Ao Piper Liping Liu |
| author_sort | Xinshu Zhao |
| collection | DOAJ |
| description | Abstract Background Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters’ random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies. Methods We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested. Results The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r 2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott’s π, Cohen’s κ and Krippendorff’s α, underperformed all other indices, reporting directional r 2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet’s AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al’s S ranked behind AC1, and Perreault and Leigh’s Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating. Conclusion The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement. |
| first_indexed | 2024-04-11T21:25:23Z |
| format | Article |
| id | doaj.art-ec86610990584a9fb21c76caf72aeb96 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-04-11T21:25:23Z |
| publishDate | 2022-08-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-ec86610990584a9fb21c76caf72aeb962022-12-22T04:02:28ZengBMCBMC Medical Research Methodology1471-22882022-08-0122111910.1186/s12874-022-01707-5Interrater reliability estimators tested against true interrater reliabilitiesXinshu Zhao0Guangchao Charles Feng1Song Harris Ao2Piper Liping Liu3Department of Communication, Faculty of Social Sciences, University of MacauDepartment of Communication, Faculty of Social Sciences, University of MacauDepartment of Communication, Faculty of Social Sciences, University of MacauDepartment of Communication, Faculty of Social Sciences, University of MacauAbstract Background Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters’ random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies. Methods We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested. Results The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r 2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott’s π, Cohen’s κ and Krippendorff’s α, underperformed all other indices, reporting directional r 2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet’s AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al’s S ranked behind AC1, and Perreault and Leigh’s Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating. Conclusion The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.https://doi.org/10.1186/s12874-022-01707-5Intercoder reliabilityInterrater reliabilityReconstructed experimentCohen’s kappaKrippendorff’s alpha |
| spellingShingle | Xinshu Zhao Guangchao Charles Feng Song Harris Ao Piper Liping Liu Interrater reliability estimators tested against true interrater reliabilities BMC Medical Research Methodology Intercoder reliability Interrater reliability Reconstructed experiment Cohen’s kappa Krippendorff’s alpha |
| title | Interrater reliability estimators tested against true interrater reliabilities |
| title_full | Interrater reliability estimators tested against true interrater reliabilities |
| title_fullStr | Interrater reliability estimators tested against true interrater reliabilities |
| title_full_unstemmed | Interrater reliability estimators tested against true interrater reliabilities |
| title_short | Interrater reliability estimators tested against true interrater reliabilities |
| title_sort | interrater reliability estimators tested against true interrater reliabilities |
| topic | Intercoder reliability Interrater reliability Reconstructed experiment Cohen’s kappa Krippendorff’s alpha |
| url | https://doi.org/10.1186/s12874-022-01707-5 |
| work_keys_str_mv | AT xinshuzhao interraterreliabilityestimatorstestedagainsttrueinterraterreliabilities AT guangchaocharlesfeng interraterreliabilityestimatorstestedagainsttrueinterraterreliabilities AT songharrisao interraterreliabilityestimatorstestedagainsttrueinterraterreliabilities AT piperlipingliu interraterreliabilityestimatorstestedagainsttrueinterraterreliabilities |