Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers
Background: Educational researchers have long espoused the virtues of writing with regard to student cognitive skills. However, research on the reliability of the grades assigned to written papers reveals a high degree of contradiction, with some researchers concluding that the grades assig...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Evaluation Center at Western Michigan University
2012-09-01
|
| Series: | Journal of MultiDisciplinary Evaluation |
| Online Access: | https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/362 |
| _version_ | 1797812690897338368 |
|---|---|
| author | Mihaiela R. Gugiu Paul C. Gugiu Robert Baldus |
| author_facet | Mihaiela R. Gugiu Paul C. Gugiu Robert Baldus |
| author_sort | Mihaiela R. Gugiu |
| collection | DOAJ |
| description | Background: Educational researchers have long espoused the virtues of writing with regard to student cognitive skills. However, research on the reliability of the grades assigned to written papers reveals a high degree of contradiction, with some researchers concluding that the grades assigned are very reliable whereas others suggesting that they are so unreliable that random assignment of grades would have been almost as helpful.
Purpose: The primary purpose of the study was to investigate the reliability of grades assigned to written reports. The secondary purpose was to illustrate the use of Generalizability Theory, specifically the fully-crossed two-facet model, for computing interrater reliability coefficients.
Setting: The participants for this study were 29 undergraduate students enrolled in an introductory-level course on Political Behavior in Spring 2011 at a Midwest university.
Intervention: Not applicable.
Research Design: Students were randomly assigned to one of nine groups. Two-facet fully crossed G-study and D-study designs were used wherein two raters graded four assignments for 9 student groups—72 evaluations in total. The universe of admissible observations was deemed to be random for both raters and assignments, whereas the universe of generalization was deemed to be mixed (random for two raters but fixed for four assignments).
Data Collection and Analysis: The semester-long project was assigned to groups consisting of an annotated bibliography, survey development, sampling design, and analysis and final report. Four grading rubrics were developed and utilized to evaluate the quality of each written report. Two-facet generalizability analyses were conducted to assess interrater reliability using software developed by one of the authors.
Findings: This study found a very high interrater reliability coefficient (0.929) for only two raters who received no training in how to use the four grading rubrics.
Keywords: grading; reliability; Generalizability Theory; writing
|
| first_indexed | 2024-03-13T07:42:14Z |
| format | Article |
| id | doaj.art-db0e703975a84648a5912076a5d6b488 |
| institution | Directory Open Access Journal |
| issn | 1556-8180 |
| language | English |
| last_indexed | 2024-03-13T07:42:14Z |
| publishDate | 2012-09-01 |
| publisher | The Evaluation Center at Western Michigan University |
| record_format | Article |
| series | Journal of MultiDisciplinary Evaluation |
| spelling | doaj.art-db0e703975a84648a5912076a5d6b4882023-06-03T07:06:22ZengThe Evaluation Center at Western Michigan UniversityJournal of MultiDisciplinary Evaluation1556-81802012-09-0181910.56645/jmde.v8i19.362Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research PapersMihaiela R. GugiuPaul C. GugiuRobert Baldus Background: Educational researchers have long espoused the virtues of writing with regard to student cognitive skills. However, research on the reliability of the grades assigned to written papers reveals a high degree of contradiction, with some researchers concluding that the grades assigned are very reliable whereas others suggesting that they are so unreliable that random assignment of grades would have been almost as helpful. Purpose: The primary purpose of the study was to investigate the reliability of grades assigned to written reports. The secondary purpose was to illustrate the use of Generalizability Theory, specifically the fully-crossed two-facet model, for computing interrater reliability coefficients. Setting: The participants for this study were 29 undergraduate students enrolled in an introductory-level course on Political Behavior in Spring 2011 at a Midwest university. Intervention: Not applicable. Research Design: Students were randomly assigned to one of nine groups. Two-facet fully crossed G-study and D-study designs were used wherein two raters graded four assignments for 9 student groups—72 evaluations in total. The universe of admissible observations was deemed to be random for both raters and assignments, whereas the universe of generalization was deemed to be mixed (random for two raters but fixed for four assignments). Data Collection and Analysis: The semester-long project was assigned to groups consisting of an annotated bibliography, survey development, sampling design, and analysis and final report. Four grading rubrics were developed and utilized to evaluate the quality of each written report. Two-facet generalizability analyses were conducted to assess interrater reliability using software developed by one of the authors. Findings: This study found a very high interrater reliability coefficient (0.929) for only two raters who received no training in how to use the four grading rubrics. Keywords: grading; reliability; Generalizability Theory; writing https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/362 |
| spellingShingle | Mihaiela R. Gugiu Paul C. Gugiu Robert Baldus Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers Journal of MultiDisciplinary Evaluation |
| title | Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers |
| title_full | Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers |
| title_fullStr | Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers |
| title_full_unstemmed | Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers |
| title_short | Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers |
| title_sort | utilizing generalizability theory to investigate the reliability of grades assigned to undergraduate research papers |
| url | https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/362 |
| work_keys_str_mv | AT mihaielargugiu utilizinggeneralizabilitytheorytoinvestigatethereliabilityofgradesassignedtoundergraduateresearchpapers AT paulcgugiu utilizinggeneralizabilitytheorytoinvestigatethereliabilityofgradesassignedtoundergraduateresearchpapers AT robertbaldus utilizinggeneralizabilitytheorytoinvestigatethereliabilityofgradesassignedtoundergraduateresearchpapers |