Norming a VALUE rubric to assess graduate information literacy skills
Objective: The study evaluated whether a modified version of the information literacy Valid Assessment of Learning in Undergraduate Education (VALUE) rubric would be useful for assessing the information literacy skills of graduate health sciences students. Methods: Through facilitated calibration w...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
University Library System, University of Pittsburgh
2016-09-01
|
| Series: | Journal of the Medical Library Association |
| Subjects: | |
| Online Access: | http://jmla.pitt.edu/ojs/jmla/article/view/13 |
| _version_ | 1818417023525322752 |
|---|---|
| author | David J. Turbow Julie Evener |
| author_facet | David J. Turbow Julie Evener |
| author_sort | David J. Turbow |
| collection | DOAJ |
| description | Objective: The study evaluated whether a modified version of the information literacy Valid Assessment of Learning in Undergraduate Education (VALUE) rubric would be useful for assessing the information literacy skills of graduate health sciences students.
Methods: Through facilitated calibration workshops, an interdepartmental six-person team of librarians and faculty engaged in guided discussion about the meaning of the rubric criteria. They applied the rubric to score student work for a peer-review essay assignment in the ‘‘Information Literacy for Evidence-Based Practice’’ course. To determine inter-rater reliability, the raters participated in a follow-up exercise in which they independently applied the rubric to ten samples of work from a research project in the doctor of physical therapy program: the patient case report assignment.
Results: For the peer-review essay, a high level of consistency in scoring was achieved for the second workshop, with statistically significant intra-class correlation coefficients above 0.8 for 3 criteria: ‘‘Determine the extent of evidence needed,’’ ‘‘Use evidence effectively to accomplish a specific purpose,’’ and ‘‘Access the needed evidence.’’ Participants concurred that the essay prompt and rubric criteria adequately discriminated the quality of student work for the peer-review essay assignment. When raters independently scored the patient case report assignment, inter-rater agreement was low and statistically insignificant for all rubric criteria (kappa¼[1]0.16, p.0.05–kappa¼0.12, p.0.05).
Conclusions: While the peer-review essay assignment lent itself well to rubric calibration, scorers had a difficult time with the patient case report. Lack of familiarity among some raters with the specifics of the patient case report assignment and subject matter might have accounted for low inter-rater reliability. When norming, it is important to hold conversations about search strategies and expectations of performance. Overall, the authors found the rubric to be appropriate for assessing information literacy skills of graduate health sciences students. |
| first_indexed | 2024-12-14T12:00:11Z |
| format | Article |
| id | doaj.art-de978d92cf58412ca279f339b0ba5a7b |
| institution | Directory Open Access Journal |
| issn | 1536-5050 1558-9439 |
| language | English |
| last_indexed | 2024-12-14T12:00:11Z |
| publishDate | 2016-09-01 |
| publisher | University Library System, University of Pittsburgh |
| record_format | Article |
| series | Journal of the Medical Library Association |
| spelling | doaj.art-de978d92cf58412ca279f339b0ba5a7b2022-12-21T23:02:00ZengUniversity Library System, University of PittsburghJournal of the Medical Library Association1536-50501558-94392016-09-01104310.5195/jmla.2016.137Norming a VALUE rubric to assess graduate information literacy skillsDavid J. Turbow0Julie Evener1Outcomes Assessment Coordinator, University of St. Augustine for Health Sciences, 700 Windy Point Drive, San Marcos, CA 92069MLIS, Director of Library Services, University of St. Augustine for Health Sciences, 1 University Boulevard, St. Augustine, FL 32086Objective: The study evaluated whether a modified version of the information literacy Valid Assessment of Learning in Undergraduate Education (VALUE) rubric would be useful for assessing the information literacy skills of graduate health sciences students. Methods: Through facilitated calibration workshops, an interdepartmental six-person team of librarians and faculty engaged in guided discussion about the meaning of the rubric criteria. They applied the rubric to score student work for a peer-review essay assignment in the ‘‘Information Literacy for Evidence-Based Practice’’ course. To determine inter-rater reliability, the raters participated in a follow-up exercise in which they independently applied the rubric to ten samples of work from a research project in the doctor of physical therapy program: the patient case report assignment. Results: For the peer-review essay, a high level of consistency in scoring was achieved for the second workshop, with statistically significant intra-class correlation coefficients above 0.8 for 3 criteria: ‘‘Determine the extent of evidence needed,’’ ‘‘Use evidence effectively to accomplish a specific purpose,’’ and ‘‘Access the needed evidence.’’ Participants concurred that the essay prompt and rubric criteria adequately discriminated the quality of student work for the peer-review essay assignment. When raters independently scored the patient case report assignment, inter-rater agreement was low and statistically insignificant for all rubric criteria (kappa¼[1]0.16, p.0.05–kappa¼0.12, p.0.05). Conclusions: While the peer-review essay assignment lent itself well to rubric calibration, scorers had a difficult time with the patient case report. Lack of familiarity among some raters with the specifics of the patient case report assignment and subject matter might have accounted for low inter-rater reliability. When norming, it is important to hold conversations about search strategies and expectations of performance. Overall, the authors found the rubric to be appropriate for assessing information literacy skills of graduate health sciences students.http://jmla.pitt.edu/ojs/jmla/article/view/13Information LiteracyCalibrationEducational MeasurementEducation, GraduateInterdepartmental RelationsCooperative Behavior |
| spellingShingle | David J. Turbow Julie Evener Norming a VALUE rubric to assess graduate information literacy skills Journal of the Medical Library Association Information Literacy Calibration Educational Measurement Education, Graduate Interdepartmental Relations Cooperative Behavior |
| title | Norming a VALUE rubric to assess graduate information literacy skills |
| title_full | Norming a VALUE rubric to assess graduate information literacy skills |
| title_fullStr | Norming a VALUE rubric to assess graduate information literacy skills |
| title_full_unstemmed | Norming a VALUE rubric to assess graduate information literacy skills |
| title_short | Norming a VALUE rubric to assess graduate information literacy skills |
| title_sort | norming a value rubric to assess graduate information literacy skills |
| topic | Information Literacy Calibration Educational Measurement Education, Graduate Interdepartmental Relations Cooperative Behavior |
| url | http://jmla.pitt.edu/ojs/jmla/article/view/13 |
| work_keys_str_mv | AT davidjturbow normingavaluerubrictoassessgraduateinformationliteracyskills AT julieevener normingavaluerubrictoassessgraduateinformationliteracyskills |