A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool
Introduction Quality improvement (QI) competencies for health professions trainees were developed to address health care quality. Strategies to integrate QI into curricula exist, but methods for assessing interdisciplinary learners’ competency are less developed. We refined the Knowledge section sco...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Association of American Medical Colleges
2022-12-01
|
| Series: | MedEdPORTAL |
| Subjects: | |
| Online Access: | http://www.mededportal.org/doi/10.15766/mep_2374-8265.11290 |
| _version_ | 1811198196773289984 |
|---|---|
| author | Corrine Abraham Krysta Johnson-Martinez Anne Tomolo |
| author_facet | Corrine Abraham Krysta Johnson-Martinez Anne Tomolo |
| author_sort | Corrine Abraham |
| collection | DOAJ |
| description | Introduction Quality improvement (QI) competencies for health professions trainees were developed to address health care quality. Strategies to integrate QI into curricula exist, but methods for assessing interdisciplinary learners’ competency are less developed. We refined the Knowledge section scoring rubric of the Systems Quality Improvement Training and Assessment Tool (SQI TAT) and examined its validity evidence. Methods In 2017, the SQI TAT Knowledge section was expanded to cover seven core QI concepts, and the scoring rubric was refined. Three coders independently scored 35 SQI TAT Knowledge sections (18 pretests, 17 posttests). Interrater reliability was assessed by percent agreement and Cohen's kappa for individual variables and by Lin's concordance correlation for total scores for knowledge and application. Concurrent validity was assessed by comparing responses from two groups with different QI exposure and evaluating whether differences in exposure were measured. Results Total-score interrater reliability average measures of concordance were .89 for all coders and >.70 for six of seven concept scores. The total score discriminated the two groups (p <. 05), and five of seven concept scores were higher for the group with more QI experience. Total scores were significantly higher posttest than pretest (p < .001), with improvement in posttest knowledge scores. Discussion The SQI TAT Knowledge section provides a comprehensive assessment of QI knowledge. The scoring rubric was able to discriminate QI knowledge along a continuum. The SQI TAT Knowledge section is not linked to a clinical context, making it useful for assessing interprofessional learners and varying education levels. |
| first_indexed | 2024-04-12T01:27:52Z |
| format | Article |
| id | doaj.art-daae5e9dec584899b52ba5ab819df44c |
| institution | Directory Open Access Journal |
| issn | 2374-8265 |
| language | English |
| last_indexed | 2024-04-12T01:27:52Z |
| publishDate | 2022-12-01 |
| publisher | Association of American Medical Colleges |
| record_format | Article |
| series | MedEdPORTAL |
| spelling | doaj.art-daae5e9dec584899b52ba5ab819df44c2022-12-22T03:53:36ZengAssociation of American Medical CollegesMedEdPORTAL2374-82652022-12-011810.15766/mep_2374-8265.11290A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment ToolCorrine Abraham0Krysta Johnson-Martinez1Anne Tomolo2Associate Professor, Nell Hodgson Woodruff School of Nursing at Emory University; Coordinator, Evidence-Based Practice and Innovation, and Co-Director, VA Quality Scholars Fellowship Program, Atlanta VA Health Care SystemSpecialty Care Lead and Chief Medical Informatics Officer, VISN 8 VA Sunshine Healthcare NetworkPhysician, National Center for Patient Safety; Associate Professor, Emory University School of MedicineIntroduction Quality improvement (QI) competencies for health professions trainees were developed to address health care quality. Strategies to integrate QI into curricula exist, but methods for assessing interdisciplinary learners’ competency are less developed. We refined the Knowledge section scoring rubric of the Systems Quality Improvement Training and Assessment Tool (SQI TAT) and examined its validity evidence. Methods In 2017, the SQI TAT Knowledge section was expanded to cover seven core QI concepts, and the scoring rubric was refined. Three coders independently scored 35 SQI TAT Knowledge sections (18 pretests, 17 posttests). Interrater reliability was assessed by percent agreement and Cohen's kappa for individual variables and by Lin's concordance correlation for total scores for knowledge and application. Concurrent validity was assessed by comparing responses from two groups with different QI exposure and evaluating whether differences in exposure were measured. Results Total-score interrater reliability average measures of concordance were .89 for all coders and >.70 for six of seven concept scores. The total score discriminated the two groups (p <. 05), and five of seven concept scores were higher for the group with more QI experience. Total scores were significantly higher posttest than pretest (p < .001), with improvement in posttest knowledge scores. Discussion The SQI TAT Knowledge section provides a comprehensive assessment of QI knowledge. The scoring rubric was able to discriminate QI knowledge along a continuum. The SQI TAT Knowledge section is not linked to a clinical context, making it useful for assessing interprofessional learners and varying education levels.http://www.mededportal.org/doi/10.15766/mep_2374-8265.11290AssessmentCompetency-Based Medical Education (Competencies, Milestones, EPAs)Curriculum DevelopmentInterprofessional EducationProgram EvaluationQuality Improvement/Patient Safety |
| spellingShingle | Corrine Abraham Krysta Johnson-Martinez Anne Tomolo A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool MedEdPORTAL Assessment Competency-Based Medical Education (Competencies, Milestones, EPAs) Curriculum Development Interprofessional Education Program Evaluation Quality Improvement/Patient Safety |
| title | A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool |
| title_full | A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool |
| title_fullStr | A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool |
| title_full_unstemmed | A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool |
| title_short | A Scoring Rubric for the Knowledge Section of the Systems Quality Improvement Training and Assessment Tool |
| title_sort | scoring rubric for the knowledge section of the systems quality improvement training and assessment tool |
| topic | Assessment Competency-Based Medical Education (Competencies, Milestones, EPAs) Curriculum Development Interprofessional Education Program Evaluation Quality Improvement/Patient Safety |
| url | http://www.mededportal.org/doi/10.15766/mep_2374-8265.11290 |
| work_keys_str_mv | AT corrineabraham ascoringrubricfortheknowledgesectionofthesystemsqualityimprovementtrainingandassessmenttool AT krystajohnsonmartinez ascoringrubricfortheknowledgesectionofthesystemsqualityimprovementtrainingandassessmenttool AT annetomolo ascoringrubricfortheknowledgesectionofthesystemsqualityimprovementtrainingandassessmenttool AT corrineabraham scoringrubricfortheknowledgesectionofthesystemsqualityimprovementtrainingandassessmenttool AT krystajohnsonmartinez scoringrubricfortheknowledgesectionofthesystemsqualityimprovementtrainingandassessmenttool AT annetomolo scoringrubricfortheknowledgesectionofthesystemsqualityimprovementtrainingandassessmenttool |