Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students
Salah Eldin Kassab,1 Mubarak Bidmos,1 Michail Nomikos,1 Suhad Daher-Nashif,2 Tanya Kane,2 Srikant Sarangi,3 Marwan Abu-Hijleh1 1Department of Basic Medical Sciences, College of Medicine, QU Health, Qatar University, Doha, Qatar; 2Department of Population Medicine, College of Medicine, Qatar Universi...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Dove Medical Press
2020-06-01
|
| Series: | Advances in Medical Education and Practice |
| Subjects: | |
| Online Access: | https://www.dovepress.com/construct-validity-of-an-instrument-for-assessment-of-reflective-writi-peer-reviewed-article-AMEP |
| _version_ | 1818292511013076992 |
|---|---|
| author | Kassab SE Bidmos M Nomikos M Daher-Nashif S Kane T Sarangi S Abu-Hijleh M |
| author_facet | Kassab SE Bidmos M Nomikos M Daher-Nashif S Kane T Sarangi S Abu-Hijleh M |
| author_sort | Kassab SE |
| collection | DOAJ |
| description | Salah Eldin Kassab,1 Mubarak Bidmos,1 Michail Nomikos,1 Suhad Daher-Nashif,2 Tanya Kane,2 Srikant Sarangi,3 Marwan Abu-Hijleh1 1Department of Basic Medical Sciences, College of Medicine, QU Health, Qatar University, Doha, Qatar; 2Department of Population Medicine, College of Medicine, Qatar University, QU Health, Doha, Qatar; 3Danish Institute of Humanities and Medicine (DIHM), Aalborg University, Aalborg, DenmarkCorrespondence: Salah Eldin KassabPhysiology & Medical Education, College of Medicine, QU Health, Qatar University, PO Box 2713, Doha, QatarTel +974 – 4403 7843Email skassab@qu.edu.qaPurpose: Assessment of reflective writing for medical students is challenging, and there is lack of an available instrument with good psychometric properties. The authors developed a new instrument for assessment of reflective writing-based portfolios and examined the construct validity of this instrument.Methods: After an extensive literature review and pilot testing of the instrument, two raters assessed the reflective writing-based portfolios from years 2 and 3 medical students (n=135) on three occasions. The instrument consists of three criteria: organization, description of an experience and reflection on the experience. We calculated the reliability of scores using generalizability theory with a fully crossed design and two facets (raters and occasions). In addition, we measured criterion validity by testing correlations with students’ scores using other assessment methods.Results: The dependability (Φ) coefficient of the portfolio scores was 0.75 using two raters on three occasions. Students’ portfolio scores represented 46.6% of the total variance across all score comparisons. The variance due to occasions was negligible, while the student–occasion interaction was small. The variance due to student–rater interaction represented 17.7%, and the remaining 27.7% of the variance was due to unexplained sources of error. The decision (D) study suggested that an acceptable dependability (Φ = 0.70 and 0.72) can be achieved by using two raters for one and two occasions, respectively. Finally, we found moderate to large effect-size correlations between students’ scores in reflective writing-based portfolios and communication skills (r = 0.47) and PBL tutorials (r = 0.50).Conclusion: We demonstrated the presence of different sources of evidence that support construct validity of the study instrument. Further studies are warranted before utilizing this instrument for summative assessment of students’ reflective writing-based portfolios in other medical schools.Keywords: G-theory, validity, reflective writing, portfolio, reliability, student assessment |
| first_indexed | 2024-12-13T03:01:07Z |
| format | Article |
| id | doaj.art-042e1ba156f645a2a8df2952aafa222c |
| institution | Directory Open Access Journal |
| issn | 1179-7258 |
| language | English |
| last_indexed | 2024-12-13T03:01:07Z |
| publishDate | 2020-06-01 |
| publisher | Dove Medical Press |
| record_format | Article |
| series | Advances in Medical Education and Practice |
| spelling | doaj.art-042e1ba156f645a2a8df2952aafa222c2022-12-22T00:01:49ZengDove Medical PressAdvances in Medical Education and Practice1179-72582020-06-01Volume 1139740454242Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical StudentsKassab SEBidmos MNomikos MDaher-Nashif SKane TSarangi SAbu-Hijleh MSalah Eldin Kassab,1 Mubarak Bidmos,1 Michail Nomikos,1 Suhad Daher-Nashif,2 Tanya Kane,2 Srikant Sarangi,3 Marwan Abu-Hijleh1 1Department of Basic Medical Sciences, College of Medicine, QU Health, Qatar University, Doha, Qatar; 2Department of Population Medicine, College of Medicine, Qatar University, QU Health, Doha, Qatar; 3Danish Institute of Humanities and Medicine (DIHM), Aalborg University, Aalborg, DenmarkCorrespondence: Salah Eldin KassabPhysiology & Medical Education, College of Medicine, QU Health, Qatar University, PO Box 2713, Doha, QatarTel +974 – 4403 7843Email skassab@qu.edu.qaPurpose: Assessment of reflective writing for medical students is challenging, and there is lack of an available instrument with good psychometric properties. The authors developed a new instrument for assessment of reflective writing-based portfolios and examined the construct validity of this instrument.Methods: After an extensive literature review and pilot testing of the instrument, two raters assessed the reflective writing-based portfolios from years 2 and 3 medical students (n=135) on three occasions. The instrument consists of three criteria: organization, description of an experience and reflection on the experience. We calculated the reliability of scores using generalizability theory with a fully crossed design and two facets (raters and occasions). In addition, we measured criterion validity by testing correlations with students’ scores using other assessment methods.Results: The dependability (Φ) coefficient of the portfolio scores was 0.75 using two raters on three occasions. Students’ portfolio scores represented 46.6% of the total variance across all score comparisons. The variance due to occasions was negligible, while the student–occasion interaction was small. The variance due to student–rater interaction represented 17.7%, and the remaining 27.7% of the variance was due to unexplained sources of error. The decision (D) study suggested that an acceptable dependability (Φ = 0.70 and 0.72) can be achieved by using two raters for one and two occasions, respectively. Finally, we found moderate to large effect-size correlations between students’ scores in reflective writing-based portfolios and communication skills (r = 0.47) and PBL tutorials (r = 0.50).Conclusion: We demonstrated the presence of different sources of evidence that support construct validity of the study instrument. Further studies are warranted before utilizing this instrument for summative assessment of students’ reflective writing-based portfolios in other medical schools.Keywords: G-theory, validity, reflective writing, portfolio, reliability, student assessmenthttps://www.dovepress.com/construct-validity-of-an-instrument-for-assessment-of-reflective-writi-peer-reviewed-article-AMEPg-theoryvalidityreflective writingportfolioreliabilitystudent assessment. |
| spellingShingle | Kassab SE Bidmos M Nomikos M Daher-Nashif S Kane T Sarangi S Abu-Hijleh M Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students Advances in Medical Education and Practice g-theory validity reflective writing portfolio reliability student assessment. |
| title | Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students |
| title_full | Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students |
| title_fullStr | Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students |
| title_full_unstemmed | Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students |
| title_short | Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students |
| title_sort | construct validity of an instrument for assessment of reflective writing based portfolios of medical students |
| topic | g-theory validity reflective writing portfolio reliability student assessment. |
| url | https://www.dovepress.com/construct-validity-of-an-instrument-for-assessment-of-reflective-writi-peer-reviewed-article-AMEP |
| work_keys_str_mv | AT kassabse constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents AT bidmosm constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents AT nomikosm constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents AT dahernashifs constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents AT kanet constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents AT sarangis constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents AT abuhijlehm constructvalidityofaninstrumentforassessmentofreflectivewritingbasedportfoliosofmedicalstudents |