Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ?
The onset of the COVID-19 pandemic brought with it the closure of many bricks-and-mortar testing centres and a concomitant move to the remote (online) proctoring of many credentialing exams that were scheduled to take place throughout 2020. In effect, this meant changing from a standardized to a non...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2021-12-01
|
| Series: | Computers and Education Open |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S266655732100032X |
| _version_ | 1819174332703703040 |
|---|---|
| author | Gemma Cherry Michael O'Leary Oksana Naumenko Li-Ann Kuan Linda Waters |
| author_facet | Gemma Cherry Michael O'Leary Oksana Naumenko Li-Ann Kuan Linda Waters |
| author_sort | Gemma Cherry |
| collection | DOAJ |
| description | The onset of the COVID-19 pandemic brought with it the closure of many bricks-and-mortar testing centres and a concomitant move to the remote (online) proctoring of many credentialing exams that were scheduled to take place throughout 2020. In effect, this meant changing from a standardized to a non-standardized administration of tests given that candidates could take tests at home using personal desktop or laptop devices. This in turn begs the question: do outcomes across tests taken in testing centres and via live remote proctoring differ? Due to the relative novelty of remote proctoring, research addressing this question is currently thin on the ground, thus justifying the research described in this paper. The study uses data from eleven professional licensure examinations taken by 14,097 candidates across four US States to compare outcomes for tests proctored either in test centres or remotely in real-time using live remote proctoring (LRP) software. Candidate outcomes were compared using average percent correct and passing rates. Test psychometric properties were compared using KR-20 reliability, decision consistency (Subkoviak's c), item difficulty, item discrimination and time taken to complete tests. At the individual test level, while some statistically significant differences were observed in outcomes across the two proctoring modes, most effect sizes were small. Overall, with the exception of some observable differences in metrics pertaining to item discrimination and timing, no detectable pattern was observed in favour of either mode. The study is significant in that it provides early evidence supporting the use of LRP in high stakes contexts. |
| first_indexed | 2024-12-22T20:37:18Z |
| format | Article |
| id | doaj.art-8627eb04c211464a974a49a63e7aa507 |
| institution | Directory Open Access Journal |
| issn | 2666-5573 |
| language | English |
| last_indexed | 2024-12-22T20:37:18Z |
| publishDate | 2021-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education Open |
| spelling | doaj.art-8627eb04c211464a974a49a63e7aa5072022-12-21T18:13:26ZengElsevierComputers and Education Open2666-55732021-12-012100061Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ?Gemma Cherry0Michael O'Leary1Oksana Naumenko2Li-Ann Kuan3Linda Waters4Centre for Assessment Research, Policy and Practice in Education (CARPE), Institute of Education, Dublin City University, D09Y00, IrelandCentre for Assessment Research, Policy and Practice in Education (CARPE), Institute of Education, Dublin City University, D09Y00, Ireland; Corresponding author.Proemtric Inc, 501 South Clinton St, Baltimore, MD 21224, United StatesProemtric Inc, 501 South Clinton St, Baltimore, MD 21224, United StatesProemtric Inc, 501 South Clinton St, Baltimore, MD 21224, United StatesThe onset of the COVID-19 pandemic brought with it the closure of many bricks-and-mortar testing centres and a concomitant move to the remote (online) proctoring of many credentialing exams that were scheduled to take place throughout 2020. In effect, this meant changing from a standardized to a non-standardized administration of tests given that candidates could take tests at home using personal desktop or laptop devices. This in turn begs the question: do outcomes across tests taken in testing centres and via live remote proctoring differ? Due to the relative novelty of remote proctoring, research addressing this question is currently thin on the ground, thus justifying the research described in this paper. The study uses data from eleven professional licensure examinations taken by 14,097 candidates across four US States to compare outcomes for tests proctored either in test centres or remotely in real-time using live remote proctoring (LRP) software. Candidate outcomes were compared using average percent correct and passing rates. Test psychometric properties were compared using KR-20 reliability, decision consistency (Subkoviak's c), item difficulty, item discrimination and time taken to complete tests. At the individual test level, while some statistically significant differences were observed in outcomes across the two proctoring modes, most effect sizes were small. Overall, with the exception of some observable differences in metrics pertaining to item discrimination and timing, no detectable pattern was observed in favour of either mode. The study is significant in that it provides early evidence supporting the use of LRP in high stakes contexts.http://www.sciencedirect.com/science/article/pii/S266655732100032X |
| spellingShingle | Gemma Cherry Michael O'Leary Oksana Naumenko Li-Ann Kuan Linda Waters Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? Computers and Education Open |
| title | Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? |
| title_full | Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? |
| title_fullStr | Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? |
| title_full_unstemmed | Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? |
| title_short | Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? |
| title_sort | do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ |
| url | http://www.sciencedirect.com/science/article/pii/S266655732100032X |
| work_keys_str_mv | AT gemmacherry dooutcomesfromhighstakesexaminationstakenintestcentresandvialiveremoteproctoringdiffer AT michaeloleary dooutcomesfromhighstakesexaminationstakenintestcentresandvialiveremoteproctoringdiffer AT oksananaumenko dooutcomesfromhighstakesexaminationstakenintestcentresandvialiveremoteproctoringdiffer AT liannkuan dooutcomesfromhighstakesexaminationstakenintestcentresandvialiveremoteproctoringdiffer AT lindawaters dooutcomesfromhighstakesexaminationstakenintestcentresandvialiveremoteproctoringdiffer |