Standard setting anchor statements: a double cross-over trial of two different methods
Context: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item....
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
F1000 Research Ltd
2021-02-01
|
| Series: | MedEdPublish |
| Subjects: | |
| Online Access: | https://www.mededpublish.org/Manuscripts/3529 |
| _version_ | 1819047870342365184 |
|---|---|
| author | Steven Burr Theresa Martin James Edwards Colin Ferguson Kerry Gilbert Christian Gray Adele Hill Joanne Hosking Karen Johnstone Jolanta Kisielewska Chloe Milsom Siobhan Moyes Ann Rigby-Jones Iain Robinson Nick Toms Helen Watson Daniel Zahra |
| author_facet | Steven Burr Theresa Martin James Edwards Colin Ferguson Kerry Gilbert Christian Gray Adele Hill Joanne Hosking Karen Johnstone Jolanta Kisielewska Chloe Milsom Siobhan Moyes Ann Rigby-Jones Iain Robinson Nick Toms Helen Watson Daniel Zahra |
| author_sort | Steven Burr |
| collection | DOAJ |
| description | Context: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item.
Methods: The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods.
Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%).
Conclusion: This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method. |
| first_indexed | 2024-12-21T11:07:14Z |
| format | Article |
| id | doaj.art-667a7ce34f7641c386ed59643c8d6fc0 |
| institution | Directory Open Access Journal |
| issn | 2312-7996 |
| language | English |
| last_indexed | 2024-12-21T11:07:14Z |
| publishDate | 2021-02-01 |
| publisher | F1000 Research Ltd |
| record_format | Article |
| series | MedEdPublish |
| spelling | doaj.art-667a7ce34f7641c386ed59643c8d6fc02022-12-21T19:06:11ZengF1000 Research LtdMedEdPublish2312-79962021-02-01101Standard setting anchor statements: a double cross-over trial of two different methodsSteven Burr0Theresa Martin1James Edwards2Colin Ferguson3Kerry Gilbert4Christian Gray5Adele Hill6Joanne Hosking7Karen Johnstone8Jolanta Kisielewska9Chloe Milsom10Siobhan Moyes11Ann Rigby-Jones12Iain Robinson13Nick Toms14Helen Watson15Daniel Zahra16Peninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthSchool of Clinical Medicine, University of QueenslandPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthPeninsula Medical School, University of PlymouthContext: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods: The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods. Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion: This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.https://www.mededpublish.org/Manuscripts/3529Angoffstandard settingassessment |
| spellingShingle | Steven Burr Theresa Martin James Edwards Colin Ferguson Kerry Gilbert Christian Gray Adele Hill Joanne Hosking Karen Johnstone Jolanta Kisielewska Chloe Milsom Siobhan Moyes Ann Rigby-Jones Iain Robinson Nick Toms Helen Watson Daniel Zahra Standard setting anchor statements: a double cross-over trial of two different methods MedEdPublish Angoff standard setting assessment |
| title | Standard setting anchor statements: a double cross-over trial of two different methods |
| title_full | Standard setting anchor statements: a double cross-over trial of two different methods |
| title_fullStr | Standard setting anchor statements: a double cross-over trial of two different methods |
| title_full_unstemmed | Standard setting anchor statements: a double cross-over trial of two different methods |
| title_short | Standard setting anchor statements: a double cross-over trial of two different methods |
| title_sort | standard setting anchor statements a double cross over trial of two different methods |
| topic | Angoff standard setting assessment |
| url | https://www.mededpublish.org/Manuscripts/3529 |
| work_keys_str_mv | AT stevenburr standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT theresamartin standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT jamesedwards standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT colinferguson standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT kerrygilbert standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT christiangray standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT adelehill standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT joannehosking standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT karenjohnstone standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT jolantakisielewska standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT chloemilsom standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT siobhanmoyes standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT annrigbyjones standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT iainrobinson standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT nicktoms standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT helenwatson standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods AT danielzahra standardsettinganchorstatementsadoublecrossovertrialoftwodifferentmethods |