Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study
Purpose Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee method...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Korea Health Personnel Licensing Examination Institute
2022-10-01
|
| Series: | Journal of Educational Evaluation for Health Professions |
| Subjects: | |
| Online Access: | http://www.jeehp.org/upload/jeehp-19-27.pdf |
| _version_ | 1797787263192530944 |
|---|---|
| author | Ying-Ying Yang Pin-Hsiang Huang Ling-Yu Yang Chia-Chang Huang Chih-Wei Liu Shiau-Shian Huang Chen-Huan Chen Fa-Yauh Lee Shou-Yen Kao Boaz Shulruf |
| author_facet | Ying-Ying Yang Pin-Hsiang Huang Ling-Yu Yang Chia-Chang Huang Chih-Wei Liu Shiau-Shian Huang Chen-Huan Chen Fa-Yauh Lee Shou-Yen Kao Boaz Shulruf |
| author_sort | Ying-Ying Yang |
| collection | DOAJ |
| description | Purpose Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method. Methods The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station. Results On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts. Conclusion The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills. |
| first_indexed | 2024-03-13T01:20:23Z |
| format | Article |
| id | doaj.art-78d7a53a6f484ce2955bd16be495035a |
| institution | Directory Open Access Journal |
| issn | 1975-5937 |
| language | English |
| last_indexed | 2024-03-13T01:20:23Z |
| publishDate | 2022-10-01 |
| publisher | Korea Health Personnel Licensing Examination Institute |
| record_format | Article |
| series | Journal of Educational Evaluation for Health Professions |
| spelling | doaj.art-78d7a53a6f484ce2955bd16be495035a2023-07-05T04:44:27ZengKorea Health Personnel Licensing Examination InstituteJournal of Educational Evaluation for Health Professions1975-59372022-10-011910.3352/jeehp.2022.19.27452Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation studyYing-Ying Yang0Pin-Hsiang Huang1Ling-Yu Yang2Chia-Chang Huang3Chih-Wei Liu4Shiau-Shian Huang5Chen-Huan Chen6Fa-Yauh Lee7Shou-Yen Kao8Boaz Shulruf9Division of Clinical Skills Training Center, Taipei Veterans General Hospital, Taipei, TaiwanNational Yang-Ming University, Taipei, TaiwanDepartment of Medical Education, Taipei Veterans General Hospital, Taipei, TaiwanDivision of Clinical Skills Training Center, Taipei Veterans General Hospital, Taipei, TaiwanDivision of Clinical Skills Training Center, Taipei Veterans General Hospital, Taipei, TaiwanDivision of Clinical Skills Training Center, Taipei Veterans General Hospital, Taipei, TaiwanNational Yang-Ming University, Taipei, TaiwanDepartment of Medical Education, Taipei Veterans General Hospital, Taipei, TaiwanDepartment of Medical Education, Taipei Veterans General Hospital, Taipei, TaiwanUniversity of New South Wales, Sydney, AustraliaPurpose Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method. Methods The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station. Results On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts. Conclusion The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills.http://www.jeehp.org/upload/jeehp-19-27.pdfcomputer simulationeducational measurementstandardstaiwanundergraduate medical education |
| spellingShingle | Ying-Ying Yang Pin-Hsiang Huang Ling-Yu Yang Chia-Chang Huang Chih-Wei Liu Shiau-Shian Huang Chen-Huan Chen Fa-Yauh Lee Shou-Yen Kao Boaz Shulruf Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study Journal of Educational Evaluation for Health Professions computer simulation educational measurement standards taiwan undergraduate medical education |
| title | Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study |
| title_full | Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study |
| title_fullStr | Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study |
| title_full_unstemmed | Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study |
| title_short | Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study |
| title_sort | equal z standard setting method to estimate the minimum number of panelists for a medical school s objective structured clinical examination in taiwan a simulation study |
| topic | computer simulation educational measurement standards taiwan undergraduate medical education |
| url | http://www.jeehp.org/upload/jeehp-19-27.pdf |
| work_keys_str_mv | AT yingyingyang equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT pinhsianghuang equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT lingyuyang equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT chiachanghuang equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT chihweiliu equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT shiaushianhuang equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT chenhuanchen equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT fayauhlee equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT shouyenkao equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy AT boazshulruf equalzstandardsettingmethodtoestimatetheminimumnumberofpanelistsforamedicalschoolsobjectivestructuredclinicalexaminationintaiwanasimulationstudy |