Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response
Abstract Background A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delp...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2020-06-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s12874-020-01029-4 |
| _version_ | 1828383604405698560 |
|---|---|
| author | Rebekka Veugelers Menno I. Gaakeer Peter Patka Robbert Huijsman |
| author_facet | Rebekka Veugelers Menno I. Gaakeer Peter Patka Robbert Huijsman |
| author_sort | Rebekka Veugelers |
| collection | DOAJ |
| description | Abstract Background A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delphi methodology issues are rare. Since the introduction of electronic surveys, details on response times remain scarce. We aim to bridge a number of gaps by providing a real world example covering methodological choices and response times in detail. Methods The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We opted for a two-part design with explicit decision rules. Part 1 focused on gathering and defining items; Part 2 addressed the main research question using an online survey tool. A two-person consensus rule was applied throughout: even after consensus on specific items was reached, panellists could reopen the discussion as long as at least two panellists argued similarly. Per round, the number of reminders sent and individual response times were noted. We also recorded the methodological considerations and evaluations made by the research team prior to as well as during the study. Results The study was performed in eight rounds and an additional confirmation round. Response rates were 100% in all rounds, resulting in 100% consensus in Part 1 and 96% consensus in Part 2. Our decision rules proved to be stable and easily applicable. Items with negative advice required more rounds before consensus was reached. Response delays were mostly due to late starts, but once panellists started, they nearly always finished the questionnaire on the same day. Reminders often yielded rapid responses. Intra-individual differences in response time were large, but quick responders remained quick. Conclusions We advise those considering Delphi study to follow the CREDES guideline, consider a two-part design, invest in personal commitment of the panellists, set clear decision rules, use a consistent lay-out and send out your reminders early. Adopting this overall approach may assist researchers in future Delphi studies and may help to improve the quality of Delphi designs in terms of improved rigor and higher response rates. |
| first_indexed | 2024-12-10T04:52:03Z |
| format | Article |
| id | doaj.art-3e1b15f6cfbc4ca2b0293fd81ebd585c |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-12-10T04:52:03Z |
| publishDate | 2020-06-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-3e1b15f6cfbc4ca2b0293fd81ebd585c2022-12-22T02:01:35ZengBMCBMC Medical Research Methodology1471-22882020-06-0120111510.1186/s12874-020-01029-4Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% responseRebekka Veugelers0Menno I. Gaakeer1Peter Patka2Robbert Huijsman3Emergency Department, Adrz hospitalEmergency Department, Adrz hospitalEmergency Department, Erasmus MCErasmus School of Health Policy & ManagementAbstract Background A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delphi methodology issues are rare. Since the introduction of electronic surveys, details on response times remain scarce. We aim to bridge a number of gaps by providing a real world example covering methodological choices and response times in detail. Methods The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We opted for a two-part design with explicit decision rules. Part 1 focused on gathering and defining items; Part 2 addressed the main research question using an online survey tool. A two-person consensus rule was applied throughout: even after consensus on specific items was reached, panellists could reopen the discussion as long as at least two panellists argued similarly. Per round, the number of reminders sent and individual response times were noted. We also recorded the methodological considerations and evaluations made by the research team prior to as well as during the study. Results The study was performed in eight rounds and an additional confirmation round. Response rates were 100% in all rounds, resulting in 100% consensus in Part 1 and 96% consensus in Part 2. Our decision rules proved to be stable and easily applicable. Items with negative advice required more rounds before consensus was reached. Response delays were mostly due to late starts, but once panellists started, they nearly always finished the questionnaire on the same day. Reminders often yielded rapid responses. Intra-individual differences in response time were large, but quick responders remained quick. Conclusions We advise those considering Delphi study to follow the CREDES guideline, consider a two-part design, invest in personal commitment of the panellists, set clear decision rules, use a consistent lay-out and send out your reminders early. Adopting this overall approach may assist researchers in future Delphi studies and may help to improve the quality of Delphi designs in terms of improved rigor and higher response rates.http://link.springer.com/article/10.1186/s12874-020-01029-4E-DelphiDelphi techniqueEmergency medicineEmergency serviceHospitalResponse rate |
| spellingShingle | Rebekka Veugelers Menno I. Gaakeer Peter Patka Robbert Huijsman Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response BMC Medical Research Methodology E-Delphi Delphi technique Emergency medicine Emergency service Hospital Response rate |
| title | Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response |
| title_full | Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response |
| title_fullStr | Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response |
| title_full_unstemmed | Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response |
| title_short | Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response |
| title_sort | improving design choices in delphi studies in medicine the case of an exemplary physician multi round panel study with 100 response |
| topic | E-Delphi Delphi technique Emergency medicine Emergency service Hospital Response rate |
| url | http://link.springer.com/article/10.1186/s12874-020-01029-4 |
| work_keys_str_mv | AT rebekkaveugelers improvingdesignchoicesindelphistudiesinmedicinethecaseofanexemplaryphysicianmultiroundpanelstudywith100response AT mennoigaakeer improvingdesignchoicesindelphistudiesinmedicinethecaseofanexemplaryphysicianmultiroundpanelstudywith100response AT peterpatka improvingdesignchoicesindelphistudiesinmedicinethecaseofanexemplaryphysicianmultiroundpanelstudywith100response AT robberthuijsman improvingdesignchoicesindelphistudiesinmedicinethecaseofanexemplaryphysicianmultiroundpanelstudywith100response |