Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention
ObjectiveWe examined the effect of a simple Delphi-method feedback on visual identification of high frequency oscillations (HFOs) in the ripple (80–250 Hz) band, and assessed the impact of this training intervention on the interrater reliability and generalizability of HFO evaluations.MethodsWe empl...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-02-01
|
| Series: | Frontiers in Neurology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fneur.2022.794668/full |
| _version_ | 1819279061405401088 |
|---|---|
| author | Aaron M. Spring Aaron M. Spring Aaron M. Spring Daniel J. Pittman Daniel J. Pittman Arsalan Rizwan Arsalan Rizwan Yahya Aghakhani Jeffrey Jirsch Mary Connolly Samuel Wiebe Samuel Wiebe Juan Pablo Appendino Juan Pablo Appendino Anita Datta Trevor Steve Neelan Pillay Manouchehr Javidan Morris Scantlebury Morris Scantlebury Chantelle Hrazdil Colin Bruce Josephson Colin Bruce Josephson Cyrus Boelman Donald Gross Shaily Singh Luis Bello-Espinosa Luis Bello-Espinosa Linda Huh Nathalie Jetté Paolo Federico Paolo Federico Paolo Federico |
| author_facet | Aaron M. Spring Aaron M. Spring Aaron M. Spring Daniel J. Pittman Daniel J. Pittman Arsalan Rizwan Arsalan Rizwan Yahya Aghakhani Jeffrey Jirsch Mary Connolly Samuel Wiebe Samuel Wiebe Juan Pablo Appendino Juan Pablo Appendino Anita Datta Trevor Steve Neelan Pillay Manouchehr Javidan Morris Scantlebury Morris Scantlebury Chantelle Hrazdil Colin Bruce Josephson Colin Bruce Josephson Cyrus Boelman Donald Gross Shaily Singh Luis Bello-Espinosa Luis Bello-Espinosa Linda Huh Nathalie Jetté Paolo Federico Paolo Federico Paolo Federico |
| author_sort | Aaron M. Spring |
| collection | DOAJ |
| description | ObjectiveWe examined the effect of a simple Delphi-method feedback on visual identification of high frequency oscillations (HFOs) in the ripple (80–250 Hz) band, and assessed the impact of this training intervention on the interrater reliability and generalizability of HFO evaluations.MethodsWe employed a morphology detector to identify potential HFOs at two thresholds and presented them to visual reviewers to assess the probability of each epoch containing an HFO. We recruited 19 board-certified epileptologists with various levels of experience to complete a series of HFO evaluations during three sessions. A Delphi-style intervention was used to provide feedback on the performance of each reviewer relative to their peers. A delayed-intervention paradigm was used, in which reviewers received feedback either before or after the second session. ANOVAs were used to assess the effect of the intervention on the reviewers' evaluations. Generalizability theory was used to assess the interrater reliability before and after the intervention.ResultsThe intervention, regardless of when it occurred, resulted in a significant reduction in the variability between reviewers in both groups (pGroupDI = 0.037, pGroupEI = 0.003). Prior to the delayed-intervention, the group receiving the early intervention showed a significant reduction in variability (pGroupEI = 0.041), but the delayed-intervention group did not (pGroupDI = 0.414). Following the intervention, the projected number of reviewers required to achieve strong generalizability decreased from 35 to 16.SignificanceThis study shows a robust effect of a Delphi-style intervention on the interrater variability, reliability, and generalizability of HFO evaluations. The observed decreases in HFO marking discrepancies across 14 of the 15 reviewers are encouraging: they are necessarily associated with an increase in interrater reliability, and therefore with a corresponding decrease in the number of reviewers required to achieve strong generalizability. Indeed, the reliability of all reviewers following the intervention was similar to that of experienced reviewers prior to intervention. Therefore, a Delphi-style intervention could be implemented either to sufficiently train any reviewer, or to further refine the interrater reliability of experienced reviewers. In either case, a Delphi-style intervention would help facilitate the standardization of HFO evaluations and its implementation in clinical care. |
| first_indexed | 2024-12-24T00:21:55Z |
| format | Article |
| id | doaj.art-74f9f315f5dc4ef38f83be3306633fba |
| institution | Directory Open Access Journal |
| issn | 1664-2295 |
| language | English |
| last_indexed | 2024-12-24T00:21:55Z |
| publishDate | 2022-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurology |
| spelling | doaj.art-74f9f315f5dc4ef38f83be3306633fba2022-12-21T17:24:34ZengFrontiers Media S.A.Frontiers in Neurology1664-22952022-02-011310.3389/fneur.2022.794668794668Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style InterventionAaron M. Spring0Aaron M. Spring1Aaron M. Spring2Daniel J. Pittman3Daniel J. Pittman4Arsalan Rizwan5Arsalan Rizwan6Yahya Aghakhani7Jeffrey Jirsch8Mary Connolly9Samuel Wiebe10Samuel Wiebe11Juan Pablo Appendino12Juan Pablo Appendino13Anita Datta14Trevor Steve15Neelan Pillay16Manouchehr Javidan17Morris Scantlebury18Morris Scantlebury19Chantelle Hrazdil20Colin Bruce Josephson21Colin Bruce Josephson22Cyrus Boelman23Donald Gross24Shaily Singh25Luis Bello-Espinosa26Luis Bello-Espinosa27Linda Huh28Nathalie Jetté29Paolo Federico30Paolo Federico31Paolo Federico32Department of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaHotchkiss Brain Institute, University of Calgary, Calgary, AB, CanadaSeaman Family MR Research Centre, Foothills Medical Centre, Calgary, AB, CanadaHotchkiss Brain Institute, University of Calgary, Calgary, AB, CanadaSeaman Family MR Research Centre, Foothills Medical Centre, Calgary, AB, CanadaSeaman Family MR Research Centre, Foothills Medical Centre, Calgary, AB, CanadaDepartment of Medicine, Queen's University, Kingston, ON, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaDepartment of Medicine, University of Alberta, Edmonton, AB, CanadaDepartment of Pediatrics, University of British Columbia, Vancouver, BC, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaHotchkiss Brain Institute, University of Calgary, Calgary, AB, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaDepartment of Pediatrics, University of Calgary, Calgary, AB, CanadaDepartment of Pediatrics, University of British Columbia, Vancouver, BC, CanadaDepartment of Medicine, University of Alberta, Edmonton, AB, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaDepartment of Medicine, University of British Columbia, Vancouver, BC, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaDepartment of Pediatrics, University of Calgary, Calgary, AB, CanadaDepartment of Medicine, University of British Columbia, Vancouver, BC, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaHotchkiss Brain Institute, University of Calgary, Calgary, AB, CanadaDepartment of Pediatrics, University of British Columbia, Vancouver, BC, CanadaDepartment of Medicine, University of Alberta, Edmonton, AB, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaDepartment of Pediatrics, University of Calgary, Calgary, AB, CanadaDepartment of Pediatrics, University of British Columbia, Vancouver, BC, CanadaDepartment of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, United StatesDepartment of Clinical Neurosciences, University of Calgary, Calgary, AB, CanadaHotchkiss Brain Institute, University of Calgary, Calgary, AB, CanadaSeaman Family MR Research Centre, Foothills Medical Centre, Calgary, AB, CanadaObjectiveWe examined the effect of a simple Delphi-method feedback on visual identification of high frequency oscillations (HFOs) in the ripple (80–250 Hz) band, and assessed the impact of this training intervention on the interrater reliability and generalizability of HFO evaluations.MethodsWe employed a morphology detector to identify potential HFOs at two thresholds and presented them to visual reviewers to assess the probability of each epoch containing an HFO. We recruited 19 board-certified epileptologists with various levels of experience to complete a series of HFO evaluations during three sessions. A Delphi-style intervention was used to provide feedback on the performance of each reviewer relative to their peers. A delayed-intervention paradigm was used, in which reviewers received feedback either before or after the second session. ANOVAs were used to assess the effect of the intervention on the reviewers' evaluations. Generalizability theory was used to assess the interrater reliability before and after the intervention.ResultsThe intervention, regardless of when it occurred, resulted in a significant reduction in the variability between reviewers in both groups (pGroupDI = 0.037, pGroupEI = 0.003). Prior to the delayed-intervention, the group receiving the early intervention showed a significant reduction in variability (pGroupEI = 0.041), but the delayed-intervention group did not (pGroupDI = 0.414). Following the intervention, the projected number of reviewers required to achieve strong generalizability decreased from 35 to 16.SignificanceThis study shows a robust effect of a Delphi-style intervention on the interrater variability, reliability, and generalizability of HFO evaluations. The observed decreases in HFO marking discrepancies across 14 of the 15 reviewers are encouraging: they are necessarily associated with an increase in interrater reliability, and therefore with a corresponding decrease in the number of reviewers required to achieve strong generalizability. Indeed, the reliability of all reviewers following the intervention was similar to that of experienced reviewers prior to intervention. Therefore, a Delphi-style intervention could be implemented either to sufficiently train any reviewer, or to further refine the interrater reliability of experienced reviewers. In either case, a Delphi-style intervention would help facilitate the standardization of HFO evaluations and its implementation in clinical care.https://www.frontiersin.org/articles/10.3389/fneur.2022.794668/fullhigh frequency oscillations (HFO)generalizability theoryDelphi methodtrainingfeedbackinterrater reliability |
| spellingShingle | Aaron M. Spring Aaron M. Spring Aaron M. Spring Daniel J. Pittman Daniel J. Pittman Arsalan Rizwan Arsalan Rizwan Yahya Aghakhani Jeffrey Jirsch Mary Connolly Samuel Wiebe Samuel Wiebe Juan Pablo Appendino Juan Pablo Appendino Anita Datta Trevor Steve Neelan Pillay Manouchehr Javidan Morris Scantlebury Morris Scantlebury Chantelle Hrazdil Colin Bruce Josephson Colin Bruce Josephson Cyrus Boelman Donald Gross Shaily Singh Luis Bello-Espinosa Luis Bello-Espinosa Linda Huh Nathalie Jetté Paolo Federico Paolo Federico Paolo Federico Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention Frontiers in Neurology high frequency oscillations (HFO) generalizability theory Delphi method training feedback interrater reliability |
| title | Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention |
| title_full | Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention |
| title_fullStr | Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention |
| title_full_unstemmed | Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention |
| title_short | Effect of Training on Visual Identification of High Frequency Oscillations—A Delphi-Style Intervention |
| title_sort | effect of training on visual identification of high frequency oscillations a delphi style intervention |
| topic | high frequency oscillations (HFO) generalizability theory Delphi method training feedback interrater reliability |
| url | https://www.frontiersin.org/articles/10.3389/fneur.2022.794668/full |
| work_keys_str_mv | AT aaronmspring effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT aaronmspring effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT aaronmspring effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT danieljpittman effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT danieljpittman effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT arsalanrizwan effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT arsalanrizwan effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT yahyaaghakhani effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT jeffreyjirsch effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT maryconnolly effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT samuelwiebe effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT samuelwiebe effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT juanpabloappendino effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT juanpabloappendino effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT anitadatta effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT trevorsteve effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT neelanpillay effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT manouchehrjavidan effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT morrisscantlebury effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT morrisscantlebury effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT chantellehrazdil effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT colinbrucejosephson effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT colinbrucejosephson effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT cyrusboelman effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT donaldgross effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT shailysingh effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT luisbelloespinosa effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT luisbelloespinosa effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT lindahuh effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT nathaliejette effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT paolofederico effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT paolofederico effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention AT paolofederico effectoftrainingonvisualidentificationofhighfrequencyoscillationsadelphistyleintervention |