Reproducible machine learning research in mental workload classification using EEG
This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2024-04-01
|
| Series: | Frontiers in Neuroergonomics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnrgo.2024.1346794/full |
| _version_ | 1797215885963821056 |
|---|---|
| author | Güliz Demirezen Tuğba Taşkaya Temizel Anne-Marie Brouwer Anne-Marie Brouwer |
| author_facet | Güliz Demirezen Tuğba Taşkaya Temizel Anne-Marie Brouwer Anne-Marie Brouwer |
| author_sort | Güliz Demirezen |
| collection | DOAJ |
| description | This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes. |
| first_indexed | 2024-04-24T11:37:11Z |
| format | Article |
| id | doaj.art-5848944514ab454e90b615f3c3d61696 |
| institution | Directory Open Access Journal |
| issn | 2673-6195 |
| language | English |
| last_indexed | 2024-04-24T11:37:11Z |
| publishDate | 2024-04-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroergonomics |
| spelling | doaj.art-5848944514ab454e90b615f3c3d616962024-04-10T04:39:59ZengFrontiers Media S.A.Frontiers in Neuroergonomics2673-61952024-04-01510.3389/fnrgo.2024.13467941346794Reproducible machine learning research in mental workload classification using EEGGüliz Demirezen0Tuğba Taşkaya Temizel1Anne-Marie Brouwer2Anne-Marie Brouwer3Department of Information Systems, Graduate School of Informatics, Middle East Technical University, Ankara, TürkiyeDepartment of Data Informatics, Graduate School of Informatics, Middle East Technical University, Ankara, TürkiyeHuman Performance, Netherlands Organisation for Applied Scientific Research (TNO), Soesterberg, NetherlandsDonders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, NetherlandsThis study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes.https://www.frontiersin.org/articles/10.3389/fnrgo.2024.1346794/fullneuroergonomicsreproducibilityEEGphysiological measurementmental workloadmachine learning |
| spellingShingle | Güliz Demirezen Tuğba Taşkaya Temizel Anne-Marie Brouwer Anne-Marie Brouwer Reproducible machine learning research in mental workload classification using EEG Frontiers in Neuroergonomics neuroergonomics reproducibility EEG physiological measurement mental workload machine learning |
| title | Reproducible machine learning research in mental workload classification using EEG |
| title_full | Reproducible machine learning research in mental workload classification using EEG |
| title_fullStr | Reproducible machine learning research in mental workload classification using EEG |
| title_full_unstemmed | Reproducible machine learning research in mental workload classification using EEG |
| title_short | Reproducible machine learning research in mental workload classification using EEG |
| title_sort | reproducible machine learning research in mental workload classification using eeg |
| topic | neuroergonomics reproducibility EEG physiological measurement mental workload machine learning |
| url | https://www.frontiersin.org/articles/10.3389/fnrgo.2024.1346794/full |
| work_keys_str_mv | AT gulizdemirezen reproduciblemachinelearningresearchinmentalworkloadclassificationusingeeg AT tugbataskayatemizel reproduciblemachinelearningresearchinmentalworkloadclassificationusingeeg AT annemariebrouwer reproduciblemachinelearningresearchinmentalworkloadclassificationusingeeg AT annemariebrouwer reproduciblemachinelearningresearchinmentalworkloadclassificationusingeeg |