Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research
Machine learning (ML) is a technique that learns to detect patterns and trends in data. However, the quality of reporting ML in research is often suboptimal, leading to inaccurate conclusions and hindering progress in the field, especially if disseminated in literature reviews that provide researche...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10251957/ |
| _version_ | 1797649765231493120 |
|---|---|
| author | Ryan G. L. Koh Md Asif Khan Sajjad Rashidiani Samah Hassan Victoria Tucci Theodore Liu Karlo Nesovic Dinesh Kumbhare Thomas E. Doyle |
| author_facet | Ryan G. L. Koh Md Asif Khan Sajjad Rashidiani Samah Hassan Victoria Tucci Theodore Liu Karlo Nesovic Dinesh Kumbhare Thomas E. Doyle |
| author_sort | Ryan G. L. Koh |
| collection | DOAJ |
| description | Machine learning (ML) is a technique that learns to detect patterns and trends in data. However, the quality of reporting ML in research is often suboptimal, leading to inaccurate conclusions and hindering progress in the field, especially if disseminated in literature reviews that provide researchers with an overview of a field, current knowledge gaps, and future directions. While various tools are available to assess the quality and risk-of-bias of studies, there is currently no generalized tool for assessing the reporting quality of ML in the literature. To address this, this study presents a new screening tool called STAR-ML (Screening Tool for Assessing Reporting of Machine Learning), accompanied by a guide to using it. A pilot scoping review looking at ML in chronic pain was used to investigate the tool. The time it took to screen papers and how the selection of the threshold affected the papers included were explored. The tool provides researchers with a reliable and systematic way to evaluate the quality of reporting of ML studies and to make informed decisions about the inclusion of studies in scoping or systematic reviews. In addition, this study provides recommendations for authors on how to choose the threshold for inclusion and use the tool proficiently. Lastly, the STAR-ML tool can serve as a checklist for researchers seeking to develop or implement ML techniques effectively. |
| first_indexed | 2024-03-11T15:50:43Z |
| format | Article |
| id | doaj.art-2d61da8d88e44c83a243badebe623f15 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T15:50:43Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-2d61da8d88e44c83a243badebe623f152023-10-25T23:01:25ZengIEEEIEEE Access2169-35362023-01-011110156710157910.1109/ACCESS.2023.331601910251957Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in ResearchRyan G. L. Koh0https://orcid.org/0000-0001-8662-1008Md Asif Khan1https://orcid.org/0000-0001-8395-347XSajjad Rashidiani2https://orcid.org/0009-0008-1318-698XSamah Hassan3https://orcid.org/0000-0003-2526-4515Victoria Tucci4https://orcid.org/0000-0002-2344-2560Theodore Liu5https://orcid.org/0000-0001-7334-8129Karlo Nesovic6https://orcid.org/0000-0002-1520-954XDinesh Kumbhare7https://orcid.org/0000-0003-3889-7557Thomas E. Doyle8https://orcid.org/0000-0003-1059-110XKITE Research Institute, Toronto Rehabilitation Institute, University Health Network (UHN), Toronto, ON, CanadaDepartment of Electrical and Computer Engineering, McMaster University, Hamilton, ON, CanadaDepartment of Electrical and Computer Engineering, McMaster University, Hamilton, ON, CanadaThe Institute of Education Research (TIER), UHN, Toronto, ON, CanadaFaculty of Health Sciences, McMaster University, Hamilton, ON, CanadaDepartment of Electrical and Computer Engineering, McMaster University, Hamilton, ON, CanadaKITE Research Institute, Toronto Rehabilitation Institute, University Health Network (UHN), Toronto, ON, CanadaKITE Research Institute, Toronto Rehabilitation Institute, University Health Network (UHN), Toronto, ON, CanadaDepartment of Electrical and Computer Engineering, McMaster University, Hamilton, ON, CanadaMachine learning (ML) is a technique that learns to detect patterns and trends in data. However, the quality of reporting ML in research is often suboptimal, leading to inaccurate conclusions and hindering progress in the field, especially if disseminated in literature reviews that provide researchers with an overview of a field, current knowledge gaps, and future directions. While various tools are available to assess the quality and risk-of-bias of studies, there is currently no generalized tool for assessing the reporting quality of ML in the literature. To address this, this study presents a new screening tool called STAR-ML (Screening Tool for Assessing Reporting of Machine Learning), accompanied by a guide to using it. A pilot scoping review looking at ML in chronic pain was used to investigate the tool. The time it took to screen papers and how the selection of the threshold affected the papers included were explored. The tool provides researchers with a reliable and systematic way to evaluate the quality of reporting of ML studies and to make informed decisions about the inclusion of studies in scoping or systematic reviews. In addition, this study provides recommendations for authors on how to choose the threshold for inclusion and use the tool proficiently. Lastly, the STAR-ML tool can serve as a checklist for researchers seeking to develop or implement ML techniques effectively.https://ieeexplore.ieee.org/document/10251957/Checklistliterature reviewmachine learningquality scoringreporting assessmentresearch methodology |
| spellingShingle | Ryan G. L. Koh Md Asif Khan Sajjad Rashidiani Samah Hassan Victoria Tucci Theodore Liu Karlo Nesovic Dinesh Kumbhare Thomas E. Doyle Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research IEEE Access Checklist literature review machine learning quality scoring reporting assessment research methodology |
| title | Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research |
| title_full | Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research |
| title_fullStr | Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research |
| title_full_unstemmed | Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research |
| title_short | Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research |
| title_sort | check it before you wreck it a guide to star ml for screening machine learning reporting in research |
| topic | Checklist literature review machine learning quality scoring reporting assessment research methodology |
| url | https://ieeexplore.ieee.org/document/10251957/ |
| work_keys_str_mv | AT ryanglkoh checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT mdasifkhan checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT sajjadrashidiani checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT samahhassan checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT victoriatucci checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT theodoreliu checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT karlonesovic checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT dineshkumbhare checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch AT thomasedoyle checkitbeforeyouwreckitaguidetostarmlforscreeningmachinelearningreportinginresearch |