Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection
Unlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)—involving automation of dataset labelling—represents a potential avenue to achieve this. However, many aspects of NLP for data...
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-11-01
|
| Series: | Frontiers in Radiology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fradi.2023.1251825/full |
| _version_ | 1797450879417188352 |
|---|---|
| author | Matthew Benger David A. Wood Sina Kafiabadi Aisha Al Busaidi Emily Guilhem Jeremy Lynch Matthew Townend Antanas Montvila Juveria Siddiqui Naveen Gadapa Gareth Barker Sebastian Ourselin James H. Cole James H. Cole Thomas C. Booth Thomas C. Booth |
| author_facet | Matthew Benger David A. Wood Sina Kafiabadi Aisha Al Busaidi Emily Guilhem Jeremy Lynch Matthew Townend Antanas Montvila Juveria Siddiqui Naveen Gadapa Gareth Barker Sebastian Ourselin James H. Cole James H. Cole Thomas C. Booth Thomas C. Booth |
| author_sort | Matthew Benger |
| collection | DOAJ |
| description | Unlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)—involving automation of dataset labelling—represents a potential avenue to achieve this. However, many aspects of NLP for dataset labelling remain unvalidated. Expert radiologists manually labelled over 5,000 MRI head reports in order to develop a deep learning-based neuroradiology NLP report classifier. Our results demonstrate that binary labels (normal vs. abnormal) showed high rates of accuracy, even when only two MRI sequences (T2-weighted and those based on diffusion weighted imaging) were employed as opposed to all sequences in an examination. Meanwhile, the accuracy of more specific labelling for multiple disease categories was variable and dependent on the category. Finally, resultant model performance was shown to be dependent on the expertise of the original labeller, with worse performance seen with non-expert vs. expert labellers. |
| first_indexed | 2024-03-09T14:46:54Z |
| format | Article |
| id | doaj.art-def2b449c49f4c6d9c76d5026cafd894 |
| institution | Directory Open Access Journal |
| issn | 2673-8740 |
| language | English |
| last_indexed | 2024-03-09T14:46:54Z |
| publishDate | 2023-11-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Radiology |
| spelling | doaj.art-def2b449c49f4c6d9c76d5026cafd8942023-11-27T04:34:09ZengFrontiers Media S.A.Frontiers in Radiology2673-87402023-11-01310.3389/fradi.2023.12518251251825Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detectionMatthew Benger0David A. Wood1Sina Kafiabadi2Aisha Al Busaidi3Emily Guilhem4Jeremy Lynch5Matthew Townend6Antanas Montvila7Juveria Siddiqui8Naveen Gadapa9Gareth Barker10Sebastian Ourselin11James H. Cole12James H. Cole13Thomas C. Booth14Thomas C. Booth15Department of Neuroradiology, Kings College Hospital, London, United KingdomSchool of Biomedical Engineering & Imaging Sciences, Kings College London, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomSchool of Biomedical Engineering & Imaging Sciences, Kings College London, London, United KingdomSchool of Biomedical Engineering & Imaging Sciences, Kings College London, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomInstitute of Psychiatry, Psychology & Neuroscience, Kings College London, London, United KingdomSchool of Biomedical Engineering & Imaging Sciences, Kings College London, London, United KingdomInstitute of Psychiatry, Psychology & Neuroscience, Kings College London, London, United KingdomCentre for Medical Image Computing, Dementia Research, University College London, London, United KingdomDepartment of Neuroradiology, Kings College Hospital, London, United KingdomSchool of Biomedical Engineering & Imaging Sciences, Kings College London, London, United KingdomUnlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)—involving automation of dataset labelling—represents a potential avenue to achieve this. However, many aspects of NLP for dataset labelling remain unvalidated. Expert radiologists manually labelled over 5,000 MRI head reports in order to develop a deep learning-based neuroradiology NLP report classifier. Our results demonstrate that binary labels (normal vs. abnormal) showed high rates of accuracy, even when only two MRI sequences (T2-weighted and those based on diffusion weighted imaging) were employed as opposed to all sequences in an examination. Meanwhile, the accuracy of more specific labelling for multiple disease categories was variable and dependent on the category. Finally, resultant model performance was shown to be dependent on the expertise of the original labeller, with worse performance seen with non-expert vs. expert labellers.https://www.frontiersin.org/articles/10.3389/fradi.2023.1251825/fulldeep learningcomputer vision systemlabellingneuroradiologyMRI |
| spellingShingle | Matthew Benger David A. Wood Sina Kafiabadi Aisha Al Busaidi Emily Guilhem Jeremy Lynch Matthew Townend Antanas Montvila Juveria Siddiqui Naveen Gadapa Gareth Barker Sebastian Ourselin James H. Cole James H. Cole Thomas C. Booth Thomas C. Booth Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection Frontiers in Radiology deep learning computer vision system labelling neuroradiology MRI |
| title | Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection |
| title_full | Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection |
| title_fullStr | Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection |
| title_full_unstemmed | Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection |
| title_short | Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection |
| title_sort | factors affecting the labelling accuracy of brain mri studies relevant for deep learning abnormality detection |
| topic | deep learning computer vision system labelling neuroradiology MRI |
| url | https://www.frontiersin.org/articles/10.3389/fradi.2023.1251825/full |
| work_keys_str_mv | AT matthewbenger factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT davidawood factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT sinakafiabadi factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT aishaalbusaidi factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT emilyguilhem factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT jeremylynch factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT matthewtownend factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT antanasmontvila factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT juveriasiddiqui factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT naveengadapa factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT garethbarker factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT sebastianourselin factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT jameshcole factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT jameshcole factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT thomascbooth factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection AT thomascbooth factorsaffectingthelabellingaccuracyofbrainmristudiesrelevantfordeeplearningabnormalitydetection |