Task design for crowdsourced glioma cell annotation in microscopy images
Abstract Crowdsourcing has been used in computational pathology to generate cell and cell nuclei annotations for machine learning. Herein, we broaden its scope to the previously unsolved challenging task of glioma cell detection. This requires multiplexed immunofluorescence microscopy due to diffuse...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-01-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-024-51995-8 |
| _version_ | 1797276560698376192 |
|---|---|
| author | Svea Schwarze Nadine S. Schaadt Viktor M. G. Sobotta Nicolai Spicher Thomas Skripuletz Majid Esmaeilzadeh Joachim K. Krauss Christian Hartmann Thomas M. Deserno Friedrich Feuerhake |
| author_facet | Svea Schwarze Nadine S. Schaadt Viktor M. G. Sobotta Nicolai Spicher Thomas Skripuletz Majid Esmaeilzadeh Joachim K. Krauss Christian Hartmann Thomas M. Deserno Friedrich Feuerhake |
| author_sort | Svea Schwarze |
| collection | DOAJ |
| description | Abstract Crowdsourcing has been used in computational pathology to generate cell and cell nuclei annotations for machine learning. Herein, we broaden its scope to the previously unsolved challenging task of glioma cell detection. This requires multiplexed immunofluorescence microscopy due to diffuse invasiveness and exceptional similarity between glioma cells and reactive astrocytes. In four pilot experiments, we iteratively developed a task design enabling high-quality annotations by crowdworkers on Amazon Mechanical Turk. We applied majority or weighted vote and validated them against ground truth in the final setting. On the base of a YOLO convolutional neural network architecture, we used these consensus labels for training with different image representations regarding colors, intensities, and immmunohistochemical marker combinations. A crowd of 712 workers defined aggregated point annotations in 235 images with an average $$F_1$$ F 1 score of 0.627 for majority vote. The networks resulted in acceptable $$F_1$$ F 1 scores up to 0.69 for YOLOv8 on average and indicated first evidence for transferability to images lacking tumor markers, especially in IDH-wildtype glioblastoma. Our work confirms feasibility of crowdsourcing to generate labels suitable for training of machine learning tools in the challenging and clinically relevant use case of glioma microenvironment. |
| first_indexed | 2024-03-07T15:29:55Z |
| format | Article |
| id | doaj.art-a0cba2ebc3c348469af40d2a0df25edb |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-03-07T15:29:55Z |
| publishDate | 2024-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-a0cba2ebc3c348469af40d2a0df25edb2024-03-05T16:30:08ZengNature PortfolioScientific Reports2045-23222024-01-0114111210.1038/s41598-024-51995-8Task design for crowdsourced glioma cell annotation in microscopy imagesSvea Schwarze0Nadine S. Schaadt1Viktor M. G. Sobotta2Nicolai Spicher3Thomas Skripuletz4Majid Esmaeilzadeh5Joachim K. Krauss6Christian Hartmann7Thomas M. Deserno8Friedrich Feuerhake9Department of Neuropathology, Institute for Pathology, Hannover Medical SchoolDepartment of Neuropathology, Institute for Pathology, Hannover Medical SchoolPeter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical SchoolPeter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical SchoolDepartment of Neurology, Hannover Medical SchoolDepartment of Neurosurgery, Hannover Medical SchoolDepartment of Neurosurgery, Hannover Medical SchoolDepartment of Neuropathology, Institute for Pathology, Hannover Medical SchoolPeter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical SchoolDepartment of Neuropathology, Institute for Pathology, Hannover Medical SchoolAbstract Crowdsourcing has been used in computational pathology to generate cell and cell nuclei annotations for machine learning. Herein, we broaden its scope to the previously unsolved challenging task of glioma cell detection. This requires multiplexed immunofluorescence microscopy due to diffuse invasiveness and exceptional similarity between glioma cells and reactive astrocytes. In four pilot experiments, we iteratively developed a task design enabling high-quality annotations by crowdworkers on Amazon Mechanical Turk. We applied majority or weighted vote and validated them against ground truth in the final setting. On the base of a YOLO convolutional neural network architecture, we used these consensus labels for training with different image representations regarding colors, intensities, and immmunohistochemical marker combinations. A crowd of 712 workers defined aggregated point annotations in 235 images with an average $$F_1$$ F 1 score of 0.627 for majority vote. The networks resulted in acceptable $$F_1$$ F 1 scores up to 0.69 for YOLOv8 on average and indicated first evidence for transferability to images lacking tumor markers, especially in IDH-wildtype glioblastoma. Our work confirms feasibility of crowdsourcing to generate labels suitable for training of machine learning tools in the challenging and clinically relevant use case of glioma microenvironment.https://doi.org/10.1038/s41598-024-51995-8 |
| spellingShingle | Svea Schwarze Nadine S. Schaadt Viktor M. G. Sobotta Nicolai Spicher Thomas Skripuletz Majid Esmaeilzadeh Joachim K. Krauss Christian Hartmann Thomas M. Deserno Friedrich Feuerhake Task design for crowdsourced glioma cell annotation in microscopy images Scientific Reports |
| title | Task design for crowdsourced glioma cell annotation in microscopy images |
| title_full | Task design for crowdsourced glioma cell annotation in microscopy images |
| title_fullStr | Task design for crowdsourced glioma cell annotation in microscopy images |
| title_full_unstemmed | Task design for crowdsourced glioma cell annotation in microscopy images |
| title_short | Task design for crowdsourced glioma cell annotation in microscopy images |
| title_sort | task design for crowdsourced glioma cell annotation in microscopy images |
| url | https://doi.org/10.1038/s41598-024-51995-8 |
| work_keys_str_mv | AT sveaschwarze taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT nadinesschaadt taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT viktormgsobotta taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT nicolaispicher taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT thomasskripuletz taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT majidesmaeilzadeh taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT joachimkkrauss taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT christianhartmann taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT thomasmdeserno taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages AT friedrichfeuerhake taskdesignforcrowdsourcedgliomacellannotationinmicroscopyimages |