Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope
Several machine learning algorithms have demonstrated high predictive capability in the identification of cancer within digitized pathology slides. The Augmented Reality Microscope (ARM) has allowed these algorithms to be seamlessly integrated within the pathology workflow by overlaying their infere...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-01-01
|
| Series: | Journal of Pathology Informatics |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2153353922007362 |
| _version_ | 1797977022960500736 |
|---|---|
| author | David Jin Joseph H. Rosenthal Elaine E. Thompson Jared Dunnmon Arash Mohtashamian Daniel Ward Ryan Austin Hassan Tetteh Niels H. Olson |
| author_facet | David Jin Joseph H. Rosenthal Elaine E. Thompson Jared Dunnmon Arash Mohtashamian Daniel Ward Ryan Austin Hassan Tetteh Niels H. Olson |
| author_sort | David Jin |
| collection | DOAJ |
| description | Several machine learning algorithms have demonstrated high predictive capability in the identification of cancer within digitized pathology slides. The Augmented Reality Microscope (ARM) has allowed these algorithms to be seamlessly integrated within the pathology workflow by overlaying their inferences onto its microscopic field of view in real time. We present an independent assessment of the LYmph Node Assistant (LYNA) models, state-of-the-art algorithms for the identification of breast cancer metastases in lymph node biopsies, optimized for usage on the ARM. We assessed the models on 40 whole slide images at the commonly used objective magnifications of 10×, 20×, and 40×. We analyzed their performance across clinically relevant subclasses of tissue, including breast cancer, lymphocytes, histiocytes, blood, and fat. Each model obtained overall AUC values of approximately 0.98, accuracy values of approximately 0.94, and sensitivity values above 0.88 at classifying small regions of a field of view as benign or cancerous. Across tissue subclasses, the models performed most accurately on fat and blood, and least accurately on histiocytes, germinal centers, and sinus. The models also struggled with the identification of isolated tumor cells, especially at lower magnifications. After testing, we reviewed the discrepancies between model predictions and ground truth to understand the causes of error. We introduce a distinction between proper and improper ground truth for analysis in cases of uncertain annotations. Taken together, these methods comprise a novel approach for exploratory model analysis over complex anatomic pathology data in which precise ground truth is difficult to establish. |
| first_indexed | 2024-04-11T05:00:22Z |
| format | Article |
| id | doaj.art-eaee99a16e8f4ed09b784c7643b09831 |
| institution | Directory Open Access Journal |
| issn | 2153-3539 |
| language | English |
| last_indexed | 2024-04-11T05:00:22Z |
| publishDate | 2022-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Journal of Pathology Informatics |
| spelling | doaj.art-eaee99a16e8f4ed09b784c7643b098312022-12-26T04:08:59ZengElsevierJournal of Pathology Informatics2153-35392022-01-0113100142Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality MicroscopeDavid Jin0Joseph H. Rosenthal1Elaine E. Thompson2Jared Dunnmon3Arash Mohtashamian4Daniel Ward5Ryan Austin6Hassan Tetteh7Niels H. Olson8The MITRE Corporation, 7525 Colshire Dr, McLean, VA, USAThe Henry M. Jackson Foundation for the Advancement of Military Medicine, 6720A Rockledge Dr, Bethesda, MD, USAThe Henry M. Jackson Foundation for the Advancement of Military Medicine, 6720A Rockledge Dr, Bethesda, MD, USADefense Innovation Unit, 230 RT Jones Rd, Mountain View, CA, USANaval Hospital Camp Pendleton, 200 Mercy Cir, Oceanside, CA, USANaval Medical Center San Diego, 34800 Bob Wilson Dr, San Diego, CA, USANaval Hospital Camp Pendleton, 200 Mercy Cir, Oceanside, CA, USADoD Chief Digital and AI Office, 5615 Columbia Pike, Falls Church, VA 22041, USADefense Innovation Unit, 230 RT Jones Rd, Mountain View, CA, USA; Corresponding author.Several machine learning algorithms have demonstrated high predictive capability in the identification of cancer within digitized pathology slides. The Augmented Reality Microscope (ARM) has allowed these algorithms to be seamlessly integrated within the pathology workflow by overlaying their inferences onto its microscopic field of view in real time. We present an independent assessment of the LYmph Node Assistant (LYNA) models, state-of-the-art algorithms for the identification of breast cancer metastases in lymph node biopsies, optimized for usage on the ARM. We assessed the models on 40 whole slide images at the commonly used objective magnifications of 10×, 20×, and 40×. We analyzed their performance across clinically relevant subclasses of tissue, including breast cancer, lymphocytes, histiocytes, blood, and fat. Each model obtained overall AUC values of approximately 0.98, accuracy values of approximately 0.94, and sensitivity values above 0.88 at classifying small regions of a field of view as benign or cancerous. Across tissue subclasses, the models performed most accurately on fat and blood, and least accurately on histiocytes, germinal centers, and sinus. The models also struggled with the identification of isolated tumor cells, especially at lower magnifications. After testing, we reviewed the discrepancies between model predictions and ground truth to understand the causes of error. We introduce a distinction between proper and improper ground truth for analysis in cases of uncertain annotations. Taken together, these methods comprise a novel approach for exploratory model analysis over complex anatomic pathology data in which precise ground truth is difficult to establish.http://www.sciencedirect.com/science/article/pii/S2153353922007362Machine learningMedical imagingBreast cancer metastasis |
| spellingShingle | David Jin Joseph H. Rosenthal Elaine E. Thompson Jared Dunnmon Arash Mohtashamian Daniel Ward Ryan Austin Hassan Tetteh Niels H. Olson Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope Journal of Pathology Informatics Machine learning Medical imaging Breast cancer metastasis |
| title | Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope |
| title_full | Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope |
| title_fullStr | Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope |
| title_full_unstemmed | Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope |
| title_short | Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope |
| title_sort | independent assessment of a deep learning system for lymph node metastasis detection on the augmented reality microscope |
| topic | Machine learning Medical imaging Breast cancer metastasis |
| url | http://www.sciencedirect.com/science/article/pii/S2153353922007362 |
| work_keys_str_mv | AT davidjin independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT josephhrosenthal independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT elaineethompson independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT jareddunnmon independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT arashmohtashamian independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT danielward independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT ryanaustin independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT hassantetteh independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope AT nielsholson independentassessmentofadeeplearningsystemforlymphnodemetastasisdetectionontheaugmentedrealitymicroscope |