Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center
PurposeDespite the availability of commercial artificial intelligence (AI) tools for large vessel occlusion (LVO) detection, there is paucity of data comparing traditional machine learning and deep learning solutions in a real-world setting. The purpose of this study is to compare and validate the p...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-10-01
|
| Series: | Frontiers in Neurology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fneur.2022.1026609/full |
| _version_ | 1811226121258139648 |
|---|---|
| author | Jacob Schlossman Jacob Schlossman Daniel Ro Daniel Ro Shirin Salehi Shirin Salehi Daniel Chow Daniel Chow Wengui Yu Peter D. Chang Peter D. Chang Jennifer E. Soun Jennifer E. Soun |
| author_facet | Jacob Schlossman Jacob Schlossman Daniel Ro Daniel Ro Shirin Salehi Shirin Salehi Daniel Chow Daniel Chow Wengui Yu Peter D. Chang Peter D. Chang Jennifer E. Soun Jennifer E. Soun |
| author_sort | Jacob Schlossman |
| collection | DOAJ |
| description | PurposeDespite the availability of commercial artificial intelligence (AI) tools for large vessel occlusion (LVO) detection, there is paucity of data comparing traditional machine learning and deep learning solutions in a real-world setting. The purpose of this study is to compare and validate the performance of two AI-based tools (RAPID LVO and CINA LVO) for LVO detection.Materials and methodsThis was a retrospective, single center study performed at a comprehensive stroke center from December 2020 to June 2021. CT angiography (n = 263) for suspected stroke were evaluated for LVO. RAPID LVO is a traditional machine learning model which primarily relies on vessel density threshold assessment, while CINA LVO is an end-to-end deep learning tool implemented with multiple neural networks for detection and localization tasks. Reasons for errors were also recorded.ResultsThere were 29 positive and 224 negative LVO cases by ground truth assessment. RAPID LVO demonstrated an accuracy of 0.86, sensitivity of 0.90, specificity of 0.86, positive predictive value of 0.45, and negative predictive value of 0.98, while CINA demonstrated an accuracy of 0.96, sensitivity of 0.76, specificity of 0.98, positive predictive value of 0.85, and negative predictive value of 0.97.ConclusionBoth tools successfully detected most anterior circulation occlusions. RAPID LVO had higher sensitivity while CINA LVO had higher accuracy and specificity. Interestingly, both tools were able to detect some, but not all M2 MCA occlusions. This is the first study to compare traditional and deep learning LVO tools in the clinical setting. |
| first_indexed | 2024-04-12T09:20:03Z |
| format | Article |
| id | doaj.art-120ff40fda3544b79bffecf3224f4087 |
| institution | Directory Open Access Journal |
| issn | 1664-2295 |
| language | English |
| last_indexed | 2024-04-12T09:20:03Z |
| publishDate | 2022-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurology |
| spelling | doaj.art-120ff40fda3544b79bffecf3224f40872022-12-22T03:38:41ZengFrontiers Media S.A.Frontiers in Neurology1664-22952022-10-011310.3389/fneur.2022.10266091026609Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke centerJacob Schlossman0Jacob Schlossman1Daniel Ro2Daniel Ro3Shirin Salehi4Shirin Salehi5Daniel Chow6Daniel Chow7Wengui Yu8Peter D. Chang9Peter D. Chang10Jennifer E. Soun11Jennifer E. Soun12Center for Artificial Intelligence in Diagnostic Medicine, University of California, Irvine, Irvine, CA, United StatesUniversity of California Irvine School of Medicine, Irvine, CA, United StatesCenter for Artificial Intelligence in Diagnostic Medicine, University of California, Irvine, Irvine, CA, United StatesDepartment of Neurology, University of California, Irvine, Irvine, CA, United StatesCenter for Artificial Intelligence in Diagnostic Medicine, University of California, Irvine, Irvine, CA, United StatesUniversity of California Irvine School of Medicine, Irvine, CA, United StatesCenter for Artificial Intelligence in Diagnostic Medicine, University of California, Irvine, Irvine, CA, United StatesDepartment of Radiological Sciences, University of California, Irvine, Irvine, CA, United StatesDepartment of Neurology, University of California, Irvine, Irvine, CA, United StatesCenter for Artificial Intelligence in Diagnostic Medicine, University of California, Irvine, Irvine, CA, United StatesDepartment of Radiological Sciences, University of California, Irvine, Irvine, CA, United StatesCenter for Artificial Intelligence in Diagnostic Medicine, University of California, Irvine, Irvine, CA, United StatesDepartment of Radiological Sciences, University of California, Irvine, Irvine, CA, United StatesPurposeDespite the availability of commercial artificial intelligence (AI) tools for large vessel occlusion (LVO) detection, there is paucity of data comparing traditional machine learning and deep learning solutions in a real-world setting. The purpose of this study is to compare and validate the performance of two AI-based tools (RAPID LVO and CINA LVO) for LVO detection.Materials and methodsThis was a retrospective, single center study performed at a comprehensive stroke center from December 2020 to June 2021. CT angiography (n = 263) for suspected stroke were evaluated for LVO. RAPID LVO is a traditional machine learning model which primarily relies on vessel density threshold assessment, while CINA LVO is an end-to-end deep learning tool implemented with multiple neural networks for detection and localization tasks. Reasons for errors were also recorded.ResultsThere were 29 positive and 224 negative LVO cases by ground truth assessment. RAPID LVO demonstrated an accuracy of 0.86, sensitivity of 0.90, specificity of 0.86, positive predictive value of 0.45, and negative predictive value of 0.98, while CINA demonstrated an accuracy of 0.96, sensitivity of 0.76, specificity of 0.98, positive predictive value of 0.85, and negative predictive value of 0.97.ConclusionBoth tools successfully detected most anterior circulation occlusions. RAPID LVO had higher sensitivity while CINA LVO had higher accuracy and specificity. Interestingly, both tools were able to detect some, but not all M2 MCA occlusions. This is the first study to compare traditional and deep learning LVO tools in the clinical setting.https://www.frontiersin.org/articles/10.3389/fneur.2022.1026609/fullartificial intelligencelarge vessel occlusionmachine learningdeep learningstroke |
| spellingShingle | Jacob Schlossman Jacob Schlossman Daniel Ro Daniel Ro Shirin Salehi Shirin Salehi Daniel Chow Daniel Chow Wengui Yu Peter D. Chang Peter D. Chang Jennifer E. Soun Jennifer E. Soun Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center Frontiers in Neurology artificial intelligence large vessel occlusion machine learning deep learning stroke |
| title | Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center |
| title_full | Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center |
| title_fullStr | Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center |
| title_full_unstemmed | Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center |
| title_short | Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center |
| title_sort | head to head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center |
| topic | artificial intelligence large vessel occlusion machine learning deep learning stroke |
| url | https://www.frontiersin.org/articles/10.3389/fneur.2022.1026609/full |
| work_keys_str_mv | AT jacobschlossman headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT jacobschlossman headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT danielro headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT danielro headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT shirinsalehi headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT shirinsalehi headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT danielchow headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT danielchow headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT wenguiyu headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT peterdchang headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT peterdchang headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT jenniferesoun headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter AT jenniferesoun headtoheadcomparisonofcommercialartificialintelligencesolutionsfordetectionoflargevesselocclusionatacomprehensivestrokecenter |