Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images
Abstract Programmed death-ligand 1 (PD-L1) expression is currently used in the clinic to assess eligibility for immune-checkpoint inhibitors via the tumor proportion score (TPS), but its efficacy is limited by high interobserver variability. Multiple papers have presented systems for the automatic q...
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-03-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-024-57067-1 |
| _version_ | 1827301224276295680 |
|---|---|
| author | Leander van Eekelen Joey Spronck Monika Looijen-Salamon Shoko Vos Enrico Munari Ilaria Girolami Albino Eccher Balazs Acs Ceren Boyaci Gabriel Silva de Souza Muradije Demirel-Andishmand Luca Dulce Meesters Daan Zegers Lieke van der Woude Willemijn Theelen Michel van den Heuvel Katrien Grünberg Bram van Ginneken Jeroen van der Laak Francesco Ciompi |
| author_facet | Leander van Eekelen Joey Spronck Monika Looijen-Salamon Shoko Vos Enrico Munari Ilaria Girolami Albino Eccher Balazs Acs Ceren Boyaci Gabriel Silva de Souza Muradije Demirel-Andishmand Luca Dulce Meesters Daan Zegers Lieke van der Woude Willemijn Theelen Michel van den Heuvel Katrien Grünberg Bram van Ginneken Jeroen van der Laak Francesco Ciompi |
| author_sort | Leander van Eekelen |
| collection | DOAJ |
| description | Abstract Programmed death-ligand 1 (PD-L1) expression is currently used in the clinic to assess eligibility for immune-checkpoint inhibitors via the tumor proportion score (TPS), but its efficacy is limited by high interobserver variability. Multiple papers have presented systems for the automatic quantification of TPS, but none report on the task of determining cell-level PD-L1 expression and often reserve their evaluation to a single PD-L1 monoclonal antibody or clinical center. In this paper, we report on a deep learning algorithm for detecting PD-L1 negative and positive tumor cells at a cellular level and evaluate it on a cell-level reference standard established by six readers on a multi-centric, multi PD-L1 assay dataset. This reference standard also provides for the first time a benchmark for computer vision algorithms. In addition, in line with other papers, we also evaluate our algorithm at slide-level by measuring the agreement between the algorithm and six pathologists on TPS quantification. We find a moderately low interobserver agreement at cell-level level (mean reader-reader F1 score = 0.68) which our algorithm sits slightly under (mean reader-AI F1 score = 0.55), especially for cases from the clinical center not included in the training set. Despite this, we find good AI-pathologist agreement on quantifying TPS compared to the interobserver agreement (mean reader-reader Cohen’s kappa = 0.54, 95% CI 0.26–0.81, mean reader-AI kappa = 0.49, 95% CI 0.27—0.72). In conclusion, our deep learning algorithm demonstrates promise in detecting PD-L1 expression at a cellular level and exhibits favorable agreement with pathologists in quantifying the tumor proportion score (TPS). We publicly release our models for use via the Grand-Challenge platform. |
| first_indexed | 2024-04-24T16:19:21Z |
| format | Article |
| id | doaj.art-001f7eaf59b84b3c94b65d25972e094d |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-04-24T16:19:21Z |
| publishDate | 2024-03-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-001f7eaf59b84b3c94b65d25972e094d2024-03-31T11:17:14ZengNature PortfolioScientific Reports2045-23222024-03-0114111010.1038/s41598-024-57067-1Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide imagesLeander van Eekelen0Joey Spronck1Monika Looijen-Salamon2Shoko Vos3Enrico Munari4Ilaria Girolami5Albino Eccher6Balazs Acs7Ceren Boyaci8Gabriel Silva de Souza9Muradije Demirel-Andishmand10Luca Dulce Meesters11Daan Zegers12Lieke van der Woude13Willemijn Theelen14Michel van den Heuvel15Katrien Grünberg16Bram van Ginneken17Jeroen van der Laak18Francesco Ciompi19Department of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterPathology Unit, Department of Molecular and Translational Medicine, University of BresciaDepartment of Pathology, Provincial Hospital of Bolzano (SABES-ASDAA)Department of Pathology and Diagnostics, University and Hospital Trust of VeronaDepartment of Clinical Pathology and Cancer Diagnostics, Karolinska University HospitalDepartment of Clinical Pathology and Cancer Diagnostics, Karolinska University HospitalDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Thoracic Oncology, Netherlands Cancer InstituteRespiratory Diseases Department, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Radiology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterDepartment of Pathology, Radboud University Medical CenterAbstract Programmed death-ligand 1 (PD-L1) expression is currently used in the clinic to assess eligibility for immune-checkpoint inhibitors via the tumor proportion score (TPS), but its efficacy is limited by high interobserver variability. Multiple papers have presented systems for the automatic quantification of TPS, but none report on the task of determining cell-level PD-L1 expression and often reserve their evaluation to a single PD-L1 monoclonal antibody or clinical center. In this paper, we report on a deep learning algorithm for detecting PD-L1 negative and positive tumor cells at a cellular level and evaluate it on a cell-level reference standard established by six readers on a multi-centric, multi PD-L1 assay dataset. This reference standard also provides for the first time a benchmark for computer vision algorithms. In addition, in line with other papers, we also evaluate our algorithm at slide-level by measuring the agreement between the algorithm and six pathologists on TPS quantification. We find a moderately low interobserver agreement at cell-level level (mean reader-reader F1 score = 0.68) which our algorithm sits slightly under (mean reader-AI F1 score = 0.55), especially for cases from the clinical center not included in the training set. Despite this, we find good AI-pathologist agreement on quantifying TPS compared to the interobserver agreement (mean reader-reader Cohen’s kappa = 0.54, 95% CI 0.26–0.81, mean reader-AI kappa = 0.49, 95% CI 0.27—0.72). In conclusion, our deep learning algorithm demonstrates promise in detecting PD-L1 expression at a cellular level and exhibits favorable agreement with pathologists in quantifying the tumor proportion score (TPS). We publicly release our models for use via the Grand-Challenge platform.https://doi.org/10.1038/s41598-024-57067-1 |
| spellingShingle | Leander van Eekelen Joey Spronck Monika Looijen-Salamon Shoko Vos Enrico Munari Ilaria Girolami Albino Eccher Balazs Acs Ceren Boyaci Gabriel Silva de Souza Muradije Demirel-Andishmand Luca Dulce Meesters Daan Zegers Lieke van der Woude Willemijn Theelen Michel van den Heuvel Katrien Grünberg Bram van Ginneken Jeroen van der Laak Francesco Ciompi Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images Scientific Reports |
| title | Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images |
| title_full | Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images |
| title_fullStr | Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images |
| title_full_unstemmed | Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images |
| title_short | Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images |
| title_sort | comparing deep learning and pathologist quantification of cell level pd l1 expression in non small cell lung cancer whole slide images |
| url | https://doi.org/10.1038/s41598-024-57067-1 |
| work_keys_str_mv | AT leandervaneekelen comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT joeyspronck comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT monikalooijensalamon comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT shokovos comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT enricomunari comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT ilariagirolami comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT albinoeccher comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT balazsacs comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT cerenboyaci comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT gabrielsilvadesouza comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT muradijedemirelandishmand comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT lucadulcemeesters comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT daanzegers comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT liekevanderwoude comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT willemijntheelen comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT michelvandenheuvel comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT katriengrunberg comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT bramvanginneken comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT jeroenvanderlaak comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages AT francescociompi comparingdeeplearningandpathologistquantificationofcelllevelpdl1expressioninnonsmallcelllungcancerwholeslideimages |