Improving feature extraction from histopathological images through a fine-tuning ImageNet model
Background: Due to lack of annotated pathological images, transfer learning has been the predominant approach in the field of digital pathology. Pre-trained neural networks based on ImageNet database are often used to extract “off-the-shelf” features, achieving great success in predicting tissue typ...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-01-01
|
| Series: | Journal of Pathology Informatics |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S215335392200709X |
| _version_ | 1797977018091962368 |
|---|---|
| author | Xingyu Li Min Cen Jinfeng Xu Hong Zhang Xu Steven Xu |
| author_facet | Xingyu Li Min Cen Jinfeng Xu Hong Zhang Xu Steven Xu |
| author_sort | Xingyu Li |
| collection | DOAJ |
| description | Background: Due to lack of annotated pathological images, transfer learning has been the predominant approach in the field of digital pathology. Pre-trained neural networks based on ImageNet database are often used to extract “off-the-shelf” features, achieving great success in predicting tissue types, molecular features, and clinical outcomes, etc. We hypothesize that fine-tuning the pre-trained models using histopathological images could further improve feature extraction, and downstream prediction performance. Methods: We used 100 000 annotated H&E image patches for colorectal cancer (CRC) to fine-tune a pre-trained Xception model via a 2-step approach. The features extracted from fine-tuned Xception (FTX-2048) model and Image-pretrained (IMGNET-2048) model were compared through: (1) tissue classification for H&E images from CRC, same image type that was used for fine-tuning; (2) prediction of immune-related gene expression, and (3) gene mutations for lung adenocarcinoma (LUAD). Five-fold cross validation was used for model performance evaluation. Each experiment was repeated 50 times. Findings: The extracted features from the fine-tuned FTX-2048 exhibited significantly higher accuracy (98.4%) for predicting tissue types of CRC compared to the “off-the-shelf” features directly from Xception based on ImageNet database (96.4%) (P value = 2.2 × 10−6). Particularly, FTX-2048 markedly improved the accuracy for stroma from 87% to 94%. Similarly, features from FTX-2048 boosted the prediction of transcriptomic expression of immune-related genes in LUAD. For the genes that had significant relationships with image features (P < 0.05, n = 171), the features from the fine-tuned model improved the prediction for the majority of the genes (139; 81%). In addition, features from FTX-2048 improved prediction of mutation for 5 out of 9 most frequently mutated genes (STK11, TP53, LRP1B, NF1, and FAT1) in LUAD. Conclusions: We proved the concept that fine-tuning the pretrained ImageNet neural networks with histopathology images can produce higher quality features and better prediction performance for not only the same-cancer tissue classification where similar images from the same cancer are used for fine-tuning, but also cross-cancer prediction for gene expression and mutation at patient level. |
| first_indexed | 2024-04-11T05:00:17Z |
| format | Article |
| id | doaj.art-de717a0937ab4f59b208c034d2930cfb |
| institution | Directory Open Access Journal |
| issn | 2153-3539 |
| language | English |
| last_indexed | 2024-04-11T05:00:17Z |
| publishDate | 2022-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Journal of Pathology Informatics |
| spelling | doaj.art-de717a0937ab4f59b208c034d2930cfb2022-12-26T04:08:47ZengElsevierJournal of Pathology Informatics2153-35392022-01-0113100115Improving feature extraction from histopathological images through a fine-tuning ImageNet modelXingyu Li0Min Cen1Jinfeng Xu2Hong Zhang3Xu Steven Xu4Department of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei, Anhui 230026, ChinaDepartment of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei, Anhui 230026, ChinaDepartment of Statistics and Actuarial Science, The University of Hong Kong, Hong KongDepartment of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei, Anhui 230026, China; Corresponding authors.Clinical Pharmacology and Quantitative Science, Genmab Inc., Princeton, New Jersey, USA; Corresponding authors.Background: Due to lack of annotated pathological images, transfer learning has been the predominant approach in the field of digital pathology. Pre-trained neural networks based on ImageNet database are often used to extract “off-the-shelf” features, achieving great success in predicting tissue types, molecular features, and clinical outcomes, etc. We hypothesize that fine-tuning the pre-trained models using histopathological images could further improve feature extraction, and downstream prediction performance. Methods: We used 100 000 annotated H&E image patches for colorectal cancer (CRC) to fine-tune a pre-trained Xception model via a 2-step approach. The features extracted from fine-tuned Xception (FTX-2048) model and Image-pretrained (IMGNET-2048) model were compared through: (1) tissue classification for H&E images from CRC, same image type that was used for fine-tuning; (2) prediction of immune-related gene expression, and (3) gene mutations for lung adenocarcinoma (LUAD). Five-fold cross validation was used for model performance evaluation. Each experiment was repeated 50 times. Findings: The extracted features from the fine-tuned FTX-2048 exhibited significantly higher accuracy (98.4%) for predicting tissue types of CRC compared to the “off-the-shelf” features directly from Xception based on ImageNet database (96.4%) (P value = 2.2 × 10−6). Particularly, FTX-2048 markedly improved the accuracy for stroma from 87% to 94%. Similarly, features from FTX-2048 boosted the prediction of transcriptomic expression of immune-related genes in LUAD. For the genes that had significant relationships with image features (P < 0.05, n = 171), the features from the fine-tuned model improved the prediction for the majority of the genes (139; 81%). In addition, features from FTX-2048 improved prediction of mutation for 5 out of 9 most frequently mutated genes (STK11, TP53, LRP1B, NF1, and FAT1) in LUAD. Conclusions: We proved the concept that fine-tuning the pretrained ImageNet neural networks with histopathology images can produce higher quality features and better prediction performance for not only the same-cancer tissue classification where similar images from the same cancer are used for fine-tuning, but also cross-cancer prediction for gene expression and mutation at patient level.http://www.sciencedirect.com/science/article/pii/S215335392200709XDeep learningWhole slide imagesTCGA datasetH&E imageColorectal cancerFine-tuning |
| spellingShingle | Xingyu Li Min Cen Jinfeng Xu Hong Zhang Xu Steven Xu Improving feature extraction from histopathological images through a fine-tuning ImageNet model Journal of Pathology Informatics Deep learning Whole slide images TCGA dataset H&E image Colorectal cancer Fine-tuning |
| title | Improving feature extraction from histopathological images through a fine-tuning ImageNet model |
| title_full | Improving feature extraction from histopathological images through a fine-tuning ImageNet model |
| title_fullStr | Improving feature extraction from histopathological images through a fine-tuning ImageNet model |
| title_full_unstemmed | Improving feature extraction from histopathological images through a fine-tuning ImageNet model |
| title_short | Improving feature extraction from histopathological images through a fine-tuning ImageNet model |
| title_sort | improving feature extraction from histopathological images through a fine tuning imagenet model |
| topic | Deep learning Whole slide images TCGA dataset H&E image Colorectal cancer Fine-tuning |
| url | http://www.sciencedirect.com/science/article/pii/S215335392200709X |
| work_keys_str_mv | AT xingyuli improvingfeatureextractionfromhistopathologicalimagesthroughafinetuningimagenetmodel AT mincen improvingfeatureextractionfromhistopathologicalimagesthroughafinetuningimagenetmodel AT jinfengxu improvingfeatureextractionfromhistopathologicalimagesthroughafinetuningimagenetmodel AT hongzhang improvingfeatureextractionfromhistopathologicalimagesthroughafinetuningimagenetmodel AT xustevenxu improvingfeatureextractionfromhistopathologicalimagesthroughafinetuningimagenetmodel |