Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution
Single image super-resolution is an important computer vision task with applications including remote sensing, medical imaging, and surveillance. Modern work on super-resolution utilizes deep learning to synthesize high resolution (HR) images from low resolution images (LR). With the increased utili...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-01-01
|
| Series: | Journal of Pathology Informatics |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2153353922007428 |
| _version_ | 1828086691872636928 |
|---|---|
| author | Cyrus Manuel Philip Zehnder Sertan Kaya Ruth Sullivan Fangyao Hu |
| author_facet | Cyrus Manuel Philip Zehnder Sertan Kaya Ruth Sullivan Fangyao Hu |
| author_sort | Cyrus Manuel |
| collection | DOAJ |
| description | Single image super-resolution is an important computer vision task with applications including remote sensing, medical imaging, and surveillance. Modern work on super-resolution utilizes deep learning to synthesize high resolution (HR) images from low resolution images (LR). With the increased utilization of digitized whole slide images (WSI) in pathology workflows, digital pathology has emerged as a promising domain for super-resolution. Despite extensive existing research into super-resolution, there remain challenges specific to digital pathology. Here, we investigated image augmentation techniques for hematoxylin and eosin (H&E) WSI super-resolution and model generalizability across diverse tissue types. In addition, we investigated shortcomings with common quality metrics (peak signal-to-noise ratio (PSNR), structure similarity index (SSIM)) by conducting a perceptual quality survey for super-resolved pathology images. High performing deep super-resolution models were used to generate 20X HR images from LR images (5X or 10X equivalent) for 11 different tissues and 30 human evaluators were asked to score the quality of the generated versus the ground truth 20X HR images. The scores given by a human rater and the PSNR or the SSIM were compared to investigate the correlation between model training parameters. We found that models trained on multiple tissues generalized better than those trained on a single tissue type. We also found that PSNR correlated with perceptual quality (R = 0.26) less accurately than did SSIM (R = 0.64), suggesting that the SSIM quality metric is insufficient. The methods proposed in this study can be used to virtually magnify H&E images with better perceptual quality than interpolation methods (i.e., bicubic interpolation) commonly implemented in digital pathology software. The impact of deep SISR methods is more notable when scaling to 4X is needed, such as in the case of super-resolving a low magnification WSI from 10X to 40X. |
| first_indexed | 2024-04-11T04:59:52Z |
| format | Article |
| id | doaj.art-63ce9d8d8bcb4e548914c311aa7357ff |
| institution | Directory Open Access Journal |
| issn | 2153-3539 |
| language | English |
| last_indexed | 2024-04-11T04:59:52Z |
| publishDate | 2022-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Journal of Pathology Informatics |
| spelling | doaj.art-63ce9d8d8bcb4e548914c311aa7357ff2022-12-26T04:09:02ZengElsevierJournal of Pathology Informatics2153-35392022-01-0113100148Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolutionCyrus Manuel0Philip Zehnder1Sertan Kaya2Ruth Sullivan3Fangyao Hu4Genentech, South San Francisco, CA, USAGenentech, South San Francisco, CA, USAGenentech, South San Francisco, CA, USAGenentech, South San Francisco, CA, USACorresponding author at: Genentech, 1 DNA Way, South San Francisco, CA 94080, USA.; Genentech, South San Francisco, CA, USASingle image super-resolution is an important computer vision task with applications including remote sensing, medical imaging, and surveillance. Modern work on super-resolution utilizes deep learning to synthesize high resolution (HR) images from low resolution images (LR). With the increased utilization of digitized whole slide images (WSI) in pathology workflows, digital pathology has emerged as a promising domain for super-resolution. Despite extensive existing research into super-resolution, there remain challenges specific to digital pathology. Here, we investigated image augmentation techniques for hematoxylin and eosin (H&E) WSI super-resolution and model generalizability across diverse tissue types. In addition, we investigated shortcomings with common quality metrics (peak signal-to-noise ratio (PSNR), structure similarity index (SSIM)) by conducting a perceptual quality survey for super-resolved pathology images. High performing deep super-resolution models were used to generate 20X HR images from LR images (5X or 10X equivalent) for 11 different tissues and 30 human evaluators were asked to score the quality of the generated versus the ground truth 20X HR images. The scores given by a human rater and the PSNR or the SSIM were compared to investigate the correlation between model training parameters. We found that models trained on multiple tissues generalized better than those trained on a single tissue type. We also found that PSNR correlated with perceptual quality (R = 0.26) less accurately than did SSIM (R = 0.64), suggesting that the SSIM quality metric is insufficient. The methods proposed in this study can be used to virtually magnify H&E images with better perceptual quality than interpolation methods (i.e., bicubic interpolation) commonly implemented in digital pathology software. The impact of deep SISR methods is more notable when scaling to 4X is needed, such as in the case of super-resolving a low magnification WSI from 10X to 40X.http://www.sciencedirect.com/science/article/pii/S2153353922007428Artificial intelligenceDeep learningDigital pathologyGenerative adversarial networksHistopathologyImage processing |
| spellingShingle | Cyrus Manuel Philip Zehnder Sertan Kaya Ruth Sullivan Fangyao Hu Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution Journal of Pathology Informatics Artificial intelligence Deep learning Digital pathology Generative adversarial networks Histopathology Image processing |
| title | Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution |
| title_full | Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution |
| title_fullStr | Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution |
| title_full_unstemmed | Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution |
| title_short | Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution |
| title_sort | impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution |
| topic | Artificial intelligence Deep learning Digital pathology Generative adversarial networks Histopathology Image processing |
| url | http://www.sciencedirect.com/science/article/pii/S2153353922007428 |
| work_keys_str_mv | AT cyrusmanuel impactofcoloraugmentationandtissuetypeindeeplearningforhematoxylinandeosinimagesuperresolution AT philipzehnder impactofcoloraugmentationandtissuetypeindeeplearningforhematoxylinandeosinimagesuperresolution AT sertankaya impactofcoloraugmentationandtissuetypeindeeplearningforhematoxylinandeosinimagesuperresolution AT ruthsullivan impactofcoloraugmentationandtissuetypeindeeplearningforhematoxylinandeosinimagesuperresolution AT fangyaohu impactofcoloraugmentationandtissuetypeindeeplearningforhematoxylinandeosinimagesuperresolution |