Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images
Abstract Background Applying deep learning to digital histopathology is hindered by the scarcity of manually annotated datasets. While data augmentation can ameliorate this obstacle, its methods are far from standardized. Our aim was to systematically explore the effects of skipping data augmentatio...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-03-01
|
| Series: | BMC Bioinformatics |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12859-023-05199-y |
| _version_ | 1797863322028081152 |
|---|---|
| author | Yusra A. Ameen Dalia M. Badary Ahmad Elbadry I. Abonnoor Khaled F. Hussain Adel A. Sewisy |
| author_facet | Yusra A. Ameen Dalia M. Badary Ahmad Elbadry I. Abonnoor Khaled F. Hussain Adel A. Sewisy |
| author_sort | Yusra A. Ameen |
| collection | DOAJ |
| description | Abstract Background Applying deep learning to digital histopathology is hindered by the scarcity of manually annotated datasets. While data augmentation can ameliorate this obstacle, its methods are far from standardized. Our aim was to systematically explore the effects of skipping data augmentation; applying data augmentation to different subsets of the whole dataset (training set, validation set, test set, two of them, or all of them); and applying data augmentation at different time points (before, during, or after dividing the dataset into three subsets). Different combinations of the above possibilities resulted in 11 ways to apply augmentation. The literature contains no such comprehensive systematic comparison of these augmentation ways. Results Non-overlapping photographs of all tissues on 90 hematoxylin-and-eosin-stained urinary bladder slides were obtained. Then, they were manually classified as either inflammation (5948 images), urothelial cell carcinoma (5811 images), or invalid (3132 images; excluded). If done, augmentation was eight-fold by flipping and rotation. Four convolutional neural networks (Inception-v3, ResNet-101, GoogLeNet, and SqueezeNet), pre-trained on the ImageNet dataset, were fine-tuned to binary classify images of our dataset. This task was the benchmark for our experiments. Model testing performance was evaluated using accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve. Model validation accuracy was also estimated. The best testing performance was achieved when augmentation was done to the remaining data after test-set separation, but before division into training and validation sets. This leaked information between the training and the validation sets, as evidenced by the optimistic validation accuracy. However, this leakage did not cause the validation set to malfunction. Augmentation before test-set separation led to optimistic results. Test-set augmentation yielded more accurate evaluation metrics with less uncertainty. Inception-v3 had the best overall testing performance. Conclusions In digital histopathology, augmentation should include both the test set (after its allocation), and the remaining combined training/validation set (before being split into separate training and validation sets). Future research should try to generalize our results. |
| first_indexed | 2024-04-09T22:34:49Z |
| format | Article |
| id | doaj.art-47720d2ff3684460adaae3ba6a6553a0 |
| institution | Directory Open Access Journal |
| issn | 1471-2105 |
| language | English |
| last_indexed | 2024-04-09T22:34:49Z |
| publishDate | 2023-03-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Bioinformatics |
| spelling | doaj.art-47720d2ff3684460adaae3ba6a6553a02023-03-22T12:33:15ZengBMCBMC Bioinformatics1471-21052023-03-0124112810.1186/s12859-023-05199-yWhich data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology imagesYusra A. Ameen0Dalia M. Badary1Ahmad Elbadry I. Abonnoor2Khaled F. Hussain3Adel A. Sewisy4Department of Computer Science, Faculty of Computers and Information, Assiut UniversityDepartment of Pathology, Faculty of Medicine, Assiut UniversityUrology and Nephrology Hospital, Faculty of Medicine, Assiut UniversityDepartment of Computer Science, Faculty of Computers and Information, Assiut UniversityDepartment of Computer Science, Faculty of Computers and Information, Assiut UniversityAbstract Background Applying deep learning to digital histopathology is hindered by the scarcity of manually annotated datasets. While data augmentation can ameliorate this obstacle, its methods are far from standardized. Our aim was to systematically explore the effects of skipping data augmentation; applying data augmentation to different subsets of the whole dataset (training set, validation set, test set, two of them, or all of them); and applying data augmentation at different time points (before, during, or after dividing the dataset into three subsets). Different combinations of the above possibilities resulted in 11 ways to apply augmentation. The literature contains no such comprehensive systematic comparison of these augmentation ways. Results Non-overlapping photographs of all tissues on 90 hematoxylin-and-eosin-stained urinary bladder slides were obtained. Then, they were manually classified as either inflammation (5948 images), urothelial cell carcinoma (5811 images), or invalid (3132 images; excluded). If done, augmentation was eight-fold by flipping and rotation. Four convolutional neural networks (Inception-v3, ResNet-101, GoogLeNet, and SqueezeNet), pre-trained on the ImageNet dataset, were fine-tuned to binary classify images of our dataset. This task was the benchmark for our experiments. Model testing performance was evaluated using accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve. Model validation accuracy was also estimated. The best testing performance was achieved when augmentation was done to the remaining data after test-set separation, but before division into training and validation sets. This leaked information between the training and the validation sets, as evidenced by the optimistic validation accuracy. However, this leakage did not cause the validation set to malfunction. Augmentation before test-set separation led to optimistic results. Test-set augmentation yielded more accurate evaluation metrics with less uncertainty. Inception-v3 had the best overall testing performance. Conclusions In digital histopathology, augmentation should include both the test set (after its allocation), and the remaining combined training/validation set (before being split into separate training and validation sets). Future research should try to generalize our results.https://doi.org/10.1186/s12859-023-05199-yConvolutional neural networkData augmentationDeep learningHistopathologyUrothelial cell carcinoma |
| spellingShingle | Yusra A. Ameen Dalia M. Badary Ahmad Elbadry I. Abonnoor Khaled F. Hussain Adel A. Sewisy Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images BMC Bioinformatics Convolutional neural network Data augmentation Deep learning Histopathology Urothelial cell carcinoma |
| title | Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images |
| title_full | Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images |
| title_fullStr | Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images |
| title_full_unstemmed | Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images |
| title_short | Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images |
| title_sort | which data subset should be augmented for deep learning a simulation study using urothelial cell carcinoma histopathology images |
| topic | Convolutional neural network Data augmentation Deep learning Histopathology Urothelial cell carcinoma |
| url | https://doi.org/10.1186/s12859-023-05199-y |
| work_keys_str_mv | AT yusraaameen whichdatasubsetshouldbeaugmentedfordeeplearningasimulationstudyusingurothelialcellcarcinomahistopathologyimages AT daliambadary whichdatasubsetshouldbeaugmentedfordeeplearningasimulationstudyusingurothelialcellcarcinomahistopathologyimages AT ahmadelbadryiabonnoor whichdatasubsetshouldbeaugmentedfordeeplearningasimulationstudyusingurothelialcellcarcinomahistopathologyimages AT khaledfhussain whichdatasubsetshouldbeaugmentedfordeeplearningasimulationstudyusingurothelialcellcarcinomahistopathologyimages AT adelasewisy whichdatasubsetshouldbeaugmentedfordeeplearningasimulationstudyusingurothelialcellcarcinomahistopathologyimages |