Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri
Performing classification on noisy, crowdsourced image datasets can prove challenging even for the best neural networks. Two issues which complicate the problem on such datasets are class imbalance and ground-truth uncertainty in labeling. The AL-ALL and AL-PUB datasets - consisting of tightly cropp...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nicolas Turenne
2024-02-01
|
| Series: | Journal of Data Mining and Digital Humanities |
| Subjects: | |
| Online Access: | https://jdmdh.episciences.org/10297/pdf |
| _version_ | 1797268185224839168 |
|---|---|
| author | Graham West Matthew I. Swindall Ben Keener Timothy Player Alex C. Williams James H. Brusuelas John F. Wallin |
| author_facet | Graham West Matthew I. Swindall Ben Keener Timothy Player Alex C. Williams James H. Brusuelas John F. Wallin |
| author_sort | Graham West |
| collection | DOAJ |
| description | Performing classification on noisy, crowdsourced image datasets can prove
challenging even for the best neural networks. Two issues which complicate the
problem on such datasets are class imbalance and ground-truth uncertainty in
labeling. The AL-ALL and AL-PUB datasets - consisting of tightly cropped,
individual characters from images of ancient Greek papyri - are strongly
affected by both issues. The application of ensemble modeling to such datasets
can help identify images where the ground-truth is questionable and quantify
the trustworthiness of those samples. As such, we apply stacked generalization
consisting of nearly identical ResNets with different loss functions: one
utilizing sparse cross-entropy (CXE) and the other Kullback-Liebler Divergence
(KLD). Both networks use labels drawn from a crowd-sourced consensus. This
consensus is derived from a Normalized Distribution of Annotations (NDA) based
on all annotations for a given character in the dataset. For the second
network, the KLD is calculated with respect to the NDA. For our ensemble model,
we apply a k-nearest neighbors model to the outputs of the CXE and KLD
networks. Individually, the ResNet models have approximately 93% accuracy,
while the ensemble model achieves an accuracy of > 95%, increasing the
classification trustworthiness. We also perform an analysis of the Shannon
entropy of the various models' output distributions to measure classification
uncertainty. Our results suggest that entropy is useful for predicting model
misclassifications. |
| first_indexed | 2024-04-25T01:28:27Z |
| format | Article |
| id | doaj.art-ddda7a46187541e1bd55ce683f39a3dd |
| institution | Directory Open Access Journal |
| issn | 2416-5999 |
| language | English |
| last_indexed | 2024-04-25T01:28:27Z |
| publishDate | 2024-02-01 |
| publisher | Nicolas Turenne |
| record_format | Article |
| series | Journal of Data Mining and Digital Humanities |
| spelling | doaj.art-ddda7a46187541e1bd55ce683f39a3dd2024-03-08T15:27:53ZengNicolas TurenneJournal of Data Mining and Digital Humanities2416-59992024-02-012024Digital humanities in...10.46298/jdmdh.1029710297Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek PapyriGraham West0https://orcid.org/0000-0002-7095-1894Matthew I. Swindall1https://orcid.org/0000-0002-2507-6963Ben Keener2Timothy Player3https://orcid.org/0000-0002-8397-995XAlex C. Williams4James H. Brusuelas5John F. Wallin6https://orcid.org/0000-0001-5678-8325Middle Tennessee State UniversityMiddle Tennessee State UniversityUniversity of Tennessee at KnoxvilleUniversity of Tennessee at KnoxvilleAmazon (United States)University of KentuckyMiddle Tennessee State UniversityPerforming classification on noisy, crowdsourced image datasets can prove challenging even for the best neural networks. Two issues which complicate the problem on such datasets are class imbalance and ground-truth uncertainty in labeling. The AL-ALL and AL-PUB datasets - consisting of tightly cropped, individual characters from images of ancient Greek papyri - are strongly affected by both issues. The application of ensemble modeling to such datasets can help identify images where the ground-truth is questionable and quantify the trustworthiness of those samples. As such, we apply stacked generalization consisting of nearly identical ResNets with different loss functions: one utilizing sparse cross-entropy (CXE) and the other Kullback-Liebler Divergence (KLD). Both networks use labels drawn from a crowd-sourced consensus. This consensus is derived from a Normalized Distribution of Annotations (NDA) based on all annotations for a given character in the dataset. For the second network, the KLD is calculated with respect to the NDA. For our ensemble model, we apply a k-nearest neighbors model to the outputs of the CXE and KLD networks. Individually, the ResNet models have approximately 93% accuracy, while the ensemble model achieves an accuracy of > 95%, increasing the classification trustworthiness. We also perform an analysis of the Shannon entropy of the various models' output distributions to measure classification uncertainty. Our results suggest that entropy is useful for predicting model misclassifications.https://jdmdh.episciences.org/10297/pdfcomputer science - computer vision and pattern recognitioncomputer science - machine learning |
| spellingShingle | Graham West Matthew I. Swindall Ben Keener Timothy Player Alex C. Williams James H. Brusuelas John F. Wallin Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri Journal of Data Mining and Digital Humanities computer science - computer vision and pattern recognition computer science - machine learning |
| title | Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri |
| title_full | Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri |
| title_fullStr | Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri |
| title_full_unstemmed | Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri |
| title_short | Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri |
| title_sort | incorporating crowdsourced annotator distributions into ensemble modeling to improve classification trustworthiness for ancient greek papyri |
| topic | computer science - computer vision and pattern recognition computer science - machine learning |
| url | https://jdmdh.episciences.org/10297/pdf |
| work_keys_str_mv | AT grahamwest incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri AT matthewiswindall incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri AT benkeener incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri AT timothyplayer incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri AT alexcwilliams incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri AT jameshbrusuelas incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri AT johnfwallin incorporatingcrowdsourcedannotatordistributionsintoensemblemodelingtoimproveclassificationtrustworthinessforancientgreekpapyri |