Deep Neural Network Ensembles Using Class-vs-Class Weighting
Ensembling is a popular and powerful technique to utilize predictions from several different machine learning models. The fundamental precondition of a well-working ensemble model is a diverse set of combined constituents. Rapid development in the deep learning field provides an ever-increasing pale...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10190625/ |
| _version_ | 1797768437124038656 |
|---|---|
| author | Rene Fabricius Ondrej Such Peter Tarabek |
| author_facet | Rene Fabricius Ondrej Such Peter Tarabek |
| author_sort | Rene Fabricius |
| collection | DOAJ |
| description | Ensembling is a popular and powerful technique to utilize predictions from several different machine learning models. The fundamental precondition of a well-working ensemble model is a diverse set of combined constituents. Rapid development in the deep learning field provides an ever-increasing palette of diverse model architectures. This rich variety of models provides an ideal situation to improve classification accuracy by ensembling. In this regard, we propose a novel weighted ensembling classification approach with unique weights for each combined classifier and each pair of classes. The novel weighting scheme allows us to account for the different abilities of individual classifiers to distinguish between pairs of classes. First, we analyze a theoretical scenario, in which our approach yields optimal classification. Second, we test its practical applicability on computer vision benchmark datasets. We evaluate the effectiveness of our proposed method and averaging ensemble baseline on an image classification task using the CIFAR-100 and ImageNet1k benchmarks. We use deep convolutional neural networks, vision transformers, and an MLP-Mixer as ensemble constituents. Statistical tests show that our proposed method provides higher accuracy gains than a popular baseline ensemble on both datasets. On the CIFAR-100 dataset, the proposed method attains accuracy improvements ranging from 2% to 5% compared to the best ensemble constituent. On the Imagenet dataset, these improvements range from 1% to 3% in most cases. Additionally, we show that when constituent classifiers are well -calibrated and have similar performance, the simple averaging ensemble yields good results. |
| first_indexed | 2024-03-12T20:53:29Z |
| format | Article |
| id | doaj.art-d0636749f91d41928d1ed1e185cc952b |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-12T20:53:29Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-d0636749f91d41928d1ed1e185cc952b2023-07-31T23:01:25ZengIEEEIEEE Access2169-35362023-01-0111777037771510.1109/ACCESS.2023.329805710190625Deep Neural Network Ensembles Using Class-vs-Class WeightingRene Fabricius0https://orcid.org/0000-0002-6886-2315Ondrej Such1Peter Tarabek2https://orcid.org/0000-0002-4815-3933Faculty of Management Sciences and Informatics, University of Žilina, Žilina, SlovakiaMathematical Institute of Slovak Academy of Sciences, Banská Bystrica, SlovakiaFaculty of Management Sciences and Informatics, University of Žilina, Žilina, SlovakiaEnsembling is a popular and powerful technique to utilize predictions from several different machine learning models. The fundamental precondition of a well-working ensemble model is a diverse set of combined constituents. Rapid development in the deep learning field provides an ever-increasing palette of diverse model architectures. This rich variety of models provides an ideal situation to improve classification accuracy by ensembling. In this regard, we propose a novel weighted ensembling classification approach with unique weights for each combined classifier and each pair of classes. The novel weighting scheme allows us to account for the different abilities of individual classifiers to distinguish between pairs of classes. First, we analyze a theoretical scenario, in which our approach yields optimal classification. Second, we test its practical applicability on computer vision benchmark datasets. We evaluate the effectiveness of our proposed method and averaging ensemble baseline on an image classification task using the CIFAR-100 and ImageNet1k benchmarks. We use deep convolutional neural networks, vision transformers, and an MLP-Mixer as ensemble constituents. Statistical tests show that our proposed method provides higher accuracy gains than a popular baseline ensemble on both datasets. On the CIFAR-100 dataset, the proposed method attains accuracy improvements ranging from 2% to 5% compared to the best ensemble constituent. On the Imagenet dataset, these improvements range from 1% to 3% in most cases. Additionally, we show that when constituent classifiers are well -calibrated and have similar performance, the simple averaging ensemble yields good results.https://ieeexplore.ieee.org/document/10190625/Pairwise couplingmulti-class classificationdeep neural networksdeep ensembleslinear discriminant analysishomoscedastic data |
| spellingShingle | Rene Fabricius Ondrej Such Peter Tarabek Deep Neural Network Ensembles Using Class-vs-Class Weighting IEEE Access Pairwise coupling multi-class classification deep neural networks deep ensembles linear discriminant analysis homoscedastic data |
| title | Deep Neural Network Ensembles Using Class-vs-Class Weighting |
| title_full | Deep Neural Network Ensembles Using Class-vs-Class Weighting |
| title_fullStr | Deep Neural Network Ensembles Using Class-vs-Class Weighting |
| title_full_unstemmed | Deep Neural Network Ensembles Using Class-vs-Class Weighting |
| title_short | Deep Neural Network Ensembles Using Class-vs-Class Weighting |
| title_sort | deep neural network ensembles using class vs class weighting |
| topic | Pairwise coupling multi-class classification deep neural networks deep ensembles linear discriminant analysis homoscedastic data |
| url | https://ieeexplore.ieee.org/document/10190625/ |
| work_keys_str_mv | AT renefabricius deepneuralnetworkensemblesusingclassvsclassweighting AT ondrejsuch deepneuralnetworkensemblesusingclassvsclassweighting AT petertarabek deepneuralnetworkensemblesusingclassvsclassweighting |