Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification
Recent research in deep Convolutional Neural Networks(CNN) faces the challenges of vanishing/exploding gradient issues, training instability, and feature redundancy. Orthogonality Regularization(OR), which introduces a penalty function considering the orthogonality of neural networks, could be a rem...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9804718/ |
| _version_ | 1811309347729309696 |
|---|---|
| author | Taehyeon Kim Se-Young Yun |
| author_facet | Taehyeon Kim Se-Young Yun |
| author_sort | Taehyeon Kim |
| collection | DOAJ |
| description | Recent research in deep Convolutional Neural Networks(CNN) faces the challenges of vanishing/exploding gradient issues, training instability, and feature redundancy. Orthogonality Regularization(OR), which introduces a penalty function considering the orthogonality of neural networks, could be a remedy to these challenges but is surprisingly not popular in the literature. This work revisits the OR approaches and empirically answer the question: <italic>Even when comparing various regularizations like weight decay and spectral norm regularization, which is the most powerful OR technique?</italic> We begin by introducing the improvements of various regularization techniques, specifically focusing on OR approaches over a variety of architectures. After that, we disentangle the benefits of OR in the comparison of other regularization approaches with a connection on how they affect norm preservation effects and feature redundancy in the forward and backward propagation. Our investigations show that Kernel Orthogonality Regularization(KOR) approaches, which directly penalize the orthogonality of convolutional kernel matrices, consistently outperform other techniques. We propose a simple KOR method considering both row- and column- orthogonality, of which empirical performance is the most effective in mitigating the aforementioned challenges. We further discuss several circumstances in the recent CNN models on various benchmark datasets, wherein KOR gains more effectiveness. |
| first_indexed | 2024-04-13T09:40:40Z |
| format | Article |
| id | doaj.art-13ce91d222d74e2eac7701112655f1b2 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-13T09:40:40Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-13ce91d222d74e2eac7701112655f1b22022-12-22T02:51:55ZengIEEEIEEE Access2169-35362022-01-0110697416974910.1109/ACCESS.2022.31856219804718Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image ClassificationTaehyeon Kim0Se-Young Yun1https://orcid.org/0000-0001-6675-5113Kim Jaechul Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology (KAIST), Seoul, South KoreaKim Jaechul Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology (KAIST), Seoul, South KoreaRecent research in deep Convolutional Neural Networks(CNN) faces the challenges of vanishing/exploding gradient issues, training instability, and feature redundancy. Orthogonality Regularization(OR), which introduces a penalty function considering the orthogonality of neural networks, could be a remedy to these challenges but is surprisingly not popular in the literature. This work revisits the OR approaches and empirically answer the question: <italic>Even when comparing various regularizations like weight decay and spectral norm regularization, which is the most powerful OR technique?</italic> We begin by introducing the improvements of various regularization techniques, specifically focusing on OR approaches over a variety of architectures. After that, we disentangle the benefits of OR in the comparison of other regularization approaches with a connection on how they affect norm preservation effects and feature redundancy in the forward and backward propagation. Our investigations show that Kernel Orthogonality Regularization(KOR) approaches, which directly penalize the orthogonality of convolutional kernel matrices, consistently outperform other techniques. We propose a simple KOR method considering both row- and column- orthogonality, of which empirical performance is the most effective in mitigating the aforementioned challenges. We further discuss several circumstances in the recent CNN models on various benchmark datasets, wherein KOR gains more effectiveness.https://ieeexplore.ieee.org/document/9804718/Deep neural network (DNN)kernelorthogonality regularizationconvolutional neural network (CNN)regularizationimage classification |
| spellingShingle | Taehyeon Kim Se-Young Yun Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification IEEE Access Deep neural network (DNN) kernel orthogonality regularization convolutional neural network (CNN) regularization image classification |
| title | Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification |
| title_full | Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification |
| title_fullStr | Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification |
| title_full_unstemmed | Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification |
| title_short | Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification |
| title_sort | revisiting orthogonality regularization a study for convolutional neural networks in image classification |
| topic | Deep neural network (DNN) kernel orthogonality regularization convolutional neural network (CNN) regularization image classification |
| url | https://ieeexplore.ieee.org/document/9804718/ |
| work_keys_str_mv | AT taehyeonkim revisitingorthogonalityregularizationastudyforconvolutionalneuralnetworksinimageclassification AT seyoungyun revisitingorthogonalityregularizationastudyforconvolutionalneuralnetworksinimageclassification |