Geometric prior guided hybrid deep neural network for facial beauty analysis
Abstract Facial beauty analysis is an important topic in human society. It may be used as a guidance for face beautification applications such as cosmetic surgery. Deep neural networks (DNNs) have recently been adopted for facial beauty analysis and have achieved remarkable performance. However, mos...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2024-04-01
|
| Series: | CAAI Transactions on Intelligence Technology |
| Subjects: | |
| Online Access: | https://doi.org/10.1049/cit2.12197 |
| _version_ | 1797201356522520576 |
|---|---|
| author | Tianhao Peng Mu Li Fangmei Chen Yong Xu David Zhang |
| author_facet | Tianhao Peng Mu Li Fangmei Chen Yong Xu David Zhang |
| author_sort | Tianhao Peng |
| collection | DOAJ |
| description | Abstract Facial beauty analysis is an important topic in human society. It may be used as a guidance for face beautification applications such as cosmetic surgery. Deep neural networks (DNNs) have recently been adopted for facial beauty analysis and have achieved remarkable performance. However, most existing DNN‐based models regard facial beauty analysis as a normal classification task. They ignore important prior knowledge in traditional machine learning models which illustrate the significant contribution of the geometric features in facial beauty analysis. To be specific, landmarks of the whole face and facial organs are introduced to extract geometric features to make the decision. Inspired by this, we introduce a novel dual‐branch network for facial beauty analysis: one branch takes the Swin Transformer as the backbone to model the full face and global patterns, and another branch focuses on the masked facial organs with the residual network to model the local patterns of certain facial parts. Additionally, the designed multi‐scale feature fusion module can further facilitate our network to learn complementary semantic information between the two branches. In model optimisation, we propose a hybrid loss function, where especially geometric regulation is introduced by regressing the facial landmarks and it can force the extracted features to convey facial geometric features. Experiments performed on the SCUT‐FBP5500 dataset and the SCUT‐FBP dataset demonstrate that our model outperforms the state‐of‐the‐art convolutional neural networks models, which proves the effectiveness of the proposed geometric regularisation and dual‐branch structure with the hybrid network. To the best of our knowledge, this is the first study to introduce a Vision Transformer into the facial beauty analysis task. |
| first_indexed | 2024-04-24T07:46:15Z |
| format | Article |
| id | doaj.art-e2a5663f845f40fcb134cb2f23d09ce6 |
| institution | Directory Open Access Journal |
| issn | 2468-2322 |
| language | English |
| last_indexed | 2024-04-24T07:46:15Z |
| publishDate | 2024-04-01 |
| publisher | Wiley |
| record_format | Article |
| series | CAAI Transactions on Intelligence Technology |
| spelling | doaj.art-e2a5663f845f40fcb134cb2f23d09ce62024-04-19T03:11:29ZengWileyCAAI Transactions on Intelligence Technology2468-23222024-04-019246748010.1049/cit2.12197Geometric prior guided hybrid deep neural network for facial beauty analysisTianhao Peng0Mu Li1Fangmei Chen2Yong Xu3David Zhang4The School of Computer Science and Technology Guizhou University Guiyang ChinaThe School of Computer Science and Technology Harbin Institute of Technology Shenzhen Shenzhen ChinaThe Information and Communication Engineering Department Dalian Minzu University Dalian ChinaThe School of Computer Science and Technology Harbin Institute of Technology Shenzhen Shenzhen ChinaThe School of Data Science The Chinese University of Hong Kong Shenzhen Shenzhen ChinaAbstract Facial beauty analysis is an important topic in human society. It may be used as a guidance for face beautification applications such as cosmetic surgery. Deep neural networks (DNNs) have recently been adopted for facial beauty analysis and have achieved remarkable performance. However, most existing DNN‐based models regard facial beauty analysis as a normal classification task. They ignore important prior knowledge in traditional machine learning models which illustrate the significant contribution of the geometric features in facial beauty analysis. To be specific, landmarks of the whole face and facial organs are introduced to extract geometric features to make the decision. Inspired by this, we introduce a novel dual‐branch network for facial beauty analysis: one branch takes the Swin Transformer as the backbone to model the full face and global patterns, and another branch focuses on the masked facial organs with the residual network to model the local patterns of certain facial parts. Additionally, the designed multi‐scale feature fusion module can further facilitate our network to learn complementary semantic information between the two branches. In model optimisation, we propose a hybrid loss function, where especially geometric regulation is introduced by regressing the facial landmarks and it can force the extracted features to convey facial geometric features. Experiments performed on the SCUT‐FBP5500 dataset and the SCUT‐FBP dataset demonstrate that our model outperforms the state‐of‐the‐art convolutional neural networks models, which proves the effectiveness of the proposed geometric regularisation and dual‐branch structure with the hybrid network. To the best of our knowledge, this is the first study to introduce a Vision Transformer into the facial beauty analysis task.https://doi.org/10.1049/cit2.12197deep neural networksface analysisface biometricsimage analysis |
| spellingShingle | Tianhao Peng Mu Li Fangmei Chen Yong Xu David Zhang Geometric prior guided hybrid deep neural network for facial beauty analysis CAAI Transactions on Intelligence Technology deep neural networks face analysis face biometrics image analysis |
| title | Geometric prior guided hybrid deep neural network for facial beauty analysis |
| title_full | Geometric prior guided hybrid deep neural network for facial beauty analysis |
| title_fullStr | Geometric prior guided hybrid deep neural network for facial beauty analysis |
| title_full_unstemmed | Geometric prior guided hybrid deep neural network for facial beauty analysis |
| title_short | Geometric prior guided hybrid deep neural network for facial beauty analysis |
| title_sort | geometric prior guided hybrid deep neural network for facial beauty analysis |
| topic | deep neural networks face analysis face biometrics image analysis |
| url | https://doi.org/10.1049/cit2.12197 |
| work_keys_str_mv | AT tianhaopeng geometricpriorguidedhybriddeepneuralnetworkforfacialbeautyanalysis AT muli geometricpriorguidedhybriddeepneuralnetworkforfacialbeautyanalysis AT fangmeichen geometricpriorguidedhybriddeepneuralnetworkforfacialbeautyanalysis AT yongxu geometricpriorguidedhybriddeepneuralnetworkforfacialbeautyanalysis AT davidzhang geometricpriorguidedhybriddeepneuralnetworkforfacialbeautyanalysis |