On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition
The purpose of this paper was to investigate the effect of a training state-of-the-art convolution neural network (CNN) for millimeter-wave radar-based hand gesture recognition (MR-HGR). Focusing on the small training dataset problem in MR-HGR, this paper first proposed to transfer the knowledge wit...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-01-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/1/259 |
| _version_ | 1827698575934488576 |
|---|---|
| author | Kang Zhang Shengchang Lan Guiyuan Zhang |
| author_facet | Kang Zhang Shengchang Lan Guiyuan Zhang |
| author_sort | Kang Zhang |
| collection | DOAJ |
| description | The purpose of this paper was to investigate the effect of a training state-of-the-art convolution neural network (CNN) for millimeter-wave radar-based hand gesture recognition (MR-HGR). Focusing on the small training dataset problem in MR-HGR, this paper first proposed to transfer the knowledge with the CNN models in computer vision to MR-HGR by fine-tuning the models with radar data samples. Meanwhile, for the different data modality in MR-HGR, a parameterized representation of temporal space-velocity (TSV) spectrogram was proposed as an integrated data modality of the time-evolving hand gesture features in the radar echo signals. The TSV spectrograms representing six common gestures in human–computer interaction (HCI) from nine volunteers were used as the data samples in the experiment. The evaluated models included ResNet with 50, 101, and 152 layers, DenseNet with 121, 161 and 169 layers, as well as light-weight MobileNet V2 and ShuffleNet V2, mostly proposed by many latest publications. In the experiment, not only self-testing (ST), but also more persuasive cross-testing (CT), were implemented to evaluate whether the fine-tuned models generalize to the radar data samples. The CT results show that the best fine-tuned models can reach to an average accuracy higher than 93% with a comparable ST average accuracy almost 100%. Moreover, in order to alleviate the problem caused by private gesture habits, an auxiliary test was performed by augmenting four shots of the gestures with the heaviest misclassifications into the training set. This enriching test is similar with the scenario that a tablet reacts to a new user. The results of two different volunteer in the enriching test shows that the average accuracy of the enriched gesture can be improved from 55.59% and 65.58% to 90.66% and 95.95% respectively. Compared with some baseline work in MR-HGR, the investigation by this paper can be beneficial in promoting MR-HGR in future industry applications and consumer electronic design. |
| first_indexed | 2024-03-10T13:32:09Z |
| format | Article |
| id | doaj.art-a8fae6e696e242768714f4e47d507bed |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T13:32:09Z |
| publishDate | 2021-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-a8fae6e696e242768714f4e47d507bed2023-11-21T07:49:47ZengMDPI AGSensors1424-82202021-01-0121125910.3390/s21010259On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture RecognitionKang Zhang0Shengchang Lan1Guiyuan Zhang2Department of Microwave Engineering, Harbin Institute of Technology, Harbin 150001, ChinaDepartment of Microwave Engineering, Harbin Institute of Technology, Harbin 150001, ChinaDepartment of Microwave Engineering, Harbin Institute of Technology, Harbin 150001, ChinaThe purpose of this paper was to investigate the effect of a training state-of-the-art convolution neural network (CNN) for millimeter-wave radar-based hand gesture recognition (MR-HGR). Focusing on the small training dataset problem in MR-HGR, this paper first proposed to transfer the knowledge with the CNN models in computer vision to MR-HGR by fine-tuning the models with radar data samples. Meanwhile, for the different data modality in MR-HGR, a parameterized representation of temporal space-velocity (TSV) spectrogram was proposed as an integrated data modality of the time-evolving hand gesture features in the radar echo signals. The TSV spectrograms representing six common gestures in human–computer interaction (HCI) from nine volunteers were used as the data samples in the experiment. The evaluated models included ResNet with 50, 101, and 152 layers, DenseNet with 121, 161 and 169 layers, as well as light-weight MobileNet V2 and ShuffleNet V2, mostly proposed by many latest publications. In the experiment, not only self-testing (ST), but also more persuasive cross-testing (CT), were implemented to evaluate whether the fine-tuned models generalize to the radar data samples. The CT results show that the best fine-tuned models can reach to an average accuracy higher than 93% with a comparable ST average accuracy almost 100%. Moreover, in order to alleviate the problem caused by private gesture habits, an auxiliary test was performed by augmenting four shots of the gestures with the heaviest misclassifications into the training set. This enriching test is similar with the scenario that a tablet reacts to a new user. The results of two different volunteer in the enriching test shows that the average accuracy of the enriched gesture can be improved from 55.59% and 65.58% to 90.66% and 95.95% respectively. Compared with some baseline work in MR-HGR, the investigation by this paper can be beneficial in promoting MR-HGR in future industry applications and consumer electronic design.https://www.mdpi.com/1424-8220/21/1/259hand gesture recognitionFMCW radarFine tunemillimeter-wave band |
| spellingShingle | Kang Zhang Shengchang Lan Guiyuan Zhang On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition Sensors hand gesture recognition FMCW radar Fine tune millimeter-wave band |
| title | On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition |
| title_full | On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition |
| title_fullStr | On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition |
| title_full_unstemmed | On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition |
| title_short | On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition |
| title_sort | on the effect of training convolution neural network for millimeter wave radar based hand gesture recognition |
| topic | hand gesture recognition FMCW radar Fine tune millimeter-wave band |
| url | https://www.mdpi.com/1424-8220/21/1/259 |
| work_keys_str_mv | AT kangzhang ontheeffectoftrainingconvolutionneuralnetworkformillimeterwaveradarbasedhandgesturerecognition AT shengchanglan ontheeffectoftrainingconvolutionneuralnetworkformillimeterwaveradarbasedhandgesturerecognition AT guiyuanzhang ontheeffectoftrainingconvolutionneuralnetworkformillimeterwaveradarbasedhandgesturerecognition |