Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion
Global aging leads to a surge in neurological diseases. Quantitative gait analysis for the early detection of neurological diseases can effectively reduce the impact of the diseases. Recently, extensive research has focused on gait-abnormality-recognition algorithms using a single type of portable s...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/22/9101 |
| _version_ | 1797457844653522944 |
|---|---|
| author | Jing Li Weisheng Liang Xiyan Yin Jun Li Weizheng Guan |
| author_facet | Jing Li Weisheng Liang Xiyan Yin Jun Li Weizheng Guan |
| author_sort | Jing Li |
| collection | DOAJ |
| description | Global aging leads to a surge in neurological diseases. Quantitative gait analysis for the early detection of neurological diseases can effectively reduce the impact of the diseases. Recently, extensive research has focused on gait-abnormality-recognition algorithms using a single type of portable sensor. However, these studies are limited by the sensor’s type and the task specificity, constraining the widespread application of quantitative gait recognition. In this study, we propose a multimodal gait-abnormality-recognition framework based on a Convolutional Neural Network-Bidirectional Long Short-Term Memory (CNN-BiLSTM) network. The as-established framework effectively addresses the challenges arising from smooth data interference and lengthy time series by employing an adaptive sliding window technique. Then, we convert the time series into time–frequency plots to capture the characteristic variations in different abnormality gaits and achieve a unified representation of the multiple data types. This makes our signal processing method adaptable to several types of sensors. Additionally, we use a pre-trained Deep Convolutional Neural Network (DCNN) for feature extraction, and the consequently established CNN-BiLSTM network can achieve high-accuracy recognition by fusing and classifying the multi-sensor input data. To validate the proposed method, we conducted diversified experiments to recognize the gait abnormalities caused by different neuropathic diseases, such as amyotrophic lateral sclerosis (ALS), Parkinson’s disease (PD), and Huntington’s disease (HD). In the PDgait dataset, the framework achieved an accuracy of 98.89% in the classification of Parkinson’s disease severity, surpassing DCLSTM’s 96.71%. Moreover, the recognition accuracy of ALS, PD, and HD on the PDgait dataset was 100%, 96.97%, and 95.43% respectively, surpassing the majority of previously reported methods. These experimental results strongly demonstrate the potential of the proposed multimodal framework for gait abnormality identification. Due to the advantages of the framework, such as its suitability for different types of sensors and fewer training parameters, it is more suitable for gait monitoring in daily life and the customization of medical rehabilitation schedules, which will help more patients alleviate the harm caused by their diseases. |
| first_indexed | 2024-03-09T16:28:36Z |
| format | Article |
| id | doaj.art-9823841254ee4d308c7bbe0b2ad3d298 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T16:28:36Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-9823841254ee4d308c7bbe0b2ad3d2982023-11-24T15:05:20ZengMDPI AGSensors1424-82202023-11-012322910110.3390/s23229101Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data FusionJing Li0Weisheng Liang1Xiyan Yin2Jun Li3Weizheng Guan4School of Mechanical Engineering and Hubei Modern Manufacturing Quality Engineering Key Laboratory, Hubei University of Technology, Wuhan 430068, ChinaSchool of Mechanical Engineering and Hubei Modern Manufacturing Quality Engineering Key Laboratory, Hubei University of Technology, Wuhan 430068, ChinaSchool of Mechanical Engineering and Hubei Modern Manufacturing Quality Engineering Key Laboratory, Hubei University of Technology, Wuhan 430068, ChinaDetroit Green Technology Institute, Hubei University of Technology, Wuhan 430068, ChinaDetroit Green Technology Institute, Hubei University of Technology, Wuhan 430068, ChinaGlobal aging leads to a surge in neurological diseases. Quantitative gait analysis for the early detection of neurological diseases can effectively reduce the impact of the diseases. Recently, extensive research has focused on gait-abnormality-recognition algorithms using a single type of portable sensor. However, these studies are limited by the sensor’s type and the task specificity, constraining the widespread application of quantitative gait recognition. In this study, we propose a multimodal gait-abnormality-recognition framework based on a Convolutional Neural Network-Bidirectional Long Short-Term Memory (CNN-BiLSTM) network. The as-established framework effectively addresses the challenges arising from smooth data interference and lengthy time series by employing an adaptive sliding window technique. Then, we convert the time series into time–frequency plots to capture the characteristic variations in different abnormality gaits and achieve a unified representation of the multiple data types. This makes our signal processing method adaptable to several types of sensors. Additionally, we use a pre-trained Deep Convolutional Neural Network (DCNN) for feature extraction, and the consequently established CNN-BiLSTM network can achieve high-accuracy recognition by fusing and classifying the multi-sensor input data. To validate the proposed method, we conducted diversified experiments to recognize the gait abnormalities caused by different neuropathic diseases, such as amyotrophic lateral sclerosis (ALS), Parkinson’s disease (PD), and Huntington’s disease (HD). In the PDgait dataset, the framework achieved an accuracy of 98.89% in the classification of Parkinson’s disease severity, surpassing DCLSTM’s 96.71%. Moreover, the recognition accuracy of ALS, PD, and HD on the PDgait dataset was 100%, 96.97%, and 95.43% respectively, surpassing the majority of previously reported methods. These experimental results strongly demonstrate the potential of the proposed multimodal framework for gait abnormality identification. Due to the advantages of the framework, such as its suitability for different types of sensors and fewer training parameters, it is more suitable for gait monitoring in daily life and the customization of medical rehabilitation schedules, which will help more patients alleviate the harm caused by their diseases.https://www.mdpi.com/1424-8220/23/22/9101neurodegenerative diseasesgait abnormality recognitionmulti-sensortime–frequency plotsCNN-BiLSTM network |
| spellingShingle | Jing Li Weisheng Liang Xiyan Yin Jun Li Weizheng Guan Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion Sensors neurodegenerative diseases gait abnormality recognition multi-sensor time–frequency plots CNN-BiLSTM network |
| title | Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion |
| title_full | Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion |
| title_fullStr | Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion |
| title_full_unstemmed | Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion |
| title_short | Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion |
| title_sort | multimodal gait abnormality recognition using a convolutional neural network bidirectional long short term memory cnn bilstm network based on multi sensor data fusion |
| topic | neurodegenerative diseases gait abnormality recognition multi-sensor time–frequency plots CNN-BiLSTM network |
| url | https://www.mdpi.com/1424-8220/23/22/9101 |
| work_keys_str_mv | AT jingli multimodalgaitabnormalityrecognitionusingaconvolutionalneuralnetworkbidirectionallongshorttermmemorycnnbilstmnetworkbasedonmultisensordatafusion AT weishengliang multimodalgaitabnormalityrecognitionusingaconvolutionalneuralnetworkbidirectionallongshorttermmemorycnnbilstmnetworkbasedonmultisensordatafusion AT xiyanyin multimodalgaitabnormalityrecognitionusingaconvolutionalneuralnetworkbidirectionallongshorttermmemorycnnbilstmnetworkbasedonmultisensordatafusion AT junli multimodalgaitabnormalityrecognitionusingaconvolutionalneuralnetworkbidirectionallongshorttermmemorycnnbilstmnetworkbasedonmultisensordatafusion AT weizhengguan multimodalgaitabnormalityrecognitionusingaconvolutionalneuralnetworkbidirectionallongshorttermmemorycnnbilstmnetworkbasedonmultisensordatafusion |