From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition
The advancement of deep learning in human activity recognition (HAR) using 3D skeleton data is critical for applications in healthcare, security, sports, and human–computer interaction. This paper tackles a well-known gap in the field, which is the lack of testing in the applicability and reliabilit...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/6/1940 |
| _version_ | 1797239341560365056 |
|---|---|
| author | Kimji N. Pellano Inga Strümke Espen A. F. Ihlen |
| author_facet | Kimji N. Pellano Inga Strümke Espen A. F. Ihlen |
| author_sort | Kimji N. Pellano |
| collection | DOAJ |
| description | The advancement of deep learning in human activity recognition (HAR) using 3D skeleton data is critical for applications in healthcare, security, sports, and human–computer interaction. This paper tackles a well-known gap in the field, which is the lack of testing in the applicability and reliability of XAI evaluation metrics in the skeleton-based HAR domain. We have tested established XAI metrics, namely faithfulness and stability on Class Activation Mapping (CAM) and Gradient-weighted Class Activation Mapping (Grad-CAM) to address this problem. This study introduces a perturbation method that produces variations within the error tolerance of motion sensor tracking, ensuring the resultant skeletal data points remain within the plausible output range of human movement as captured by the tracking device. We used the NTU RGB+D 60 dataset and the EfficientGCN architecture for HAR model training and testing. The evaluation involved systematically perturbing the 3D skeleton data by applying controlled displacements at different magnitudes to assess the impact on XAI metric performance across multiple action classes. Our findings reveal that faithfulness may not consistently serve as a reliable metric across all classes for the EfficientGCN model, indicating its limited applicability in certain contexts. In contrast, stability proves to be a more robust metric, showing dependability across different perturbation magnitudes. Additionally, CAM and Grad-CAM yielded almost identical explanations, leading to closely similar metric outcomes. This suggests a need for the exploration of additional metrics and the application of more diverse XAI methods to broaden the understanding and effectiveness of XAI in skeleton-based HAR. |
| first_indexed | 2024-04-24T17:50:00Z |
| format | Article |
| id | doaj.art-e6530833737243f7ad584f4143f471fa |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-04-24T17:50:00Z |
| publishDate | 2024-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-e6530833737243f7ad584f4143f471fa2024-03-27T14:04:11ZengMDPI AGSensors1424-82202024-03-01246194010.3390/s24061940From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity RecognitionKimji N. Pellano0Inga Strümke1Espen A. F. Ihlen2Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, 7034 Trondheim, NorwayDepartment of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, 7034 Trondheim, NorwayDepartment of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, 7034 Trondheim, NorwayThe advancement of deep learning in human activity recognition (HAR) using 3D skeleton data is critical for applications in healthcare, security, sports, and human–computer interaction. This paper tackles a well-known gap in the field, which is the lack of testing in the applicability and reliability of XAI evaluation metrics in the skeleton-based HAR domain. We have tested established XAI metrics, namely faithfulness and stability on Class Activation Mapping (CAM) and Gradient-weighted Class Activation Mapping (Grad-CAM) to address this problem. This study introduces a perturbation method that produces variations within the error tolerance of motion sensor tracking, ensuring the resultant skeletal data points remain within the plausible output range of human movement as captured by the tracking device. We used the NTU RGB+D 60 dataset and the EfficientGCN architecture for HAR model training and testing. The evaluation involved systematically perturbing the 3D skeleton data by applying controlled displacements at different magnitudes to assess the impact on XAI metric performance across multiple action classes. Our findings reveal that faithfulness may not consistently serve as a reliable metric across all classes for the EfficientGCN model, indicating its limited applicability in certain contexts. In contrast, stability proves to be a more robust metric, showing dependability across different perturbation magnitudes. Additionally, CAM and Grad-CAM yielded almost identical explanations, leading to closely similar metric outcomes. This suggests a need for the exploration of additional metrics and the application of more diverse XAI methods to broaden the understanding and effectiveness of XAI in skeleton-based HAR.https://www.mdpi.com/1424-8220/24/6/1940explainable AICAMGrad-CAMskeleton datahuman activity recognition |
| spellingShingle | Kimji N. Pellano Inga Strümke Espen A. F. Ihlen From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition Sensors explainable AI CAM Grad-CAM skeleton data human activity recognition |
| title | From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition |
| title_full | From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition |
| title_fullStr | From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition |
| title_full_unstemmed | From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition |
| title_short | From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition |
| title_sort | from movements to metrics evaluating explainable ai methods in skeleton based human activity recognition |
| topic | explainable AI CAM Grad-CAM skeleton data human activity recognition |
| url | https://www.mdpi.com/1424-8220/24/6/1940 |
| work_keys_str_mv | AT kimjinpellano frommovementstometricsevaluatingexplainableaimethodsinskeletonbasedhumanactivityrecognition AT ingastrumke frommovementstometricsevaluatingexplainableaimethodsinskeletonbasedhumanactivityrecognition AT espenafihlen frommovementstometricsevaluatingexplainableaimethodsinskeletonbasedhumanactivityrecognition |