Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis
Due to the growing interest in climbing, increasing importance has been given to research in the field of non-invasive, camera-based motion analysis. While existing work uses invasive technologies such as wearables or modified walls and holds, or focuses on competitive sports, we for the first time...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/19/8216 |
| _version_ | 1797575135390072832 |
|---|---|
| author | Raul Beltrán Beltrán Julia Richter Guido Köstermeyer Ulrich Heinkel |
| author_facet | Raul Beltrán Beltrán Julia Richter Guido Köstermeyer Ulrich Heinkel |
| author_sort | Raul Beltrán Beltrán |
| collection | DOAJ |
| description | Due to the growing interest in climbing, increasing importance has been given to research in the field of non-invasive, camera-based motion analysis. While existing work uses invasive technologies such as wearables or modified walls and holds, or focuses on competitive sports, we for the first time present a system that uses video analysis to automatically recognize six movement errors that are typical for novices with limited climbing experience. Climbing a complete route consists of three repetitive climbing phases. Therefore, a characteristic joint arrangement may be detected as an error in a specific climbing phase, while this exact arrangement may not considered to be an error in another climbing phase. That is why we introduced a finite state machine to determine the current phase and to check for errors that commonly occur in the current phase. The transition between the phases depends on which joints are being used. To capture joint movements, we use a fourth-generation iPad Pro with LiDAR to record climbing sequences in which we convert the climber’s 2-D skeleton provided by the Vision framework from Apple into 3-D joints using the LiDAR depth information. Thereupon, we introduced a method that derives whether a joint moves or not, determining the current phase. Finally, the 3-D joints are analyzed with respect to defined characteristic joint arrangements to identify possible motion errors. To present the feedback to the climber, we imitate a virtual mentor by realizing an application on the iPad that creates an analysis immediately after the climber has finished the route by pointing out the detected errors and by giving suggestions for improvement. Quantitative tests with three experienced climbers that were able to climb reference routes without any errors and intentionally with errors resulted in precision–recall curves evaluating the error detection performance. The results demonstrate that while the number of false positives is still in an acceptable range, the number of detected errors is sufficient to provide climbing novices with adequate suggestions for improvement. Moreover, our study reveals limitations that mainly originate from incorrect joint localizations caused by the LiDAR sensor range. With human pose estimation becoming increasingly reliable and with the advance of sensor capabilities, these limitations will have a decreasing impact on our system performance. |
| first_indexed | 2024-03-10T21:34:15Z |
| format | Article |
| id | doaj.art-2a3dabb55ff54c09979c76c2f56f0125 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T21:34:15Z |
| publishDate | 2023-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-2a3dabb55ff54c09979c76c2f56f01252023-11-19T15:04:24ZengMDPI AGSensors1424-82202023-10-012319821610.3390/s23198216Climbing Technique Evaluation by Means of Skeleton Video Stream AnalysisRaul Beltrán Beltrán0Julia Richter1Guido Köstermeyer2Ulrich Heinkel3Professorship Circuit and System Design, Chemnitz University of Technology, Reichenhainer Straße 70, 09126 Chemnitz, GermanyProfessorship Circuit and System Design, Chemnitz University of Technology, Reichenhainer Straße 70, 09126 Chemnitz, GermanyDepartment Sportwissenschaft und Sport, Friedrich-Alexander-Universität Erlangen-Nürnberg, Schlossplatz 4, 91054 Erlangen, GermanyProfessorship Circuit and System Design, Chemnitz University of Technology, Reichenhainer Straße 70, 09126 Chemnitz, GermanyDue to the growing interest in climbing, increasing importance has been given to research in the field of non-invasive, camera-based motion analysis. While existing work uses invasive technologies such as wearables or modified walls and holds, or focuses on competitive sports, we for the first time present a system that uses video analysis to automatically recognize six movement errors that are typical for novices with limited climbing experience. Climbing a complete route consists of three repetitive climbing phases. Therefore, a characteristic joint arrangement may be detected as an error in a specific climbing phase, while this exact arrangement may not considered to be an error in another climbing phase. That is why we introduced a finite state machine to determine the current phase and to check for errors that commonly occur in the current phase. The transition between the phases depends on which joints are being used. To capture joint movements, we use a fourth-generation iPad Pro with LiDAR to record climbing sequences in which we convert the climber’s 2-D skeleton provided by the Vision framework from Apple into 3-D joints using the LiDAR depth information. Thereupon, we introduced a method that derives whether a joint moves or not, determining the current phase. Finally, the 3-D joints are analyzed with respect to defined characteristic joint arrangements to identify possible motion errors. To present the feedback to the climber, we imitate a virtual mentor by realizing an application on the iPad that creates an analysis immediately after the climber has finished the route by pointing out the detected errors and by giving suggestions for improvement. Quantitative tests with three experienced climbers that were able to climb reference routes without any errors and intentionally with errors resulted in precision–recall curves evaluating the error detection performance. The results demonstrate that while the number of false positives is still in an acceptable range, the number of detected errors is sufficient to provide climbing novices with adequate suggestions for improvement. Moreover, our study reveals limitations that mainly originate from incorrect joint localizations caused by the LiDAR sensor range. With human pose estimation becoming increasingly reliable and with the advance of sensor capabilities, these limitations will have a decreasing impact on our system performance.https://www.mdpi.com/1424-8220/23/19/8216climbing motion analysissports and computer sciencevideo analysiskey point detectionhuman pose estimation |
| spellingShingle | Raul Beltrán Beltrán Julia Richter Guido Köstermeyer Ulrich Heinkel Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis Sensors climbing motion analysis sports and computer science video analysis key point detection human pose estimation |
| title | Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis |
| title_full | Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis |
| title_fullStr | Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis |
| title_full_unstemmed | Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis |
| title_short | Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis |
| title_sort | climbing technique evaluation by means of skeleton video stream analysis |
| topic | climbing motion analysis sports and computer science video analysis key point detection human pose estimation |
| url | https://www.mdpi.com/1424-8220/23/19/8216 |
| work_keys_str_mv | AT raulbeltranbeltran climbingtechniqueevaluationbymeansofskeletonvideostreamanalysis AT juliarichter climbingtechniqueevaluationbymeansofskeletonvideostreamanalysis AT guidokostermeyer climbingtechniqueevaluationbymeansofskeletonvideostreamanalysis AT ulrichheinkel climbingtechniqueevaluationbymeansofskeletonvideostreamanalysis |