Sign Language Motion Generation from Sign Characteristics
This paper proposes, analyzes, and evaluates a deep learning architecture based on transformers for generating sign language motion from sign phonemes (represented using HamNoSys: a notation system developed at the University of Hamburg). The sign phonemes provide information about sign characterist...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/23/9365 |
| _version_ | 1797399552691535872 |
|---|---|
| author | Manuel Gil-Martín María Villa-Monedero Andrzej Pomirski Daniel Sáez-Trigueros Rubén San-Segundo |
| author_facet | Manuel Gil-Martín María Villa-Monedero Andrzej Pomirski Daniel Sáez-Trigueros Rubén San-Segundo |
| author_sort | Manuel Gil-Martín |
| collection | DOAJ |
| description | This paper proposes, analyzes, and evaluates a deep learning architecture based on transformers for generating sign language motion from sign phonemes (represented using HamNoSys: a notation system developed at the University of Hamburg). The sign phonemes provide information about sign characteristics like hand configuration, localization, or movements. The use of sign phonemes is crucial for generating sign motion with a high level of details (including finger extensions and flexions). The transformer-based approach also includes a stop detection module for predicting the end of the generation process. Both aspects, motion generation and stop detection, are evaluated in detail. For motion generation, the dynamic time warping distance is used to compute the similarity between two landmarks sequences (ground truth and generated). The stop detection module is evaluated considering detection accuracy and ROC (receiver operating characteristic) curves. The paper proposes and evaluates several strategies to obtain the system configuration with the best performance. These strategies include different padding strategies, interpolation approaches, and data augmentation techniques. The best configuration of a fully automatic system obtains an average DTW distance per frame of 0.1057 and an area under the ROC curve (AUC) higher than 0.94. |
| first_indexed | 2024-03-09T01:42:50Z |
| format | Article |
| id | doaj.art-305cda5287cd43049dde00d55eb551ef |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T01:42:50Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-305cda5287cd43049dde00d55eb551ef2023-12-08T15:25:39ZengMDPI AGSensors1424-82202023-11-012323936510.3390/s23239365Sign Language Motion Generation from Sign CharacteristicsManuel Gil-Martín0María Villa-Monedero1Andrzej Pomirski2Daniel Sáez-Trigueros3Rubén San-Segundo4Grupo de Tecnología del Habla y Aprendizaje Automático (T.H.A.U. Group), Information Processing and Telecommunications Center, Escuela Técnica Superior de Ingenieros de Telecomunicación, Universidad Politécnica de Madrid, 28040 Madrid, SpainGrupo de Tecnología del Habla y Aprendizaje Automático (T.H.A.U. Group), Information Processing and Telecommunications Center, Escuela Técnica Superior de Ingenieros de Telecomunicación, Universidad Politécnica de Madrid, 28040 Madrid, SpainAlexa AI, Aleja Grunwaldzka 472, 80-309 Gdańsk, PolandAlexa AI, C. de Ramírez de Prado 5, 28045 Madrid, SpainGrupo de Tecnología del Habla y Aprendizaje Automático (T.H.A.U. Group), Information Processing and Telecommunications Center, Escuela Técnica Superior de Ingenieros de Telecomunicación, Universidad Politécnica de Madrid, 28040 Madrid, SpainThis paper proposes, analyzes, and evaluates a deep learning architecture based on transformers for generating sign language motion from sign phonemes (represented using HamNoSys: a notation system developed at the University of Hamburg). The sign phonemes provide information about sign characteristics like hand configuration, localization, or movements. The use of sign phonemes is crucial for generating sign motion with a high level of details (including finger extensions and flexions). The transformer-based approach also includes a stop detection module for predicting the end of the generation process. Both aspects, motion generation and stop detection, are evaluated in detail. For motion generation, the dynamic time warping distance is used to compute the similarity between two landmarks sequences (ground truth and generated). The stop detection module is evaluated considering detection accuracy and ROC (receiver operating characteristic) curves. The paper proposes and evaluates several strategies to obtain the system configuration with the best performance. These strategies include different padding strategies, interpolation approaches, and data augmentation techniques. The best configuration of a fully automatic system obtains an average DTW distance per frame of 0.1057 and an area under the ROC curve (AUC) higher than 0.94.https://www.mdpi.com/1424-8220/23/23/9365motion generationmotion datasetsign languagesign phonemesHamNoSyslandmarks extraction |
| spellingShingle | Manuel Gil-Martín María Villa-Monedero Andrzej Pomirski Daniel Sáez-Trigueros Rubén San-Segundo Sign Language Motion Generation from Sign Characteristics Sensors motion generation motion dataset sign language sign phonemes HamNoSys landmarks extraction |
| title | Sign Language Motion Generation from Sign Characteristics |
| title_full | Sign Language Motion Generation from Sign Characteristics |
| title_fullStr | Sign Language Motion Generation from Sign Characteristics |
| title_full_unstemmed | Sign Language Motion Generation from Sign Characteristics |
| title_short | Sign Language Motion Generation from Sign Characteristics |
| title_sort | sign language motion generation from sign characteristics |
| topic | motion generation motion dataset sign language sign phonemes HamNoSys landmarks extraction |
| url | https://www.mdpi.com/1424-8220/23/23/9365 |
| work_keys_str_mv | AT manuelgilmartin signlanguagemotiongenerationfromsigncharacteristics AT mariavillamonedero signlanguagemotiongenerationfromsigncharacteristics AT andrzejpomirski signlanguagemotiongenerationfromsigncharacteristics AT danielsaeztrigueros signlanguagemotiongenerationfromsigncharacteristics AT rubensansegundo signlanguagemotiongenerationfromsigncharacteristics |