AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE
Inability to use speech interfaces greatly limits the deaf and hearing impaired people in the possibility of human-machine interaction. To solve this problem and to increase the accuracy and reliability of the automatic Russian sign language recognition system it is proposed to use lip-reading in ad...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2019-05-01
|
| Series: | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Online Access: | https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W12/97/2019/isprs-archives-XLII-2-W12-97-2019.pdf |
| _version_ | 1811280902565658624 |
|---|---|
| author | D. Ivanko D. Ryumin A. Karpov |
| author_facet | D. Ivanko D. Ryumin A. Karpov |
| author_sort | D. Ivanko |
| collection | DOAJ |
| description | Inability to use speech interfaces greatly limits the deaf and hearing impaired people in the possibility of human-machine interaction. To solve this problem and to increase the accuracy and reliability of the automatic Russian sign language recognition system it is proposed to use lip-reading in addition to hand gestures recognition. Deaf and hearing impaired people use sign language as the main way of communication in everyday life. Sign language is a structured form of hand gestures and lips movements involving visual motions and signs, which is used as a communication system. Since sign language includes not only hand gestures, but also lip movements that mimic vocalized pronunciation, it is of interest to investigate how accurately such a visual speech can be recognized by a lip-reading system, especially considering the fact that the visual speech of hearing impaired people is often characterized with hyper-articulation, which should potentially facilitate its recognition. For this purpose, thesaurus of Russian sign language (TheRusLan) collected in SPIIRAS in 2018–19 was used. The database consists of color optical FullHD video recordings of 13 native Russian sign language signers (11 females and 2 males) from “Pavlovsk boarding school for the hearing impaired”. Each of the signers demonstrated 164 phrases for 5 times. This work covers the initial stages of this research, including data collection, data labeling, region-of-interest detection and methods for informative features extraction. The results of this study can later be used to create assistive technologies for deaf or hearing impaired people. |
| first_indexed | 2024-04-13T01:24:10Z |
| format | Article |
| id | doaj.art-41e60a01c8d14c8a99e0732df1839d27 |
| institution | Directory Open Access Journal |
| issn | 1682-1750 2194-9034 |
| language | English |
| last_indexed | 2024-04-13T01:24:10Z |
| publishDate | 2019-05-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| spelling | doaj.art-41e60a01c8d14c8a99e0732df1839d272022-12-22T03:08:41ZengCopernicus PublicationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences1682-17502194-90342019-05-01XLII-2-W129710110.5194/isprs-archives-XLII-2-W12-97-2019AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLED. Ivanko0D. Ryumin1A. Karpov2St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, SPIIRAS, Saint-Petersburg, Russian FederationSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, SPIIRAS, Saint-Petersburg, Russian FederationSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, SPIIRAS, Saint-Petersburg, Russian FederationInability to use speech interfaces greatly limits the deaf and hearing impaired people in the possibility of human-machine interaction. To solve this problem and to increase the accuracy and reliability of the automatic Russian sign language recognition system it is proposed to use lip-reading in addition to hand gestures recognition. Deaf and hearing impaired people use sign language as the main way of communication in everyday life. Sign language is a structured form of hand gestures and lips movements involving visual motions and signs, which is used as a communication system. Since sign language includes not only hand gestures, but also lip movements that mimic vocalized pronunciation, it is of interest to investigate how accurately such a visual speech can be recognized by a lip-reading system, especially considering the fact that the visual speech of hearing impaired people is often characterized with hyper-articulation, which should potentially facilitate its recognition. For this purpose, thesaurus of Russian sign language (TheRusLan) collected in SPIIRAS in 2018–19 was used. The database consists of color optical FullHD video recordings of 13 native Russian sign language signers (11 females and 2 males) from “Pavlovsk boarding school for the hearing impaired”. Each of the signers demonstrated 164 phrases for 5 times. This work covers the initial stages of this research, including data collection, data labeling, region-of-interest detection and methods for informative features extraction. The results of this study can later be used to create assistive technologies for deaf or hearing impaired people.https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W12/97/2019/isprs-archives-XLII-2-W12-97-2019.pdf |
| spellingShingle | D. Ivanko D. Ryumin A. Karpov AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| title | AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE |
| title_full | AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE |
| title_fullStr | AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE |
| title_full_unstemmed | AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE |
| title_short | AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE |
| title_sort | automatic lip reading of hearing impaired people |
| url | https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W12/97/2019/isprs-archives-XLII-2-W12-97-2019.pdf |
| work_keys_str_mv | AT divanko automaticlipreadingofhearingimpairedpeople AT dryumin automaticlipreadingofhearingimpairedpeople AT akarpov automaticlipreadingofhearingimpairedpeople |