Lipreading Using Liquid State Machine with STDP-Tuning
Lipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker’s lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of comput...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/20/10484 |
| _version_ | 1797475536545513472 |
|---|---|
| author | Xuhu Yu Zhong Wan Zehao Shi Lei Wang |
| author_facet | Xuhu Yu Zhong Wan Zehao Shi Lei Wang |
| author_sort | Xuhu Yu |
| collection | DOAJ |
| description | Lipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker’s lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of computing resources, which is not conducive to the migration of the system to edge devices. Inspired by the work of Spiking Neural Networks (SNNs) in recognizing human actions and gestures, we propose a lipreading system based on SNNs. Specifically, we construct the front-end feature extractor of the system using Liquid State Machine (LSM). On the other hand, a heuristic algorithm is used to select appropriate parameters for the classifier in the backend. On small-scale lipreading datasets, our recognition accuracy achieves good results. We claim that our network performs better in terms of accuracy and ratio of learned parameters compared to other networks, and has superior advantages in terms of network complexity and training cost. On the AVLetters dataset, our model achieves a 5% improvement in accuracy over traditional methods and a 90% reduction in parameters over the state-of-the-art. |
| first_indexed | 2024-03-09T20:46:31Z |
| format | Article |
| id | doaj.art-418d7e7e32544139b318a5c9fe065317 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T20:46:31Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-418d7e7e32544139b318a5c9fe0653172023-11-23T22:45:27ZengMDPI AGApplied Sciences2076-34172022-10-0112201048410.3390/app122010484Lipreading Using Liquid State Machine with STDP-TuningXuhu Yu0Zhong Wan1Zehao Shi2Lei Wang3The College of Computer Science, National University of Defence Technology, Changsha 410073, ChinaThe College of Computer Science, National University of Defence Technology, Changsha 410073, ChinaThe College of Computer Science, National University of Defence Technology, Changsha 410073, ChinaThe College of Computer Science, National University of Defence Technology, Changsha 410073, ChinaLipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker’s lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of computing resources, which is not conducive to the migration of the system to edge devices. Inspired by the work of Spiking Neural Networks (SNNs) in recognizing human actions and gestures, we propose a lipreading system based on SNNs. Specifically, we construct the front-end feature extractor of the system using Liquid State Machine (LSM). On the other hand, a heuristic algorithm is used to select appropriate parameters for the classifier in the backend. On small-scale lipreading datasets, our recognition accuracy achieves good results. We claim that our network performs better in terms of accuracy and ratio of learned parameters compared to other networks, and has superior advantages in terms of network complexity and training cost. On the AVLetters dataset, our model achieves a 5% improvement in accuracy over traditional methods and a 90% reduction in parameters over the state-of-the-art.https://www.mdpi.com/2076-3417/12/20/10484lipreadingliquid state machineSTDP |
| spellingShingle | Xuhu Yu Zhong Wan Zehao Shi Lei Wang Lipreading Using Liquid State Machine with STDP-Tuning Applied Sciences lipreading liquid state machine STDP |
| title | Lipreading Using Liquid State Machine with STDP-Tuning |
| title_full | Lipreading Using Liquid State Machine with STDP-Tuning |
| title_fullStr | Lipreading Using Liquid State Machine with STDP-Tuning |
| title_full_unstemmed | Lipreading Using Liquid State Machine with STDP-Tuning |
| title_short | Lipreading Using Liquid State Machine with STDP-Tuning |
| title_sort | lipreading using liquid state machine with stdp tuning |
| topic | lipreading liquid state machine STDP |
| url | https://www.mdpi.com/2076-3417/12/20/10484 |
| work_keys_str_mv | AT xuhuyu lipreadingusingliquidstatemachinewithstdptuning AT zhongwan lipreadingusingliquidstatemachinewithstdptuning AT zehaoshi lipreadingusingliquidstatemachinewithstdptuning AT leiwang lipreadingusingliquidstatemachinewithstdptuning |