Learning spoken language through vision
Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018.
| Glavni avtor: | |
|---|---|
| Drugi avtorji: | |
| Format: | Thesis |
| Jezik: | eng |
| Izdano: |
Massachusetts Institute of Technology
2018
|
| Teme: | |
| Online dostop: | http://hdl.handle.net/1721.1/118081 |
| _version_ | 1826190754201993216 |
|---|---|
| author | Harwath, David F. (David Frank) |
| author2 | James R. Glass. |
| author_facet | James R. Glass. Harwath, David F. (David Frank) |
| author_sort | Harwath, David F. (David Frank) |
| collection | MIT |
| description | Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018. |
| first_indexed | 2024-09-23T08:44:51Z |
| format | Thesis |
| id | mit-1721.1/118081 |
| institution | Massachusetts Institute of Technology |
| language | eng |
| last_indexed | 2024-09-23T08:44:51Z |
| publishDate | 2018 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1180812019-04-10T07:18:48Z Learning spoken language through vision Harwath, David F. (David Frank) James R. Glass. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Electrical Engineering and Computer Science. Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018. Cataloged from PDF version of thesis. Includes bibliographical references (pages 145-159). Humans learn language at an early age by simply observing the world around them. Why can't computers do the same? Conventional automatic speech recognition systems have a long history and have recently made great strides thanks to the revival of deep neural networks. However, their reliance on highly supervised (and therefore expensive) training paradigms has restricted their application to the major languages of the world, accounting for a small fraction of the more than 7,000 human languages spoken worldwide. This thesis introduces datasets, models, and methodologies for grounding continuous speech signals at the raw waveform level to natural image scenes. The context and constraint provided by the visual information enables our models to efficiently learn linguistic units, such as words, along with their visual semantics. For example, our models are able to recognize instances of the spoken word "water" within spoken captions and associate them with image regions containing bodies of water. Further, we demonstrate that our models are capable of learning cross-lingual semantics by using the visual space as an interlingua to perform speech-to-speech retrieval between English and Hindi. In all cases, this learning is done without linguistic transcriptions or conventional speech recognition - yet we show that our methods achieve retrieval scores close to what is possible when transcriptions are available. This offers a promising new direction for speech processing that only requires speakers to provide narrations of what they see. by David Frank Harwath. Ph. D. 2018-09-17T15:56:40Z 2018-09-17T15:56:40Z 2018 2018 Thesis http://hdl.handle.net/1721.1/118081 1052123724 eng MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 159 pages application/pdf Massachusetts Institute of Technology |
| spellingShingle | Electrical Engineering and Computer Science. Harwath, David F. (David Frank) Learning spoken language through vision |
| title | Learning spoken language through vision |
| title_full | Learning spoken language through vision |
| title_fullStr | Learning spoken language through vision |
| title_full_unstemmed | Learning spoken language through vision |
| title_short | Learning spoken language through vision |
| title_sort | learning spoken language through vision |
| topic | Electrical Engineering and Computer Science. |
| url | http://hdl.handle.net/1721.1/118081 |
| work_keys_str_mv | AT harwathdavidfdavidfrank learningspokenlanguagethroughvision |