Image sense disambiguation : a multimodal approach
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | eng |
| Published: |
Massachusetts Institute of Technology
2010
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/54651 |
| _version_ | 1826203243329355776 |
|---|---|
| author | Saenko, Ekaterina, 1976- |
| author2 | Trevor Darrell. |
| author_facet | Trevor Darrell. Saenko, Ekaterina, 1976- |
| author_sort | Saenko, Ekaterina, 1976- |
| collection | MIT |
| description | Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009. |
| first_indexed | 2024-09-23T12:33:52Z |
| format | Thesis |
| id | mit-1721.1/54651 |
| institution | Massachusetts Institute of Technology |
| language | eng |
| last_indexed | 2024-09-23T12:33:52Z |
| publishDate | 2010 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/546512019-04-12T14:39:03Z Image sense disambiguation : a multimodal approach Saenko, Ekaterina, 1976- Trevor Darrell. Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science. Electrical Engineering and Computer Science. Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009. Cataloged from PDF version of thesis. Includes bibliographical references (p. 131-136). If a picture is worth a thousand words, can a thousand words be worth a training image? Most successful object recognition algorithms require manually annotated images of objects to be collected for training. The amount of human effort required to collect training data has limited most approaches to the several hundred object categories available in the labeled datasets. While human-annotated image data is scarce, additional sources of information can be used as weak labels, reducing the need for human supervision. In this thesis, we use three types of information to learn models of object categories: speech, text and dictionaries. We demonstrate that our use of non-traditional information sources facilitates automatic acquisition of visual object models for arbitrary words without requiring any labeled image examples. Spoken object references occur in many scenarios: interaction with an assistant robot, voice-tagging of photos, etc. Existing reference resolution methods are unimodal, relying either only on image features, or only on speech recognition. We propose a method that uses both the image of the object and the speech segment referring to it to disambiguate the underlying object label. We show that even noisy speech input helps visual recognition, and vice versa. We also explore two sources of linguistic sense information: the words surrounding images on web pages, and dictionary entries for nouns that refer to objects. Keywords that index images on the web have been used as weak object labels, but these tend to produce noisy datasets with many unrelated images. We use unlabeled text, dictionary definitions, and semantic relations between concepts to learn a refined model of image sense. Our model can work with as little supervision as a single English word. We apply this model to a dataset of web images indexed by polysemous keywords, and show that it improves both retrieval of specific senses, and the resulting object classifiers. by Kate Saenko. Ph.D. 2010-04-28T17:15:03Z 2010-04-28T17:15:03Z 2009 2009 Thesis http://hdl.handle.net/1721.1/54651 606593245 eng M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission. http://dspace.mit.edu/handle/1721.1/7582 136 p. application/pdf Massachusetts Institute of Technology |
| spellingShingle | Electrical Engineering and Computer Science. Saenko, Ekaterina, 1976- Image sense disambiguation : a multimodal approach |
| title | Image sense disambiguation : a multimodal approach |
| title_full | Image sense disambiguation : a multimodal approach |
| title_fullStr | Image sense disambiguation : a multimodal approach |
| title_full_unstemmed | Image sense disambiguation : a multimodal approach |
| title_short | Image sense disambiguation : a multimodal approach |
| title_sort | image sense disambiguation a multimodal approach |
| topic | Electrical Engineering and Computer Science. |
| url | http://hdl.handle.net/1721.1/54651 |
| work_keys_str_mv | AT saenkoekaterina1976 imagesensedisambiguationamultimodalapproach |