Image sense disambiguation : a multimodal approach

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.

Bibliographic Details
Main Author: Saenko, Ekaterina, 1976-
Other Authors: Trevor Darrell.
Format: Thesis
Language:eng
Published: Massachusetts Institute of Technology 2010
Subjects:
Online Access:http://hdl.handle.net/1721.1/54651
_version_ 1826203243329355776
author Saenko, Ekaterina, 1976-
author2 Trevor Darrell.
author_facet Trevor Darrell.
Saenko, Ekaterina, 1976-
author_sort Saenko, Ekaterina, 1976-
collection MIT
description Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.
first_indexed 2024-09-23T12:33:52Z
format Thesis
id mit-1721.1/54651
institution Massachusetts Institute of Technology
language eng
last_indexed 2024-09-23T12:33:52Z
publishDate 2010
publisher Massachusetts Institute of Technology
record_format dspace
spelling mit-1721.1/546512019-04-12T14:39:03Z Image sense disambiguation : a multimodal approach Saenko, Ekaterina, 1976- Trevor Darrell. Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science. Electrical Engineering and Computer Science. Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009. Cataloged from PDF version of thesis. Includes bibliographical references (p. 131-136). If a picture is worth a thousand words, can a thousand words be worth a training image? Most successful object recognition algorithms require manually annotated images of objects to be collected for training. The amount of human effort required to collect training data has limited most approaches to the several hundred object categories available in the labeled datasets. While human-annotated image data is scarce, additional sources of information can be used as weak labels, reducing the need for human supervision. In this thesis, we use three types of information to learn models of object categories: speech, text and dictionaries. We demonstrate that our use of non-traditional information sources facilitates automatic acquisition of visual object models for arbitrary words without requiring any labeled image examples. Spoken object references occur in many scenarios: interaction with an assistant robot, voice-tagging of photos, etc. Existing reference resolution methods are unimodal, relying either only on image features, or only on speech recognition. We propose a method that uses both the image of the object and the speech segment referring to it to disambiguate the underlying object label. We show that even noisy speech input helps visual recognition, and vice versa. We also explore two sources of linguistic sense information: the words surrounding images on web pages, and dictionary entries for nouns that refer to objects. Keywords that index images on the web have been used as weak object labels, but these tend to produce noisy datasets with many unrelated images. We use unlabeled text, dictionary definitions, and semantic relations between concepts to learn a refined model of image sense. Our model can work with as little supervision as a single English word. We apply this model to a dataset of web images indexed by polysemous keywords, and show that it improves both retrieval of specific senses, and the resulting object classifiers. by Kate Saenko. Ph.D. 2010-04-28T17:15:03Z 2010-04-28T17:15:03Z 2009 2009 Thesis http://hdl.handle.net/1721.1/54651 606593245 eng M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission. http://dspace.mit.edu/handle/1721.1/7582 136 p. application/pdf Massachusetts Institute of Technology
spellingShingle Electrical Engineering and Computer Science.
Saenko, Ekaterina, 1976-
Image sense disambiguation : a multimodal approach
title Image sense disambiguation : a multimodal approach
title_full Image sense disambiguation : a multimodal approach
title_fullStr Image sense disambiguation : a multimodal approach
title_full_unstemmed Image sense disambiguation : a multimodal approach
title_short Image sense disambiguation : a multimodal approach
title_sort image sense disambiguation a multimodal approach
topic Electrical Engineering and Computer Science.
url http://hdl.handle.net/1721.1/54651
work_keys_str_mv AT saenkoekaterina1976 imagesensedisambiguationamultimodalapproach