University of Illinois Urbana-Champaign

Grounding natural language phrases in images and video

Plummer, Bryan A.

Content Files
PLUMMER-DISSERTATION-2018.pdf

Permalink

https://hdl.handle.net/2142/100977

Description

Title
Grounding natural language phrases in images and video
Author(s)
Plummer, Bryan A.
Issue Date
2018-04-16
Director of Research (if dissertation) or Advisor (if thesis)
Lazebnik, Svetlana
Doctoral Committee Chair(s)
Lazebnik, Svetlana
Committee Member(s)
  • Hockenmaier, Julia
  • Hoiem, Derek
  • Brown, Matthew
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Date of Ingest
2018-09-04T20:27:05Z
Keyword(s)
Computer Vision, Natural Language Processing, Phrase Grounding
Abstract
Grounding language in images has shown it can help improve performance on many image-language tasks. To spur research on this topic, this dissertation introduces a new dataset which provides the ground truth annotations of the location of noun phrase chunks in image captions. I begin by introducing a constituent task termed phrase localization, where the goal is to localize an entity known to exist in an image when provided with a natural language query. To address this task, I introduce a model which learns a set of models, each of which capture a different concept which is useful in our task. These concepts can be predefined, such as attributes gleamed from the adjectives, as well as those which are automatically learned in a single-end-to-end neural network. I also address the more challenging detection style task, where the goal is to localize a phrase and determine if it is associated with an image. Multiple applications of the models presented in this work demonstrate their value beyond the phrase localization task.
Graduation Semester
2018-05
Type of Resource
text
Permalink
http://hdl.handle.net/2142/100977
Copyright and License Information
Copyright 2018 Bryan A. Plummer

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois
Dissertations and Theses - Computer Science
Dissertations and Theses from the Siebel School of Computer Science