Toward ontological visual understanding

Abstract

Lack of human prior knowledge is one of the main reasons that the semantic gap still remains when it comes to automatic multimedia understanding. One difference between the human cognition system and state-of-the-art machine vision algorithms is that the former possesses and uses high-level semantic knowledge, or ontology. In this thesis, we present our work on image-level annotation and album-level event recognition, both emphasizing the ontological structure among concepts including object, scene, and event. The inference and learning make use of mutual relations among these concepts, and are general for any concept and initial concept recognition. Our experiments show that the proposed frameworks are able to perform the respective visual recognition tasks better than other methods that are also based on middle-level recognition with or without ontology, and better than methods based purely on low-level features, thus validating the use of ontology in recognizing high-level and abstract concepts.