Inferring object states and articulation modes from egocentric videos

Abstract

We develop algorithms for understanding objects from the point of view of interacting with them. There are two key aspects to obtaining such an understanding. First, objects can occur in different states and we need features that are sensitive to such states. Second, different objects can be articulated in different ways and we need to understand how to correctly infer their modes of articulation. We propose self and weakly supervised techniques to obtain such an understanding of objects purely through observation of how humans interact with the world around them through their hands. Our experiments on the challenging EPIC- KITCHENS dataset show the merits of using human hands as a probe for understanding objects.