Human Pose Retrieval for Image and Video collections

Abstract

With overwhelming amount of visual data on the internet, it is beyond doubt that a search capability for this data is needed. In this thesis, we will demonstrate that images and videos can be retrieved using the pose of the humans present in them. Here pose is the 2D/3D spatial arrangement of anatomical body parts like arms and legs. Retrieving humans using pose has commercial implications in domains such as dance (query being a dance pose) and sports (query being a shot). In this thesis, we propose three pose representations that can be used for retrieval. newline newlineOur first pose representation is based on the output of human pose estimation algorithms (HPE). We improve the reliability of these algorithms by proposing an evaluator that predicts if a HPE algorithm has succeeded. For our second pose representation, we introduce deep poselets for pose-sensitive detection of various body parts that are built on convolutional neural network (CNN) features. Second, using these detector responses, we construct a bag-of-poselets representation. Our third pose representation learns a deep neural network which maps the input image to a very low dimensional space where similar poses are close by and dissimilar poses are farther away. We show that pose retrieval system using these low dimensional representation is on par with the deep poselet representation. newline newlineFinally, we describe a method for real time video retrieval where the task is to match the 2D human pose of a query. The method is scalable and is applied to a dataset of 22 movies totaling more than three million frames. Apart from the query modalities, we introduce two other areas of novelty. First, we show that pose retrieval can proceed using a low dimensional representation. Second, we show that the precision of the results can be improved substantially by combining the outputs of independent human pose estimation algorithms. The performance of the system is assessed quantitatively over a range of pose queries.