Algorithms for Processing RGBD Images and Videos for Depth Based 3D Video Systems

Abstract

In recent times, immersive visual media such as Virtual Reality (VR), Augmented Reality (AR), 3DTV and Free Viewpoint Television (FTV) have garnered tremendous interest. Immersive visual media content typically provides interactivity and a more realistic viewing experience, thereby reducing the divide between the real and the virtual world. Such media have found applications in various fields such as gaming, education and training, entertainment etc. With increased availability of depth sensing cameras, the demand for depth-based 3D video systems is on the rise. Depth sensing cameras acquire 3D information of the scene and store it in the form of two-dimensional array known as depth map. Depth-based 3D video systems are a natural choice for immersive media. This is because the depth information not only enables the viewer to have a perception of depth but also plays an important role in enabling the viewers to view the scene by switching the viewpoints as if they are around the scene. The display systems such as 3DTV and FTV play a major role in creating the effect of immersion to the viewers. However, the efficient functioning of these display systems are tightly coupled with various aspects such as acquisition of 3D content, representation of the content in a manner suitable for processing, compression, transmission etc. All these functions together comprise the end-to-end 3D video system. In this thesis, we address few problems that are encountered in different functional blocks of the depth-based 3D video system. The problems addressed in this thesis are relevant at the acquisition, representation and display stages. In the first two contributing chapters (Chapters 2 and 3), we address the problem of depth map upsampling using a guidance image. Depth map upsampling is performed to obtain depth information corresponding to every pixel in the color image...