Design and development of a framework for visual attention based automatic image description generation with game theoretic optimization

Abstract

This thesis deals with the development of an automatic image description system newlineand its applications. We have considered the encoder-decoder architecture newlinewith a visual attention mechanism for image description generation. The newlinesystem uses a Densely connected convolutional neural network as an encoder newlineand Bidirectional LSTM as a decoder. The visual attention mechanism is also newlineincorporated in this work. The optimization of the caption is also done using a newlineCooperative game-theoretic search. Two particular applications are designed newlinewith the proposed architecture of the system. An aid for visually impaired newlinepeople and medical image captioning are the two applications implemented. newlineAn enduring vision of Artificial Intelligence is to build robots that can newlinerecognize and learn the visual world and who can speak about it in natural newlinelanguage. A significant advancement in object recognition is happening in recent newlineyears. Automatic image description generation is a demanding problem newlinein Computer Vision and Natural Language Processing. The sentence annotation newlineof image and video enhances image indexing, searching, and retrieval, newlinevital for Content-Based Image Retrieval(CBIR). The applications of image newlinedescription generation systems are in biomedicine, military, commerce, digital newlinelibraries, education, and web searching. The description should contain newlinethe scene, action, objects, etc. newlineThe automatic image description generation system proposed in this thesis newlinemainly consists of various phases, such as image feature extraction, visualattention, caption generation, and caption optimization. Image feature extraction newlineis experimented with using multiple CNNs such as VGG, Resnet, newlineDensenet, etc. Densenet gives better performance for the automatic image newlinedescription generation system. newlineThe visual attention model is used for finding salient regions in the image. newlineThe spatial attention, channel-wise attention, and layer-wise attention are newlinedesigned and developed.