Developing Efficient Deep Architectures for Classification Task

Abstract

Deep architectures are enjoying an increasing popularity due to its success in solving complex problems. In particular, deep architectures have proven to be effective in a large variety of classification tasks. Contrary to previous research, which required engineered feature representations, designed by experts, in order to succeed, deep architectures attempt to learn representation hierarchies automatically from data. The multi-layer architecture of these networks is particularly useful in capturing the hierarchical structure of the given data: simple features are detected at lower layers and fed into higher layers for extracting abstract representations. Despite the remarkable representational power of deep networks, training these models is computationally expensive. In addition, considering the lack of enough labeled training data in many applications, over-fitting is a serious threat for deep models with large number of free parameters. Also, there are innate issues with the gradient-based optimization procedure used for parameter learning in these models. Therefore, the search for algorithms to optimize the learning of deep architectures is extensive and ongoing. newlineIn this thesis, we tackle the challenging problem of optimizing the learning of deep architectures. We first draw up a state-of-the-art review of the deep architectures specifically multilayered feed forward networks aiming to understand the various techniques currently used to train deep architectures. We then propose and explore the use of two phase strategy in training of deep architectures. We propose a deep architecture where successive layers of units are pretrained using unsupervised learning. The second phase involves supervised fine tuning of various layers. For performing supervised fine tuning we introduce a set of algorithms that alleviate the problems associated with the conventional approach of fine tuning. We study the applicability and potential of the proposed architecture on a number of benchmark datasets, highlighting the .....