MULTIMODAL EMOTION RECOGNITION FOR ENHANCING HUMAN COMPUTER INTERACTION

Abstract

Emotions are fundamental to human lives and play an important role in newlinecommunication between people. This interactional phenomenon has been extensively newlinestudied by several research groups worldwide. Enabling computers to read emotions newlinehas been motivated by the potentially wide range of applications which involve newlinehuman-computer interfaces. Such an affective computer system is an active area of newlineresearch, which include recognizing emotions as well as generating appropriate newlineresponses. This dissertation addresses the emerging area of research on emotions and newlinehuman computer interaction. newlineWe present approaches for recognition of emotions by considering static image of newlineface, speech and keyboard stroke pattern modalities. Each of these modalities has newlinebeen investigated separately as unimodal systems. Recognizing the inadequacy of newlineunimodal approaches, the study also explores multimodal emotion recognition newlineframework by combining these modalities. The work involves identifying and newlineextracting relevant features from each of these modalities and using various newlineclassification algorithms for classifying them into emotions. newlineThe features extracted from the face include the position of eyes, eyebrows, mouth, newlineand nose and various distances involving them. Features of interest in speech are newlinepitch, formant frequencies, parameters related to voiced and unvoiced region and newlineMFCC. The features extracted from keyboard interaction are typing speed, number of newlinebackspaces used, number of errors made, etc. Each of these features was analyzed newlineindividually to see its variation across emotions. We have used classification newlinealgorithms such as neural networks, bayesian classification etc. to classify the newlineemotions. Various studies involving different subsets of features and different newlineclassification algorithms were carried out to identify relevant features and newlineclassification framework. The performances were also evaluated for gender newlinedependency across emotions for different subset of features and classification newlinealgorithms. newlineThe