Development of multimodal system for emotion recognition

Abstract

Due to the significance of human behavioral intelligence in computing devices, this thesis focused on the facial expressions and speech of humans for their emotion recognition in multimodal signals. A comprehensive literature survey is conducted to study and analyze the existing multimodal datasets and state-of-the-art methods for emotion recognition. Considering various research issues and challenges, two main directions of research is aimed in this thesis work. In the first direction, Panjab University Multilingual Audio and Video Facial Expression (PUMAVE) dataset is proposed for multimodal emotion recognition in Indian context. The PUMAVE dataset is the first multilingual tone-sensitive audio and video facial expression dataset which provides emotional labeled clips of subjects who speak the native language (Punjabi and Hindi) and foreign language (English). The second direction of research is the analysis and development of a multimodal system for human emotion recognition through facial expressions and tone-sensitive speech. The experiment analysis is performed on various feature extraction methods, temporal feature aggregation methods and fusion strategies to determine the best approach for emotion recognition through facial expressions and tone-sensitive speech. The findings of the experimental analysis established the foundation for the development of a multimodal system through fusion of the peak stage behavior of facial expression and speech for emotion recognition in multimodal signals. The proposed multimodal system contributes a novel method for the determination of peak stage frame through analyzing the behavior of Facial Action Units (FAUs) and speech with Improved-Technique for Order of Preference by Similarity to Ideal Solution (I-TOPSIS) method. newline