An enhanced framework for identifying human emotions from multimodal signals

Abstract

The notion of emotions is very vital when one person wants to communicate with another person. When a child is sorrowful, it tends to cry. The father or the mother of the child will try to make the child happy by giving what the child needs and the child laughs, which makes the father and mother happy. This happiness cannot be expressed because it will be beyond the measure. Similarly emotions play an important role when a human wants to interact with a machine. Take for example an automated vehicle assist system, making the vehicle understand the emotions of the driver will make the driving experience better. Another example is automated telephony wherein the emotions of the caller can be analyzed in order to improve the feedback of the system. Understanding emotions from image, speech signal, and video is of major concern in this research work. In this thesis we first work on image processing, for identifying emotions. Second we work on speech signal for emotion identification. Finally we use video to identify emotions. In the first framework the emotion of a person is identified from an image with the help of tensorflow. It is an upcoming technology in the field of artificial intelligence. Tensorflow has grown to help users to determine what they need with respect to their daily life. One of the most talked about tensorflow applications is the RankBrain developed by Google corporation which allows users to find the relevant pages with the help of deep neural nets. For our purpose we use tensorflow to identify emotions from an image using the features extracted from the image. Basically the feature extraction is done by identifying the geometry of the face after detecting the landmarks of the eyes and mouth of the face. The landmarks are constructed by using the proposed modified eyemap-mouthmap algorithm on an enhanced image which uses discrete wavelet transform and fuzzy for enhancement. Results of classification show that the proposed methodology is better when tensorflow is used. The second framework on em