A multimodal approach to develop robust speech recognition systems

Abstract

In adverse conditions, building a robust Automatic Speech Recognition (ASR) system is a challenging task. The performance of an ASR system degrades due to environmental noise, speaker variability, and transmission variability, among others. One way to improve the performance of such a system is using a multimodal approach. This thesis focuses on building a robust ASR system by combining the complementary evidence present is the multiple modalities through which speech is expressed. These complementary evidences of speech, as recorded using a Normal Microphone (NM), a Throat Microphone (TM), and a camera, respectively, are combined for building a robust multimodal speech recognition system for the highly confusable Consonant-Vowel (CV) units of Hindi language. These CV sound units differ in their place and manner of articulation, yet have high acoustic similarity and hence are highly confusable. A comparative study on the recognition performance of the trimodal ASR system (built based on the three modalities) and the bimodal ASR systems that are built both with and without NM data (using only TM and Visual Lip Reading (VLR)) is carried out to analyze the extent to which the audio and visual cues provide complementary evidence in recognizing the acoustically similar, highly confusable sound units. Such a study would help in designing robust multimodal ASR systems even for highly confusable datasets. First, unimodal systems are built for NM, TM, and VLR speech for the syllabic units. The syllabic units of Hindi language are categorized into three groups (Vowel, Place of Articulation (POA), and Manner of Articulation(MOA)). Mel Frequency Cepstrum Coefficient (MFCC) features are extracted from NM and TM signal. Pixel-based transforms such as Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT), and motion-based features such as Motion History Image (MHI) based DCT, DWT, and Zernike Moments (ZM) are used as features extracted from visual speech signals. A simple Left to Right (L-R) Hidden Markov Model - Ga