Crossmodal fusion of multiple input modalities for human computer interaction

Abstract

Modern technologies provide efficient implementation of a large newlinevariety of human-centered ubiquitous computer applications in the emergent newlinee-Society such as intelligent networked spaces for living, working, health and newlineelderly care, shopping and entertainment. In order to augment the newlinefunctionality and user-satisfaction of these e-Society applications, it newlinenecessitates to develop more accessible and unobtrusive, user-oriented and newlineuser friendly, human-computer interaction modalities allowing people with newlinediverse abilities to seamlessly access information anytime, anywhere and to newlineinteract naturally and intuitively with these new intelligent spaces. The prime newlineobjective of this dissertation is the design of a generic scalable multimodal newlineframework that enables rapid development of multimodal human computer newlineinteraction based applications using variety of modalities and methods for newlineambiguity resolution. newlineThe multimodal interaction framework proposed in this dissertation newlinetakes multimodal utterances involving speech and continuous double hand newlinegestures with elbow movement and facial cues from the user. Semantic partial newlineframes of the likely user intentions are generated by the individual modality newlinerecognizer and disseminated to the inference layers. The inference layer takes newlinethe semantic partial frames from various recognizers and fuses their results newlinewith the help of application context into a complete semantic frame for the newlineapplication. Cross modal fusion technique is then used to fuse the features newlineemerging from different modality sources and a multimodal learning system newlineis used to infer the semantic interpretation for the higher level application newlineevents. The application accepts semantic frames and provides feedback to the newlineuser through multimodal output. newline