Computer Vision based Approach for Indian Sign Language Recognition and Translation using Deep Learning

Abstract

Sign language, a language used by the Deaf, is an entirely visual language with its own grammar and differs mainly from spoken language. Translation between two spoken languages is much smoother than translation between a spoken language and a sign language due to several factors. Deaf people find it very difficult to express their feelings to the rest of society since other people lack the knowledge of the sign language used by the Deaf community. The conception of the system that would be able to translate the sign language into text would facilitate the understanding of the sign language to the rest of society. Such a system would eliminate the dependency on the translators to understand the sign language. newlineThere is significant variation between sign languages of different countries worldwide, although there are many similarities. Therefore, it may not be preferred for Indian Sign Language to fully adopt such approaches for sign language recognition (ISL). After nearly 30 years of research, ISL recognition is still in its infancy when compared to other international sign languages. Many device-based and vision-based approaches have been used for ISL recognition from images of signs. Still, most of these approaches focus on regional versions or manual components of signs having one or two words. It is challenging to handle continuous sign language sentence recognition and translation into the text as no large dataset and corpora are available for videos of ISL sentences. newlineIn our first proposed approach, we have attempted to answer one of the research questions: how to use deep learning to deal with very small amounts of input videos to incorporate both left-handed and right-handed signs without affecting ISL sentence recognition performance. Deep learning gives promising results in gesture recognition, but it requires large datasets to avoid overfitting. To avoid this problem, the concept of image augmentation and pretrained model is used to increase dataset size and reduce over fitting.