Identification of named entities from different language families

Abstract

The objective of the study is to develop the Generic Named Entity system which can automatically extract the generic features for identification of named entities from the given dataset. The features used in this work do not depend on any language or domain. The 12 languages studied in this work belong to different language families. The languages belonging to Dravidian language family are Tamil, Telugu and Malayalam, languages coming under Indo-Aryan family are Hindi, Marathi, Punjabi and Bengali, languages belonging to Germanic family are English, Dutch and German, language form Romance language family is Spanish and language coming under Uralic family is Hungarian are considered for anlaysis in this work. Named Entity Recognition (NER) is defined as the process of automatic identification of proper nouns and classifies the identified entities into the predefined categories such as person, location, organization, facilities, products, temporal or numeric expressions etc. The machine learning technique Conditional Random Fields (CRFs) is used to identify the named entities in the present work. The linguistic analysis has done to observe the part of speech patterns immediately preceding or following named entities and found the common patterns of named entities for all the 12 languages. The lexical level features such as word, first word, trigrams and bigrams of suffix and prefix information of the current token, syntactic level features such as POS and chunk information, dynamic features such as POS patterns preceding and following NE, preceding word and POS information of NE, following word and POS information of NE are used for the system development. The dynamic features obtained using the generic feature selection methodology is validated using the K-means++ clustering algorithm. newline