Duplicate Record Detection Using Soft Computing Approaches

Abstract

The abundance of data produced and the requirement to merge newlinethem from different sources have resulted in the challenge of making the newlineefficient detection of the duplicate records in databases Since the data sources newlineare independent they may adopt their own conventions and often integrating newlinedata from different sources invariably leads to erroneous duplication of data newlineTo ensure high quality data the database must validate and cleanse the newlineincoming data from the external sources In this regard data cleaning has newlinebecome essential to ensure the quality of the data stored in the real world newlinedatabases The process of identifying the record pairs that represent the same newlineentity is commonly known as duplicate record detection This is one of the newlineimportant tasks of data cleaning newlineThe proposed work suggests several new approaches to improve the newlineaccuracy of the duplicate record detection process along with other wellknown newlinemeasures The first part of the work adopts Adaptive Neuro Fuzzy newlineInference Systems ANFIS for duplicate record detection by means of newlinesimilarity functions It is not only to reduce the time taken for making the newlinedecision for duplicate detection but also to reduce the time to hard code the newlinematching rules that ANFIS used It is necessary to adapt a similarity measure newlinefor each field of the database with respect to the particular data domain for newlineattaining the accurate similarity computations Consequently the proposed newlineapproach combines these similarity values obtained from different similarity newlinemeasures to compute the distance between any two records newlineIn traditional approach each record is selected and compared with newlinethe rest of the tuples one by one making it a time consuming process In order newlineto reduce the computational time the cleaned records are clustered by Kmeans newlinealgorithm by grouping the records most likely to be duplicates in one newlinegroup Thus all the possible pairs from each cluster are selected comparing newlineonly the records within each cluster ANFIS uses similarities for newlinecomparing a pair of records and detecting the dupl