Information theory based data dependency extraction and its application in data integration

Abstract

As a huge volume of data is getting generated every day data integration becomes important to provide a uniform view over the data collected from multiple resources Effective data analysis methods are required to work with such massive integrated data and efficient data quality tools are also vital to ensure the correctness of the integrated data The data store for many organizations is the databases and integrity constraints are the primary means for ensuring data integrity in databases Functional Dependencies are the common type of integrity constraints that are specified by the database designer when the database schema is designed In the quest for capturing more information from data in the form of constraints functional dependencies are extended in several ways Among various data dependencies conditional functional dependencies fuzzy functional dependencies and matching dependencies are widely used for data cleaning operations In addition to the constraints specified by the designer there are other constraints hidden in data and it becomes essential to mine dependencies from data Several dependency discovery methods are proposed in this work to extract functional dependencies and its extensions newlineThe proposed work also uses information theory measure to extract various types of data dependencies from data newline newline