Development of optimized algorithm for extract transform load process using soft computing techniques

Abstract

Extract Transform Load refers to a database process framework entrusted with the task of extraction, transformation, and loading of data in data warehouse. Web Data Extraction algorithm is proposed wherein web templates are identified by developing feature-based web data extraction algorithm by clustering the similar web pages together based on feature similarity of their DOM structure. The hybrid transformation technique is proposed that employs token-wise sentence sorting alongwith Levenshtein distance for noise reduction. The RDBMS is replaced with distributed failsafe data clusters as data warehouse using Hadoop based techniques. This delimits the constraint of data processing, storage and retrieval of large data structure. The developed algorithm is validated on USPTO web site. newline