Multilevel Association Rule Mining in Distributed Environment

Abstract

Multilevel association rule discovers knowledge from conceptual hierarchical data newlineset and thus provides more significant information than single level association rule. However, newlineexisting multilevel association rule mining algorithms have limitation of processing newlinespeed while analyzing big data. To overcome this, Hadoop-based distributed multilevel newlineassociation rule mining approach is proposed which process the transactional dataset into newlinepartitions then transfers each task to all participating nodes. Thus, it reduces inter node newlinemessage passing in the cluster. newlineThe proposed methodology is applied in two phases. In the first phase, the transactional newlinedataset is generated from big sales dataset using Hadoop MapReduce framework. newlineThen, a proposed distributed multilevel frequent pattern mining algorithms MR-MLAB newline(MapReduce based Multilevel Apriori using Bottom-up Approach) and MR-MLAT (MapReduce newlinebased Multilevel Apriori using Top-down Approach) are used to generate level-crossing newlinefrequent itemset for each level of concept hierarchy. Performance of the system is compared newlinebased on minimum support threshold at different level of concept hierarchy and also newlineby varying dataset size. Moreover, time efficiency of proposed algorithms is compared newlinewith existing Traditional Multilevel Apriori (TMLA) algorithm. Due to ancestor relationship, newlinethis proposed distributed multilevel frequent pattern mining algorithm generates huge newlineamount of hierarchical redundancy. Thus, to improve the performance of the system, such newlinehierarchical redundancy needs to be eliminated. In second phase, distributed multilevel newlinefrequent pattern mining algorithm is applied on regional transactional dataset to generate newlinefrequent k-itemsets for each region. Then, multilevel association rules are generated for newlineeach region. These generated regional multilevel rules are so large that it becomes complex newlineto analyze it using traditional methods. Hence, MR-MCIRD (MapReduce based Multilevel newlineConsistent and Inconsistent Rule Detection) algorithm is proposed to derive cons